3D model generation apparatus, generation method, and program

Abstract

To continuously generate a three-dimensional model even when an object goes out of a three-dimensional model generation area.SOLUTION: A generation device generates a three-dimensional model representing a three-dimensional shape of an object existing in an area where image capturing ranges of a plurality of imaging devices overlap using a plurality of moving images obtained by capturing the object by the plurality of imaging devices. When the object corresponding to the generated three-dimensional model goes out of the area, the generation device calculates a velocity vector when the object crosses a border of the area. Then, the generation device estimates a position of the object after the object goes out of the area on the basis of the calculated velocity vector and the position at the time of the object crossing the border of the area. Further, the generation device estimates a three-dimensional model representing a three-dimensional shape of the object at the estimated position.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a technology for generating a 3D model of a subject (object).

Background Art

[0002] There is a technology that generates a virtual viewpoint image from any arbitrary virtual viewpoint by using multi-viewpoint captured images obtained by synchronized imaging performed by a plurality of imaging apparatuses. Generation of this virtual viewpoint image can be realized, for example, by generating three-dimensional shape data (3D model) of an object appearing in the multi-viewpoint captured images using a method such as visual hull, and performing rendering in accordance with the virtual viewpoint. In this case, to generate a 3D model, the object needs to appear in each of the multi-viewpoint captured images.

[0003] Patent Document 1 describes a technology that suppresses distortion of the shape or chipping of the contour of the 3D model of a target object when the target object is hidden behind another object from the viewpoint of any one of the imaging apparatuses, for example.

Prior Art Literature

Patent Literature

[0004]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0005] For example, in ball sports like baseball, the ball as an object may fly higher than expected and go outside the 3D model generation area (for example, the imaging range covered by multiple imaging devices). When an object goes outside the 3D model generation area in this way, the technology described in Patent Document 1 cannot be used to address the issue, and the generation of the 3D model will cease. As a result, an unnatural virtual viewpoint image may be generated in which, for example, the ball that was previously visible suddenly disappears from the screen during viewing. [Means for solving the problem]

[0006] The generation apparatus according to this disclosure comprises: generation means for generating a 3D model representing the three-dimensional shape of an object located in an area where the imaging ranges of the multiple imaging devices overlap, using multiple moving images obtained by imaging an object with multiple imaging devices; calculation means for calculating a first velocity vector when the object, corresponding to the 3D model generated by the generation means, crosses the boundary of the area when the object leaves the area; first estimation means for estimating the position of the object after it has left the area, based on the first velocity vector calculated by the calculation means and the position of the object when it crosses the boundary of the area; and second estimation means for estimating a 3D model representing the three-dimensional shape of the object at the position estimated by the first estimation means, wherein the second estimation means associates a copy of the 3D model generated by the generation means with the estimated position. [Effects of the Invention]

[0007] According to the technology disclosed herein, 3D model generation can be continued even if an object moves outside the 3D model generation area. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example configuration of a virtual viewpoint image generation system. [Figure 2]A diagram explaining the visual volume cross-section method. [Figure 3] A diagram showing an example of the hardware configuration of various devices that make up a virtual viewpoint image generation system. [Figure 4] A flowchart illustrating the flow of 3D model generation processing on the front-end server. [Figure 5] (a) and (b) are diagrams showing, in chronological order, what happens when a ball hit by a batter leaves the 3D model generation area and then returns to the 3D model generation area. [Figure 6] (a) and (b) are explanatory diagrams for Modification Example 3. [Modes for carrying out the invention]

[0009] The following embodiment will be described with reference to the drawings. Note that the following embodiment is not necessarily limiting to the present invention. Furthermore, not all combinations of features described in this embodiment are essential to the solution of the present invention.

[0010] [Embodiment 1] Figure 1 shows an example of the configuration of a virtual viewpoint image generation system according to this embodiment. The camera group 101 consists of multiple imaging devices (cameras) and all cameras synchronize to capture video for the purpose of generating data representing the three-dimensional shape of a subject (object) (generally called a "3D model"). Multiple video images with different viewpoints obtained by the synchronized imaging of multiple cameras are linked to a time code indicating the time of imaging for each frame and transmitted to the front-end server 102. For the sake of explanation in the following description, multiple video images with different viewpoints obtained by the synchronized imaging of multiple cameras will be collectively referred to as "camera images".

[0011] The front end server 102 includes a 3D model generation unit 110, a data buffer 111, a boundary crossing determination unit 112, a speed calculation unit 113, a position estimation unit 114, a 3D model estimation unit 115, and an output unit 116.

[0012] The 3D model generation unit 110 uses camera images input from the camera group 101 to generate 3D models of moving objects such as people and balls captured in the camera images, frame by frame, for example, using the viewing volume cross-section method. Figure 2 illustrates the viewing volume cross-section method. The frustum extending in three-dimensional space from the camera's imaging center, passing through each point on the silhouette contour in the mask image (foreground silhouette image), is called the "viewing volume" of the camera. The three-dimensional shape of an object is reproduced by finding the common region of the viewing volumes of multiple cameras. In this embodiment, the three-dimensional shape is represented by a collection of tiny cubes called voxels. The generated 3D model is stored in the data buffer 111, linked to the time code of the corresponding frame, and transmitted to the database 103.

[0013] The data buffer 111 stores the 3D models generated by the 3D model generation unit 110 in chronological order according to the time code. The stored 3D models are used for various image processing tasks, such as boundary crossing detection, velocity calculation, position estimation, and 3D model estimation, which will be described later. The data buffer 111 has enough memory to store the required number of 3D models for each image processing task.

[0014] The boundary crossing determination unit 112 determines whether an object corresponding to a 3D model stored in the data buffer 111 has crossed the boundary of a pre-set 3D model generation area and gone outside the area. The 3D model generation area is a three-dimensional area (common imaging area) where the imaging ranges of multiple cameras constituting the camera group 101 overlap, which has been set by an operator or the like after confirming that it is covered by a sufficient number of cameras to obtain a 3D model of a certain quality. If the determination result shows that the object has gone outside the model generation area, the speed calculation unit 113 is notified of this fact along with the time code of the frame immediately before the boundary was crossed (hereinafter referred to as "boundary crossing time information").

[0015] The velocity calculation unit 113 calculates the velocity vector when an object leaves the 3D model generation area based on the boundary crossing time information notified by the boundary crossing determination unit 112. This velocity vector is calculated by referring to the boundary crossing time information to identify two or more frames immediately before the crossing, and obtaining the 3D models corresponding to those two or more frames from the data buffer 111. The calculated velocity vector is sent to the position estimation unit 114 along with the boundary crossing time information.

[0016] The position estimation unit 114 obtains position information associated with the 3D model immediately before crossing the boundary from the data buffer 111 based on the input boundary crossing time information, and estimates the object's position outside the 3D model generation area based on the input velocity vector. In this embodiment, the position estimation unit 114 performs position estimation at predetermined intervals for the period from when the target object leaves the 3D model generation area until when it returns to the 3D model generation area. The predetermined interval is approximately every 16.6ms, for example, when the frame rate of the camera group 101 performing synchronous imaging is 60fps and the 3D model is generated every frame. It is desirable to perform position estimation every frame. However, if the virtual viewpoint image is not displayed in real time, the user may set an arbitrary interval for position estimation, such as once every few frames. The estimated position of one or more objects (estimated position) is then associated with the time code of the corresponding frame and sent to the 3D model estimation unit 115 along with the boundary crossing time information.

[0017] The 3D model estimation unit 115 estimates a 3D model of an object at each estimated position based on the position estimation result input from the position estimation unit 114. This estimation can be achieved by duplicating the 3D model corresponding to the time code indicated by the crossing time information and arranging the duplicated 3D model at the estimated position (associating the duplicated 3D model with the time code of the frame corresponding to the estimated position). In the present embodiment, the 3D model estimation unit 115 estimates the 3D model at predetermined intervals for the period from when the object leaves the 3D model generation region to when the object returns back into the 3D model generation region. The predetermined interval in this case is the same as the predetermined interval used for position estimation.

[0018] The output unit 116 outputs the 3D model generated by the 3D model generation unit 110 and the 3D model estimated by the 3D model estimation unit 115 to the database 103 in association with the time codes of the corresponding frames, respectively.

[0019] The database 103 stores each 3D model received from the front-end server 102 in association with the time code of the corresponding frame.

[0020] The virtual viewpoint controller 104 is a control device that allows a user to set a virtual viewpoint by operating a virtual camera in an imaging space where the camera group 101 performs synchronized imaging. The virtual viewpoint controller is configured of a display, a joystick, a keyboard, and the like. For example, the user specifies camera parameters such as the position and orientation, and angle of view of the virtual camera by operating the joystick on a user interface screen (UI screen) displaying a CG space that reproduces the imaging space. The camera parameters of the virtual camera specified by the user are associated with the corresponding time code, and transmitted to the back-end server 105 as virtual viewpoint information.

[0021] The backend server 105 retrieves a 3D model from the database 103 based on the virtual viewpoint information received from the virtual viewpoint controller 104, and generates a virtual viewpoint image corresponding to the specified virtual camera's position, orientation, and field of view. The generated virtual viewpoint image is sent to the display 106 and displayed on the display 106. Note that the output destination of the virtual viewpoint image does not necessarily have to be a display. For example, the virtual viewpoint image could be sent to a distribution provider (not shown) and published on the web by the distribution provider.

[0022] <Hardware Configuration> Figure 3 shows an example of the hardware configuration of various devices that constitute the virtual viewpoint image generation system. Specifically, the front-end server 102, database 103, virtual viewpoint controller 104, and back-end server 105 can be realized by the information processing device 200 shown in Figure 2. The information processing device 200 has a CPU 201, ROM 202, RAM 203, auxiliary storage device 204, display unit 205, operation unit 206, communication I / F 207, and system bus 208.

[0023] The CPU 201 controls the entire information processing device 200 using computer programs and data stored in the ROM 202 and RAM 203, thereby realizing each function of the virtual viewpoint image generation system shown in Figure 1. The information processing device 200 may have one or more dedicated hardware components separate from the CPU 201, and at least a portion of the processing performed by the CPU 201 may be executed by the dedicated hardware. Examples of such dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0024] ROM202 stores programs and other data that do not require modification. RAM203 temporarily stores programs and data supplied from auxiliary storage device 204, as well as data supplied from external sources via communication interface 207. Auxiliary storage device 204 is composed of, for example, a hard disk drive and stores various types of data such as image data and audio data.

[0025] The display unit 205 is composed of, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) for the user to give instructions to the information processing device 200. The operation unit 206 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and receives various instructions from the user and inputs them to the CPU 201. The CPU 201 operates as a display control unit that controls the display unit 205, and as an operation control unit that controls the operation unit 206.

[0026] Communication I / F 207 is used for communication with external devices of the information processing device 200, such as the camera group 101 and the microphone group (not shown). If the information processing device 200 has a function to communicate wirelessly with external devices, communication I / F 207 is equipped with an antenna. System bus 208 connects the various parts of the information processing device 200 and transmits information.

[0027] In the configuration example shown in Figure 3, the display unit 205 and the operation unit 206 are assumed to be located inside the information processing device 200. However, at least one of the display unit 205 and the operation unit 206 may be located outside the information processing device 200 as a separate device.

[0028] <3D model generation process in front-end server 102> Figure 4 is a flowchart showing the process flow by which the front end server 102 generates a 3D model of an object. As described above, even if the object moves outside the 3D model generation area, the front end server 102 of this embodiment estimates the subsequent position of the object and generates a 3D model at the estimated position. The following explanation follows the flow shown in Figure 4. In the following explanation, the symbol "S" represents a step.

[0029] In S401, the 3D model generation unit 110 generates 3D models of objects using the view volume cross-eyed method based on camera images input from the camera group 101. Here, a 3D model for each object present within a pre-defined 3D model generation area is generated for each frame that makes up the camera image. A specific example is shown. Figure 5(a) is a time-series diagram showing the process from when a baseball hit by batter 500 leaves the 3D model generation area and then returns to the 3D model generation area 501, assuming the target event for imaging is a baseball game. In Figure 5(a), ball marks 510 to 525 show the trajectory of the ball from immediately after it is hit by batter 500, representing the position and orientation of the ball in each frame. Currently, the batter and the ball are present within the 3D model generation area 501, and 3D models of them will be generated. Position information will also be attached to the generated 3D models. In this case, the positional information is expressed in a world coordinate system, and is given as three-dimensional coordinates (x,y,z) with the ground in the real space being imaged as the X and Y axes, and the vertical direction as the Z axis. The 3D models corresponding to ball marks 510-512 and 522-525 in Figure 5(a) will be assigned positional information as shown in Table 1 below.

[0030] [Table 1]

[0031] In S402, the 3D models generated for each object in S401 are linked to the timecode of the corresponding frame and stored chronologically in the data buffer 111. In the example of Figure 5(a) mentioned above, at this point, 3D models for all frames have been obtained for the batter, and 3D models corresponding to ball marks 510-512 and 522-525 have been obtained for the ball. Then, the timecode of the frame corresponding to each 3D model is linked and stored in the data buffer 111.

[0032] In S403, the boundary crossing determination unit 112 determines whether each object for which a 3D model was generated in S401 has gone outside the 3D model generation area. The determination method is not particularly limited, but for example, it can be determined by the following method. First, the trajectory of the three-dimensional position of the object of interest is determined based on the 3D model of the object of interest stored in the data buffer 111 in chronological order, and the timing at which the object of interest crosses the boundary of the 3D model generation area is predicted from the obtained trajectory. Then, if the predicted timing matches the timing at which the 3D model of the object of interest disappears from the data buffer 111, it is determined that the object of interest has gone outside the 3D model generation area. If it is determined that only a part of the object exists within the 3D model generation area (when a 3D model representing the three-dimensional shape of a part of the object exists in the data buffer 111), it can be handled as follows, for example. That is, in this case, a 3D model with a missing part of the shape (a partial 3D model) is generated. Therefore, if it is possible to identify that the partial 3D model relates to the same object as the 3D model of the previous frame, it is not determined that it has gone outside the 3D model generation area. On the other hand, if a partial 3D model cannot be identified as relating to the same object as the 3D model in the previous frame, it is determined that the object has moved outside the 3D model generation area. In the example of Figure 5(a) above, ball marks 510 to 512 are inside the 3D model generation area 501, so the complete 3D models for each corresponding frame are stored in the data buffer 111. On the other hand, ball mark 513 is outside the 3D model generation area 501, so the corresponding 3D model is not stored in the data buffer 111. Here, based on the trajectory of the positions of ball marks 510 to 512, it can be predicted that the timing when the ball crosses the boundary of the 3D model generation area 501 is when it reaches the position of ball mark 513. And since the 3D model of the ball disappears from the data buffer 111 at this predicted timing, it is determined that the ball has moved outside the 3D model generation area 501.In this case, the timecodes of the frames immediately preceding the ball crossing the boundary, for example, the timecodes for two frames corresponding to ball marks 512 and 511, will be sent to the speed calculation unit 113 and the position estimation unit 114 as boundary crossing time information.

[0033] In S404, a distribution process is performed based on the result of the judgment in S403. Specifically, if there are any objects that have moved outside the 3D model generation area, S405 is executed next. On the other hand, if there are no objects that have moved outside the 3D model generation area, this process is terminated.

[0034] In S405, the velocity calculation unit 113 calculates the velocity vector at the moment just before the target object crosses the boundary, based on the trajectory of the target object's movement up to the boundary crossing, based on the boundary crossing time information input from the boundary crossing determination unit 112. In the example of Figure 5(a) above, the velocity vector when the ball is at the position of ball mark 512 is calculated as follows, based on the 3D models for two frames corresponding to ball marks 512 and 511. Let (x1, y1, z1) be the coordinates of the position indicated by ball mark 512 in the frame just before the ball crosses the boundary, and (x2, y2, z2) be the coordinates of the position indicated by ball mark 511 in the frame immediately before that. Here, in Figure 5(a), home plate is the origin (0, 0, 0), the line connecting home plate and first base is the X-axis (positive direction is towards first base), the line connecting home plate and third base is the Y-axis (positive direction is towards third base), and the vertical direction is the Z-axis (positive direction is towards the sky). In this case, the velocity vector v (m / s) we are looking for is expressed by the following equation. Velocity vector v = ((x1-x2) / Δt, (y1-y2) / Δt, (z1-z2) / Δt) ... Equation (1)

[0035] In equation (1) above, Δt represents the time elapsed between frames. Thus, the velocity vector v at ball mark 512 with the ball center as the origin is 512=(10, 10, 1.1) is obtained. Note that there are no particular limitations on the method for determining the trajectory of object movement, but one example is to identify the center of gravity of each 3D model from the three-dimensional shape represented by the 3D model, and use the trajectory of the identified center of gravity as the trajectory of the 3D model. The velocity vector calculated in this way is input to the position estimation unit 114.

[0036] In S406, the position estimation unit 114 estimates the three-dimensional position of the object outside the 3D model generation area based on the boundary crossing time information input from the boundary crossing determination unit 112 and the velocity vector input from the velocity calculation unit 113. In the example of Figure 5(a) above, first the three-dimensional position information of the ball mark 512 is obtained from the 3D model linked to the time code indicated by the boundary crossing time information. Then, the obtained three-dimensional position (x,y,z)=(30,30,5.4) and the velocity vector v calculated in S405 are used. 512 Based on =(10, 10, 1.1), the position of the ball corresponding to each frame outside the 3D model generation area is estimated. This results in the example in Figure 5(a) above, where the estimated positions corresponding to ball marks 513 to 521 are obtained, as shown in Table 2 below.

[0037] [Table 2]

[0038] The method for estimating the position is not particularly limited, but one example is to estimate it using a quadratic curve derived from the position of the ball mark 512 just before it crosses the boundary, its velocity vector, and the acceleration due to gravity. Furthermore, estimation may also be performed taking wind speed into consideration. Alternatively, machine learning may be performed using many actual values ​​from similar scenes as training data, and the resulting trained model may be used for position estimation. Each estimated position is associated with the time code of the corresponding frame and input into the 3D model estimation unit 115.

[0039] In S407, the 3D model estimation unit 115 estimates a 3D model for each estimated position of the target object based on the position estimation results input from the position estimation unit 114. Specifically, it performs the process of placing a copy of the 3D model corresponding to the time code of the frame immediately before crossing the boundary at each estimated position. In the example of Figure 5(a) above, a copy of the 3D model corresponding to the ball mark 512 will be placed at all the positions estimated for each frame in S406. When placing the 3D model at the estimated three-dimensional position, the centroid of the 3D model at the time immediately before crossing the boundary should be found, and the 3D model should be placed so that the found centroid coincides with the estimated position. In this way, the 3D models corresponding to ball marks 513 to 521 in Figure 5(a) above are obtained as estimation results.

[0040] In S408, the output unit 116 outputs all 3D models generated in S402 and all 3D models estimated in S407, each associated with the time code of the corresponding frame. In this case, the time code is associated with the estimated 3D model based on the time code associated with the estimated position. In this embodiment, the data is transmitted to the database 103 and stored in the database 103.

[0041] The above describes the 3D model generation process in the front end server 102. While the explanation uses the example of inputting camera images that have already been captured and saved, it is also possible to input and process images being captured in real time.

[0042] <Example 1> If a 3D model is duplicated just before it leaves the 3D model generation area and placed at the estimated position, its orientation (orientation) will be the same as that of the duplicated 3D model. In this case, there is no problem if the set virtual viewpoint is far from the object, but it becomes a problem if it is close to the object. For example, in the ball example above, if the orientation of all the 3D models placed at each estimated position is the same, the ball, which appears large in the virtual viewpoint image, will be displayed as if it is not rotating, which will cause discomfort to the viewer. This discomfort is even greater for objects that are not perfectly spherical, such as rugby balls. Also, since the ball rotates as it flies, the effect of the air changes, and the trajectory of its movement changes depending on the angular velocity. Therefore, it may be possible to calculate the angular velocity vector (a vector indicating the axis of rotation with the absolute value of the angular velocity ω as its magnitude) based on the time code of the 3D model just before it leaves the 3D model generation area, and then place the 3D model at the estimated position while also considering the orientation of the object.

[0043] In this modified example, the velocity calculation unit 113 calculates not only the velocity vector of the 3D model just before it leaves the 3D model generation area, but also the angular velocity vector. This angular velocity vector is also calculated based on 3D models for two or more frames stored in the data buffer 111. In the example of Figure 5(a) above, the angular velocity vector ω512 when the ball leaves the 3D model generation area (ball mark 512) is calculated based on the rotation trajectory at ball marks 510 to 512. The method for calculating the angular velocity vector is not particularly limited; for example, the center of gravity and feature points other than the center of gravity can be extracted from the shape of the 3D model, and the angular velocity vector can be calculated by finding the rotation angle of the feature points centered on the center of gravity. Now, if the angular velocity vector ω512 = (ωx, ωy, ωz), then ωx is positive when the rotation direction is rotated in the direction toward first base with the line connecting home plate and first base as the axis of rotation. Similarly, ωy is defined as positive when a right-handed screw is rotated toward third base, with the line connecting home plate and third base as the axis of rotation. Similarly, ωz is defined as positive when a right-handed screw is rotated toward the sky, with the vertical direction as the axis of rotation. Note that the components of each velocity vector represent angular velocity (rad / sec). In this case, depending on the relationship between the object's rotation speed and the frame rate, there is a possibility of miscalculating rotation in the opposite direction to the actual rotation. Therefore, a separate determination means may be provided to determine whether the rotation direction indicated by the calculated angular velocity vector is correct, by identifying the object's rotation direction based on camera images captured at a higher frame rate than the camera images used for 3D model generation. The calculated angular velocity vector is input to the position estimation unit 114 along with the velocity vector.

[0044] The position estimation unit 114 in this modified example then estimates the object's position based on the velocity vector and angular velocity vector input from the velocity calculation unit 113. Specifically, it first reads the 3D model from the data buffer 111 at the point just before crossing the boundary and obtains its position and orientation. Then, based on the obtained 3D model position and the input velocity vector and angular velocity vector, it estimates the positions of all the balls (ball marks 513 to 521) outside the 3D model generation area. At this time, the estimated position may be corrected by calculating the lift force obtained from the angular velocity vector for the quadratic curve obtained from the position, velocity vector and gravitational acceleration when the ball leaves the 3D model generation area (ball mark 512). The positions thus estimated are input to the 3D model estimation unit 115.

[0045] The 3D model estimation unit 115 duplicates and places the 3D model at the input estimated positions. In doing so, the 3D model estimation unit 115 corrects the orientation of the duplicated 3D model based on the input angular velocity vector and then places it at each estimated position. The method of correcting the orientation is not particularly limited, but for example, it is conceivable to determine and correct the orientation of the 3D model at each estimated position under the assumption that the 3D model continues to rotate according to the calculated angular velocity vector. As a result, as shown in Figure 5(b), for example, a duplicate of the 3D model corresponding to ball mark 512 is placed with its orientation corrected, and 3D models corresponding to ball marks 513' to 521' are obtained. Compared with Figure 5(a), it can be seen that the rotation of the ball is taken into consideration in Figure 5(b).

[0046] As described above, it is also possible to calculate not only the velocity vector but also the angular velocity vector, and to correct the estimated position or the orientation of the 3D model placed at the estimated position based on the angular velocity vector. This makes it possible to obtain a virtual viewpoint image that looks less unnatural even when the virtual viewpoint approaches an object such as a ball.

[0047] <Modification 2> Furthermore, while the above embodiment uses only the velocity vector when the object leaves the 3D model generation area for position estimation, it is not limited to this. If the object returns to the 3D model generation area, the velocity vector at the time of return can be calculated, and position estimation can be performed using both the velocity vectors when the object left and when it returned. In this case, assuming that the velocity vector from Vout to Vin is linear, the position estimation can be performed assuming that the velocity vector from Vout to Vin is linear, given that Vout is the velocity vector when the object leaves and Vin is the velocity vector when it returns. Similarly, in the above-described modification 1, the angular velocity vector when the object returns to the 3D model generation area can also be calculated and used for position estimation and estimation of the 3D model for the estimated position. That is, in the example of Figure 5(b) above, the estimated position and orientation of the 3D model to be placed corresponding to ball marks 513'~521' can be corrected based on the velocity vector, angular velocity vector, position, and orientation of the ball (ball mark 522) when it returns to the area. Furthermore, if the input camera image has already been captured and saved, the 3D model generated immediately after the object returns to the 3D model generation area may be duplicated and placed at the estimated position. In the examples of Figures 5(a) and (b) above, the ball, as an object, returns to the area through a natural free fall, following a smooth parabolic trajectory after leaving the 3D model generation area. However, it is not a necessary requirement for this embodiment to apply that an object that has left the 3D model generation area returns to the area through a natural free fall. For example, this embodiment can be applied even if the ball disappears out of bounds or if its trajectory changes after hitting an obstacle. In this case, additional processing, such as discarding the estimated 3D model, is performed prior to S408, and the output unit 116 does not output the 3D model obtained by estimation. Whether or not the ball has disappeared out of bounds can be determined, for example, by whether or not the object returns to the 3D model generation area within a certain period of time. Furthermore, determining whether an object has collided with an obstacle and changed its trajectory can be done based on the object's three-dimensional position and orientation when it returns to the 3D model generation area.For example, in the example shown in Figure 5(a) above, if the position and orientation of the ball when it returns to the 3D model generation area deviate by more than a predetermined threshold, it should be determined that the change in trajectory was due to hitting an obstacle, rather than due to air resistance or wind. Alternatively, instead of discarding the estimated 3D model when an object disappears out of bounds or its trajectory changes due to hitting an obstacle, the system may allow the user to select whether or not to keep the 3D model via a UI screen or similar.

[0048] <Variation 3> Furthermore, for objects within the camera's field of view, it is possible to correct the estimated position and the orientation of the 3D model to be placed based on the captured image. In this case, the captured image may be an image taken by any of the cameras constituting the camera group 101, or an image taken by a different camera not included in the camera group 101. This will be explained with reference to Figures 6(a) and 6(b). First, candidate object positions are determined from the position and orientation of the camera corresponding to the arbitrary captured image. Here, candidate object positions are represented by straight lines in virtual space. Now, the captured image 600 shown in Figure 6(a) is an image obtained by the camera 610 shown in Figure 6(b), and the captured image 600 shows a ball indicated by the ball mark 601. In Figure 6(b), the dotted straight line 611 extending from camera 610 indicates candidate positions of the ball in virtual space, estimated based on the position of the ball mark 601. That is, the ball will be located at one of the positions on the straight line 611. If there is a discrepancy between the line representing the candidate object position obtained in this way and the estimated position, the estimated position is corrected to the closest position on the line. In Figure 6(b), the ball mark 612 indicates the estimated position and the orientation of the 3D model placed there, and it can be seen that the position is misaligned with the line 611. In this case, the estimated position is corrected to the closest position on the line 611, and the orientation of the 3D model at the corrected estimated position as seen from the camera 610 is corrected to match the orientation indicated by the ball mark 601. As a result, the position and orientation of the 3D model corresponding to the ball mark 612 are corrected to the position and orientation indicated by the ball mark 613.

[0049] The above example described the case where correction is performed using a single captured image. However, when using two or more captured images obtained from multiple cameras, the object's position is determined from the corresponding positions and orientations of the multiple cameras. If there is a discrepancy between the determined object position and the estimated position, the position of the 3D model is corrected to the determined object position. Furthermore, the orientation of the 3D model at the corrected estimated position as seen from each of the multiple cameras is corrected to match the orientation of the object in each captured image. Thus, although limited to cases where the target object is within the camera's field of view, the estimated position and the orientation of the 3D model to be placed may be corrected based on the captured images.

[0050] As described above, in this embodiment, if an object moves outside the 3D model generation area, the position of the object outside the area is estimated based on the velocity vector immediately before that, and a copy of the 3D model as it was when the object was inside the area is duplicated and placed. This allows for the continuous supply of 3D models even if an object moves outside the 3D model generation area, preventing the object from suddenly disappearing in the virtual viewpoint image.

[0051] (Other embodiments) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0052] Furthermore, the disclosure of this embodiment includes the following configurations and methods.

[0053] (Composition 1) A generation means for generating a 3D model representing the three-dimensional shape of an object located in a region where the imaging ranges of multiple imaging devices overlap, using multiple moving images obtained by imaging an object with multiple imaging devices, When the object corresponding to the 3D model generated by the generation means goes outside the region, a calculation means calculates a first velocity vector when the object crosses the boundary of the region, A first estimation means for estimating the position of the object after it has left the region, based on the first velocity vector calculated by the calculation means and the position of the object when it crosses the boundary of the region, A second estimation means for estimating a 3D model representing the three-dimensional shape of the object at the position estimated by the first estimation means, It has, The second estimation means associates a copy of the 3D model generated by the generation means with the estimated position. A generating apparatus characterized by the following features.

[0054] (Configuration 2) The calculation means further calculates a first angular velocity vector when the object crosses the boundary of the region, The second estimation means determines the orientation of the replicated 3D model and associates it with the estimated position, based on the assumption that the object continues to rotate according to the first angular velocity vector calculated by the calculation means. The generating apparatus according to configuration 1, characterized by the above.

[0055] (Composition 3) The second estimation means described above is: Based on the first angular velocity vector calculated by the calculation means, the position estimated by the first estimation means is corrected. A copy of the 3D model generated by the generation means immediately before the object generated by the generation means leaves the area is linked to the corrected position. The generating apparatus according to configuration 2, characterized in that it is a generating apparatus.

[0056] (Composition 4) The generation apparatus according to any one of configurations 1 to 3, characterized in that the first estimation means estimates the position of the object during the period from when the object leaves the area until it returns to the area.

[0057] (Composition 5) The calculation means further determines a second velocity vector when the object that has left the region returns to the region, The first estimation means estimates the position of the object based on the first velocity vector and the second velocity vector calculated by the calculation means. The generating apparatus according to configuration 4, characterized by the features described above.

[0058] (Composition 6) The calculation means further calculates a second angular velocity vector when the object that has left the region returns to the region, The second estimation means described above is: Based on the second angular velocity vector calculated by the calculation means, the position estimated by the first estimation means is corrected. The replicated 3D model corresponding to the point just before crossing the boundary generated by the generation means is linked to the corrected position. The generating apparatus according to configuration 5, characterized by the features described herein.

[0059] (Composition 7) The second estimation means described above is: Based on images of the object captured by one or more imaging devices, the position estimated by the first estimation means is corrected. A copy of the 3D model generated by the generation means immediately before the object generated by the generation means leaves the area is linked to the corrected position. The generating apparatus according to configuration 1, characterized by the above.

[0060] (Composition 8) The second estimation means described above is: Based on the position of the object as seen in the image captured by the one or more imaging devices, candidate positions of the object in the virtual space are calculated. If there is a discrepancy between the calculated candidate position and the position estimated by the first estimation means, the estimated position is corrected to the closest position among the candidate positions. The generating apparatus according to configuration 7, characterized by the features described above.

[0061] (Composition 9) The generation apparatus according to configuration 8, characterized in that the second estimation means corrects the orientation of the 3D model associated with the corrected position so that it matches the orientation of the object in an image captured by the one or more imaging devices.

[0062] (Composition 10) The system further includes a determination means for determining whether the object corresponding to the 3D model generated by the generation means has moved outside the region. The calculation means calculates the first velocity vector when the determination means determines that the object has moved outside the area. A generating apparatus according to any one of configurations 1 to 9, characterized by the above.

[0063] (Composition 11) The system further includes a storage means for storing the 3D model generated by the generation means, The determination means is, From the trajectory of the 3D model's position stored in the storage means in chronological order, the timing at which the 3D model crosses the boundary of the region is predicted. The system determines that the object has left the area when the predicted timing coincides with the timing at which the 3D model disappears from the storage means. The generating apparatus according to configuration 10, characterized by the above.

[0064] (Composition 12) The generation means generates the 3D model in units of frames that constitute the plurality of images, The first estimation means estimates the position of the object after it leaves the region on a frame-by-frame basis that constitutes the plurality of images, The second estimation means estimates a 3D model representing the three-dimensional shape of the object at each position estimated on a frame-by-frame basis by the first estimation means. A generating apparatus according to any one of configurations 1 to 11, characterized by the features described herein.

[0065] (Composition 13) The generation apparatus according to any one of configurations 1 to 12, characterized in that the second estimation means associates a copy of the 3D model generated by the generation means immediately before the object leaves the area.

[0066] (Composition 14) The generation apparatus according to any one of configurations 1 to 13, further comprising an output means for outputting the 3D model generated by the generation means and the 3D model estimated by the second estimation means.

[0067] (Composition 15) The generation apparatus according to configuration 14, characterized in that the output means does not output the 3D model estimated by the second estimation means if the object determined to have gone outside the area does not return to the area, or if it returns to the area but its trajectory has changed.

[0068] (Method 1) A generation step of generating a 3D model representing the three-dimensional shape of an object located in a region where the imaging ranges of the multiple imaging devices overlap, using multiple moving images obtained by imaging an object with multiple imaging devices, If the object corresponding to the 3D model generated in the generation step goes outside the region, a calculation step is performed to calculate a first velocity vector when the object crosses the boundary of the region. A first estimation step in which the position of the object after it has left the region is estimated based on the first velocity vector calculated in the calculation step and the position of the object when it crosses the boundary of the region, A second estimation step involves estimating a 3D model representing the three-dimensional shape of the object at the position estimated in the first estimation step, Includes, In the second estimation step, a copy of the 3D model generated in the generation step is associated with the estimated position. A method for generating products characterized by the following features.

[0069] (Composition 17) A program for causing a computer to function as a generating device as described in any one of configurations 1 to 15.

Claims

1. A generation means for generating a 3D model representing the three-dimensional shape of an object using multiple moving images obtained by imaging the object with multiple imaging devices, When the object corresponding to the 3D model generated by the generation means moves outside the region for which the 3D model of the object is to be generated, a calculation means calculates a first velocity vector when the object crosses the boundary of the region, A first estimation means for estimating the position of the object after it has left the region, based on the first velocity vector calculated by the calculation means and the position of the object when it crosses the boundary of the region, A second estimation means for estimating a 3D model representing the three-dimensional shape of the object at the position estimated by the first estimation means, It has, The second estimation means associates the 3D model generated by the generation means with the estimated position. A generating apparatus characterized by the following features.

2. The calculation means further calculates a first angular velocity vector when the object crosses the boundary of the region, The second estimation means determines the orientation of the 3D model based on the assumption that the object continues to rotate according to the first angular velocity vector calculated by the calculation means, and associates it with the estimated position. The generating apparatus according to feature 1.

3. The second estimation means described above is: Based on the first angular velocity vector calculated by the calculation means, the position estimated by the first estimation means is corrected. The 3D model generated by the generation means immediately before the object generated by the generation means leaves the region is associated with the corrected position. The generating apparatus according to feature 2.

4. The generation apparatus according to any one of claims 1 to 3, characterized in that the first estimation means estimates the position of the object during the period from when the object leaves the area until it returns to the area.

5. The calculation means further determines a second velocity vector when the object that has left the region returns to the region, The first estimation means estimates the position of the object based on the first velocity vector and the second velocity vector calculated by the calculation means. The generating apparatus according to feature 4.

6. The calculation means further calculates a second angular velocity vector when the object that has left the region returns to the region, The second estimation means described above is: Based on the second angular velocity vector calculated by the calculation means, the position estimated by the first estimation means is corrected. The 3D model corresponding to the point just before crossing the boundary generated by the generation means is associated with the corrected position. The generating apparatus according to feature 5.

7. The second estimation means described above is: Based on images of the object captured by one or more imaging devices, the position estimated by the first estimation means is corrected. The 3D model generated by the generation means immediately before the object generated by the generation means leaves the region is associated with the corrected position. The generating apparatus according to feature 1.

8. The second estimation means described above is: Based on the position of the object as seen in the image captured by the one or more imaging devices, candidate positions of the object in the virtual space are calculated. If there is a discrepancy between the calculated candidate position and the position estimated by the first estimation means, the estimated position is corrected to the closest position among the candidate positions. The generating apparatus according to feature 7.

9. The generation apparatus according to claim 8, characterized in that the second estimation means corrects the orientation of the 3D model associated with the corrected position to match the orientation of the object in an image of the object captured by the one or more imaging devices.

10. The system further includes a determination means for determining whether the object corresponding to the 3D model generated by the generation means has moved outside the region. The calculation means calculates the first velocity vector when the determination means determines that the object has moved outside the area. The generating apparatus according to feature 1.

11. The system further includes a storage means for storing the 3D model generated by the generation means, The determination means is, From the trajectory of the 3D model's position stored in chronological order in the storage means, the timing at which the 3D model crosses the boundary of the region is predicted. The system determines that the object has left the region when the predicted timing coincides with the timing at which the 3D model disappears from the storage means. The generating apparatus according to feature 10.

12. The generation means generates the 3D model in units of frames that constitute the plurality of moving images, The first estimation means estimates the position of the object after it has left the region, frame by frame, which constitutes the plurality of moving images. The second estimation means estimates a 3D model representing the three-dimensional shape of the object at each position estimated frame by the first estimation means. The generating apparatus according to feature 1.

13. The generation apparatus according to claim 1, characterized in that the second estimation means associates the 3D model generated by the generation means immediately before the object leaves the region.

14. The generation apparatus according to claim 1, further comprising an output means for outputting the 3D model generated by the generation means and the 3D model estimated by the second estimation means.

15. The generation apparatus according to claim 14, characterized in that the output means does not output the 3D model estimated by the second estimation means if the object determined to have gone outside the area does not return to the area, or if it returns to the area but its trajectory has changed.

16. A generation method performed by one or more processors, A generation step of generating a 3D model representing the three-dimensional shape of an object using multiple moving images obtained by capturing an object with multiple imaging devices, If the object corresponding to the 3D model generated in the generation step moves outside the region for which the 3D model of the object is to be generated, a calculation step is made to calculate a first velocity vector when the object crosses the boundary of the region. A first estimation step in which the position of the object after it has left the region is estimated based on the first velocity vector calculated in the calculation step and the position of the object when it crosses the boundary of the region, A second estimation step involves estimating a 3D model representing the three-dimensional shape of the object at the position estimated in the first estimation step, Includes, In the second estimation step, the 3D model generated in the generation step is associated with the estimated position. A method for generating products characterized by the following features.

17. A program for causing a computer to perform the generation method described in claim 16.

Images