Three-dimensional model generation device, generation method, and program

Abstract

To continuously generate a three-dimensional model even when an object goes out of a three-dimensional model generation area.SOLUTION: A generation device generates a three-dimensional model representing a three-dimensional shape of an object existing in an area where image capturing ranges of a plurality of imaging devices overlap using a plurality of moving images obtained by capturing the object by the plurality of imaging devices. When the object corresponding to the generated three-dimensional model goes out of the area, the generation device calculates a velocity vector when the object crosses a border of the area. Then, the generation device estimates a position of the object after the object goes out of the area on the basis of the calculated velocity vector and the position at the time of the object crossing the border of the area. Further, the generation device estimates a three-dimensional model representing a three-dimensional shape of the object at the estimated position.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a technique for generating a 3D model of a subject (object). [Background technology]

[0002] There is a technology that generates a virtual viewpoint image from an arbitrary virtual viewpoint using multi-viewpoint captured images obtained by synchronous imaging with multiple imaging devices. This virtual viewpoint image can be generated, for example, by generating three-dimensional shape data (3D model) of an object reflected in the multi-viewpoint captured images using a volume intersection method or the like, and performing rendering according to the virtual viewpoint. In this case, to generate a 3D model, the object needs to be reflected in each of the multi-viewpoint captured images.

[0003] Patent Document 1 describes a technique for preventing the shape of a 3D model of an object of interest from being distorted or its contours from being lost, for example, when the object of interest is hidden behind another object from the viewpoint of one of the imaging devices. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-101845 A Summary of the Invention [Problem to be solved by the invention]

[0005] For example, in ball sports such as baseball, a ball as an object may fly higher than expected and may go outside the generation area of ​​the 3D model (for example, the imaging range covered by multiple imaging devices). When the object goes outside the generation area of ​​the 3D model in this way, the technology of Patent Document 1 cannot handle it, and the generation of the 3D model stops. As a result, for example, an unnatural virtual viewpoint image is generated in which the ball that had been displayed until then suddenly disappears from the screen while viewing. [Means for solving the problem]

[0006] A generation device according to the present disclosure includes a generation means for generating a 3D model representing a three-dimensional shape of an object existing in an area where the imaging ranges of a plurality of imaging devices overlap, using a plurality of moving images obtained by capturing an object with the plurality of imaging devices; a calculation means for calculating a first velocity vector when the object corresponding to the 3D model generated by the generation means goes outside the area, when the object crosses a boundary of the area; a first estimation means for estimating a position of the object after it goes outside the area based on the first velocity vector calculated by the calculation means and a position of the object when it crosses the boundary of the area; and a second estimation means for estimating a 3D model representing the three-dimensional shape of the object at the position estimated by the first estimation means, wherein the second estimation means links a copy of the 3D model generated by the generation means to the estimated position. Effect of the Invention

[0007] According to the technology of the present disclosure, it is possible to continuously generate a 3D model even if an object falls outside the generation area of ​​the 3D model. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a virtual viewpoint image generating system. [Diagram 2]FIG. [Diagram 3] FIG. 2 is a diagram showing an example of the hardware configuration of various devices constituting the virtual viewpoint image generation system. [Figure 4] 11 is a flowchart showing the flow of a 3D model generation process in the front-end server. [Diagram 5] (a) and (b) are diagrams showing the timeline of when a ball hit by a batter leaves the 3D model generation area and then returns to the 3D model generation area. [Figure 6] 13A and 13B are explanatory diagrams of a third modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the present embodiment will be described with reference to the drawings. Note that the following embodiment does not necessarily limit the present invention. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0010] [Embodiment 1] 1 is a diagram showing an example of the configuration of a virtual viewpoint image generating system according to this embodiment. The camera group 101 is composed of multiple imaging devices (cameras), and all of the cameras are synchronized to capture moving images in order to generate data (generally called a "3D model") that represents a three-dimensional shape of a subject (object). Multiple moving images with different viewpoints obtained by the multiple cameras performing synchronized imaging are linked to a time code indicating the imaging time of each frame and transmitted to a front-end server 102. For convenience of explanation, in the following explanation, multiple moving images with different viewpoints obtained by the multiple cameras performing synchronized imaging are collectively referred to as "camera images."

[0011] The front-end server 102 includes a 3D model generation unit 110, a data buffer 111, a border crossing determination unit 112, a speed calculation unit 113, a position estimation unit 114, a 3D model estimation unit 115, and an output unit 116.

[0012] The 3D model generation unit 110 uses the camera images input from the camera group 101 to generate 3D models for moving objects such as people and balls captured in the camera images, for example, by a volume intersection method, on a frame-by-frame basis. FIG. 2 is a diagram for explaining the volume intersection method. A frustum extending into a three-dimensional space from the imaging center of the camera so as to pass through each point on the silhouette contour in the mask image (foreground silhouette image) is called the “view volume” of the camera. The three-dimensional shape of the object is reproduced by determining the common area of ​​the view volumes of the multiple cameras. In this embodiment, the three-dimensional shape is expressed by a collection of tiny cubes called voxels. The generated 3D model is linked to the time code of the corresponding frame, stored in the data buffer 111, and transmitted to the database 103.

[0013] The data buffer 111 stores the 3D models generated by the 3D model generation unit 110 in the chronological order of the time codes. The stored 3D models are used for various image processing such as border crossing determination, speed calculation, position estimation, and 3D model estimation, which will be described later. The data buffer 111 has a storage area large enough to store the 3D models with the necessary number of frames to perform various image processing.

[0014] The border crossing determination unit 112 determines whether an object corresponding to a 3D model stored in the data buffer 111 has crossed the boundary of a preset 3D model generation area and gone outside the area. The 3D model generation area is a three-dimensional area (common imaging range) where the imaging ranges of the multiple cameras constituting the camera group 101 overlap, and is set by an operator or the like after confirming that it is covered by a sufficient number of cameras to obtain a 3D model of a certain quality. If the result of the determination is that the object has gone outside the model generation area, the speed calculation unit 113 is notified of this fact together with the time code of the frame immediately before the object crossed the boundary (hereinafter referred to as "border crossing time information").

[0015] The speed calculation unit 113 calculates the speed vector when the object goes outside the 3D model generation area based on the border crossing time information notified from the border crossing determination unit 112. This speed vector is calculated by identifying two or more frames immediately before the border crossing with reference to the border crossing time information, and acquiring the 3D model corresponding to the two or more frames from the data buffer 111. The calculated speed vector is sent to the position estimation unit 114 together with the border crossing time information.

[0016] The position estimation unit 114 acquires position information associated with the 3D model immediately before crossing the boundary from the data buffer 111 based on the input border crossing time information, and estimates the object position outside the 3D model generation area based on the input velocity vector. In this embodiment, the position estimation unit 114 performs position estimation at a predetermined interval for the period from the time when the target object goes outside the 3D model generation area to the time when it returns to the 3D model generation area. The predetermined interval at this time is, for example, about every 16.6 ms when the frame rate when the camera group 101 performs synchronous imaging is 60 fps and the generation of the 3D model is performed every frame. Note that it is desirable to perform position estimation every frame. However, when the virtual viewpoint image is not displayed in real time, the user may set the interval for performing position estimation at any interval, such as once every several frames. Then, the estimated one or more object positions (estimated positions) are linked to the time code of the corresponding frame and sent to the 3D model estimation unit 115 together with the border crossing time information.

[0017] The 3D model estimation unit 115 estimates a 3D model of the object at each estimated position based on the result of position estimation input from the position estimation unit 114. This estimation can be realized by duplicating the 3D model corresponding to the time code indicated by the border crossing time information and arranging it at the estimated position (linking the copied 3D model to the time code of the frame corresponding to the estimated position). In this embodiment, the 3D model estimation unit 115 estimates the 3D model at a predetermined interval during the period from when the object leaves the 3D model generation area to when it returns to the 3D model generation area. The predetermined interval in this case is the same as the predetermined interval used for position estimation.

[0018] The output unit 116 outputs the 3D model generated by the 3D model generation unit 110 and the 3D model estimated by the 3D model estimation unit 115 to the database 103 by linking them with the time codes of the corresponding frames.

[0019] The database 103 stores the 3D models received from the front-end server 102 in association with the time codes of the corresponding frames.

[0020] The virtual viewpoint controller 104 is a control device for a user to set a virtual viewpoint by operating a virtual camera in an imaging space where the camera group 101 performs synchronous imaging. The virtual viewpoint controller is composed of a display, a joystick, a keyboard, and the like. For example, the user operates a joystick on a user interface screen (UI screen) that displays a CG space that reproduces the imaging space to specify camera parameters such as the position, attitude, and angle of view of the virtual camera. The camera parameters of the virtual camera specified by the user are linked to the corresponding time code and transmitted to the backend server 105 as virtual viewpoint information.

[0021] The back-end server 105 acquires a 3D model from the database 103 based on the virtual viewpoint information received from the virtual viewpoint controller 104, and generates a virtual viewpoint image according to the position, attitude, and angle of view of the specified virtual camera. The generated virtual viewpoint image is transmitted to the display 106 and displayed on the display 106. Note that the output destination of the virtual viewpoint image does not necessarily have to be a display. For example, the virtual viewpoint image may be transmitted to a distributor (not shown), and the distributor may publish it on the Web.

[0022] <Hardware configuration> Fig. 3 is a diagram showing an example of the hardware configuration of various devices constituting the virtual viewpoint image generation system. That is, the front-end server 102, the database 103, the virtual viewpoint controller 104, and the back-end server 105 can be realized by an information processing device 200 shown in Fig. 2. The information processing device 200 has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication I / F 207, and a system bus 208.

[0023] The CPU 201 realizes each function of the virtual viewpoint image generating system shown in Fig. 1 by controlling the entire information processing device 200 using computer programs and data stored in the ROM 202 and the RAM 203. Note that the information processing device 200 may have one or more pieces of dedicated hardware different from the CPU 201, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of such dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0024] The ROM 202 stores programs that do not require modification. The RAM 203 temporarily stores programs and data supplied from the auxiliary storage device 204, and data supplied from the outside via the communication I / F 207. The auxiliary storage device 204 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

[0025] The display unit 205 is composed of, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) etc. for the user to give instructions to the information processing device 200. The operation unit 206 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by the user to input various instructions to the CPU 201. The CPU 201 operates as a display control unit that controls the display unit 205, and as an operation control unit that controls the operation unit 206.

[0026] The communication I / F 207 is used for communication with devices external to the information processing device 200, such as the camera group 101 and a microphone group (not shown). When the information processing device 200 has a function of wireless communication with external devices, the communication I / F 207 includes an antenna. The system bus 208 connects each unit of the information processing device 200 to transmit information.

[0027] In the configuration example shown in FIG. 3, the display unit 205 and the operation unit 206 are assumed to exist inside the information processing device 200, but at least one of the display unit 205 and the operation unit 206 may be present as a separate device outside the information processing device 200.

[0028] <3D model generation process in the front-end server 102> Fig. 4 is a flowchart showing the flow of processing in which the front-end server 102 generates a 3D model of an object. As described above, even if the object goes outside the 3D model generation area, the front-end server 102 of this embodiment estimates the subsequent object position and generates a 3D model at the estimated position. The following description will be given along with the flow of Fig. 4. In the following description, the symbol "S" means step.

[0029] In S401, the 3D model generating unit 110 generates a 3D model of an object using a volume intersection method based on the camera image input from the camera group 101. Here, a 3D model for each object present in a preset 3D model generation area is generated for each frame constituting the camera image. Here, a specific example is shown. FIG. 5(a) is a diagram showing in time series the state of a ball hit by a batter 500 going out of the 3D model generation area until it returns to the 3D model generation area 501 when the event to be imaged is a baseball game. In FIG. 5(a), ball marks 510 to 525 show the trajectory of the ball immediately after the batter 500 hits it, and represent the position and direction of the ball in each frame. Now, a batter and a ball are present in the 3D model generation area 501, and 3D models for them will be generated. In addition, position information is attached to the generated 3D model. In this case, the position information is expressed in the world coordinate system and is given as three-dimensional coordinates (x, y, z) with the ground in the real space to be imaged as the X-axis and Y-axis, and the vertical direction as the Z-axis. The position information shown in the following Table 1 is assigned to each of the 3D models corresponding to the ball marks 510 to 512 and 522 to 525 in Fig. 5(a).

[0030] [Table 1]

[0031] In S402, the 3D model for each frame generated for each object in S401 is linked to the time code of the corresponding frame and stored in chronological order in the data buffer 111. In the example of FIG. 5(a) described above, at this point, 3D models for all frames are obtained for the batter, and 3D models corresponding to the ball marks 510 to 512 and 522 to 525 are obtained for the ball. Then, each 3D model is stored in the data buffer 111 linked to the time code of the corresponding frame.

[0032] In S403, the border crossing determination unit 112 determines whether each object for which a 3D model was generated in S401 has gone outside the 3D model generation area. The determination method is not particularly limited, but may be, for example, the following method. First, the trajectory of the three-dimensional position of the target object is obtained based on the 3D model of the target object stored in the data buffer 111 in chronological order, and the timing at which the target object crosses the boundary of the 3D model generation area is predicted from the obtained trajectory. If the predicted timing coincides with the timing at which the 3D model of the target object disappears from the data buffer 111, the target object is determined to have gone outside the 3D model generation area. Note that, when it is determined that only a part of the object exists within the 3D model generation area (when a 3D model representing a three-dimensional shape of a part of the object exists in the data buffer 111), it may be handled as follows, for example. That is, in this case, a 3D model with a part of the shape missing (partial 3D model) is generated. Therefore, when the partial 3D model can be identified as being related to the same object as the 3D model of the previous frame, it is not determined that the target object has gone outside the 3D model generation area. On the other hand, if it is not possible to identify that the partial 3D model is related to the same object as the 3D model of the previous frame, it is determined that the ball has gone outside the 3D model generation area. In the example of FIG. 5(a) described above, since the ball marks 510 to 512 are inside the 3D model generation area 501, the 3D models without any missing parts in the corresponding frames are stored in the data buffer 111. On the other hand, since the ball mark 513 is outside the 3D model generation area 501, the corresponding 3D model is not stored in the data buffer 111. Here, based on the trajectory of the positions of the ball marks 510 to 512, it is possible to predict that the timing at which the ball will cross the boundary of the 3D model generation area 501 is the timing at which the ball will arrive at the position of the ball mark 513. Then, since the 3D model of the ball disappears from the data buffer 111 at the predicted timing, it is determined that the ball has gone outside the 3D model generation area 501.In this case, the time codes of the frames immediately before the ball crosses the boundary, for example, the time codes of two frames corresponding to ball mark 512 and ball mark 511, are sent to speed calculation unit 113 and position estimation unit 114 as boundary crossing time information.

[0033] In S404, a distribution process is performed according to the result of the determination in S403. Specifically, if an object has gone outside the 3D model generation area, S405 is executed next. On the other hand, if no object has gone outside the 3D model generation area, this process is terminated.

[0034] In S405, the speed calculation unit 113 calculates the speed vector immediately before the target object crosses the boundary based on the trajectory of the target object's movement until crossing the boundary, based on the border crossing time information input from the border crossing determination unit 112. In the example of Fig. 5(a) described above, the speed vector when the ball is at the position of ball mark 512 is calculated as follows based on two frames of 3D models corresponding to ball marks 512 and 511. Now, the coordinates of the position indicated by ball mark 512 in the frame immediately before the ball crosses the boundary are (x1, y1, z1), and the coordinates of the position indicated by ball mark 511 in the frame immediately before that are (x2, y2, z2). Here, home base in Figure 5(a) is the origin (0,0,0), the line connecting home base and first base is the X-axis (the direction toward first base is the positive direction), the line connecting home base and third base is the Y-axis (the direction toward third base is the positive direction), and the vertical direction is the Z-axis (the direction toward the sky is the positive direction). In this case, the velocity vector v (m / s) to be calculated is expressed by the following formula. Velocity vector v = ((x1-x2) / Δt, (y1-y2) / Δt, (z1-z2) / Δt) Equation (1)

[0035] In the above formula (1), Δt represents the time it takes between frames. Thus, the velocity vector v of the ball mark 512 with the ball center as the origin is 512= (10, 10, 1.1). The method of calculating the trajectory of the object's movement is not particularly limited, but one example is a method of identifying the center of gravity of each 3D model from the three-dimensional shape represented by the 3D model, and using the trajectory of the identified center of gravity as the trajectory of the 3D model. The velocity vector calculated in this manner is input to the position estimation unit 114.

[0036] In S406, the position estimation unit 114 estimates the three-dimensional position of the object outside the 3D model generation area based on the border crossing time information input from the border crossing determination unit 112 and the velocity vector input from the velocity calculation unit 113. In the example of FIG. 5(a) described above, first, information on the three-dimensional position of the ball mark 512 is acquired from the 3D model linked to the time code indicated by the border crossing time information. Then, the acquired three-dimensional position (x, y, z)=(30, 30, 5.4) and the velocity vector v calculated in S405 are used to estimate the three-dimensional position of the object outside the 3D model generation area. 512 Based on the above-mentioned 3D model generation area, the ball position corresponding to each frame outside the 3D model generation area is estimated based on the above-mentioned 3D model generation area. In this way, in the example of FIG. 5(a), the estimated positions corresponding to the ball marks 513 to 521 are obtained as shown in Table 2 below.

[0037] [Table 2]

[0038] The method of position estimation is not particularly limited, but an example is a method of estimating using a quadratic curve calculated from the position of the ball mark 512 immediately before crossing the boundary, its velocity vector, and gravitational acceleration. Furthermore, estimation may be performed taking wind speed into consideration. Alternatively, machine learning may be performed using many actual results from similar scenes as training data, and the obtained trained model may be used to estimate the position. Each estimated position is associated with the time code of the corresponding frame and input to the 3D model estimation unit 115.

[0039] In S407, the 3D model estimation unit 115 estimates a 3D model at each estimated position for the target object based on the result of the position estimation input from the position estimation unit 114. Specifically, a process is performed in which a copy of the 3D model corresponding to the time code of the frame immediately before the boundary is crossed is placed at each estimated position. In the example of FIG. 5(a) described above, copies of the 3D model corresponding to the ball mark 512 are placed at all positions estimated for each frame in S406. In addition, in placing the 3D model at the estimated three-dimensional position, the center of gravity of the 3D model at the time immediately before the boundary is crossed is obtained, and the 3D model is placed so that the center of gravity thus obtained coincides with the estimated position. In this way, the 3D models corresponding to the ball marks 513 to 521 in FIG. 5(a) described above are obtained as the estimation results.

[0040] In S408, the output unit 116 outputs all the 3D models generated in S402 and all the 3D models estimated in S407, linking them to the time codes of the corresponding frames. At this time, the linking of the time codes to the estimated 3D models is performed based on the time code linked to the estimated position. In this embodiment, the time codes are transmitted to the database 103 and stored in the database 103.

[0041] The above is the content of the 3D model generation process in the front-end server 102. Note that, although the above description has been given taking an example in which camera images that have already been captured and stored are input and processed, images being captured may be input and processed sequentially in real time.

[0042] <Variation 1> If a 3D model is duplicated just before it goes out of the 3D model generation area and placed in the estimated position as it is, its orientation (posture) will be the same as that of the duplicated 3D model. In this case, there is no problem if the set virtual viewpoint position is far from the object, but if it is close to the object, it becomes a problem. For example, in the above-mentioned ball example, if the orientations of the 3D models placed at each estimated position are all the same, the ball that appears large on the virtual viewpoint image will be displayed without rotation, which will give the viewer a sense of incongruity. In particular, if the shape is not a perfect sphere, such as a rugby ball, the viewer's sense of incongruity will be even greater. In addition, since the ball rotates while flying, the influence of the air also changes, so the trajectory of the movement also changes depending on the angular velocity. Therefore, an angular velocity vector (a vector indicating the direction of the rotation axis with the absolute value of the angular velocity ω as its magnitude) may also be calculated based on the time code of the 3D model just before it goes out of the 3D model generation area, and the 3D model at the estimated position may be placed taking into account the orientation of the object.

[0043] In this modified example, the velocity calculation unit 113 calculates an angular velocity vector in addition to the velocity vector of the 3D model immediately before it goes out of the 3D model generation area. This calculation of the angular velocity vector is also performed based on two or more frames of the 3D model stored in the data buffer 111. In the example of FIG. 5(a) described above, the angular velocity vector ω512 when the ball goes out of the 3D model generation area (ball mark 512) is calculated based on the rotation trajectory of the ball marks 510 to 512. The method of calculating the angular velocity vector is not particularly limited, and for example, the center of gravity and feature points other than the center of gravity are extracted from the shape of the 3D model, and the angular velocity vector is calculated by finding the rotation angle of the feature points centered on the center of gravity. Now, if the angular velocity vector ω512 = (ωx, ωy, ωz), ωx is positive in the rotation direction when a right-handed screw is turned toward first base with the line connecting home base and first base as the rotation axis. Similarly, ωy is positive when the rotation direction is a right-handed screw rotated toward third base with the line connecting home base and third base as the axis of rotation. Similarly, ωz is positive when the rotation direction is a right-handed screw rotated toward the sky with the vertical direction as the axis of rotation. Note that the components of each velocity vector indicate angular velocity (rad / sec). In this case, depending on the relationship between the rotation speed of the object and the frame rate, there is a possibility that the rotation is calculated in the opposite direction to the actual rotation. Therefore, a determination means may be provided separately for determining whether the rotation direction of the object is correct based on a camera image captured at a frame rate higher than the camera image for generating the 3D model. The calculated angular velocity vector is input to the position estimation unit 114 together with the velocity vector.

[0044] Then, the position estimation unit 114 of this modified example estimates the position of the object based on the velocity vector and angular velocity vector input from the velocity calculation unit 113. Specifically, first, the 3D model immediately before crossing the boundary is read from the data buffer 111, and its position and orientation are acquired. Then, based on the acquired position of the 3D model and the input velocity vector and angular velocity vector, the positions of the ball (ball marks 513 to 521) outside the 3D model generation area are all estimated. At this time, the estimated position may be corrected by calculating the lift force calculated from the angular velocity vector for the quadratic curve calculated from the position, velocity vector, and gravitational acceleration when the ball (ball mark 512) goes out of the 3D model generation area. The position estimated in this way is input to the 3D model estimation unit 115.

[0045] The 3D model estimation unit 115 copies and places the 3D model at the input estimated position. At this time, the 3D model estimation unit 115 corrects the orientation of the copied 3D model based on the input angular velocity vector and places it at each estimated position. The method of correcting the orientation is not particularly limited, but for example, it is possible to determine and correct the orientation of the 3D model at each estimated position under the assumption that the 3D model continues to rotate according to the calculated angular velocity vector. As a result, as shown in FIG. 5(b), for example, a 3D model corresponding to ball mark 512 is copied and placed after correcting its orientation, and a 3D model corresponding to ball mark 513' to 521' is obtained. Compared with FIG. 5(a), it can be seen that the rotation of the ball is taken into consideration in FIG. 5(b).

[0046] As described above, it is possible to calculate not only the velocity vector but also the angular velocity vector, and correct the estimated position based on the angular velocity vector, or correct the orientation of the 3D model to be placed at the estimated position. This makes it possible to obtain a virtual viewpoint image that does not look unnatural even when the virtual viewpoint approaches an object such as a ball.

[0047] <Variation 2> In the above embodiment, the position estimation is performed using only the velocity vector when the object goes out of the 3D model generation area, but the present invention is not limited to this. When the object returns to the 3D model generation area, the velocity vector when the object returns may be further obtained, and the position estimation may be performed using both the velocity vectors when the object went out and when the object returned. In this case, the position estimation may be performed under the assumption that the velocity vector when the object goes out is Vout and the velocity vector when the object returns is Vin, and the velocity vector when the object returns is linear from Vout to Vin. Similarly, in the above-mentioned modified example 1, the angular velocity vector when the object returns to the 3D model generation area may be calculated and used for position estimation and estimation of a 3D model for the estimated position. That is, in the example of FIG. 5(b) described above, the estimated position corresponding to the ball marks 513' to 521' and the orientation of the 3D model to be placed may be corrected based on the velocity vector, angular velocity vector, position, and orientation of the ball (ball mark 522) when the ball returns to the area. Also, if the input camera image has already been captured and saved, the 3D model generated immediately after the object returns to the 3D model generation area may be duplicated and placed at the estimated position. In the above-mentioned examples of Figs. 5(a) and (b), the trajectory of the ball as an object after it leaves the 3D model generation area is a natural falling motion that draws a smooth parabola and returns to the area. However, it is not a necessary requirement for applying this embodiment that the object that has gone outside the 3D model generation area returns to the area by a natural falling motion. For example, this embodiment is applicable even when the ball has disappeared outside the play area or when it has hit an obstacle and changed its trajectory. In this case, for example, additional processing such as discarding the estimated 3D model is performed prior to S408, and the output unit 116 does not output the 3D model obtained by estimation. Whether or not the ball has disappeared outside the play area may be determined, for example, by whether or not the object has returned to the 3D model generation area within a certain period of time. Furthermore, whether or not the object has collided with an obstacle and changed its trajectory can be determined based on the three-dimensional position and orientation of the object at the time it returns to the 3D model generation area.For example, in the example of Fig. 5(a) described above, if the position or orientation of the ball when it returns to the 3D model generation area deviates by more than a preset threshold, it may be determined that the change in trajectory was due to hitting an obstacle, not due to air resistance or wind. Note that when an object disappears from the field or hits an obstacle and changes its trajectory, instead of discarding the estimated 3D model, a configuration may be adopted in which the user can select whether or not the 3D model is needed via a UI screen or the like.

[0048] <Modification 3> For an object existing within the angle of view of the camera, it is also possible to correct the estimated position and the orientation of the 3D model to be placed based on the captured image. The captured image in this case may be an image captured by any of the cameras constituting the camera group 101, or may be an image captured by another camera not included in the camera group 101. A description will be given with reference to (a) and (b) of FIG. 6. First, a candidate object position is obtained from the position and orientation of a camera corresponding to an arbitrary captured image. Here, the candidate object position is expressed by a straight line in a virtual space. Now, a captured image 600 shown in FIG. 6(a) is an image captured by a camera 610 shown in FIG. 6(b), and a ball indicated by a ball mark 601 is reflected in the captured image 600. In FIG. 6(b), a dotted straight line 611 extending from the camera 610 indicates a candidate ball position in the virtual space estimated based on the position of the ball mark 601. That is, the ball is present at any position on the straight line 611. If there is a deviation between the straight line representing the candidate object position thus obtained and the estimated position, the estimated position is corrected to the closest position on the straight line. In FIG. 6(b), a ball mark 612 indicates the estimated position and the orientation of the 3D model placed there, and it can be seen that the position is deviated from the straight line 611. In this case, the estimated position is corrected to the closest position on the straight line 611, and the orientation of the 3D model at the corrected estimated position as viewed from the camera 610 is corrected to match the orientation indicated by the ball mark 601. As a result, the position and orientation of the 3D model corresponding to the ball mark 612 are corrected to the position and orientation indicated by the ball mark 613.

[0049] In the above example, the correction is performed using one captured image. However, when two or more captured images obtained by capturing images using multiple cameras are used, the position of the object is identified from the positions and orientations of the multiple cameras corresponding to the captured images. Then, when there is a deviation between the identified object position and the estimated position, the position of the 3D model is corrected to the identified object position. Furthermore, the orientation of the 3D model at the estimated position after correction as viewed from each of the multiple cameras is corrected so as to match the orientation of the object in each captured image. In this way, although limited to the case where the target object exists within the angle of view of the camera, the estimated position and the orientation of the 3D model to be placed may be corrected based on the captured image.

[0050] As described above, in this embodiment, when an object goes outside the 3D model generation area, the position of the object outside the area is estimated based on the velocity vector immediately before the object goes outside the area, and the 3D model when the object was inside the area is duplicated and placed. This makes it possible to continuously supply 3D models even if the object goes outside the 3D model generation area, and prevents the object from suddenly disappearing from the virtual viewpoint image.

[0051] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0052] The disclosure of this embodiment also includes the following configurations and methods.

[0053] (Configuration 1) a generating means for generating a 3D model representing a three-dimensional shape of an object existing in an area where imaging ranges of a plurality of imaging devices overlap, using a plurality of moving images obtained by imaging the object with the plurality of imaging devices; a calculation means for calculating, when the object corresponding to the 3D model generated by the generation means goes outside the area, a first velocity vector when the object crosses a boundary of the area; a first estimation means for estimating a position of the object after it has gone outside the area based on the first velocity vector calculated by the calculation means and a position of the object when it crosses a boundary of the area; a second estimation means for estimating a 3D model representing a three-dimensional shape of the object at the position estimated by the first estimation means; having The second estimation means links a copy of the 3D model generated by the generation means to the estimated position. A generating device characterized by:

[0054] (Configuration 2) The calculation means further calculates a first angular velocity vector when the object crosses a boundary of the region; the second estimation means determines an orientation of the replicated 3D model based on an assumption that the object continues to rotate according to the first angular velocity vector calculated by the calculation means, and associates the orientation with the estimated position. 2. The generating device according to configuration 1.

[0055] (Configuration 3) The second estimation means includes: correcting the position estimated by the first estimation means based on the first angular velocity vector calculated by the calculation means; A copy of the 3D model generated by the generating means immediately before the object generated by the generating means moves out of the area is linked to the corrected position. 3. The generating device according to configuration 2.

[0056] (Configuration 4) 4. The generating device according to any one of configurations 1 to 3, wherein the first estimation means estimates a position of the object during a period from when the object leaves the area to when the object returns to the area.

[0057] (Configuration 5) The calculation means further calculates a second velocity vector when the object, which has gone outside the area, returns to within the area; the first estimation means estimates a position of the object based on the first velocity vector and the second velocity vector calculated by the calculation means. 5. The generating device according to configuration 4.

[0058] (Configuration 6) the calculation means further calculates a second angular velocity vector when the object, which has gone outside the region, returns to within the region; The second estimation means includes: correcting the position estimated by the first estimation means based on the second angular velocity vector calculated by the calculation means; The duplicated 3D model corresponding to the corrected position immediately before crossing the boundary generated by the generation means is linked to the corrected position. The generating device according to configuration 5.

[0059] (Configuration 7) The second estimation means includes: correcting the position estimated by the first estimation means based on images of the object captured by one or more imaging devices; A copy of the 3D model generated by the generating means immediately before the object generated by the generating means moves out of the area is linked to the corrected position. 2. The generating device according to configuration 1.

[0060] (Configuration 8) The second estimation means includes: calculating a candidate position of the object in a virtual space based on a position of the object shown in an image captured by the one or more imaging devices; when there is a deviation between the calculated position candidate and the position estimated by the first estimation means, correcting the estimated position to a closest position among the position candidates; 8. The generating device according to configuration 7.

[0061] (Configuration 9) The generating device according to configuration 8, wherein the second estimation means corrects an orientation of the 3D model associated with the corrected position so as to match an orientation of the object in an image captured by the one or more imaging devices.

[0062] (Configuration 10) The method further includes a determination unit that determines whether or not the object corresponding to the 3D model generated by the generation unit has gone outside the area, the calculation means calculates the first velocity vector when the determination means determines that the object has gone outside the area. 10. The generating device according to any one of configurations 1 to 9.

[0063] (Configuration 11) The apparatus further includes a storage unit for storing the 3D model generated by the generation unit, The determination means is predicting a timing when the 3D model will cross the boundary of the region from the trajectory of the position of the 3D model stored in the storage means in chronological order; determining that the object has gone outside the area when the predicted timing coincides with the timing at which the 3D model disappeared from the storage means; 11. The generating device according to configuration 10.

[0064] (Configuration 12) The generating means generates the 3D model for each frame constituting the plurality of images, The first estimation means estimates a position of the object after it leaves the region for each frame constituting the plurality of images; The second estimation means estimates a 3D model representing a three-dimensional shape of the object at each position estimated on a frame-by-frame basis by the first estimation means. 12. The generating device according to any one of configurations 1 to 11.

[0065] (Configuration 13) 13. The generating device according to any one of configurations 1 to 12, wherein the second estimation means links a copy of the 3D model generated by the generating means immediately before the object leaves the area.

[0066] (Configuration 14) 14. The generating device according to any one of configurations 1 to 13, further comprising an output means for outputting the 3D model generated by the generating means and the 3D model estimated by the second estimating means.

[0067] (Configuration 15) The generating device according to configuration 14, characterized in that the output means does not output the 3D model estimated by the second estimation means if the object determined to have gone outside the area does not return to the area, or if the object returns to the area but its trajectory has changed.

[0068] (Method 1) a generation step of generating a 3D model representing a three-dimensional shape of the object existing in an area where imaging ranges of the plurality of imaging devices overlap, using a plurality of moving images obtained by imaging the object with the plurality of imaging devices; a calculation step of calculating, when the object corresponding to the 3D model generated in the generation step goes outside the area, a first velocity vector when the object crosses a boundary of the area; a first estimation step of estimating a position of the object after it has gone outside the area based on the first velocity vector calculated in the calculation step and a position of the object when it crosses a boundary of the area; a second estimation step of estimating a 3D model representing a three-dimensional shape of the object at the position estimated in the first estimation step; Including, In the second estimation step, a copy of the 3D model generated in the generation step is linked to the estimated position. A generating method comprising:

[0069] (Configuration 17) A program for causing a computer to function as the generating device according to any one of configurations 1 to 15.

Claims

1. a generating means for generating a 3D model representing a three-dimensional shape of an object using a plurality of moving images obtained by capturing images of the object using a plurality of imaging devices; a calculation means for calculating, when the object corresponding to the 3D model generated by the generation means goes outside an area for which a 3D model of the object is to be generated, a first velocity vector when the object crosses a boundary of the area; a first estimation means for estimating a position of the object after it has left the area based on the first velocity vector calculated by the calculation means and a position of the object when it crosses the boundary of the area; a second estimation means for estimating a 3D model representing a three-dimensional shape of the object at the position estimated by the first estimation means; and the second estimation means associates the 3D model generated by the generation means with the estimated position; A generating device characterized by:

2. the calculation means further calculates a first angular velocity vector when the object crosses a boundary of the region; the second estimation means determines an orientation of the 3D model based on an assumption that the object continues to rotate according to the first angular velocity vector calculated by the calculation means, and associates the orientation with the estimated position. The generating device according to claim 1 .

3. The second estimation means correcting the position estimated by the first estimation means based on the first angular velocity vector calculated by the calculation means; Associating the corrected position with a 3D model generated by the generating means immediately before the object generated by the generating means moves out of the area. The generating device according to claim 2 .

4. 4. The generating device according to claim 1, wherein the first estimation means estimates the position of the object during a period from when the object leaves the area to when the object returns to the area.

5. the calculation means further calculates a second velocity vector when the object, which has gone out of the area, returns to the area; the first estimation means estimates a position of the object based on the first velocity vector and the second velocity vector calculated by the calculation means; The generating device according to claim 4 .

6. the calculation means further calculates a second angular velocity vector when the object, having gone out of the area, returns to the area; The second estimation means correcting the position estimated by the first estimation means based on the second angular velocity vector calculated by the calculation means; Associating the corrected position with the 3D model corresponding to the position immediately before crossing the boundary, which is generated by the generation means; The generating device according to claim 5 .

7. The second estimation means correcting the position estimated by the first estimation means based on images of the object captured by one or more imaging devices; Associating the corrected position with a 3D model generated by the generating means immediately before the object generated by the generating means moves out of the area. The generating device according to claim 1 .

8. The second estimation means calculating a candidate position of the object in a virtual space based on a position of the object shown in an image captured by the one or more imaging devices; when there is a deviation between the calculated position candidate and the position estimated by the first estimation means, correcting the estimated position to the closest position among the position candidates; The generating device according to claim 7 .

9. The generation device according to claim 8, wherein the second estimation means corrects an orientation of the 3D model associated with the corrected position so as to match an orientation of the object in an image captured by the one or more imaging devices.

10. The method further includes a determination unit that determines whether the object corresponding to the 3D model generated by the generation unit has gone outside the area, the calculation means calculates the first velocity vector when the determination means determines that the object has gone outside the area. The generating device according to claim 1 .

11. further comprising a storage means for storing the 3D model generated by the generation means; The determination means predicting a timing at which the 3D model will cross the boundary of the area from the trajectory of the position of the 3D model stored in the storage means in chronological order; determining that the object has gone outside the area when the predicted timing coincides with the timing when the 3D model disappeared from the storage means; The generating device according to claim 10 .

12. the generating means generates the 3D model for each frame constituting the plurality of images, the first estimation means estimates a position of the object after it has moved out of the area for each frame constituting the plurality of images; the second estimation means estimates a 3D model representing a three-dimensional shape of the object at each position estimated on a frame-by-frame basis by the first estimation means; The generating device according to claim 1 .

13. The generating device according to claim 1 , wherein the second estimating means associates the 3D model generated by the generating means immediately before the object moves out of the region.

14. The generating device according to claim 1 , further comprising an output unit that outputs the 3D model generated by the generating unit and the 3D model estimated by the second estimating unit.

15. 15. The generation device according to claim 14, wherein the output means does not output the 3D model estimated by the second estimation means if the object determined to have gone outside the area does not return to the area, or if the object returns to the area but its trajectory has changed.

16. a generation step of generating a 3D model representing a three-dimensional shape of the object using a plurality of moving images obtained by capturing images of the object using a plurality of imaging devices; a calculation step of calculating, when the object corresponding to the 3D model generated in the generation step goes outside an area targeted for generating a 3D model of the object, a first velocity vector when the object crosses a boundary of the area; a first estimation step of estimating a position of the object after it has left the area based on the first velocity vector calculated in the calculation step and a position of the object when it crosses a boundary of the area; a second estimation step of estimating a 3D model representing a three-dimensional shape of the object at the position estimated in the first estimation step; Including, In the second estimation step, the 3D model generated in the generation step is associated with the estimated position. A generating method characterized by:

17. A program for causing a computer to execute the generating method according to claim 16.

Images