Three-dimensional model generation device, generation method and storage medium

Abstract

The present invention provides a three-dimensional model generation device, generation method and storage medium. Even if there is a structure that blocks a part of the target object in the shooting scene, defects are prevented from occurring in the three-dimensional model to be generated. The three-dimensional model generation device includes: an acquisition component for acquiring a first mask image and a second mask image, the first mask image representing the area of ​​the structure as a stationary object in each image captured from multiple viewpoints, and the second mask image representing the area of ​​the foreground as a moving object in each image captured from multiple viewpoints; a synthesis component for synthesizing the acquired first mask image and the second mask image to generate a third mask image that integrates the area of ​​the structure and the area of ​​the foreground in the images captured from multiple viewpoints; and a generation component for using the third mask image to generate a three-dimensional model including the structure and the foreground by a view volume intersection method.

Description

[0001] (This application is a divisional application of an application filed on December 3, 2018, with application number 201880028645.2 and invention name “Device, method and program for generating three-dimensional model”.) Technical Field

[0002] The present invention relates to the generation of three-dimensional models of objects within images. Background Art

[0003] Conventionally, as a method for estimating the three-dimensional shape of an object by using multi-viewpoint images synchronously captured by multiple cameras from different viewpoints, a method called “Visual Hull” is known (Patent Document 1, Non-Patent Document 1). Figure 1 (a) to (c) of FIG. 1 are diagrams showing the basic principle of the view volume intersection method. From an image obtained by photographing a specific object, a mask image ( ) representing the two-dimensional outline of the object is obtained at the imaging plane. Figure 1 Then, consider a cone form ( Figure 1 (b)). This cone shape is called the target "visual volume" of the relevant camera. In addition, by finding the common area of ​​multiple visual volumes, that is, the intersection of the visual volumes, the three-dimensional shape (three-dimensional model) of the object is found ( Figure 1 As described above, in shape estimation by the view intersection method, the three-dimensional shape of the object is estimated by projecting sampling points in a space where the object may exist onto a mask image and verifying whether the projected points are included in the mask image common to multiple viewpoints.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2014-10805

[0007] Non-patent literature

[0008] Non-patent literature 1: Laurentini A: "The Visual Hull Concept for Silhouette-Based Image Understanding", IEEE Transcriptions Pattern Analysis and machine Intelligence, Vol. 16, No. 2, pp. 150-162, Feb, 1994 Summary of the invention

[0009] Problem that the invention aims to solve

[0010] In the above-mentioned view volume intersection method, the mask image needs to be able to correctly express the outline of the target object, and in the case where the outline on the mask image is incorrect, the generated three-dimensional shape is also incorrect. For example, in the case where a part of a person as the target object is blocked from being photographed by a stationary object such as a structure existing in front of the person, and therefore a part of the outline of the person represented by the mask image is missing, a defect occurs in the generated three-dimensional model. In addition, in the case where the mask image in which a part of the outline is missing is not used, the geometric accuracy of the obtained three-dimensional model is reduced. In particular, in the case where the part blocked from being photographed by the structure is relatively small, it is desirable to use as much as possible the mask image in which even a part of the outline is missing, because a three-dimensional model with high geometric accuracy can be obtained by using the mask image.

[0011] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to prevent defects from occurring in a generated three-dimensional model even when there is a structure or the like in an imaging scene that blocks a portion of a target object from being photographed.

[0012] Solutions for solving problems

[0013] A generating device according to the present invention includes: a first acquisition component for acquiring first area information, wherein the first area information represents an area of ​​an object within a plurality of images obtained by photographing from a plurality of photographing directions; a second acquisition component for acquiring second area information, wherein the second area information represents an area of ​​a structure that may block the object from being photographed when photographing from at least one of the plurality of photographing directions; and a generating component for generating three-dimensional shape data corresponding to the object based on both the first area information representing the area of ​​the object acquired by the first acquisition component and the second area information representing the area of ​​the structure acquired by the second acquisition component.

[0014] Effects of the Invention

[0015] According to the present invention, even when there is a structure or the like that blocks a portion of a target object from being photographed within an imaging scene, a high-quality three-dimensional model with no defects or with reduced defects can be generated.

[0016] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 (a) to (c) are diagrams showing the basic principle of the view volume intersection method;

[0018] Figure 2 (a) is a block diagram showing the structure of a virtual viewpoint image generation system, and Figure 2 (b) is a diagram showing an example of the configuration of each camera constituting the camera array;

[0019] Figure 3 is a functional block diagram showing the internal structure of the three-dimensional model generating device according to the first embodiment;

[0020] Figure 4 is a flowchart showing the flow of a three-dimensional model forming process according to the first embodiment;

[0021] Figure 5 (a) to (h) are diagrams each showing an example of an image captured by each camera;

[0022] Figure 6 (a) to (h) are each a diagram showing an example of a structure mask;

[0023] Figure 7 (a) to (h) are each a diagram showing an example of a foreground mask;

[0024] Figure 8 (a) to (h) are each a diagram showing an example of an integrated mask;

[0025] Fig. 9 is a diagram showing an example of an integrated three-dimensional model generated based on an integrated mask;

[0026] Fig.10 is a diagram showing an example of a three-dimensional model generated by a conventional method using only a foreground mask;

[0027] Fig.11 is a flowchart showing the flow of a three-dimensional model forming process according to the second embodiment;

[0028] Fig.12 (a) is a diagram showing an integrated three-dimensional model generated based on an integrated mask, Fig.12 (b) is a diagram showing a three-dimensional model of a structure generated based only on a structure mask, and Fig.12 (c) shows that Fig.12 (a) The integrated three-dimensional model and Fig.12 (b) a diagram of a three-dimensional model of only the foreground obtained by comparing the difference between the three-dimensional models of the structure;

[0029] Fig.13 is a diagram showing an example of voxels constituting a three-dimensional model according to the third embodiment;

[0030] Fig.14 is a diagram showing an example of a camera configuration of a virtual viewpoint image generating system according to a third embodiment;

[0031] Fig.15 is a diagram showing an example of a functional structure of a three-dimensional model generating apparatus according to a third embodiment;

[0032] Fig.16 is a flowchart showing the procedure of processing performed by the three-dimensional model generating apparatus according to the third embodiment;

[0033] Fig.17 is a diagram showing an example of captured images captured by a plurality of cameras according to the third embodiment;

[0034] Fig.18 is a diagram showing an example of a structure mask image according to the third embodiment;

[0035] Fig.19 is a diagram showing an example of a foreground mask image according to the third embodiment;

[0036] Fig. 20 is a diagram showing an example of an integrated mask image obtained by integrating a foreground mask image and a structure mask image according to the third embodiment;

[0037] Fig.21 is a diagram showing a voxel space as a generation target of a three-dimensional model of a gymnasium system according to a third embodiment;

[0038] Fig. 22 is a diagram showing true count (True Count) / false count (False Count) according to the third embodiment;

[0039] Fig.23 is a diagram showing an example of a three-dimensional model generated by threshold-based determination applying a false count according to the third embodiment;

[0040] Fig.24 is a diagram showing an example of a three-dimensional model generated by applying a threshold-based determination of a false count and a threshold-based determination of a true count according to the third embodiment;

[0041] Fig.25 is a diagram showing a three-dimensional model in the case where a defect occurs;

[0042] Fig.26 is a diagram showing an example of a functional structure of a three-dimensional model generating apparatus according to a fourth embodiment;

[0043] Fig. 27is a flowchart showing the procedure of processing performed by the three-dimensional model generating apparatus according to the fourth embodiment;

[0044] Fig.28 is a diagram showing an example of a camera configuration and a foreground of a virtual viewpoint image generating system according to a fourth embodiment;

[0045] Fig.29 is a diagram showing true / false counts according to a fourth embodiment;

[0046] Fig.30 is a diagram showing functional blocks of a three-dimensional model generating apparatus according to a fifth embodiment;

[0047] Fig.31 is a diagram showing a processing flow of a three-dimensional model generating apparatus according to a fifth embodiment;

[0048] Fig.32 is a diagram showing true / false counts of absence of weight addition and presence of weight addition according to the fifth embodiment;

[0049] Fig.33 is a diagram showing functional blocks of a three-dimensional model generating apparatus according to a sixth embodiment;

[0050] Fig.34 is a diagram showing a processing flow of a three-dimensional model generating apparatus according to a sixth embodiment; and

[0051] Fig.35 is a diagram showing pseudo count / construction according to the sixth embodiment. Specific embodiments

[0052] The present invention will be described in detail below based on embodiments with reference to the accompanying drawings. The structures shown in the following embodiments are merely exemplary, and the present invention is not limited to the schematically shown structures.

[0053] [First embodiment]

[0054] In this embodiment, the following aspect is described: by using, in addition to the two-dimensional outline of the foreground in the captured scene, a mask image including a two-dimensional outline of a structure that blocks at least a portion of it, a three-dimensional model is generated for the foreground that does not have defects or has a reduced degree of defects. In this aspect, a three-dimensional model including a structure that blocks a portion of the foreground is generated. In this specification, "foreground" refers to a moving object that moves (its absolute position can change) and can be viewed from a virtual viewpoint when the image is captured in a time series from the same angle and exists in the captured image. In addition, "structure" refers to a stationary object that does not move (its absolute position does not change, that is, it is stationary) and may block the foreground from being captured and exists in the captured image when the image is captured in a time series from the same angle. The three-dimensional model mentioned here is data representing a three-dimensional shape.

[0055] In the following description, it is assumed that when a virtual viewpoint image is generated by taking a soccer match as a camera scene, a part of a foreground (moving object) such as a player and a ball is blocked from being photographed by a structure (stationary object) such as a soccer goal. A virtual viewpoint image is a video image that is generated by freely manipulating the position and posture of a virtual camera by an end user and / or a designated operator, and is also referred to as a free viewpoint image and an arbitrary viewpoint image. In addition, the generated virtual viewpoint image and the multi-viewpoint image that is the source of the virtual viewpoint image may be a moving image or a still image. In each of the embodiments described below, a case where a three-dimensional model of a virtual viewpoint image for generating a moving image is generated by using a multi-viewpoint image of a moving image is described as an example.

[0056] In this embodiment, the description is given assuming that a football match is used as a shooting scene and a fixedly installed football goal is a structure, but this is not restrictive. For example, a corner flag can also be processed as a structure, and in the case of an indoor studio as a shooting scene, furniture or props can also be processed as a structure. That is, any stationary object can be processed as a structure as long as its stationary state or a state close to the stationary state continues.

[0057] (System Structure)

[0058] Figure 2(a) is a block diagram showing an example of the structure of a virtual viewpoint image generation system including a three-dimensional model generation device according to the present embodiment. The virtual viewpoint image generation system 100 includes a camera array 110 including a plurality of cameras, a control device 120, a foreground separation device 130, a three-dimensional model generation device 140, and a rendering device 150. The control device 120, the foreground separation device 130, the three-dimensional model generation device 140, and the rendering device 150 are each implemented by a general-purpose computer (information processing device) including a CPU for performing calculation processing, or a memory storing the results of calculation processing and programs, etc.

[0059] Figure 2 (b) is a diagram showing the configuration of a total of eight cameras 211 to 218 constituting the camera array 110 in a bird's-eye view when the field 200 is viewed from directly above. The cameras 211 to 218 are each installed at a predetermined height from the ground in a manner surrounding the field 200, and multi-view image data from different viewpoints is acquired by photographing the position in front of one of the goals from various angles. On the artificial turf field 200, a football field 201 is drawn (actually, with white lines), and on the left side of the football field 201, a football goal 202 is placed. In addition, the × mark 203 in front of the football goal 202 indicates the common line of sight direction (point of gaze) of the cameras 211 to 218, and the dotted circle 204 indicates the area where the cameras 211 to 218 can each photograph with the point of gaze 203 as the center. In this embodiment, the position is expressed in a coordinate system with one corner of the field 200 as the origin, the long side direction as the x-axis, the short side direction as the y-axis, and the height direction as the z-axis. The data of the multi-viewpoint image obtained by each camera of the camera array 110 is sent to the control device 120 and the foreground separation device 130. Figure 2 In (a), the cameras 211 to 218, the control device 120, and the foreground separation device 130 are connected in a star topology, but a ring topology or a bus topology connected in a daisy chain may also be used. Figure 2 In (b), an example in which the number of cameras is eight is described, but the number of cameras may be less than eight or more than eight.

[0060] The control device 120 generates camera parameters and a structure mask, and supplies both to the three-dimensional model generation device 140. The camera parameters include external parameters indicating the position and posture (line of sight direction) of each camera, and internal parameters indicating the focal length and angle of view (photographing area) of the lens included in each camera, and are obtained by calibration. Calibration is a process for finding the correspondence between a point in a three-dimensional world coordinate system obtained by using a plurality of images of a specific pattern such as a checkerboard and a two-dimensional point corresponding to the point. The structure mask is a mask image indicating the two-dimensional contour of a structure existing in each captured image acquired by each of the cameras 211 to 218. The mask image is a reference image that specifies which is the extraction target part in the captured image, and is a binary image represented by 0 and 1. In the present embodiment, the soccer goal 202 is processed as a structure, and the contour image indicating the area (two-dimensional contour) of the soccer goal 202 in the image captured by each camera from a predetermined position and at a predetermined angle is the structure mask. As a captured image used as a source of a structure mask, an image captured at a timing when a player or the like as a foreground does not exist (such as a timing before or after a game or a timing during halftime, etc.) can be used. However, there are cases where images captured before or after are not suitable because the video is affected by changes in daylight (for example, outdoors). In cases such as this case, for example, a captured image as a source of a structure mask can also be obtained by excluding the player or the like from a predetermined number of frames (for example, continuous frames corresponding to ten seconds) of a motion image in which the player or the like is captured. In this case, the structure mask can be obtained based on an image that uses the median value of each pixel value in each frame.

[0061] The foreground separation device 130 performs the following processing: for each captured image input from multiple viewpoints, the foreground corresponding to the players and the ball on the field 200 is determined to be distinguished from the background area other than the foreground area. Regarding the determination of the foreground area, a pre-prepared background image is used (which can be the same as the captured image as the source of the structure mask). Specifically, the difference with the background image is calculated for each captured image, and the area corresponding to the difference is designated as the foreground area. Due to this, a foreground mask representing the foreground area of ​​each captured image is generated. In the present embodiment, the following binary image is generated as a foreground mask, and the binary image uses "0" to represent pixels belonging to the foreground area representing the players and the ball, and uses "1" to represent pixels belonging to the background area other than the foreground.

[0062] The three-dimensional model generating device 140 generates a three-dimensional model of the object based on the camera parameters and the multi-viewpoint images. The details of the three-dimensional model generating device 140 will be described later. The data of the generated three-dimensional model is output to the rendering device 150.

[0063] The rendering device 150 generates a virtual viewpoint image based on the three-dimensional model received from the three-dimensional model generating device 140, the camera parameters received from the control device 120, the foreground image received from the foreground separation device 130, and the background image prepared in advance. Specifically, the positional relationship between the foreground image and the three-dimensional model is obtained according to the camera parameters, and a virtual viewpoint image is generated in the case where the object of interest is viewed from an arbitrary angle by mapping the foreground image corresponding to the three-dimensional model. In this way, for example, a virtual viewpoint image of a decisive scene in front of a goal where a player has scored can be obtained.

[0064] Figure 2 The structure of the virtual viewpoint image generation system shown in (a) is an example, and the structure is not limited to this. For example, one computer may include the functions of a plurality of devices (e.g., the foreground separation device 130 and the three-dimensional model generation device 140, etc.). Alternatively, a structure may be designed so that the module of each camera has the function of the foreground separation device 130, and the data of the captured image and its foreground mask are supplied from each camera.

[0065] (3D model generation device)

[0066] Figure 3 3D model generating device 140 according to the present embodiment is a functional block diagram showing the internal structure. 3D model generating device 140 includes a data receiving unit 310, a structure mask storage unit 320, a mask integration unit 330, a coordinate conversion unit 340, a 3D model generating unit 350 and a data output unit 360. Each unit is described in detail below.

[0067] The data receiving unit 310 receives the camera parameters of each camera constituting the camera array 110 and the structure mask representing the two-dimensional outline of the structure existing in the imaging scene from the control device 120. In addition, the data receiving unit 310 receives the captured images (multi-viewpoint images) obtained by each camera of the camera array 110 and the data of the foreground mask representing the two-dimensional outline of the foreground existing in each captured image from the foreground separation device 130. Among the received data, the structure mask is transmitted to the structure mask storage unit 320, the foreground mask is transmitted to the mask integration unit 330, the multi-viewpoint image is transmitted to the coordinate conversion unit 340, and the camera parameters are transmitted to the coordinate conversion unit 340 and the three-dimensional model generation unit 350.

[0068] The structure mask storage unit 320 stores the structure mask in a RAM or the like, and supplies the structure mask to the mask integration unit 330 as necessary.

[0069] The mask synthesis unit 330 reads the structure mask from the structure mask storage unit 320 and synthesizes the structure mask with the foreground mask received from the data receiving unit 310, thereby generating a mask image (hereinafter referred to as "integrated mask") that integrates the two masks into one mask. The generated integrated mask is sent to the three-dimensional model generation unit 350.

[0070] The coordinate conversion unit 340 converts the multi-viewpoint images received from the data receiving unit 310 from the camera coordinate system to the world coordinate system based on the camera parameters. Through this coordinate conversion, each captured image with different viewpoints is converted into information indicating which region in the three-dimensional space each captured image shows.

[0071] The 3D model generation unit 350 generates a 3D model of an object including a structure in the camera scene by using the multi-viewpoint image converted to the world coordinate system and the integration mask corresponding to each camera using the view volume intersection method. The data of the generated 3D model of the object is output to the rendering device 150 via the data output unit 360.

[0072] (Processing of forming a three-dimensional model)

[0073] Figure 4 1 is a flowchart showing the flow of the three-dimensional model forming process according to the present embodiment. This series of processes is realized by the CPU included in the three-dimensional model generating device 140 loading a predetermined program stored in a storage medium such as a ROM and a HDD onto a RAM and executing the program. Figure 4 The process is described below.

[0074] First, in step 401 , the data receiving unit 310 receives from the control device 120 a structure mask representing a two-dimensional outline of a structure (here, the soccer goal 202 ) when the structure is viewed from each of the cameras 211 to 218 and camera parameters of each camera. Figure 5 (a) to (h) of FIG. 1 show images captured by cameras 211 to 218 constituting camera array 110. Here, there is a player (goalkeeper) in front of soccer goal 202 on soccer field 201. Figure 5 In each of the captured images (a), (b), and (h), the soccer goal 202 exists between the camera and the player, so a part of the player is hidden by the soccer goal 202. Figure 5 For each captured image of (a) to (h), a structure mask is obtained, in which an area is represented by two values, such as the area of ​​the soccer goal 202 is 1 (white) and the area other than the area of ​​the soccer goal 202 is 0 (black). Figure 6 (a) to (h) show the Figure 5 Structural object masks corresponding to each captured image of (a) to (h).

[0075] Next, in step 402, data receiving unit 310 receives from foreground separation device 130 a foreground mask representing a two-dimensional outline of the foreground (here, the player and the ball) in the images captured by each of cameras 211 to 218, and a multi-viewpoint image serving as a source of the foreground mask. Figure 7 (a) to (h) show the Figure 5 The foreground separation device 130 extracts the area that changes over time between the images captured at the same angle as the foreground, so Figure 7 In each of (a), (b), and (h), a region of a part of the player hidden by the soccer goal 202 is not extracted as a foreground region. The received foreground mask data is sent to the mask integration unit 330.

[0076] Next, in step 403, the mask integration unit 330 performs processing for reading the data of the structure object mask from the structure object mask storage unit 320 and synthesizing the read structure object mask with the foreground mask received from the data receiving unit 310. This synthesis is a calculation process for obtaining a logical sum (OR) for each pixel of the foreground mask and the structure object mask, both of which are represented by two values ​​(white and black). Figure 8 (a) to (h) each show that Figure 6 The structural masks shown in (a) to (h) and Figure 7 The integrated mask is obtained by synthesizing the foreground masks shown in (a) to (h) of FIG. In the completed integrated mask, no defects are visible in the outline of the player.

[0077] Then, in step 404, the three-dimensional model generation unit 350 generates a three-dimensional model by using a view volume intersection method based on the integration mask obtained in step 403. As a result, a model (hereinafter referred to as an "integrated three-dimensional model") representing the three-dimensional shape of the foreground and the structure existing in the common camera area of ​​the plurality of images captured from different viewpoints is generated. In the case of the present embodiment, an integrated three-dimensional model including the soccer goal 202 in addition to the players and the ball is generated. The generation of the integrated three-dimensional model is specifically performed by the following process. First, volume data of the three-dimensional space on the field 200 filled with cubes (voxels) of a predetermined size is prepared. The values ​​of the voxels constituting the volume data are represented by 0 and 1, respectively, and "1" represents an area that contributes to the formation of the shape and "0" represents an area that does not contribute to the formation of the shape. Next, by using the camera parameters (installation position and sight line direction, etc.) of each camera 211 to 218, the three-dimensional coordinates of the voxels are converted from the world coordinate system to the camera coordinate system. Then, when the structure and foreground represented by the integrated mask exist in the camera coordinate system, a model of the three-dimensional shape of the structure and foreground is generated by voxels. The three-dimensional shape can also be represented by a collection of points (point cloud) representing the center of the voxels instead of the voxels themselves. Fig. 9 Shown based on Figure 8 The integrated three-dimensional model generated by the integrated mask shown in (a) to (h) of FIG. 1 , and symbol 901 corresponds to the three-dimensional shape of the player as the foreground, and symbol 902 corresponds to the three-dimensional shape of the soccer goal 202 as the structure. As described above, in the integrated mask, there is no defect in the outline of the player as the foreground, and therefore there is no defect in the completed integrated three-dimensional model. Fig.10 FIG. 3 shows a 3D model generated by a conventional method using only a foreground mask. As described above, Figure 7 In the foreground masks shown in (a), (b) and (h) of FIG. 1 , a part of the player is not represented as the foreground area, so the part becomes a defect in the generated 3D model. The method of this embodiment uses a mask image synthesized by the foreground mask and the structure mask, so that defects can be avoided in a part of the 3D model of the foreground.

[0078] The above is the content of the three-dimensional model formation processing according to the present embodiment. In the case of generating a virtual viewpoint image of a motion image, a three-dimensional model of each frame is generated by repeating the processing of the above steps in units of frames. However, it is only necessary to receive and store the structure mask only immediately after the start of the process (step 401), and this operation can be omitted for the second frame and subsequent frames. In addition, in the case of shooting at the same shooting position by changing the date, it is also possible to receive and store the structure mask only for the first time and store the structure mask in RAM, etc., and use the stored structure mask next time and thereafter.

[0079] As described above, according to the present embodiment, even in the case where there is a structure hiding an object as the foreground, a highly accurate three-dimensional model can be generated in which there is no defect in the foreground or the degree of the defect is reduced.

[0080] [Second embodiment]

[0081] In the first embodiment, a three-dimensional model including a structure existing in the shooting scene and a foreground without defects or with reduced degrees of defects is generated. Next, the following aspect is described as a second embodiment: a three-dimensional model with only a foreground from which structures are removed and without defects or with reduced degrees of defects is generated. The description of the contents common to the contents of the first embodiment such as the system structure is omitted or simplified, and the following mainly describes the differences.

[0082] The structure of the three-dimensional model generating device 140 of this embodiment is also similar to the structure of the three-dimensional model generating device 140 of the first embodiment (see Figure 3 ) are basically the same, but differ in the following points.

[0083] First, the structure mask stored in the structure mask storage unit 320 is read not only by the mask integration unit 330 but also by the three-dimensional model generation unit 350 . Figure 3 This is indicated by the dotted arrow in . Then, in addition to generating an integrated 3D model of the foreground and the structure using the integration mask, the 3D model of only the structure using the structure mask is generated in the 3D model generating unit 350. Then, by finding the difference between the integrated 3D model generated based on the integration mask and the 3D model of the structure generated based on the structure mask, a 3D model of only the foreground without defects or with reduced defects is extracted.

[0084] (Processing of forming a three-dimensional model)

[0085] Fig.11 1 is a flowchart showing the flow of the three-dimensional model forming process according to the present embodiment. This series of processes is realized by the CPU included in the three-dimensional model generating device 140 loading a predetermined program stored in a storage medium such as a ROM and a HDD onto a RAM and executing the program. Fig.11 The process is described below.

[0086] Steps 1101 to 1104 correspond to the steps of the first embodiment. Figure 4 , and there is no difference between steps 401 to 404 in the process, so the description is omitted.

[0087] In the subsequent step 1105 , the three-dimensional model generating unit 350 reads the structure mask from the structure mask storing unit 320 , and generates a three-dimensional model of the structure by a volume intersection method.

[0088] Next, in step S1106, the three-dimensional model generation unit 350 obtains the difference between the composite three-dimensional model of the foreground and the structure generated in step S1104 and the three-dimensional model of the structure generated in step S1105, and extracts the three-dimensional model of only the foreground. Here, the difference with the integrated three-dimensional model can also be obtained after the three-dimensional model of the structure is enlarged by, for example, about 10% in the three-dimensional space. Due to this, the part corresponding to the structure can be reliably removed from the three-dimensional integrated model. At this time, only a part of the three-dimensional model of the structure can also be enlarged. For example, the part to be enlarged according to the area can also be determined in the following manner: in the case of the soccer goal 202, since the possibility that the player is present in the soccer field 201 is high, one side of the field 201 is not enlarged and only the opposite side of the field 201 is enlarged. In addition, the enlargement ratio (enlargement rate) can also be changed according to how far the object (such as the player and the ball) as the foreground is from the structure. For example, in the case where the object as the foreground is located far from the structure, the enlargement rate is increased so that the three-dimensional model of the structure is reliably removed. In addition, by reducing the magnification rate when the foreground object is located close to the structure, a part of the three-dimensional model of the foreground is prevented from being mistakenly removed. The magnification rate at this time can also be linearly changed according to the distance from the foreground, or the magnification rate can be determined step by step by providing one or more distances as a reference.

[0089] Fig.12 (a) shows the same as described above Fig. 9 The integrated three-dimensional model generated based on the integrated mask is the same as the integrated three-dimensional model in. Fig.12 (b) shows a three-dimensional model of the structure generated based only on the structure mask. Then, Fig.12 (c) shows that Fig.12 (a) The integrated three-dimensional model and Fig.12 (b) The difference between the 3D models of the structured object and the 3D model of only the foreground is obtained.

[0090] The above is the content of the three-dimensional model formation processing according to the present embodiment. In the case of generating a virtual viewpoint image of a moving image, a three-dimensional model of each frame is generated by repeating the processing of the above steps in units of frames. However, it is only necessary to receive and store the structure mask (step 1101) and generate the three-dimensional model of the structure (step 1105) only immediately after the start of the process, and this operation can be omitted for the second frame and subsequent frames. In addition, in the case of shooting at the same shooting position by changing the date, it is also possible to receive and store the structure mask and generate the three-dimensional model of the structure only for the first time and store them in RAM, etc., and use them next time and thereafter. As described above, according to the present embodiment, even in the case of a structure that hides an object as the foreground, a three-dimensional model of only the foreground that does not include the structure and is highly accurate can be generated.

[0091] [Third embodiment]

[0092] In the first and second embodiments, a foreground-only three-dimensional model is generated by subtracting the three-dimensional model of the structure from the integrated three-dimensional model of the foreground and the structure. Next, the following aspect is described as a third embodiment: for each voxel constituting the integrated three-dimensional model of the foreground and the structure (or for each predetermined area), the voxel is counted in which mask image the voxel is included, and the portion whose count value is less than or equal to a threshold value is removed from the integrated three-dimensional model to obtain a foreground-only three-dimensional model.

[0093] In this embodiment, first, for each of the plurality of partial regions constituting the three-dimensional space, it is determined whether the following condition is satisfied: whether the number of cameras in the foreground area of ​​the region of the target object represented in the captured images of the plurality of cameras is less than or equal to a first threshold. As the first threshold, an arbitrary value less than the total number of cameras is set by taking into account the installation position and sight direction of each camera, etc. Then, a three-dimensional model of the target object including the partial region that is not determined to satisfy the condition is generated.

[0094] (Method of expressing three-dimensional models)

[0095] Fig.13 (a) shows a single cubic voxel. Fig.13 (b) shows a set of voxels representing the target space for generating a three-dimensional model. Fig.13 As shown in (b), voxels are tiny regions that make up a three-dimensional space. Then, Fig.13 (c) shows an example in which voxels in a region other than the quadrangular pyramid region are removed from the voxel set as the target space. Fig.13The set in (b) is removed to generate a set of voxels of a three-dimensional model of a quadrangular pyramid. In this embodiment, an example in which a three-dimensional space and a three-dimensional model are composed of cubic voxels is described, but this is not restrictive, and the three-dimensional space and the three-dimensional model can be composed of a point cloud, etc.

[0096] (System Structure)

[0097] A block diagram showing a configuration example of a virtual viewpoint image generation system including a three-dimensional model generation device according to the present embodiment Figure 2 The block diagram is the same as that shown in (a), so the description is omitted.

[0098] The camera array 110 is an imaging device group including a plurality of cameras 110a to 110z, and captures a subject from various angles and outputs images to the foreground separation device 130 and the control device 120. It is assumed that the cameras 110a to 110z, the foreground separation device 130, and the control device 120 are connected in a star topology, but may be connected in a ring or bus type topology using a daisy chain connection. The camera array 110 may be, for example, Fig.14 As shown, the cameras are arranged around the stadium and synchronously shoot from various angles toward a gaze point on the field common to all cameras. However, multiple gaze points may also be set, such as a gaze point that half of the cameras included in the camera array 110 face and another gaze point that the remaining half of the cameras face.

[0099] Here, the foreground is a predetermined target object (a subject that is a target for generating a three-dimensional model based on a captured image) that can be viewed from any angle at a virtual viewpoint, and in the present embodiment, refers to a person existing on the field of a gymnasium. On the other hand, the background is an area other than the foreground, and in the present embodiment, refers to the entire gymnasium (field and spectator stands, etc.). However, the foreground and background are not limited to these examples. In addition, it is assumed that the virtual viewpoint image in the present embodiment includes all images representing the appearance from a virtual viewpoint where no camera is installed, rather than just images representing the appearance from a freely specifyable viewpoint.

[0100] The control device 120 calculates the camera parameters representing the positions and postures of the cameras 110a to 110z based on the images synchronously captured by the camera array 110, and outputs the calculated camera parameters to the three-dimensional model generation device 140. Here, the camera parameters include external parameters and internal parameters. The external parameters include a rotation matrix and a translation matrix, and represent the position and posture of the camera. The internal parameters include information related to the focal length and optical center of the camera, and represent the viewing angle of the camera and the size of the camera sensor, etc.

[0101] The process of calculating the camera parameters is called calibration. The camera parameters can be found by using the correspondence between points in a three-dimensional world coordinate system (which are acquired by using multiple images obtained by photographing a specific pattern such as a checkerboard) and two-dimensional points corresponding thereto.

[0102] The control device 120 calculates a structure mask image representing a structure area that may overlap in front of the foreground in the images captured by the cameras 110a to 110z, and outputs information related to the calculated structure mask image. In this embodiment, the structure is a stationary object installed in the imaging target space, and as an example, a soccer goal is handled as a structure, and an image representing the area of ​​the goal in the images captured by each camera is a structure mask image.

[0103] The foreground separation device 130 identifies the area where the person on the field exists as the foreground and the background area other than the foreground from the images captured by the multiple cameras input from the camera array 110, and outputs a foreground mask image representing the foreground area. As a method of identifying the foreground area, a method of identifying an area where there is a difference between a pre-stored background image and a captured image as a foreground image, or a method of identifying an area of ​​a moving object as a foreground area can be used.

[0104] Here, the mask image is a reference image representing a specific portion that is desired to be extracted from the captured image, and is a binary image represented by 0 and 1. For example, the foreground mask image represents an area in the captured image where a foreground such as a player exists, and is an image in which pixels indicating the foreground area are represented by 1 and pixels other than the foreground are represented by 0 at the same resolution as the captured area. However, the format of the mask image is not limited thereto, and may be any information representing an area of ​​a specific object within the captured image.

[0105] The three-dimensional model generating device 140 has a function as an information processing device that generates a three-dimensional model by using a plurality of captured images captured by a plurality of cameras. First, the three-dimensional model generating device 140 receives camera parameters and information related to a structure mask image from the control device 120, and receives a foreground mask image from the foreground separation device 130. Then, the three-dimensional model generating device 140 generates an integrated mask image representing an integrated area by integrating the structure mask image and the foreground mask image. In addition, the three-dimensional model generating device 140 determines whether to remove each voxel based on the number of cameras that each voxel (actually, a point corresponding to a voxel, and this applies hereinafter) in the space as a generation target of the three-dimensional model of the foreground is not included in the integrated mask image, and the number of cameras that each voxel is included in the foreground mask image. Then, based on the remaining voxels after removing the voxels determined to be removed, a three-dimensional model of the foreground is generated by, for example, a view volume intersection method and the three-dimensional model is output to the rendering device 150.

[0106] The rendering device 150 receives the three-dimensional model from the three-dimensional model generating device 140, and receives the image representing the foreground from the foreground separating device 130. In addition, the rendering device 150 calculates the positional relationship between the image representing the foreground and the three-dimensional model according to the camera parameters, and pastes the foreground image corresponding to the three-dimensional model for coloring, thereby generating a virtual viewpoint image when the three-dimensional model is observed from an arbitrary viewpoint. The virtual viewpoint image may include an image of the background. That is, the rendering device 150 may also generate a virtual viewpoint image when the background and foreground are viewed from the set viewpoint by setting the model of the background, the model of the foreground, and the position of the viewpoint in the three-dimensional space.

[0107] (Functional structure of three-dimensional model generation device)

[0108] Next, refer to Fig.15 The functional structure of the three-dimensional model generating device according to the present embodiment is described below. The three-dimensional model generating device 140 includes a receiving unit 155, a structure mask storage unit 101, a camera parameter storage unit 102, a mask integration unit 103, a coordinate conversion unit 104, a mask inside / outside judgment unit 105, a threshold setting unit 106, a foreground model generating unit 107, and an output unit 108.

[0109] The receiving unit 155 receives the camera parameters of each camera constituting the camera array 110 and the structure mask image representing the area of ​​the structure from the control device 120. In addition, the receiving unit 155 receives the image captured by each camera of the camera array 110 and the foreground mask image representing the foreground area in the image from the foreground separation device 130 at each time of shooting.

[0110] The structure object mask storage unit 101 stores the structure object mask image received by the receiving unit 155. The structure object mask image is a fixed image corresponding to the position of the camera.

[0111] The camera parameter storage unit 102 stores, as camera parameters, external parameters indicating the position and / or posture of each camera captured by the camera array 110 and internal parameters indicating the focal length and / or image size.

[0112] The mask integration unit 103 generates an integrated mask image by integrating the foreground mask image received from the foreground separation device 130 each time the camera array 110 performs imaging with the structure mask image stored in the structure mask storage unit 101. Details of the method of integrating the foreground mask image and the structure mask image will be described later.

[0113] The coordinate conversion unit 104 calculates the position and viewing angle of each captured image in the world coordinate system based on the camera parameters stored in the camera parameter storage unit 102 , and converts the two into information indicating which captured area on the three-dimensional space each captured image shows.

[0114] The mask inside / outside judgment unit 105 judges to remove the voxel when the number of cameras included in the foreground mask image for each voxel in the target voxel space is less than or equal to a threshold value. In addition, the mask inside / outside judgment unit 105 judges to remove the voxel when the number of cameras not included in the integrated mask image for each voxel in the target voxel space is greater than or equal to another threshold value.

[0115] The threshold setting unit 106 sets each threshold value for determining whether to remove a voxel using the mask inside / outside determination unit 105. The threshold value may be set according to a user operation for the three-dimensional model generation device 140, or may be automatically set by the threshold setting unit 106. The foreground model generation unit 107 removes the voxels determined to be removed by the mask inside / outside determination unit 105 from the voxels in the target voxel space, and generates a three-dimensional model based on the remaining voxels. The output unit 108 outputs the three-dimensional model generated by the foreground model generation unit 107 to the rendering device 150.

[0116] Fig.16 : is a flowchart showing the procedure of processing performed by the three-dimensional model generating device according to the present embodiment.

[0117] In S1601, the receiving unit 155 receives the structure object mask image of each camera constituting the camera array 110 from the control device 120. Here, an example of the captured image and the structure object mask image is described. Fig.171 shows an example of five captured images captured by five cameras constituting a part of the camera array 110. Here, a person exists on the field and a goal exists on the field as a structure, and Fig.17 In (b), (c) and (d), the goal as a structure exists in front of the character, so a part of the character is hidden. Fig.18 Shown with Fig.17 The structure mask image corresponding to each captured image is shown. The structure mask image is shown as a binary image in which the area of ​​the goal as the structure is 1 (white) and the area other than the structure is 0 (black).

[0118] In S1602, the receiving unit 155 receives a foreground mask image representing the foreground area from the foreground separation device 130. Here, an example of the foreground mask image is described. Fig.19 Shown with Fig.17 The foreground mask image corresponding to each captured image shown in FIG. 1 is a foreground mask image corresponding to each captured image shown in FIG. 1 . The foreground separation device 130 extracts the area that changes over time as the foreground area. Fig.19 As in (b), (c) and (d) of FIG. 1 , a part of the area of ​​the person hidden by the goal is not extracted as the foreground area. Fig.19 In (e), a portion of the person's legs that does not change over time is not extracted as the foreground area.

[0119] In S1603 , the mask integration unit 103 generates an integrated mask image by integrating the structure mask image and the foreground mask image received in S1601 and S1602 . Fig. 20 Shown as Fig.18 The structure mask image and Fig.19 An example of an integrated mask image as a result of integrating the foreground mask image shown in FIG. The integrated mask image is calculated by OR (logical sum) of the foreground mask image and the structure mask image, each of which is represented by two values.

[0120] In S1604 , the mask inside / outside judgment unit 105 selects one voxel that has not been selected yet from the target voxel space.

[0121] In S1605 , the mask inside / outside determination unit 105 counts the number of cameras whose selected one voxel is not included in the mask region of the integrated mask image of each camera (hereinafter referred to as a false count).

[0122] In S1606, the mask inside / outside judgment unit 105 judges whether the false count is greater than or equal to the threshold. In the case where the false count is greater than or equal to the threshold, it can be judged that the selected one voxel is neither the foreground nor the structure, so the process proceeds to S1607. Due to this, many voxels that are obviously non-foreground can be removed. On the other hand, in the case where the false count is less than the threshold, it can be judged that the selected one voxel is the foreground or the structure, so the process proceeds to S1608.

[0123] In S1607, the foreground model generation unit 107 removes the selected one voxel from the target voxel space. In S1608, the mask inside / outside determination unit 105 counts the number of cameras in which the selected one voxel is included in the mask area of ​​the foreground mask image of each camera (hereinafter referred to as True Count).

[0124] In S1609, the mask inside / outside judgment unit 105 judges whether the true count is less than or equal to other thresholds. If the true count is less than or equal to other thresholds, it can be judged that the selected one voxel is a structure, so the process proceeds to S1607 and the selected one voxel is removed from the target voxel space. On the other hand, if the true count exceeds other thresholds, it can be judged that the selected one voxel is a foreground, so the selected one voxel is not removed from the target voxel space.

[0125] In S1610, the mask inside / outside judgment unit 105 judges whether the processing has been completed for all voxels in the target voxel space. In the case where the processing has been completed for all voxels, the process proceeds to S1611. On the other hand, in the case where the processing has not been completed for all voxels, the process returns to S1604, and the next voxel is selected from the voxels that have not been selected, and the same process is performed thereafter.

[0126] In S1611 , the foreground model generation unit 107 generates a three-dimensional model of the foreground by using remaining voxels after performing the voxel removal determination on the target voxel space.

[0127] In S1612, the output unit 108 outputs the three-dimensional model of the foreground generated by the foreground model generation unit 107 to the rendering device 150. The above series of processes are performed on each frame captured by each camera.

[0128] Here, by using Fig.14 The virtual viewpoint image generation system of a stadium photographed by 16 cameras is shown as an example to explain the generation example of a three-dimensional model. Fig.21 is a diagram showing a voxel space as a three-dimensional model generation target of the gymnasium system according to the present embodiment, and a rectangular parallelepiped region shown by a mesh represents the target voxel space.

[0129] Fig. 22 The following are shown for the person, the person's legs, the person's head, the goal and other areas (which are used as Fig.14 The foreground, foreground not detected by some cameras, foreground hidden by structures, structures and non-foreground voxels (false count / true count) and judgment results are shown in the case where 16 cameras are used to capture the stadium. Here, it is assumed that one camera fails in foreground extraction of the legs of the person, the head of the person is hidden by the goal as a structure in three cameras, and the legs and head of the person are not extracted as the foreground by the foreground separation device 130.

[0130] In the case where the threshold of the false count is a fixed value of 10 in the judgment of S1606, the false count of the voxel located in the other area is 16 and exceeds the threshold, so the voxel is removed. As a result of this operation, for example, Fig.23 The three-dimensional model shown includes the foreground and the structure. Here, Fig.23 is a diagram showing an example of a three-dimensional model generated by threshold-based determination applying a false count.

[0131] Furthermore, in the case where the threshold value (other threshold value) of the true count in the judgment of S1609 is a fixed value of 5, the true count of the voxel located in the area of ​​the goal as a structure is 0 and is less than or equal to the threshold value, so the voxel is removed. On the other hand, the true counts of the voxels located in the areas of the person, the legs of the person, and the head are 16, 15, and 13, respectively, which exceed the second threshold value, so these voxels are not removed.

[0132] That is to say, Fig. 22 As shown in FIG. 1 , the foreground (people), a part of the foreground that is not detected (legs), and the foreground hidden by the structure (head) are judged as voxels to be retained, and the structure (goal) and non-foreground (other areas) are judged as voxels to be removed. Fig.21 The voxel set of the target space shown in FIG. 1 is finally generated as shown in FIG. Fig.24 The three-dimensional model of the character shown here does not have any defects. Here, Fig.24 is a diagram showing an example of a three-dimensional model generated by applying the threshold-based determination of false counts and the threshold-based determination of true counts.

[0133] In contrast, Fig.25 Shown only using Fig.19 The foreground mask image shown is an example of generating a 3D model using the volume intersection method. Fig.19 In (a), the entire person is photographed, but in Fig.19In the captured images shown in (b), (c) and (d) of FIG. 1 , a portion of the person's head is hidden by the goal as a structure. Fig.19 In the captured image shown in (e), the legs of the person are not extracted as the foreground. As a result, a part of the generated three-dimensional model also becomes defective.

[0134] As described above, in the present embodiment, for each voxel in the target space for generating a three-dimensional model of a target object (foreground), it is determined whether the number of cameras in which the target voxel is included in a foreground mask image of an area representing the foreground is less than or equal to a threshold value (threshold value of true count), and if the number is less than or equal to the threshold value, the voxel is removed.

[0135] According to the present embodiment, even when there are defects in the foreground mask image representing the area of ​​the target object (foreground), the defects in the three-dimensional model to be generated of the target object (foreground) can be avoided and the quality of the three-dimensional model can be improved.

[0136] In addition, an integrated mask image is generated by integrating the foreground mask image and the structure mask image, and when the number of cameras in which the target voxel is not included in the integrated mask image is greater than or equal to a threshold value (false count), it is determined that the voxel is removed. Due to this, many voxels that are obviously non-foreground can be removed, thereby making it possible to increase the speed of subsequent processing.

[0137] [Fourth embodiment]

[0138] In the third embodiment described above, it is not determined whether the voxel is within the imaging range (angle of view) of each camera. Therefore, if the voxel is outside the imaging range of many cameras, there is a possibility that the voxel representing the foreground is mistakenly removed. Fig.14 In the case of the camera configuration shown shooting a stadium, the number of cameras included in the imaging range for voxels in the area of ​​a person near the goal on the opposite side of the gaze point is three, and therefore the true count is 3. At this time, in the case where the threshold value of the true count is 5, the voxel is removed because the true count is less than the threshold value. Therefore, the following aspect is described as a fourth embodiment: a three-dimensional model is generated by setting a threshold value based on the result of the inside / outside judgment of the viewing angle so that the foreground located far from the viewing point is not removed. In the present embodiment, by calculating the threshold value of the true count based on the number of cameras included in the imaging range (viewing angle) of the voxel, erroneous removal of voxels representing the foreground is avoided even in the case where the voxel is far from the viewing point.

[0139] (Functional structure of three-dimensional model generation device)

[0140] refer to Fig.2610. The functional structure of the three-dimensional model generating device according to the present embodiment is described. The three-dimensional model generating device 140 according to the present embodiment includes a viewing angle inside / outside judging unit 109 and a threshold calculating unit 260 in addition to the receiving unit 155, the structure mask storage unit 101, the camera parameter storage unit 102, the mask integrating unit 103, the coordinate conversion unit 104, the mask inside / outside judging unit 105, the threshold setting unit 106, the foreground model generating unit 107 and the output unit 108. The basic structure of the virtual viewpoint image generating system is the same as that of the first to third embodiments, and thus the description thereof is omitted. In addition, the receiving unit 155, the structure mask storage unit 101, the camera parameter storage unit 102, the mask integrating unit 103, the coordinate conversion unit 104, the mask inside / outside judging unit 105, the threshold setting unit 106, the foreground model generating unit 107 and the output unit 108 constituting the three-dimensional model generating device 140 are the same as those of the third embodiment, and thus the description thereof is omitted.

[0141] The inside / outside viewing angle determination unit 109 determines whether each voxel in the target voxel space is within the imaging range of each camera based on the camera parameters of each camera.

[0142] The threshold value calculation unit 260 calculates a value obtained by multiplying the number of cameras determined to be within the imaging range of each voxel by a predetermined ratio as a threshold value of the true count. For example, in a case where the number of cameras within the imaging range of a particular voxel is 5 and the predetermined ratio is 60%, the threshold value of the true count for the voxel is calculated to be 3. The threshold value calculated by the threshold value calculation unit 260 is output to the threshold value setting unit 106, and the threshold value setting unit 106 sets the threshold value input from the threshold value calculation unit 260 as the threshold value of the true count.

[0143] When the number of cameras within the imaging range of a particular voxel is less than a predetermined number, it is considered that the accuracy of the three-dimensional model to be generated is reduced and no processing is required. Therefore, a structure can also be designed to set the threshold to a predetermined value when the number of such cameras is less than a predetermined number.

[0144] Fig. 27 1 is a flowchart showing the process performed by the three-dimensional model generation device according to this embodiment. Fig.16 The processing of S1601 to S1604 in the process is the same, so the description is omitted.

[0145] In S2705 , the inside / outside angle of view determination unit 109 determines whether the one voxel selected in S2704 is included in the angle of view of each camera based on the camera parameters of each camera.

[0146] In S2706, the mask inside / outside judgment unit 105 counts the number of cameras (hereinafter referred to as pseudo count) for which a selected voxel is not included in the mask area of ​​the integrated mask image of each camera but the selected voxel is included in the field of view.

[0147] The processing of S2707 to S2709 is the same as that described above. Fig.16 The processing of S1606 to S1608 in the process is the same, so the description is omitted.

[0148] In S2710 , the threshold value calculation unit 260 calculates a threshold value of the true count based on the number of cameras whose selected one voxel is included in the field of view. The threshold value setting unit 106 sets the threshold value of the true count calculated by the threshold value calculation unit 260 .

[0149] The processing of S2711 to S2714 is the same as that described above. Fig.16 The processing of S1609 to S1612 in the process is the same, so the description is omitted. Fig. 27 A series of processing in the process.

[0150] here, Fig.28 Shown as Fig.14 The method of using 16 cameras to shoot a stadium including foreground A and foreground B, wherein foreground A is composed of Fig.28 , and foreground B is represented by a black dot at a position close to the gaze point indicated by the × mark in , and foreground A is represented by a black dot at a position far from the gaze point. It is assumed that foreground A is within the field of view of all 16 cameras, and foreground B is within the field of view of only three cameras 110k, 110l and 110m.

[0151] also, Fig.29 Shown in Fig.28 An example of false counts / true counts for voxels at foreground A locations close to the gaze point and voxels at foreground B locations far from the gaze point in the camera configuration shown. Assume that the threshold for false counts is a fixed value of 10, and the threshold for true counts is 70% of the number of cameras whose voxels are included in the field of view.

[0152] The voxels located at the foreground A close to the fixation point are included in the integrated mask images of all 16 cameras, and there is no camera whose voxels are outside the integrated mask image. Therefore, the number of cameras whose voxels are outside the integrated mask image but within the view angle is 0, so the false count is 0.

[0153] Furthermore, the number of cameras in which the voxel at the foreground A located close to the gaze point is included in the field of view is also 16, and therefore the threshold value of the true count is 11.2, which is 70% of 16. Then, the voxel at the foreground A located close to the gaze point is within the foreground mask images of all cameras, and the true count is 16 and the count value is greater than or equal to the threshold value (11.2), and therefore the voxel is not removed.

[0154] The voxel at the position of the foreground B far from the gaze point is outside the view angle of 13 cameras (13 cameras other than cameras 110k, 110l and 110m) and within the view angle of three cameras (cameras 110k, 110l and 110m). In addition, the voxel is within the integrated mask image of the three cameras (cameras 110k, 110l and 110m). Therefore, the number of cameras where the voxel is outside the integrated mask image but the voxel is within the view angle is 0, and therefore the false count is 0.

[0155] Furthermore, the number of cameras in which the voxel located at the foreground B far from the gaze point is included in the field of view is three, and thus the threshold value of the true count is 2.1, which is 70% of 3. Then, the voxel located at the foreground B far from the gaze point is within the foreground mask images of the three cameras, and the true count is 3 and the count value is greater than or equal to the threshold value (2.1), and thus the voxel is not removed.

[0156] As described above, by setting the threshold of the true count based on the number of cameras in which the target voxel is included in the field of view, a three-dimensional model can be generated for a foreground that is far from the gaze point and in which the number of cameras in which the target voxel is included in the field of view is small. Therefore, even for a foreground that is far from the gaze point, a three-dimensional model that suppresses the degree of defects can be generated.

[0157] [Fifth embodiment]

[0158] In the third and fourth embodiments described above, an aspect of counting only the camera counts in which the voxels are included in the foreground mask image as the true counts of each voxel is described. However, in this case, since the true counts do not exceed the threshold value, voxels located at the position of the foreground hidden by the structure in many cameras can be removed. Therefore, the following aspect is described as a fifth embodiment: even in the case where the foreground is blocked from shooting by the structure in many cameras, a defect-free three-dimensional model is generated.

[0159] In this embodiment, even if the target voxel is outside the foreground mask image, the voxel may be the foreground under the condition that the target voxel is included in the structure mask image, so the foreground defect is avoided by adding the value obtained by multiplying the number of cameras judged as the voxel included in the structure mask image by the weight value to the true count.

[0160] Specifically, first, a weight value is set based on the number of cameras in which the target voxel is included in the structure mask image. Then, when the sum of the number of cameras in which the target voxel is included in the foreground mask image and the value obtained by multiplying the number of cameras in which the target voxel is included in the structure mask image by the weight value is less than or equal to the threshold value of the true count, it is determined that the voxel is removed.

[0161] (Functional structure of three-dimensional model generation device)

[0162] refer to Fig.30 The functional structure of the three-dimensional model generating apparatus according to the present embodiment is described below. The three-dimensional model generating apparatus 140 according to the present embodiment includes a weight setting unit 300 in addition to the structure of the three-dimensional model generating apparatus according to the fourth embodiment.

[0163] The weight setting unit 300 sets the value added to the true count when the target voxel is judged to be within the structure mask image as the weight value of each camera. The weight value is equivalent to a value indicating the possibility that the voxel is located in the foreground, and in this embodiment, the weight value of each camera is set to 0.5. Then, a value obtained by multiplying the number of cameras in which the target voxel is judged to be within the structure mask image by 0.5, which is the weight value of each camera, is added to the true count.

[0164] Fig.31 : is a flowchart showing the procedure of processing performed by the three-dimensional model generating device according to the present embodiment.

[0165] The processing of S3101 to S3104 is similar to that of the fourth embodiment. Fig. 27 The processing of S2701 to S2704 in the process of is the same. In addition, the processing of S3105 to S3108 is the same as that described above. Fig. 27 The processing of S2706 to S2709 in the process of is the same. In addition, the processing of S3109 and S3110 is the same as that described above. Fig. 27 The processing of S2705 and S2710 in the process is the same.

[0166] In S3111 , the mask inside / outside determination unit 105 counts the number of cameras in which the selected one voxel is included in the mask region of the structure object mask image of each camera.

[0167] In S3112, the weight setting unit 300 adds the value obtained by multiplying the number of cameras in which the selected voxel is included in the mask area of ​​the structure mask image by 0.5, which is the weight value of each camera, to the true count calculated in S3108. Fig. 27The processes of S2711 to S2714 in the process are the same. Fig.31 A series of processing in the process.

[0168] here, Fig.32 An example of a true count in the absence of weight addition and an example of a true count in the presence of weight addition according to the present embodiment are shown for voxels located in a specific foreground area.

[0169] Assume that the voxel is within the field of view of all 16 cameras, the number of cameras in which the target voxel is included in the foreground mask image is 7, and the number of cameras in which the target voxel is included in the structure mask image is 9. In this case, the number of cameras in which the voxel is outside the integrated mask image is 0 (the total number of cameras is 16-7-9). Therefore, the number of cameras in which the voxel is outside the integrated mask image but the voxel is within the field of view is 0, so the false count is 0.

[0170] In the absence of weight addition, the number of cameras in which the target voxel is included in the foreground mask image is 7, so the true count is 7. Assume that the threshold of the true count is 70% of the number of cameras in which the target voxel is included in the field of view. Then, the threshold is 11.2 (16×0.7), and the true count (7) < the threshold (11.2) and the true count is less than or equal to the threshold, so the voxel is removed.

[0171] On the other hand, in the case of weight addition, the number of cameras in which the target voxel is included in the foreground mask image is 7, so the true count is also 7 and the weight value is added to the true count. The number of cameras in which the target voxel is included in the structure mask image is 9 and the weight value of each camera is 0.5, so 9×0.5=4.5 is added as the weight value. The true count after adding the weight value is 11.5, and the true count (11.5)>threshold (11.2) and the true count exceeds the threshold, so the voxel is regarded as the foreground and is not removed.

[0172] In the present embodiment, it is assumed that there is a case where there is one structure, but in the case where there are multiple different structures that may overlap with the foreground, different weight values ​​may be set for each class of the structure mask image, and the value based on the weight value may be added to the true count. For example, for the structure mask image of the electronic sign installed as a sports field surrounding the gymnasium, the electronic sign is large and may overlap with the foreground, so the possibility of including the foreground becomes high, so the weight value of each camera is set to 0.5. In addition, for the structure mask image of the goal, the weight value of each camera is set to 0.3. It is believed that since the electronic sign is larger than the goal and there is no gap in the electronic sign, the possibility of the electronic sign overlapping with the foreground (person) is higher than that of the goal, so the weight value for the electronic sign is set to a value larger than the weight value for the goal.

[0173] In addition, different weight values ​​may be set according to the voxel position, scene, size and shape of the mask area, and the area of ​​the target stadium being photographed.

[0174] As described above, in this embodiment, after adding the weight based on the number of cameras whose target voxels are included in the mask area of ​​the structure mask image to the true count, the judgment based on the threshold is performed. Due to this, even in the case where the foreground is blocked by the structure in many cameras, the generation of a defect-free three-dimensional model can be achieved.

[0175] As described above, according to the first to fifth embodiments, even in the case where there is a structure hiding an object as the foreground, a highly accurate three-dimensional model of only the foreground excluding the structure can be generated.

[0176] [Sixth embodiment]

[0177] Next, the following aspect is described as a sixth embodiment: instead of the number of cameras (true count) in which the target voxel is included in the foreground mask image used in the third embodiment, the number of cameras in which the target voxel is included in the structure mask image is used. In the third embodiment, for the three-dimensional model generated based on the foreground mask image and the structure mask image, the foreground mask image is updated each time and it is determined whether the voxel constituting the three-dimensional model is included in the foreground mask image, so there is a case where the processing is complicated. Therefore, for the three-dimensional model generated based on the foreground mask image and the structure mask image, the three-dimensional model of the foreground that does not include the structure is generated by counting the number of cameras in which the target voxel is included in the fixed structure mask image.

[0178] (Functional structure and hardware structure of three-dimensional model generation device)

[0179] Fig.33is a diagram showing the structure of the three-dimensional model generating device 140 in the present embodiment. The structure of the three-dimensional model generating device 140 in the present embodiment is basically the same as that of the third embodiment, and therefore the description of the blocks that perform the same processing is omitted. The three-dimensional model generating device 140 according to the present embodiment includes a mask inside / outside judgment unit 3300 instead of the mask inside / outside judgment unit 105. The mask inside / outside judgment unit 3300 counts the number of cameras in which each voxel in the target voxel space is included in the integrated mask image and in the area of ​​the structure mask image, determines whether to remove the target voxel through a threshold-based judgment, and outputs the judgment result to the foreground model generating unit 107. The hardware structure of the three-dimensional model generating device 140 in the present embodiment is similar to that of the third embodiment. Fig.15 The hardware structure is the same as in , so the description is omitted.

[0180] Fig.34 The flowchart shows the process performed by the three-dimensional model generating device 140 in this embodiment. The processes of S3401 to S3407 and S3410 to S3412 are the same as those in the third embodiment. Fig.16 The processes of S1601 to S1607 and S1610 to S1612 are the same, so the description is omitted and the necessary parts are mainly described.

[0181] In S3406, the mask inside / outside determination unit 3300 determines whether the false count is greater than or equal to a threshold value. If the false count is less than the threshold value, it can be determined that the selected voxel is a foreground or a structure, and the process proceeds to S3408.

[0182] In S3408 , the mask inside / outside determination unit 3300 counts the number of cameras whose pixels or regions corresponding to the selected one voxel are included in the mask region of the structure object mask image of each camera (hereinafter referred to as structure object count).

[0183] In S3409, the mask inside / outside judgment unit 3300 judges whether the structure count is greater than or equal to the threshold. If the structure count is greater than or equal to the threshold, it can be judged that the selected voxel is a structure, so the process proceeds to S3407 and the selected voxel is removed from the target voxel space. On the other hand, if the structure count is less than the threshold, it can be judged that the selected voxel is a foreground, so the selected voxel is not removed from the target voxel space.

[0184] Here, by using Fig.14 The virtual viewpoint image generation system of a stadium photographed by 16 cameras is shown as an example to explain the generation example of a three-dimensional model. Fig.35 The following diagrams are shown for the person, the person's legs, the person's head, the goal and other areas (which are respectively Fig.14 1 and 2. An example of the false count / structure count of voxels and the judgment result of the foreground, foreground not detected by some cameras, foreground hidden by structures, structures and non-foreground in the virtual viewpoint image generation system shown. Here, it is assumed that one camera fails in foreground extraction of the legs of the person, the head of the person is hidden by the goal as a structure in three cameras, and the legs and head of the person are not extracted as the foreground by the foreground separation device 130.

[0185] In the case where the threshold of the false count is a fixed value of 10 in the judgment of S3404, the false count of the voxel located in the area other than the person, legs, head, and goal posts as structures is 16 and exceeds the threshold, so the voxel is removed. The three-dimensional model generated by applying the threshold-based judgment of the false count is different from the above-mentioned Fig.23 The three-dimensional models shown are the same.

[0186] Furthermore, in the case where the threshold of the structure count in the judgment shown in S3408 is a fixed value of 3, the structure count of the voxel located in the area of ​​the goal as a structure is 5 and is greater than or equal to the threshold, so the voxel is removed. On the other hand, the structure count of each voxel located in the area of ​​the person, the legs and the head of the person is 0 and less than the threshold, so these voxels are not removed. Therefore, the above-mentioned structure count is generated. Fig.24 A 3D model of a flawless character is shown.

[0187] Through the above processing, by threshold-based determination of the number of cameras (structure count) whose target voxel is included in the structure mask, generation of a flawless three-dimensional model can be achieved even when the foreground is blocked from shooting by a structure.

[0188] (Other embodiments)

[0189] The present invention can also be implemented by the following process: a program that implements one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or storage medium, and one or more processors in a computer of the system or device read and execute the program. In addition, the present invention can also be implemented by a circuit (e.g., ASIC) that implements one or more functions.

[0190] The present invention has been described so far with reference to the embodiments, but it is needless to say that the present invention is not limited to the above-described embodiments. The scope of the appended claims is consistent with the broadest description to include all such modifications, equivalent structures and functions. The present invention is not limited to the above-described embodiments, and various changes and modifications may exist without departing from the spirit and scope of the present invention. Therefore, in order to disclose the scope of the present invention, the appended claims are added.

[0191] This application claims the benefit of Japanese Patent Application No. 2017-239891 filed on December 14, 2017, Japanese Patent Application No. 2018-089467 filed on May 7, 2018, and Japanese Patent Application No. 2018-209196 filed on November 6, 2018, which are hereby incorporated by reference herein in their entirety.

Claims

1. A generating device for generating three-dimensional shape data corresponding to an object, the generating device comprising: an acquisition unit for acquiring image data representing a region of the object within a plurality of captured images obtained by a plurality of imaging devices for capturing images from a plurality of imaging directions, and image data representing a region of a structure that blocks the object when the image is captured from at least one of the plurality of imaging directions; a designation means for designating, with respect to an element forming a three-dimensional space for generating three-dimensional shape data corresponding to the object, the number of imaging devices for obtaining a captured image including pixels or an area corresponding to the element in an area of ​​the object; as well as A generating component is used to extract three-dimensional shape data corresponding to the object, excluding three-dimensional shape data corresponding to the structure, based on the image data representing the area of ​​the object acquired by the acquiring component, the image data representing the area of ​​the structure acquired by the acquiring component, and the information on the number of camera devices specified by the specifying component.

2. The generating device according to claim 1, wherein: the three-dimensional shape data corresponding to the object is the three-dimensional shape data corresponding to the object excluding the three-dimensional shape data corresponding to the structure, and The generating component performs the following operations: generating three-dimensional shape data corresponding to the object including three-dimensional shape data corresponding to the structure based on the image data representing the area of ​​the object acquired by the acquisition unit and the image data representing the area of ​​the structure acquired by the acquisition unit; as well as Based on the three-dimensional shape data corresponding to the object including the three-dimensional shape data corresponding to the structure and information on the number of imaging devices specified by the specifying means, three-dimensional shape data corresponding to the object excluding the three-dimensional shape data corresponding to the structure is generated.

3. The generating device according to claim 2, wherein: By removing data corresponding to elements in which the number of camera devices specified by the designated component is less than or equal to a threshold from the three-dimensional shape data corresponding to the object including the three-dimensional shape data corresponding to the structure, three-dimensional shape data corresponding to the object that does not include the three-dimensional shape data corresponding to the structure is generated.

4. The generating device according to claim 3, wherein: The threshold value is a value based on the plurality of imaging directions.

5. The generating device according to claim 3, wherein: The threshold value is a value based on an installation position of each of the plurality of imaging apparatuses for performing imaging from the plurality of imaging directions.

6. The generating device according to claim 3, wherein: The threshold value is a value smaller than the number of the plurality of imaging devices for performing imaging from the plurality of imaging directions.

7. The generating device according to any one of claims 1 to 6, wherein: The image data is image data obtained by synthesizing a first image representing a region of the object and a second image representing a region of the structure, and The designating component designates, for an element forming a three-dimensional space for generating three-dimensional shape data corresponding to the object, the number of imaging devices corresponding to the first image including pixels or areas corresponding to the elements in the area of ​​the object as the number of imaging devices.

8. The generating device according to any one of claims 1 to 6, wherein: The image data includes image data of a first image representing a region of the object and image data of a second image representing a region of the structure, and The designating component designates, for an element forming a three-dimensional space for generating three-dimensional shape data corresponding to the object, the number of imaging devices corresponding to the first image including pixels or areas corresponding to the elements in the area of ​​the object as the number of imaging devices.

9. The generating device according to claim 1, wherein: The elements are points or voxels forming the three-dimensional space.

10. A generating device for generating three-dimensional shape data corresponding to an object, the generating device comprising: an acquisition unit for acquiring image data representing a region of the object within a plurality of captured images obtained by a plurality of imaging devices for capturing images from a plurality of imaging directions, and image data representing a region of a structure that blocks the object when the image is captured from at least one of the plurality of imaging directions; a designation means for designating, with respect to an element forming a three-dimensional space for generating three-dimensional shape data corresponding to the object, the number of imaging devices for obtaining a captured image including pixels or an area corresponding to the element in a region of the structure; as well as A generating component is used to extract three-dimensional shape data corresponding to the object, excluding three-dimensional shape data corresponding to the structure, based on the image data representing the area of ​​the object acquired by the acquiring component, the image data representing the area of ​​the structure acquired by the acquiring component, and the information on the number of camera devices specified by the specifying component.

11. The generating device according to claim 10, wherein: the three-dimensional shape data corresponding to the object is the three-dimensional shape data corresponding to the object excluding the three-dimensional shape data corresponding to the structure, and The generating component performs the following operations: generating three-dimensional shape data corresponding to the object including three-dimensional shape data corresponding to the structure based on the image data representing the area of ​​the object acquired by the acquisition unit and the image data representing the area of ​​the structure acquired by the acquisition unit; as well as Based on the three-dimensional shape data corresponding to the object including the three-dimensional shape data corresponding to the structure and information on the number of imaging devices specified by the specifying means, three-dimensional shape data corresponding to the object excluding the three-dimensional shape data corresponding to the structure is generated.

12. The generating device according to claim 11, wherein: By removing data corresponding to elements in which the number of camera devices specified by the designated component is less than or equal to a threshold from the three-dimensional shape data corresponding to the object including the three-dimensional shape data corresponding to the structure, three-dimensional shape data corresponding to the object that does not include the three-dimensional shape data corresponding to the structure is generated.

13. The generating device according to any one of claims 10 to 12, wherein: The image data is image data obtained by synthesizing a first image representing a region of the object and a second image representing a region of the structure, and The designating component designates, for an element forming a three-dimensional space for generating three-dimensional shape data corresponding to the object, the number of imaging devices corresponding to the second image including pixels or areas corresponding to the elements in the area of ​​the structure as the number of imaging devices.

14. The generating device according to any one of claims 10 to 12, wherein: The image data includes image data of a first image representing a region of the object and image data of a second image representing a region of the structure, and The designating component designates, for an element forming a three-dimensional space for generating three-dimensional shape data corresponding to the object, the number of imaging devices corresponding to the second image including pixels or areas corresponding to the elements in the area of ​​the structure as the number of imaging devices.

15. The generating device according to claim 10, wherein: The elements are points or voxels forming the three-dimensional space.

16. A method for generating three-dimensional shape data corresponding to an object, the method comprising: acquiring image data representing a region of the object within a plurality of captured images obtained by a plurality of imaging devices for capturing images from a plurality of imaging directions, and image data representing a region of a structure that blocks the object when capturing images from at least one of the plurality of imaging directions; For an element forming a three-dimensional space for generating three-dimensional shape data corresponding to the object, specifying the number of imaging devices for obtaining a captured image including pixels or an area corresponding to the element in a region of the object; as well as Based on the acquired image data representing the area of ​​the object, the acquired image data representing the area of ​​the structure, and the designated number of imaging devices, three-dimensional shape data corresponding to the object excluding three-dimensional shape data corresponding to the structure is extracted.

17. A method for generating three-dimensional shape data corresponding to an object, the method comprising: acquiring image data representing a region of the object within a plurality of captured images obtained by a plurality of imaging devices for capturing images from a plurality of imaging directions, and image data representing a region of a structure that blocks the object when the image is captured from at least one of the plurality of imaging directions; For an element forming a three-dimensional space for generating three-dimensional shape data corresponding to the object, specifying the number of imaging devices for obtaining a captured image including pixels or an area corresponding to the element in a region of the structure; as well as Based on the acquired image data representing the area of ​​the object, the acquired image data representing the area of ​​the structure, and the designated number of imaging devices, three-dimensional shape data corresponding to the object excluding three-dimensional shape data corresponding to the structure is extracted.

18. A computer-readable storage medium storing a program for causing a computer to function as the generating device according to claim 1.

19. A computer-readable storage medium storing a program for causing a computer to function as the generating device according to claim 10.

Images