Three-dimensional shape restoration device, method and program

Abstract

To restore a three-dimensional shape, using only some images picked out of a large number of captured multi-view images.SOLUTION: A three-dimensional shape restoration apparatus picks out some of multi-view images captured by cameras different in point of view, as picked images, to restore, by a computer, a three-dimensional shape of an object using a foreground image extracted from among the picked images. An image pick-out unit 30 includes: a candidate image selection unit 302 which sequentially selects candidate images for picked images out of unpicked multi-view images; a three-dimensional shape temporary restoration unit 303 which temporarily restores a three-dimensional shape of the object using the candidate images and the picked images; a third foreground image extraction unit 304 which extracts, using the temporarily restored three-dimensional shape, a model foreground image for each of unselected multi-view images which have not been picked out or selected as candidate images; and a difference average value calculation unit 305 which repeats calculating, for each candidate image, an evaluation value of the candidate image on the basis of the model foreground image. The image pick-out unit repeats picking out a candidate image with the highest evaluation value.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a three-dimensional shape restoration device, method, and program for restoring the three-dimensional shape of an object using foreground images extracted from multi-viewpoint images, and in particular to a three-dimensional shape restoration device, method, and program for restoring the three-dimensional shape using foreground images of a small number of multi-viewpoint images selected from a large number of multi-viewpoint images. [Background technology]

[0002] There are two approaches to reconstructing 3D data from images: one that uses multiple cameras to reconstruct 3D data, and one that uses multiple cameras to reconstruct 3D data.

[0003] A commonly known approach to reconstructing 3D data from multi-viewpoint images taken with multiple cameras is to extract the silhouette (foreground) of an object from its multi-viewpoint images and determine its 3D shape using the Shape-from-Silhouette (SfS) method.

[0004] Patent Document 1 discloses a method for improving SfS so that a highly accurate three-dimensional shape can be obtained even when silhouette extraction is inaccurate, and for performing three-dimensional reconstruction by globally evaluating multiple silhouette images. Patent Document 2 discloses a method for improving accuracy when reconstructing an object with some areas moving at high speed using the SfS method by adjusting parameters according to the speed of the area.

[0005] On the other hand, as a method used in an approach to perform 3D reconstruction from multi-view images captured using one moving camera, MVS (Multi-View Stereo) is known, which estimates the camera postures of each of the multiple view images of an object using the SfM (Structure-from-Motion) method (Non-Patent Document 1), generates parallax images by stereo matching between the images, and determines the 3D shape by synthesizing the parallax images. For example, Patent Document 3 discloses a 3D reconstruction method using the PatchMatch Stereo method (Non-Patent Document 2), which performs MVS at high speed by random search while considering the normal direction of the object surface.

[0006] Patent Document 4 discloses a three-dimensional restoration method that improves restoration accuracy by adjusting the stereo matching score according to the amount of blur in a captured image in order to improve the robustness of restoration against blur that occurs when an object is photographed with a moving camera. Patent Document 5 discloses a method that supports a user in moving a camera to move the camera appropriately when photographing an object with the camera. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2013-25458 A [Patent Document 2] JP 2020-35218 A [Patent Document 3] JP 2018-181047 A [Patent Document 4] JP 2020-9255 A [Patent Document 5] JP 2020-88646 A [Patent Document 6] JP 2005-078522 A [Non-patent literature]

[0008] [Non-Patent Document 1] JL Schonberger, et al., "Structure-from-Motion Revisited", Proc. of CVPR, 2016. [Non-Patent Document 2] M.Bleyer, C.Rhemann, C.Rother, "PatchMatch Stereo - Stereo Matching with Slanted Support Windows", Proc. of BMVC, 2011. [Non-Patent Document 3] Laurentini, A. "The Visual Hull Concept for Silhouette Based Image Understanding." IEEE PAMI, 16,2 (1994), 150-162 Summary of the Invention [Problem to be solved by the invention]

[0009] The prior art had a problem in that it was difficult to select the optimal image from the multi-viewpoint images of an object captured by a moving camera and restore the three-dimensional shape of the object with high accuracy. For example, Patent Documents 1 to 5 all disclose techniques for estimating a three-dimensional shape from multiple images, but they assume that all input images are used, and do not disclose a method for selecting the optimal image for restoring the three-dimensional shape.

[0010] In the prior art, the 3D shape is estimated by the volume intersection method (Non-Patent Document 3) using the silhouette image of the object. In the case of multi-view images taken with multiple cameras, there is a limit to the number of cameras that can be prepared, and in many cases, there is a technical problem that the number of images is insufficient to estimate the 3D shape with high accuracy.

[0011] On the other hand, when using a single moving camera, it is possible to prepare a large number of images, but when estimating the 3D shape with high accuracy, the images must be selected appropriately. This is because, due to the mechanism of the visual volume intersection method, the 3D shape is reduced as the number of images increases, so if an image that is inappropriate for estimating the 3D shape, for example an image where the estimation of the camera parameters fails, is included, the 3D shape of the target object will be erroneously reduced.

[0012] To address these technical issues, a method for selecting an appropriate 2D image when estimating a person's 3D shape is disclosed in Patent Document 6. However, the 2D image selection method disclosed in Patent Document 6 merely selects a 2D image facing the door portion located in front of the revolving door as a 2D image suitable for generating a 3D shape, and therefore, although it can estimate a person's 3D shape with high accuracy, it does not handle objects other than people.

[0013] An object of the present invention is to solve the above technical problems and provide a three-dimensional shape restoration device, method, and program that restores a three-dimensional shape using only a portion of multi-viewpoint images selected from a large number of multi-viewpoint images. [Means for solving the problem]

[0014] In order to achieve the above-mentioned object, the present invention provides a three-dimensional shape restoration device that has a means for selecting a portion of a multi-viewpoint image captured by cameras with different viewpoints as a selected image, and restores the three-dimensional shape of an object using a foreground image extracted from the selected image, and is characterized in that it has the following configuration.

[0015] (1) The selecting means includes means for sequentially selecting candidate images for selection from unselected multi-view images as candidate images, means for provisionally restoring a three-dimensional shape of an object using the selected candidate images and the selected images, means for extracting a model foreground image for each unselected multi-view image that has not been selected and has not been selected as a candidate image using the provisionally restored three-dimensional shape, and means for repeating the calculation of an evaluation value of the candidate image based on the model foreground image each time the candidate images are sequentially selected and the three-dimensional shape is provisionally restored, and the selecting means is configured to select a portion of the candidate images based on the evaluation value.

[0016] (2) When calculating the evaluation value of the candidate images, the difference between the foreground image of each unselected multi-view image and the model foreground image is calculated for each candidate image selected in sequence, and the statistical value of each difference is used as the evaluation value for each candidate image.

[0017] (3) The statistical value of each difference is the average value of each difference, and the average value of each difference is weighted according to the distance so that candidate images from camera positions that are farther away from the camera position of each selected image are more likely to be selected. Effect of the Invention

[0018] According to the present invention, the following effects can be achieved.

[0019] (1) From a large number of multi-viewpoint images, a portion of the images is sequentially selected from which foreground images extracted from each image are expected to enable accurate reconstruction of the three-dimensional shape. The three-dimensional shape is reconstructed using only the foreground images of the selected images, thereby enabling the three-dimensional shape of an object to be reconstructed with high accuracy and with a small computational load.

[0020] (2) For each candidate image that is selected sequentially, the difference between the foreground image of each unselected multi-view image and the model foreground image is calculated, and images are selected based on the statistical values. This makes it possible to select images that are suitable for interpolating the three-dimensional shape to be restored using the selected images.

[0021] (3) Since candidate images at camera positions farther away from the camera position of a selected image are more likely to be selected, by dispersing the viewpoints of the selected images, images suitable for interpolating three-dimensional shapes can be selected. [Brief description of the drawings]

[0022] [Figure 1] 1 is a functional block diagram of a three-dimensional shape reconstruction device according to an embodiment of the present invention. [Diagram 2] 13 is a flowchart showing a procedure for selecting an image. [Diagram 3] FIG. 1 is a schematic diagram (part 1) showing a procedure for selecting images using a three-dimensional shape provisionally reconstructed using an initial image (first selected image) and candidate images. [Figure 4] FIG. 2 is a schematic diagram (part 2) showing the procedure for selecting images using a three-dimensional shape provisionally reconstructed using an initial image (first selected image) and candidate images. [Diagram 5] 11A and 11B are diagrams illustrating a procedure for adding a selected image by using a three-dimensional shape temporarily restored by using a selected image and a candidate image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a functional block diagram showing the configuration of a three-dimensional shape reconstruction device 1 according to an embodiment of the present invention, and the main components thereof are a camera parameter estimation unit 10, a foreground image extraction unit 20 (20a, 20b), an image selection unit 30, and a three-dimensional shape reconstruction unit 40. Components not necessary for the explanation of the present invention are not shown in the figure.

[0024] The camera parameter estimation unit 10 acquires camera images from multiple viewpoints as multi-viewpoint images In (n=1,...,N) and simultaneously estimates the external parameters (information on the camera's position and orientation) Wn (n=1,...,N) and internal parameters (information on the camera's focal length, optical axis center position, etc.) An (n=1,...,N) of each camera when each multi-viewpoint image (hereinafter sometimes simply referred to as an image) In was captured. The estimation results Wn and An of each camera parameter are provided to the image selection unit 30 and the three-dimensional shape restoration unit 40.

[0025] The extrinsic parameters Wn and intrinsic parameters A n of the camera are parameters that transform three-dimensional coordinates [X, Y, Z] T in the global three-dimensional Euclidean space into two-dimensional pixel coordinates [u, v] T of each camera image, and are expressed by the following equation (1).

[0026]

number

[0027] Here, A is an internal parameter of the camera and is generally expressed by four parameters: focal lengths fx, fy, and optical axis center positions cx, cy. W is an external parameter of the camera and is generally expressed by six parameters: nine parameters of rotation matrices r11 to r33, and mutually convertible rotation vectors rx to rz (three parameters) and translation vectors tx to tz (three parameters).

[0028] A method for simultaneously restoring external and internal parameters of cameras from multi-viewpoint images, as well as the three-dimensional coordinates of feature points in the multi-viewpoint images, is disclosed as the SfM method in, for example, Non-Patent Document 1. The camera parameter estimation unit 10 simultaneously estimates the external and internal parameters of all cameras using an arbitrary SfM method. Note that the three-dimensional coordinates of feature points that are simultaneously restored are not used.

[0029] The first foreground image extraction unit 20a extracts a foreground image FIn from each multi-viewpoint image In. In general, a foreground image is an image in which pixels in a foreground region in the image are specified, and is a binary image (silhouette) in which the luminance of the foreground region is 255 (white) and the luminance of the background region is 0 (black), for example.

[0030] The first foreground image extraction unit 20a can use any method to extract the silhouette of the object. For example, the user may manually specify the foreground / background for each pixel of the image and generate the foreground image completely manually.

[0031] Alternatively, a trimap image (a foreground extraction auxiliary image in which each pixel is expressed with three values: foreground / background / unknown) can be generated by the user manually specifying one of the three types (foreground / background / unknown) for each pixel of the image, and a foreground image can be automatically generated from the trimap image using, for example, the Alpha Matting method disclosed in Non-Patent Document 3.

[0032] A trimap image is an image in which the foreground, background, and unknown (foreground and background cannot be distinguished) pixels are specified, and it is a grayscale image in which the brightness of the foreground area is 255 (white), the brightness of the background area is 0 (black), and the brightness of the unknown area is 128 (gray). By applying Alpha Matting, it is possible to automatically classify the pixels in the unknown areas of a trimap image into foreground / background, and convert the trimap image into a foreground image.

[0033] Alternatively, after automatically generating foreground images from all multi-viewpoint images using an Alpha Matting method (the background subtraction method used in Patent Documents 1 and 2 is also of this type), which generates foreground images completely automatically from input images, the user may extract only the foreground images that he or she judges to have been generated with good accuracy, and use them as the foreground images, although the accuracy is inferior to that of the method using the trimap. In this case, the user does not need to generate the trimap.

[0034] As described in detail below, the image selection unit 30 selects some of the multi-viewpoint images to be used for restoring the three-dimensional shape based on the foreground image FIn of each multi-viewpoint image In and its camera parameters Wn, An. The second foreground image extraction unit 20b extracts the foreground image from the selected multi-viewpoint images (selected images). The three-dimensional shape restoration unit 40 restores the three-dimensional shape of the object by, for example, the SfS method using the foreground image of the selected images and the camera parameters of each image.

[0035] Here, the three-dimensional shape refers to a three-dimensional model in a three-dimensional space that contains an object. Three-dimensional models generally include solid models, mesh (surface) models, wireframe models, etc. Although the present invention is not specialized for a specific model, the following description will be given assuming that the three-dimensional shape is a mesh model (a three-dimensional model that represents the surface of an object using information on vertices, lines, and faces).

[0036] Generally, in the SfS method, voxel data containing the target object is calculated by backprojecting a silhouette into a three-dimensional voxel space using a foreground image and camera parameters (this process is also called visual hull). Furthermore, the voxel data is converted into a mesh model by an algorithm such as the marching cubes method. In the marching cubes method, a cube with eight adjacent voxels as vertices is treated as one unit, and the voxel data can be converted into a three-dimensional shape model by repeating a process of converting the unit into polygons of 15 patterns predefined according to the voxel values ​​of the eight vertices. Any SfS method can be applied to the present invention.

[0037] Such a three-dimensional shape reconstruction device can be configured by implementing an application (program) that realizes each function on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

[0038] The image selection unit 30 mainly comprises an initial image selection unit 301, a candidate image selection unit 302, a temporary three-dimensional shape restoration unit 303, a third foreground image extraction unit 304 and an average difference calculation unit 305, and selects some multi-view images In to be used for restoring the three-dimensional shape from all the multi-view images In.

[0039] Next, the operation of the image selection unit 30 will be described in detail with reference to the flowchart in Fig. 2. In step S1, the first one or more selected images to be used for restoring a 3D shape from a group of multi-view images are selected as initial images by the initial image selection unit 301. For ease of understanding, the description will be continued assuming that image I1 is added as the initial image (first selected image) in ascending order of index n that identifies each multi-view image In.

[0040] In step S2, all the remaining multi-view images except for the selected images up to this point (here, only the initial image I1) are classified as unselected images. In step S3, one of the unselected images is selected as a candidate for a selected image (candidate image) by the candidate image selection unit 302. Here, the explanation will be continued assuming that image I2 is selected as the candidate image in ascending order of index n.

[0041] In step S4, the three-dimensional shape of the object is provisionally restored, for example, by the SfS method, using all currently selected images and the current candidate image by the three-dimensional shape provisional restoration unit 303. In this embodiment, the three-dimensional shape provisional restoration unit 303 has the same or equivalent functions as the three-dimensional shape restoration unit 40, and here, as shown in Fig. 3, the three-dimensional shape of the object is provisionally restored, for example, by the volume intersection method, using an initial image I1, a current candidate image I2, and external parameters W1, W2 and internal parameters A1, A2 of the cameras that captured the images I1, I2.

[0042] In step S5, from all unselected multi-view images (hereinafter sometimes referred to as unselected images) excluding the current candidate image I2 and the initial image I1, a current image of interest is selected by third foreground image extraction unit 304. Here, the explanation will be continued assuming that multi-view image I3 is selected in ascending order of index n.

[0043] In step S6, third foreground image extraction unit 304 further projects each point in three-dimensional space of the provisionally restored three-dimensional shape onto a point in two-dimensional space using the camera parameters of the current image of interest I3 to generate a subject image at that camera position, and extracts a foreground image of image of interest I3 by Alpha Matting. Hereinafter, the foreground image extracted by foreground image extraction unit 304 from image of interest In is referred to as model foreground image MFIn.

[0044] In step S7, a difference D between the model foreground image MFIi extracted by the foreground image extraction unit 304 from the current target image Ii and the foreground image FIi extracted by the foreground image extraction unit 20a is calculated. i is calculated based on the following formula (2):

[0045]

number

[0046] Here, the difference D between the foreground image FI3 extracted by the foreground image extraction unit 20a from the current image of interest I3 and the model foreground image MFI3 extracted by the foreground image extraction unit 304 based on the image of interest I3 is 3 is calculated.

[0047] In step S8, it is determined whether or not the difference D has been calculated for all unselected images as the image of interest. If there are any unselected images remaining for which the difference D has not been calculated as the image of interest, the process returns to step S5, and the above processes are repeated while switching the image of interest to one of the remaining unselected images, thereby further calculating the difference D. 4 ,D 5 ,D 6 …D N is calculated.

[0048] When the difference D is obtained for all unselected images, the process proceeds to step S9, where the difference D is obtained by repeatedly selecting the image of interest for the current candidate image I2 as shown in FIG. iThe average value Davg2 of the differences D is calculated by the following formula (3): Here, M is the number of differences D, and the average difference value Davg2 is an evaluation value for whether or not to select the candidate image I2.

[0049]

number

[0050] In addition, in order to make it easier to select a candidate image that is farther away from the selected image up to this point in time, with a higher evaluation value, it is possible to calculate, for example, the average or minimum value of the Euclidean distance between the camera position of each selected image at that time and the camera position of the candidate image, and to divide the average difference Davg2 by this average or minimum value and use the result as the evaluation value.

[0051] In step S10, it is determined whether or not there are any unselected images remaining that have not been selected as candidate images. If there are any unselected images remaining, the process returns to step S3, and the above-mentioned processes are repeated while switching the selection of the candidate images, thereby obtaining the average difference Davg for each candidate image. j is calculated as follows: Fig. 4 shows a schematic diagram of a method for calculating the average difference value Davg3 based on the following equation (4) using an unselected multi-view image I3 as a candidate image.

[0052]

number

[0053] Next, when the multi-view image I4 is selected as a candidate image, the difference average Davg4 is calculated based on the following equation (5).

[0054]

number

[0055] For all unselected images, the average difference Davg j When the calculation is completed, proceed to S11 and calculate the average difference Davg jThe candidate image Ii for which the minimum value is given is added as a new selected image.

[0056] In step S11, instead of adding as a new selected image the candidate image Ii having the smallest average difference value Davgj as described above, a predetermined number of candidate images Ii having the smallest average difference value Davgj (for example, two candidate images, the candidate image having the smallest value and the candidate image having the next smallest value) may be added as new selected images.

[0057] In step S12, it is determined whether the image selection has converged. In this embodiment, it is determined that the image selection has converged when any one or more of the following four conditions are satisfied.

[0058] Condition 1: The total number of selected images reaches the number specified by the user. Condition 2: The change in the average difference value is below a predetermined threshold. Condition 3: The difference average value increased from the previously calculated difference average value. Condition 4: The average difference value exceeds the minimum past average value by a certain threshold value or more.

[0059] If convergence has not been achieved, the process returns to step S2 to classify the currently selected image, and from step S3 onward, the above processes are repeated while switching the selection of candidate images from among the unselected images.

[0060] FIG. 5 shows a schematic diagram of the operation when candidate image I2 is added as a selected image, and in step S2, multi-view images I1 and I2 are selected as the selected images, and multi-view image I3 is again selected as the candidate image, and the above-mentioned processes are repeated.

[0061] In the above-mentioned step S12, it has been explained that when any one or more of conditions 1 to 4 are satisfied, it is determined that the image selection has converged and the process is terminated. However, for example, step S12 may be omitted by adding the required number of newly selected images in step S11.

[0062] Thus, according to this embodiment, the selection means 30 sequentially selects selected candidate images from unselected multi-viewpoint images, provisionally reconstructs the three-dimensional shape of the object using the selected candidate images and selected images, and each time a three-dimensional shape is provisionally reconstructed, the difference between the model foreground image MFIi and the foreground image FIi for each unselected multi-viewpoint image is calculated using the three-dimensional shape, and a predetermined number of selected images are obtained by repeating image selection using the statistical value of each difference for each candidate image as an evaluation value.

[0063] If selected images are selected randomly from camera images (multi-viewpoint images), there is a possibility that a camera image for which camera parameters have failed to be estimated will be selected, in which case an accurate three-dimensional shape cannot be restored. In this embodiment, however, if a camera image for which camera parameters have failed to be estimated becomes a candidate image, the three-dimensional shape will not be restored correctly, and the resulting calculated Davg will be a high value. This makes it possible to prevent such camera images from being selected as selected images, resulting in a highly accurate three-dimensional shape that is ultimately output.

[0064] According to the above embodiment, high-quality three-dimensional shapes can be generated, making it possible to provide a variety of entertainment to many people regardless of geographic or economic disparity. As a result, it is possible to contribute to the achievement of Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations. [Explanation of symbols]

[0065] 1...three-dimensional shape restoration device, 10...camera parameter estimation unit, 20 (20a, 20b)...foreground image extraction unit, 30...image selection unit, 40...three-dimensional shape restoration unit, 301...initial image selection, 302...candidate image selection unit, 303...three-dimensional shape temporary restoration unit, 304...third foreground image extraction unit, 305...difference average value calculation unit

Claims

1. A three-dimensional shape reconstruction device includes a means for selecting a part of a multi-viewpoint image captured by a camera having different viewpoints as a selected image, and reconstructs a three-dimensional shape of an object using a foreground image extracted from the selected image, The means for selecting comprises: A means for sequentially selecting candidates for a selected image as a candidate image from among unselected multi-viewpoint images; A means for provisionally reconstructing a three-dimensional shape of an object using the selected candidate images and the selected image; a means for extracting a model foreground image of the image of interest from a subject image generated by repeatedly selecting one of the unselected multi-view images that have not been screened and have not been selected as a candidate image as an image of interest and projecting the provisionally restored three-dimensional shape for each image of interest onto a two-dimensional space using camera parameters of the image of interest; a means for calculating, for each of the target images, a difference between a foreground image extracted from the target image and the model foreground image, and calculating an evaluation value of the candidate image based on a statistical value of each difference for the selected number of target images; A three-dimensional shape restoration device characterized in that the selection means repeats provisional restoration of the three-dimensional shape, extraction of the model foreground image, and calculation of the evaluation value for each selection of a candidate image, and selects a portion of the candidate images based on the evaluation value.

2. 2. The three-dimensional shape reconstruction apparatus according to claim 1, wherein the statistical value of each difference is an average value of each difference.

3. The three-dimensional shape reconstruction device according to claim 2, characterized in that the average value of each difference is weighted according to the distance so that candidate images at camera positions farther away from the camera position of each selected image are more likely to be selected.

4. 4. The three-dimensional shape reconstruction apparatus according to claim 1, further comprising a means for estimating camera parameters of each multi-viewpoint image based on the multi-viewpoint image.

5. 5. The three-dimensional shape reconstruction device according to claim 1, wherein the selection means repeats the process of selecting one of the candidate images based on the evaluation value for the remaining unselected multi-viewpoint images.

6. 6. The three-dimensional shape reconstruction apparatus according to claim 5, wherein said selecting means repeats the selection until a total number of selected images reaches a predetermined number.

7. The three-dimensional shape restoration device described in claim 2 or 3, characterized in that the selection means repeats the process of selecting one of the candidate images based on the evaluation value for the remaining unselected multi-view images until the change in the average value of the differences becomes equal to or less than a predetermined threshold value.

8. The three-dimensional shape restoration device described in claim 2 or 3, characterized in that the selection means repeats the process of selecting one of the candidate images based on the evaluation value for the remaining unselected multi-view images until the average value of the differences increases above the average value of the differences previously calculated.

9. The three-dimensional shape restoration device described in claim 2 or 3, characterized in that the selection means repeats the process of selecting one of the candidate images based on the evaluation value for the remaining unselected multi-view images until the average value of the differences exceeds the minimum value of past average values ​​by a certain threshold value or more.

10. A three-dimensional shape restoration method in which some of multi-viewpoint images taken with cameras having different viewpoints are selected as selected images, and a computer restores a three-dimensional shape of an object using a foreground image extracted from the selected images, comprising: Sequentially selecting candidates for the selected image from the unselected multi-view images as candidate images; A three-dimensional shape of the object is provisionally reconstructed using the selected candidate images and the selected image; Repeatedly selecting one of the unselected multi-viewpoint images that have not been screened and have not been selected as a candidate image as an image of interest, and extracting a model foreground image of the image of interest from a subject image generated by projecting the provisionally restored three-dimensional shape for each image of interest onto a two-dimensional space using the camera parameters of the image of interest; determining, for each of the images of interest, a difference between a foreground image extracted from the image of interest and the model foreground image, and calculating an evaluation value of the candidate image based on a statistical value of each difference for the number of images of interest selected; A three-dimensional shape restoration method characterized by repeating the provisional restoration of the three-dimensional shape, the extraction of the model foreground image, and the calculation of the evaluation value for each selection of a candidate image, and selecting a portion of the candidate images as selected images based on the evaluation value.

11. A three-dimensional shape reconstruction program for causing a computer to select a part of a multi-viewpoint image taken by a camera with different viewpoints as a selected image, and to reconstruct the three-dimensional shape of an object using a foreground image extracted from the selected image, comprising: a step of sequentially selecting candidates for the selected image from the unselected multi-view images as candidate images; A step of provisionally reconstructing a three-dimensional shape of the object using the selected candidate images and the selected image; a step of repeatedly selecting one of the unselected multi-viewpoint images that have not been screened and have not been selected as a candidate image as an image of interest, and extracting a model foreground image of the image of interest from a subject image generated by projecting the provisionally restored three-dimensional shape for each image of interest onto a two-dimensional space using camera parameters of the image of interest; a step of calculating, for each of the images of interest, a difference between a foreground image extracted from the image of interest and the model foreground image, and calculating an evaluation value of the candidate image based on a statistical value of each difference for the selected number of images of interest; A three-dimensional shape restoration program characterized by repeating the provisional restoration of the three-dimensional shape, the extraction of the model foreground image, and the calculation of the evaluation value for each selection of a candidate image, and selecting a portion of the candidate images as selected images based on the evaluation value.

Images