Three-dimensional shape restoration device, method and program

The device enhances three-dimensional shape restoration by selectively extracting and refining foreground images from a subset of multi-view images, addressing silhouette and stereo matching challenges to achieve high precision and reduce user burden.

JP7594559B2Active Publication Date: 2024-12-04KDDI CORP
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2022044234
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-03-18
Publication Date
2024-12-04
Estimated Expiration
2042-03-18

AI Technical Summary

Technical Problem

Existing methods struggle to restore the three-dimensional shape of an object with high precision from multi-view images taken by a moving camera in an environment with an arbitrary background, particularly due to challenges in silhouette extraction, camera pose determination, and stereo matching accuracy.

Method used

A three-dimensional shape restoring device that selectively extracts high-precision foreground images from a subset of multi-view images, using a feedback loop to refine the silhouette extraction and stereo matching processes, and combines SfS and point cloud data to enhance reconstruction accuracy.

Benefits of technology

Enables high-precision three-dimensional shape reconstruction with reduced user burden and system resources, suitable for environments with arbitrary backgrounds, improving accuracy for objects with complex and monochromatic patterns.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594559000002
    Figure 0007594559000002
  • Figure 0007594559000003
    Figure 0007594559000003
  • Figure 0007594559000004
    Figure 0007594559000004
Patent Text Reader

Abstract

To restore a three-dimensional shape of a target object with high accuracy using a foreground image extracted from a multi-viewpoint image captured under an arbitrary background environment.SOLUTION: A three-dimensional shape restoration device restores a three-dimensional shape of a target object using multi-viewpoint images captured by cameras with different viewpoints. A first foreground extraction unit 20a extracts a first foreground image F1k from a part of the multi-viewpoint images. A first three-dimensional shape restoration unit 40a coarsely restores a first three-dimensional shape M1 of the target object using the first foreground image F1k. A foreground extraction auxiliary image generating unit 30 uses the first three-dimensional shape M1 to generate a Trimap image for each of the multi-viewpoint images more than the part of the images. A second foreground extraction portion 20b extracts a second foreground image F2n for each of the many multi-viewpoint images using a corresponding Trimap image. A second three-dimensional shape restoration unit 40b restores a second three-dimensional shape M2 using the second foreground image F2n.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a three-dimensional shape restoration device, method, and program that restores the three-dimensional shape of an object using a foreground image extracted from a multi-viewpoint image, and in particular to a three-dimensional shape restoration device, method, and program that restores the three-dimensional shape using a foreground image extracted from a multi-viewpoint image captured in an environment with an arbitrary background. [Background technology]

[0002] There are two approaches to reconstructing 3D data from images: one is to reconstruct 3D data from multiple-viewpoint images taken with multiple fixed cameras, and the other is to reconstruct 3D data from multiple-viewpoint images taken with a single moving camera.

[0003] One commonly known approach to 3D reconstruction from multi-view images taken using multiple fixed cameras (where the relative orientation between the cameras is known through calibration, etc.) is to extract the silhouette (foreground) of an object from multiple-view images of the object and determine the 3D shape using the Shape-from-Silhouette (SfS) method.

[0004] Patent Document 1 discloses a method for improving SfS so that a highly accurate three-dimensional shape can be obtained even when silhouette extraction is inaccurate, and for performing three-dimensional reconstruction by globally evaluating multiple silhouette images. Patent Document 2 discloses a method for improving accuracy when reconstructing an object with some areas moving at high speed using the SfS method by adjusting parameters according to the speed of the area.

[0005] On the other hand, as a method used in an approach to perform 3D reconstruction from multi-view images captured using a moving camera (or multiple fixed cameras with unknown relative orientations between the cameras), a method using the MVS (Multi-View Stereo) method is also known, in which the camera orientations of the multiple view images of an object are estimated using the SfM (Structure-from-Motion) method (Non-Patent Document 1), disparity images are generated by stereo matching between the images, and the 3D shape is obtained by synthesizing the disparity images. For example, Patent Document 3 discloses a 3D reconstruction method using the PatchMatch Stereo method (Non-Patent Document 2), which performs MVS at high speed by random search while considering the normal direction of the object surface.

[0006] Patent Document 4 discloses a three-dimensional restoration method that improves restoration accuracy by adjusting the stereo matching score according to the amount of blur in a captured image in order to improve the robustness of restoration against blur that occurs when an object is photographed with a moving camera. Patent Document 5 discloses a method that supports a user in moving a camera to move the camera appropriately when photographing an object with the camera. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2013-25458 A [Patent Document 2] JP 2020-35218 A [Patent Document 3] JP 2018-181047 A [Patent Document 4] JP 2020-9255 A [Patent Document 5] JP 2020-88646 A [Patent Document 6] JP 2005-078522 A [Non-patent literature]

[0008] [Non-Patent Document 1] JL Schonberger, et al., "Structure-from-Motion Revisited", Proc. of CVPR, 2016. [Non-Patent Document 2] M.Bleyer, C.Rhemann, C.Rother, "PatchMatch Stereo - Stereo Matching with Slanted Support Windows", Proc. of BMVC, 2011. [Non-Patent Document 3] N Xu, et al.,"Deep Image Matting", Proc. of CVPR, 2017. Summary of the Invention [Problem to be solved by the invention]

[0009] In any of the prior art techniques, it was difficult to accurately reconstruct the three-dimensional shape of an object from multi-viewpoint images of the object captured by a moving camera in an environment with an arbitrary background.

[0010] For example, in the SfS method used in Patent Documents 1 and 2, it is necessary to extract the silhouette of an object from a multi-viewpoint image with high accuracy. However, it is generally difficult to extract a silhouette of an object with high accuracy from an image taken by a moving camera in an environment with an arbitrary background. In particular, the background subtraction method assumed in Patent Documents 1 and 2 is a method for extracting the foreground on the premise that the background is constant, so the accuracy of silhouette extraction deteriorates when the camera moves.

[0011] On the other hand, there are methods such as the Alpha Matting method (Non-Patent Document 3) that perform semi-automatic, highly accurate silhouette extraction based on partial user input, rather than fully automatic. However, performing user input for all multi-view images places a high burden on the user and is unrealistic.

[0012] It is also possible to take pictures in an environment with no pattern on the background, such as a green screen, so that the background does not change even if the camera moves. However, it is generally difficult to obtain the camera pose using the SfM method from multi-viewpoint images taken in such an environment, so the error in the camera pose increases, and the final 3D reconstruction accuracy tends to deteriorate.

[0013] On the other hand, the MVS methods disclosed in Patent Documents 3 to 5 do not require silhouette information of the object, and therefore do not suffer from the above-mentioned problems. However, although these techniques are generally excellent in restoring objects with complex patterns (textures), they tend to degrade in restoring objects with little pattern.

[0014] For example, in the case of an object that has no pattern at all, such as furniture made of a single-color plastic material, accurate stereo matching of the object area between stereo images cannot be performed, which tends to degrade the accuracy of the disparity images and therefore the final 3D reconstruction accuracy.

[0015] The object of the present invention is to solve the above-mentioned technical problems and, based on the SfS method, which is not limited to the pattern of the object, to reconstruct a rough three-dimensional shape of the object from a small number of silhouette images using the SfS method, automatically generate high-precision silhouette images of the remaining many viewpoint images from this three-dimensional shape, and reconstruct the final three-dimensional shape by repeatedly applying the SfS method, thereby solving the above-mentioned problems while reducing the burden on the user. [Means for solving the problem]

[0016] In order to achieve the above-mentioned object, the present invention provides a three-dimensional shape restoration device that restores the three-dimensional shape of an object using multi-viewpoint images taken by cameras with different viewpoints, and is characterized in that it has the following configuration.

[0017] (1) The device includes a first restoration means for restoring a first three-dimensional shape of an object using a first foreground image extracted from a portion of multi-view images, a means for generating a trimap image for each of the multiple multi-view images using the first three-dimensional shape and camera parameters of the multiple multi-view images more than the portion, a means for extracting a second foreground image using a trimap image corresponding to each of the multiple multi-view images more than the portion, and a second restoration means for restoring a second three-dimensional shape using the second foreground image.

[0018] (2) The second restoration means feeds back the second three-dimensional shape to the trimap image generating means until the second three-dimensional shape satisfies a predetermined convergence condition, and the trimap image generating means generates a trimap image using the fed back second three-dimensional shape instead of the first three-dimensional shape, and repeats the process of restoring the second three-dimensional shape using a second foreground image extracted using the trimap image generated using the second three-dimensional shape.

[0019] (3) The first restoration means applies the first foreground image and camera parameters corresponding to the first foreground image to an SfS method to restore a first three-dimensional shape on an SfS basis.

[0020] (4) The camera parameter estimation unit acquires three-dimensional point cloud data of the object from a number of multi-viewpoint images greater than the portion, and the first restoration means uses the three-dimensional point cloud data to restore a first three-dimensional shape on a point cloud basis.

[0021] (5) The first reconstruction means reconstructs the first three-dimensional shape by combining the SfS base of (3) and the point cloud base of (4). Effect of the Invention

[0022] According to the present invention, a three-dimensional shape reflecting the contents of a large number of multi-viewpoint images can be restored simply by selecting a small number of highly accurate foreground images from a large number of multi-viewpoint images taken under an environment with an arbitrary background, thereby making it possible to restore the three-dimensional shape of an object with high accuracy while reducing the burden on the user and system resources. [Brief description of the drawings]

[0023] [Figure 1] 1 is a functional block diagram of a three-dimensional shape reconstruction device according to a first embodiment of the present invention. [Diagram 2] 1A to 1C are diagrams illustrating a procedure in which a three-dimensional shape reconstruction device reconstructs a three-dimensional shape of an object based on multi-viewpoint images of an arbitrary background. [Diagram 3] 11A to 11C are diagrams illustrating a procedure for generating a foreground extraction auxiliary image. [Figure 4] FIG. 11 is a diagram showing a procedure for extracting a second foreground image. [Diagram 5] FIG. 11 is a functional block diagram of a three-dimensional shape reconstruction device according to a second embodiment of the present invention. [Figure 6] FIG. 11 is a functional block diagram of a three-dimensional shape reconstruction device according to a third embodiment of the present invention. [Figure 7] FIG. 13 is a functional block diagram of a three-dimensional shape reconstruction device according to a fourth embodiment of the present invention. [Figure 8] FIG. 13 is a functional block diagram of a three-dimensional shape reconstruction device according to a fifth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a functional block diagram showing the configuration of a three-dimensional shape reconstruction device 1 according to a first embodiment of the present invention, and Fig. 2 is a diagram showing a schematic diagram of a procedure in which the three-dimensional shape reconstruction device 1 finally reconstructs the three-dimensional shape of an object with high quality based on multi-viewpoint images.

[0025] The three-dimensional shape restoration device 1 mainly comprises a camera parameter estimation unit 10, two foreground extraction units 20 (20a, 20b), a foreground extraction auxiliary image generation unit 30, and two three-dimensional shape restoration units 40 (40a, 40b), and illustrations of components not necessary for explaining the present invention are omitted here.

[0026] Such a three-dimensional shape reconstruction device 1 can be configured by implementing applications (programs) that realize each function on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

[0027] The camera parameter estimation unit 10 acquires camera images from multiple viewpoints as multi-viewpoint images In (n=1,...,N) and simultaneously estimates the external parameters (information on the camera position and orientation) Wn (n=1,...,N) and internal parameters (information on the camera focal length, optical axis center position, etc.) An (n=1,...,N) of each camera when each multi-viewpoint image (hereinafter sometimes simply referred to as image) In was captured. The estimation results of each camera parameter Wn, An are provided to the first and second 3D shape restoration units 40a, 40b and the foreground extraction auxiliary image generation unit 30.

[0028] The extrinsic parameters Wn and intrinsic parameters A n of the camera are parameters that convert three-dimensional coordinates [X, Y, Z] T in the global three-dimensional Euclidean space into two-dimensional pixel coordinates [u, v] T of each camera image, and are typically expressed by the following equation (1).

[0029]

number

[0030] Here, A represents the internal parameters of the camera, which are generally represented by four parameters: the focal lengths fx, fy and the optical axis center positions cx, cy. W represents the external parameters of the camera, which are generally represented by nine parameters of the rotation matrix r11~r33, six parameters of the rotation vector rx~rz (3 parameters) and the translation vector tx~tz (3 parameters) that can be mutually converted.

[0031] A method for simultaneously restoring the external parameters and internal parameters of the camera from multi-viewpoint images, and further the three-dimensional coordinates of the feature points in the multi-viewpoint images, is disclosed as the SfM method in Non-Patent Document 1, for example. The camera parameter estimation unit 10 simultaneously estimates the external parameters and internal parameters of all cameras using an arbitrary SfM method. Note that the three-dimensional coordinates of the feature points restored simultaneously are not used.

[0032] As shown in FIGS. 2(a) and 2(b), the first foreground extraction unit 20a extracts the silhouette of the object from a part of the images Ik (where Ik is a total of K images, K < N, k ∈ {1,..., K}) selected from a large number of multi-viewpoint images, and outputs this as the first foreground image F 1 k. Generally, a foreground image refers to an image in which the pixels in the foreground area of the image are specified, and is a binary image in which the luminance of the foreground area is set to 255 (white) and the luminance of the background area is set to 0 (black), etc.

[0033] The first foreground extraction unit 20a can extract the silhouette of the object by an arbitrary method. For example, the user may manually specify the foreground / background for each pixel of the image and generate the first foreground image completely manually. Although it is unrealistic from the perspective of the user burden to generate the foreground image completely manually for all (N) multi-viewpoint images, in this embodiment, since the necessary work is limited to a part (a small number of k images), the user burden is significantly reduced.

[0034] Alternatively, a trimap image may be generated by a user manually designating each pixel of an image as one of three types: foreground / background / unknown, and a foreground image may be automatically generated from the trimap image by Alpha Matting disclosed in Non-Patent Document 3. A trimap image is an image in which pixels in the image are designated as foreground region / background region / unknown (foreground and background cannot be distinguished), and is expressed in the form of a grayscale image in which the luminance of the foreground region is 255 (white), the luminance of the background region is 0 (black), and the luminance of the unknown region is 128 (gray), etc.

[0035] By applying Alpha Matting to a trimap image, it is possible to automatically classify pixels in unknown areas of the trimap image into foreground / background and convert the trimap image into a foreground image. Since it is sufficient to roughly specify all pixels near the boundary as unknown, the burden on the user is significantly reduced compared to generating the foreground image completely manually.

[0036] Alternatively, after automatically generating foreground images from all multi-viewpoint images using an Alpha Matting method (the background subtraction method used in Patent Documents 1 and 2 is also of this type), which generates foreground images completely automatically from input images (however, the accuracy is inferior to the method using a Trimap image), only foreground images that the user judges to have been generated with high accuracy are extracted to generate the first foreground image F 1 k may be used. In this case, the load is the lowest because the user does not need to generate a trimap image.

[0037] As shown in FIG. 2C, the first 3D shape restoration unit 40a extracts a small number of first foreground images F 1 k and at least the first foreground image F 1 The three-dimensional shape M1 of the object is reconstructed based on the camera parameters of each multi-view image from which k is extracted.

[0038] Here, the three-dimensional shape refers to a three-dimensional model in a three-dimensional space that contains an object. Three-dimensional models generally include solid models, mesh (surface) models, wireframe models, etc. Although the present invention is not specialized for a specific model, the description will be continued assuming that the three-dimensional shape is a mesh model (a three-dimensional model that represents the object surface using information on vertices, lines, and faces) in this embodiment.

[0039] The first three-dimensional shape reconstruction unit 40a receives a small number (about 20) of foreground images F 1 However, in this embodiment, the second foreground extraction unit 20b extracts a large number of second foreground images F 2 n, and the second foreground image F 2 Since shape reconstruction is performed anew using n, ultimately it becomes possible to achieve highly accurate and high-quality shape reconstruction.

[0040] The foreground extraction auxiliary image generating unit 30 includes a 3D rendering unit 301, a foreground image generating unit 302, and an area dividing unit 303 for generating a trimap image, and generates a foreground extraction auxiliary image S using the first three-dimensional shape M1 restored by the first three-dimensional shape restoring unit 40a and each camera parameter of a larger number of multi-viewpoint images than the partial multi-viewpoint images, as shown in Fig. 2(d). In this embodiment, N trimap images are generated as foreground extraction auxiliary images Sn (n = 1, ..., N) using the external parameters Wn and internal parameters An of all N cameras.

[0041] FIG. 3 is a schematic diagram showing the procedure by which the foreground extraction auxiliary image generation unit 30 generates a foreground extraction auxiliary image Sn. First, the 3D rendering unit 301 performs 3D rendering processing using a first three-dimensional shape M1 and camera parameters Wn, An of N multi-viewpoint images to convert the three-dimensional shape M1 into two-dimensional image data obtained when the three-dimensional shape M1 is photographed with each camera parameter.

[0042] Next, in order to obtain a foreground image of the object from each rendering result, the foreground image generating unit 302 refers to the depth value of each pixel from the depth buffer, and treats the pixel having the depth value as a foreground pixel of the three-dimensional shape M1, thereby generating a foreground image F based on the first three-dimensional shape M1. M1 Generate n.

[0043] This M1-based foreground image F M1 n is a rough silhouette image of the first three-dimensional shape M1, so it cannot be used as is to accurately reconstruct the three-dimensional shape. M1 n is the region Z that is assumed to be definitely foreground pixels. F n, the area Z that is assumed to be background pixels B n and the region Z where the foreground / background distinction is uncertain U Divide it into n parts and make a trimap.

[0044] In this embodiment, each M1-based foreground image F M1 The region segmentation is achieved by performing expansion and contraction processing on n. First, the M1 base foreground image F M1 The foreground region resulting from the contraction process for n is Z F n. Next, the M1 base foreground image F M1 The background region resulting from the expansion process for n is Z B n. Finally, Z F n and Z B The area that is not n is Z U Let n.

[0045] Since the first three-dimensional shape M1 is expected to represent the three-dimensional shape of the object to some extent, although it contains errors, an expansion and contraction process that takes into account the expected errors can be used to reliably identify the areas that are expected to be the foreground / background.

[0046] Here, the foreground extraction auxiliary image generating unit 30 may increase the number of times of expansion processing with respect to contraction processing, or may increase the kernel size. In general, in the SfS method, the restored shape tends to be contracted more than the actual shape, so this adjustment can improve the accuracy of the foreground extraction auxiliary image Sn.

[0047] Furthermore, the foreground extraction auxiliary image generating unit 30 may increase the number of times of contraction processing or increase the kernel size of the contraction processing when the number Nk of the small number of foreground images used in the first shape restoration falls below a certain threshold, and may increase the number of times of expansion processing or increase the kernel size of the expansion processing when the number Nk of the small number of foreground images exceeds a certain threshold. Note that the kernel size of the expansion processing may be increased as well as the number of times of expansion processing.

[0048] Alternatively, within a range where the number Nk of the small number of foreground images used in the first shape recovery is below a certain threshold, the number of times of the contraction process may be increased or the kernel size of the contraction process may be increased as Nk decreases, whereas within a range where the number Nk of the foreground images exceeds a certain threshold, the number of times of the expansion process may be increased or the kernel size of the expansion process may be increased as Nk increases. Note that the kernel size of the expansion process may also be increased along with the number of times of the expansion process.

[0049] Generally, with the SfS method, when the number of viewpoints (number of foreground images) is small, there is a tendency for the reconstructed shape to have parts that bulge outward compared to the actual shape, whereas the more the number of viewpoints (number of foreground images) is, the more likely the reconstructed shape to have parts that shrink inward compared to the actual shape. Therefore, by adopting such adjustments, the accuracy of the foreground extraction auxiliary image (Trimap image) can be further improved.

[0050] The second foreground extraction unit 20b includes an Alpha Matting unit 201, and generates a number of second foreground images F as shown in FIG. 2(f) based on the foreground extraction auxiliary image (Trimap image) Sn generated by the foreground extraction auxiliary image generation unit 30 for each of the multiple (N in this embodiment) multi-viewpoint images and the multiple multi-viewpoint images In (FIG. 2(e)). 2 Extract n (n=1,…N).

[0051] FIG. 4 shows the second foreground extraction unit 20b extracting the second foreground image F 2 First, Alpha Matting unit 201 estimates the foreground / background for each pixel using the trimap image Sn corresponding to each multi-view image In. Since the estimation result is a continuous value of the probability of determining that it is the foreground, it is converted into binary data of foreground / background using a predetermined threshold T to obtain a second foreground image F 2 Get n.

[0052] The threshold T can take any value between the foreground (255) and background (0), and the user can specify a predetermined value in advance, but by setting it to an appropriate value, it is possible to improve the accuracy of the second shape reconstruction.

[0053] The threshold T is the first foreground image F 1 k and the second foreground image F 2 n (k images) may be automatically set based on the result of comparison with the first foreground image F 1 For k, the second foreground image F 2 n, and the threshold T can be set so that the average or sum of the similarities is the highest. 1 k is used to adjust the threshold to obtain the first foreground image F 1 It is expected that the second foreground extraction can be performed with similar high accuracy even for camera images in which k does not exist.

[0054] The second foreground extraction unit 20b extracts the first foreground image F 1A second remaining foreground image F is extracted from only the multi-view images from which k has not been extracted, i.e., all the remaining multi-view images excluding the part of the multi-view images. 2 nk and extract the first foreground image F 1 k and the second remaining foreground image F 2 nk to obtain the second foreground image F 2 It can also be n.

[0055] This not only reduces the processing load compared to when the second foreground extraction unit 20b extracts foreground images from all multi-viewpoint images, but also reduces the processing load of the first foreground image F 1 Although k is a small number, it is highly likely to be highly accurate, which makes it possible to further improve the accuracy of the second shape reconstruction.

[0056] The second three-dimensional shape reconstruction unit 40b generates a plurality of second foreground images F 2 Using n and the corresponding camera parameters, as shown in FIG. 2(g), the three-dimensional shape M2 of the object is reconstructed with high quality in a similar manner to that of the first three-dimensional shape reconstruction unit 40a.

[0057] According to this embodiment, it is possible to highly accurately reconstruct the three-dimensional shape of an object from multi-viewpoint images of the object captured in an environment with an arbitrary background.

[0058] FIG. 5 is a functional block diagram showing the configuration of a second embodiment of the present invention, in which the same reference numerals as those described above denote the same or equivalent parts, and therefore the description thereof will be omitted.

[0059] In this embodiment, the second three-dimensional shape reconstruction unit 40b generates a large number of second foreground images F 2 n, the second three-dimensional shape M2 of relatively high quality is reconstructed and fed back to the foreground extraction auxiliary image generating unit 30.

[0060] When the second three-dimensional shape M2 is fed back, the foreground extraction auxiliary image generation unit 30 generates an M2-based foreground image F by 3D rendering using the second three-dimensional shape M2 instead of the first three-dimensional shape M1. M2n is generated and divided into regions to generate a foreground extraction auxiliary image Sn. The second foreground extraction unit 20b extracts a second foreground image F 2 Extract n again.

[0061] The second three-dimensional shape reconstruction unit 40b converts the second foreground image F 2 The second three-dimensional shape M2 (M2_2) is reconstructed using n and the corresponding camera parameters until the convergence determination unit 403 determines that the difference between the previous (nth) second three-dimensional shape M2 (M2_n) and the current second three-dimensional shape M2 (M2_n+1) falls below a predetermined threshold.

[0062] Since the second three-dimensional shape M2 is more accurate than the first three-dimensional shape M1, the quality of the second three-dimensional shape M2 can be further improved by repeating the generation of the foreground extraction auxiliary image and the generation of the second three-dimensional shape M2 as described above.

[0063] FIG. 6 is a diagram showing the configuration of a third embodiment of the present invention, in which the same reference numerals as those described above denote the same or equivalent parts, and therefore the description thereof will be omitted.

[0064] This embodiment is characterized in that the first three-dimensional shape restoration unit 40a is equipped with an SfS-based restoration unit 401, and the first foreground extraction unit 20a restores a first three-dimensional shape M1 by the SfS method using a first foreground image extracted from the portion of the multi-view images and the camera parameters of each camera that captured the portion of the multi-view images.

[0065] In general, with the SfS method, the more foreground images (number of viewpoints) used, the more accurate the three-dimensional shape can be restored, so the first shape restoration method, which uses only a subset of images, tends not to achieve sufficient accuracy.

[0066] The SfS-based reconstruction unit 401 extracts a first foreground image F 1Using k and the corresponding camera parameters, the silhouette is backprojected into a three-dimensional voxel space to calculate voxel data encompassing the object (this process is also called visual hull).

[0067] The SfS-based restoration unit 401 further converts the voxel data into a mesh model using an algorithm such as the marching cubes method. In the marching cubes method, a cube with eight adjacent voxels as vertices is treated as one unit, and the voxel data can be converted into a three-dimensional shape model by repeating a process of converting the voxel data into polygons of 15 patterns predefined according to the values ​​of the voxels at the eight vertices.

[0068] FIG. 7 is a diagram showing the configuration of a fourth embodiment of the present invention, in which the same reference numerals as those described above denote the same or equivalent parts, and therefore the description thereof will be omitted.

[0069] This embodiment is characterized in that the first three-dimensional shape restoration unit 40a includes a point cloud-based restoration unit 402, and restores the first three-dimensional shape M1 using point cloud data extracted from all (N) camera images.

[0070] The camera parameter estimation unit 10 simultaneously estimates the external parameters Wn (n = 1, ..., N) and the internal parameters An (n = 1, ..., N) when each camera captures a multi-view image, and acquires the three-dimensional coordinates (three-dimensional point cloud data P) of feature points in the multi-view image estimated by the SfM method.

[0071] The first three-dimensional shape restoration unit 40a uses the three-dimensional point cloud data P to restore a first three-dimensional shape M1 of the object using a general algorithm (such as the Poisson Surface Reconstruction method) for restoring a three-dimensional model from three-dimensional point cloud data.

[0072] The first shape recovery is an approach similar to that of the general SfM method, so the recovery accuracy of objects with little pattern tends to deteriorate. However, in this embodiment, we can expect high-precision shape recovery by the second shape recovery using foreground images from all viewpoints.

[0073] In the above third and fourth embodiments, the first three-dimensional shape restoration unit 40a has been described as restoring the first three-dimensional shape M1 based on SfS or point cloud, but the present invention is not limited to this. As in the fifth embodiment shown in Figure 8, the first three-dimensional shape restoration unit 40a may be provided with an SfS-based restoration unit 401 and a point cloud-based restoration unit 402, and the two restoration methods may be combined.

[0074] Specifically, the first foreground image F 1 The three-dimensional shape M1 reconstructed from k by the SfS method is first converted into three-dimensional point cloud data P', and the three-dimensional point cloud data P acquired by the camera parameter estimation unit 10 is integrated with the converted three-dimensional point cloud data P', and the three-dimensional shape M1 can be reconstructed using the integrated three-dimensional point cloud.

[0075] According to each of the above embodiments, it is possible to restore the three-dimensional shape of an object from a multi-view image of the object captured under an arbitrary background environment with high accuracy while minimizing the burden on users and system resources, so that it is possible to provide a variety of three-dimensional shapes to many people regardless of geographic or economic disparity. As a result, it is possible to contribute to Goal 9 "Develop resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations. [Explanation of symbols]

[0076] 1...three-dimensional shape restoration device, 10...camera parameter estimation unit, 20 (20a, 20b)...foreground image extraction unit, 30...foreground extraction auxiliary image generation unit, 40 (40a, 40b)...three-dimensional shape restoration unit, 201...alpha matting unit, 301...3D rendering unit, 302...foreground image generation unit, 303...area division unit, 401...SfS-based restoration unit, 402...point cloud-based restoration unit, 403...convergence judgment unit

Claims

1. A three-dimensional shape reconstruction device that reconstructs a three-dimensional shape of an object using multiple viewpoint images captured by cameras with different viewpoints, means for estimating camera parameters for each multi-view image; a first reconstruction means for reconstructing a first three-dimensional shape of the object using a first foreground image extracted from a portion of the multi-viewpoint images; a means for generating a trimap image for each of the multiple multi-view images using the first three-dimensional shape and camera parameters of the multiple multi-view images that are greater than the part; A means for extracting a second foreground image using a trimap image corresponding to each of the multi-view images that is greater than the part of the multi-view images; and second restoration means for restoring a second three-dimensional shape using the second foreground image.

2. The three-dimensional shape restoration device according to claim 1, characterized in that the first restoration means restores a first three-dimensional shape by applying the first foreground image and camera parameters corresponding to the first foreground image to an SfS method.

3. The means for estimating the camera parameters acquires three-dimensional point cloud data of the object from a number of multi-viewpoint images greater than the part, 3. The three-dimensional shape apparatus according to claim 1, wherein the first reconstruction means reconstructs a first three-dimensional shape by using the three-dimensional point cloud data.

4. The second restoration means feeds back the second three-dimensional shape to the trimap image generating means until the second three-dimensional shape satisfies a predetermined convergence condition; The means for generating a trimap image generates a trimap image using the fed back second three-dimensional shape instead of the first three-dimensional shape; A three-dimensional shape restoration device as described in any one of claims 1 to 3, characterized in that the second three-dimensional shape is repeatedly restored using a second foreground image extracted using a trimap image generated using the second three-dimensional shape.

5. A three-dimensional shape restoration device as described in any one of claims 1 to 4, characterized in that the means for generating the trimap image generates a trimap image using each camera parameter of the first three-dimensional shape and a multi-view image that is greater than the portion.

6. a number of multi-view images greater than the portion include the portion of multi-view images; The means for generating a trimap image generates a trimap image for each of the remaining multi-viewpoint images excluding the part of the multi-viewpoint images from the multi-viewpoint images that are larger than the part; The means for extracting a second foreground image extracts a second remaining foreground image using a corresponding trimap image for each of the remaining multi-view images; 6. The three-dimensional shape reconstruction device according to claim 1, wherein the second reconstruction means reconstructs a second three-dimensional shape using the first and second remaining foreground images.

7. A three-dimensional shape restoration device as described in any one of claims 1 to 6, characterized in that the means for generating the trimap image generates an image divided into areas that can be distinguished as foreground pixels, areas that can be distinguished as background pixels, and areas that cannot be distinguished as either foreground or background.

8. 8. The three-dimensional shape reconstruction device according to claim 7, wherein the means for generating the trimap image performs region division using an expansion / contraction process.

9. The three-dimensional shape restoration device described in claim 8, characterized in that the means for generating the trimap image defines a foreground area resulting from an erosion process on the first foreground image as an area that can be distinguished as foreground pixels, and defines a background area resulting from an expansion process on the first foreground image as an area that can be distinguished as background pixels.

10. 10. The three-dimensional shape reconstruction device according to claim 8, wherein the means for generating the trimap image controls at least one of the number of times expansion processing is performed and a kernel size of the expansion processing in accordance with the number of first foreground images.

11. The three-dimensional shape restoration device according to claim 10, characterized in that the means for generating the trimap image increases the number of times the erosion process is performed or increases the kernel size of the erosion process when the number of first foreground images is below a predetermined threshold.

12. The three-dimensional shape restoration device according to claim 10, characterized in that the means for generating the trimap image increases the number of expansion processes or increases the kernel size of the expansion process when the number of first foreground images exceeds a predetermined threshold.

13. The three-dimensional shape restoration device according to claim 10, characterized in that the means for generating the trimap image increases the number of times of erosion processing or increases the kernel size of the erosion processing as the number of first foreground images decreases when the number of first foreground images is below a predetermined threshold.

14. The three-dimensional shape restoration device described in claim 10, characterized in that the means for generating the trimap image increases the number of expansion processes or increases the kernel size of the expansion process as the number of first foreground images exceeds a predetermined threshold.

15. A three-dimensional shape restoration device as described in any one of claims 1 to 14, characterized in that the second foreground extraction means extracts a second foreground image converted into binary data of the entire scene and background by thresholding the probability of the foreground or background obtained for each pixel by alpha matting on the trimap image.

16. The three-dimensional shape restoration device according to claim 15, characterized in that the second foreground extraction unit sets a threshold for the threshold processing so as to increase the similarity between a first foreground image and an equal number of second foreground images whose camera parameters match those of the first foreground image.

17. A three-dimensional shape reconstruction method in which a computer reconstructs a three-dimensional shape of an object using multi-viewpoint images captured by cameras with different viewpoints, comprising: Reconstructing a first three-dimensional shape of the object using a first foreground image extracted from a portion of the multi-view images; generating a trimap image for each of the multiple multi-view images using the first three-dimensional shape and camera parameters of the multiple multi-view images that are greater than the portion; Extracting a second foreground image using a trimap image corresponding to each of the multi-view images that is greater than the portion; A three-dimensional shape reconstruction method comprising: reconstructing a second three-dimensional shape using the second foreground image.

18. In a three-dimensional shape reconstruction program that reconstructs the three-dimensional shape of an object using multi-viewpoint images taken with cameras with different viewpoints, Reconstructing a first three-dimensional shape of the object using a first foreground image extracted from a portion of the multi-view images; generating a trimap image for each of the multiple multi-view images using the first three-dimensional shape and camera parameters of the multiple multi-view images that are greater than the portion; extracting a second foreground image using a trimap image corresponding to each of the multi-view images that is greater than the part; and a step of reconstructing a second three-dimensional shape using the second foreground image.

Citation Information

Patent Citations

  • Solid form information generation system, solid form information generation device, solid form information generation method, and solid form information generation program

    JP2005078522A

  • Three-dimensional shape estimation device, three-dimensional shape estimation method and three-dimensional shape estimation program

    JP2013025458A

  • Three-dimensional shape model generating device, three-dimensional shape model generating method and program

    JP2018181047A

  • Three-dimensional shape model generation device, three-dimensional model generation method and program

    JP2020009255A

  • Image processing device, method, and program

    JP2020035218A