Three-dimensional shape model generating device, three-dimensional shape model generating method, and program

Abstract

To provide a three-dimensional shape model generation device, a three-dimensional shape model generation method and a program, capable of three-dimensionally restoring a shape of a target object in which a plane structure and a line structure are mixed, with high accuracy.SOLUTION: A three-dimensional shape model generation device includes: a label map generation part configured to generate a label map with a label added, the label showing a similarity to a plane structure and a similarity to a line structure by pixel at each of multi-viewpoint images being two or more images photographing a target object from different viewpoints; and a three-dimensional information map generation part configured to generate a three-dimensional information map for generating a three-dimensional shape model of the target object having a plane structure and a line structure, according to the label map.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a three-dimensional shape model generating device, a three-dimensional shape model generating method, and a program. [Background technology]

[0002] Three-dimensional shape recovery technology, which generates a three-dimensional shape model of an object based on two or more multiple-view images (hereinafter referred to as "multi-view images") taken of the object from different viewpoints, is attracting attention not only from the computer vision research community but also from a wide range of fields, such as digital archiving of cultural assets and the entertainment industry.

[0003] For example, Patent Document 1 discloses a three-dimensional restoration method that generates a depth map indicating the distance in the depth direction from each camera of a viewpoint to an object from multi-viewpoint images and restores the three-dimensional shape of the object by integrating these depth maps. In the depth map generation process in the three-dimensional restoration method of Patent Document 1, the local shape of the object is approximated by a small rectangular three-dimensional plane (patch), and the problem is solved as to find the position and orientation of a group of patches that are most consistent between multi-viewpoint images.

[0004] On the other hand, Non-Patent Document 1 discloses a three-dimensional restoration method for restoring the three-dimensional shape of a thin structure such as a hair. In the three-dimensional restoration method of Non-Patent Document 1, a depth map is also generated from multi-viewpoint images, and the depth map is integrated to restore the three-dimensional shape of the target. However, the three-dimensional restoration method of Non-Patent Document 1 differs from the three-dimensional restoration method of Patent Document 1 in that the local shape of the target is approximated by small line segments, and the position and orientation of a line segment group that is most consistent between multi-viewpoint images is obtained. In this way, the three-dimensional restoration method of Non-Patent Document 1 realizes the restoration of a thin structure such as a hair. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-181047 A [Non-patent literature]

[0006] [Non-Patent Document 1] Giljoo Nam, Chenglei Wu, Min H. Kim, Yaser Sheikh, “Strand-accurate Multi-view Hair Capture”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp.155-164, 2019 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in a three-dimensional restoration method for restoring a surface structure such as that in Patent Document 1, if the size of the patch is sufficiently small compared to the local shape of the measurement target, the shape can be restored, but if the size of the patch is large compared to the local shape of the target, restoration fails or the accuracy drops significantly. Therefore, it is difficult to restore a structure with a thin shape such as a hair or a needle. On the other hand, in a three-dimensional restoration method for restoring a line structure such as that in Non-Patent Document 1, if the shape of the target is a curved surface, the shape of the target cannot be restored as a curved surface, and can only be restored as a set of curves (or broken lines approximating the curves). As described above, conventional 3D reconstruction methods from multi-viewpoint images are limited to either reconstructing the shape of an object as a surface or as a line, and therefore have the problem that they cannot accurately reconstruct the shape of an object that has a mixture of surface and line structures in 3D.

[0008] In view of the above-mentioned problems, an object of the present invention is to provide a three-dimensional shape model generation device, a three-dimensional shape model generation method, and a program capable of three-dimensionally restoring the shape of an object having a mixture of surface and line structures with high accuracy. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, a 3D shape model generating device according to one aspect of the present invention provides a multi-viewpoint image, which is two or more images of an object captured from different viewpoints. a first depth map generating unit that generates a first provisional depth map by stereo matching assuming a local surface structure based on the multi-view image; a second depth map generating unit that generates a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image; and a second depth map generating unit that generates a second provisional depth map by stereo matching assuming a local line structure based on information obtained in generating the first provisional depth map and the second provisional depth map. a label map generating unit that generates a label map in which a label indicating a likelihood of a surface structure and a likelihood of a line structure is assigned to each pixel in each viewpoint image; and a three-dimensional information map generating unit that generates a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map. the three-dimensional information map is a depth map including a first depth map in the area of ​​the planar structure and a second depth map in the area of ​​the line structure, the first depth map generation unit generates the first depth map by stereo matching assuming a local planar structure based on the multi-view image and the label map, the second depth map generation unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map, the label map generation unit calculates a first matching score which is a matching score of stereo matching for the first provisional depth map and a second matching score which is a matching score of stereo matching for the second provisional depth map, and determines a label indicating a high likelihood of a planar structure when the first matching score is higher than the second matching score, and determines a label indicating a high likelihood of a line structure when the second matching score is higher than the first matching score. . A three-dimensional shape model generating device according to one aspect of the present invention includes a first depth map generating unit that generates a first provisional depth map by stereo matching assuming a local surface structure based on multi-viewpoint images, which are two or more images of an object captured from different viewpoints; a second depth map generating unit that generates a second provisional depth map by stereo matching assuming a local line structure based on the multi-viewpoint images; a label map generating unit that generates a label map in which labels indicating the likelihood of a surface structure and a line structure are assigned to each pixel in each viewpoint image of the multi-viewpoint images based on information obtained in generating the first provisional depth map and the second provisional depth map; and a three-dimensional information map generating unit that generates a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map. a generation unit, wherein the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure, the first depth map generation unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map, the second depth map generation unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map, and the label map generation unit calculates a matching score by performing stereo matching on the local surface structure calculated from the first provisional depth map using a weighting function based on the line structure calculated from the second provisional depth map, and determines a label depending on the magnitude of the matching score value. A three-dimensional shape model generating device according to one aspect of the present invention includes a first depth map generating unit configured to generate a first provisional depth map by stereo matching assuming a local surface structure based on multi-viewpoint images, which are two or more images of an object captured from different viewpoints; a second depth map generating unit configured to generate a second provisional depth map by stereo matching assuming a local line structure based on the multi-viewpoint images; a label map generating unit configured to generate a label map in which a label indicating a surface structure likelihood and a line structure likelihood is assigned to each pixel in each viewpoint image of the multi-viewpoint images based on information obtained in generating the first provisional depth map and the second provisional depth map; and a three-dimensional information map generating unit configured to generate a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map, the three-dimensional information map being a data including a first depth map in an area of ​​the surface structure and a second depth map in an area of ​​the line structure. the first depth map generation unit generates the first depth map based on the multi-view image and the label map by stereo matching assuming a local surface structure, refers to the label map of the multiple viewpoints, and if the number of labels indicating a line structure-likeness at the first provisional depth of the first provisional depth map is greater than the number of labels indicating a surface structure-likeness, corrects the value of the first provisional depth to a value indicating that it is invalid, and sets the corrected value as the depth value of the first depth map; the second depth map generation unit generates the second depth map based on the multi-view image and the label map by stereo matching assuming a local surface structure, refers to the label map of the multiple viewpoints, and if the number of labels indicating a surface structure-likeness at the second provisional depth of the second provisional depth map is greater than the number of labels indicating a line structure-likeness, corrects the value of the second provisional depth to a value indicating that it is invalid, and sets the corrected value as the depth value of the second depth map. A three-dimensional shape model generating device according to one aspect of the present invention includes a label map generating unit that generates a label map in which a label indicating the likelihood of a planar structure and a line structure is assigned to each pixel in each viewpoint image of a multi-view image, which is two or more images captured of an object from different viewpoints; a three-dimensional information map generating unit that generates a three-dimensional information map that indicates three-dimensional information for generating a three-dimensional shape model of the object having a planar structure and a line structure based on the label map and is a depth map including a first depth map in an area of ​​the planar structure and a second depth map in an area of ​​the line structure; and a local information generating unit that generates a three-dimensional information map that indicates three-dimensional information for generating a three-dimensional shape model of the object having a planar structure and a line structure based on the multi-view images and the label map. The stereo image generating device includes a first depth map generation unit that generates the first depth map by stereo matching assuming a surface structure, and a second depth map generation unit that generates the second depth map based on the multi-view image and the label map by stereo matching assuming a local line structure, wherein the first depth map generation unit performs stereo matching using a weighting function that increases the weight of pixels in the label map that indicate a surface structure-likeness, and the second depth map generation unit performs stereo matching using a weighting function that increases the weight of pixels in the label map that indicate a line structure-likeness.

[0010] A three-dimensional shape model generating method according to one aspect of the present invention includes the steps of: First Depth Map Generator A multi-view image is two or more images of an object captured from different viewpoints. a step of generating a first provisional depth map by stereo matching assuming a local surface structure based on the multi-view image by a second depth map generating unit, a step of generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image by a label map generating unit, a step of generating a label map in which a label indicating a likelihood of a plane structure and a likelihood of a line structure is assigned to each pixel in each viewpoint image; and a step of a three-dimensional information map generating unit generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map. the three-dimensional information map is a depth map including a first depth map in the area of ​​the planar structure and a second depth map in the area of ​​the line structure, the first depth map generation unit generates the first depth map by stereo matching assuming a local planar structure based on the multi-view image and the label map, the second depth map generation unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map, the label map generation unit calculates a first matching score which is a matching score of stereo matching for the first provisional depth map and a second matching score which is a matching score of stereo matching for the second provisional depth map, and determines a label indicating a high likelihood of a planar structure when the first matching score is higher than the second matching score, and determines a label indicating a high likelihood of a line structure when the second matching score is higher than the first matching score. . A three-dimensional shape model generating method according to one aspect of the present invention includes a step of a first depth map generating unit generating a first provisional depth map by stereo matching assuming a local surface structure based on multi-viewpoint images, which are two or more images of an object captured from different viewpoints; a step of a second depth map generating unit generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-viewpoint images; a step of a label map generating unit generating a label map in which labels indicating a surface structure likelihood and a line structure likelihood are assigned to each pixel in each viewpoint image of the multi-viewpoint images based on information obtained in generating the first provisional depth map and the second provisional depth map; and a step of a three-dimensional information map generating unit generating three-dimensional information indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map. and a process of generating an information map, the three-dimensional information map being a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure, the first depth map generation unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map, the second depth map generation unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map, and the label map generation unit calculates a matching score by performing stereo matching on the local surface structure calculated from the first provisional depth map using a weight function based on the line structure calculated from the second provisional depth map, and determines a label depending on the magnitude of the value of the matching score. A three-dimensional shape model generating method according to one embodiment of the present invention includes the steps of: a first depth map generating unit generating a first provisional depth map by stereo matching assuming a local surface structure based on multi-view images, which are two or more images of an object captured from different viewpoints; a second depth map generating unit generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view images; a label map generating unit generating a label map in which labels indicating a surface structure likelihood and a line structure likelihood are assigned to each pixel in each viewpoint image of the multi-view image based on information obtained in generating the first provisional depth map and the second provisional depth map; and a three-dimensional information map generating unit generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map, wherein the three-dimensional information map is a first depth map in an area of ​​the surface structure and a second depth map in an area of ​​the line structure. the first depth map generation unit generates the first depth map based on the multi-view image and the label map by stereo matching assuming a local surface structure, refers to the label map of the multiple viewpoints, and if the number of labels indicating a line structure-likeness at the first provisional depth of the first provisional depth map is greater than the number of labels indicating a surface structure-likeness, corrects the value of the first provisional depth to a value indicating that it is invalid, and sets the corrected value as the depth value of the first depth map; the second depth map generation unit generates the second depth map based on the multi-view image and the label map by stereo matching assuming a local line structure, refers to the label map of the multiple viewpoints, and if the number of labels indicating a surface structure-likeness at the second provisional depth of the second provisional depth map is greater than the number of labels indicating a line structure-likeness, corrects the value of the second provisional depth to a value indicating that it is invalid, and sets the corrected value as the depth value of the second depth map. A three-dimensional shape model generating method according to one aspect of the present invention includes a step of generating a label map in which a label indicating a likelihood of a planar structure and a likelihood of a line structure is assigned to each pixel in each viewpoint image of a multi-view image, which is two or more images capturing an object from different viewpoints, by a label map generating unit; a step of generating a three-dimensional information map that indicates three-dimensional information for generating a three-dimensional shape model of the object having a planar structure and a line structure based on the label map, the three-dimensional information map being a depth map including a first depth map in an area of ​​the planar structure and a second depth map in an area of ​​the line structure; and a step of generating a three-dimensional information map that indicates three-dimensional information for generating a three-dimensional shape model of the object having a planar structure and a line structure based on the label map, the three-dimensional information map including a first depth map in an area of ​​the planar structure and a second depth map in an area of ​​the line structure. The method includes a first depth map generation process for generating the first depth map by stereo matching assuming a local surface structure, and a second depth map generation process for generating the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map by a second depth map generation unit, wherein the first depth map generation unit performs stereo matching using a weighting function that increases the weight of pixels in the label map that indicate a surface structure-likeness, and the second depth map generation unit performs stereo matching using a weighting function that increases the weight of pixels in the label map that indicate a line structure-likeness.

[0011] A program according to one aspect of the present invention causes a computer to generate a multi-viewpoint image, which is two or more images of an object captured from different viewpoints. a first depth map generating means for generating a first provisional depth map by stereo matching assuming a local surface structure based on the multi-view image; a second depth map generating means for generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image; and a second depth map generating means for generating a second provisional depth map by stereo matching assuming a local line structure based on information obtained in generating the first provisional depth map and the second provisional depth map. and a three-dimensional information map generating means for generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a planar structure and a line structure based on the label map. the three-dimensional information map is a depth map including a first depth map in the area of ​​the planar structure and a second depth map in the area of ​​the line structure, the first depth map generating means generates the first depth map by stereo matching assuming a local planar structure based on the multi-view image and the label map, the second depth map generating means generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map, the label map generating means calculates a first matching score which is a matching score of stereo matching for the first provisional depth map and a second matching score which is a matching score of stereo matching for the second provisional depth map, and determines a label indicating a high likelihood of a planar structure when the first matching score is higher than the second matching score, and determines a label indicating a high likelihood of a line structure when the second matching score is higher than the first matching score. . A program according to one aspect of the present invention includes a first depth map generating means for generating a first provisional depth map by stereo matching assuming a local surface structure based on multi-viewpoint images, which are two or more images of an object captured from different viewpoints; a second depth map generating means for generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-viewpoint images; a label map generating means for generating a label map in which labels indicating the likelihood of a surface structure and a line structure are assigned to each pixel in each viewpoint image of the multi-viewpoint images based on information obtained in generating the first provisional depth map and the second provisional depth map; and a three-dimensional information map generating means for generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map. and a means for generating a depth map based on the local surface structure of the multi-view image and the label map, the three-dimensional information map being a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure. The first depth map generating means generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map. The second depth map generating means generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map. The label map generating means calculates a matching score by performing stereo matching on the local surface structure calculated from the first provisional depth map using a weighting function based on the line structure calculated from the second provisional depth map, and determines a label depending on the magnitude of the value of the matching score. A program according to one aspect of the present invention causes a computer to function as: a first depth map generating means for generating a first provisional depth map by stereo matching assuming a local surface structure, based on multi-viewpoint images which are two or more images of an object captured from different viewpoints; a second depth map generating means for generating a second provisional depth map by stereo matching assuming a local line structure, based on the multi-viewpoint images; a label map generating means for generating a label map in which a label indicating a surface structure likelihood and a line structure likelihood is assigned to each pixel in each viewpoint image of the multi-viewpoint images, based on information obtained in generating the first provisional depth map and the second provisional depth map; and a three-dimensional information map generating means for generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure, based on the label map, the three-dimensional information map being a first depth map in an area of ​​the surface structure and a second depth map in an area of ​​the line structure. the first depth map generation means generates the first depth map based on the multi-view image and the label map by stereo matching assuming a local surface structure, refers to the label map of the multiple viewpoints, and if the number of labels indicating a line structure-likeness at the first provisional depth of the first provisional depth map is greater than the number of labels indicating a surface structure-likeness, corrects the value of the first provisional depth to a value indicating that it is invalid, and sets the corrected value as the depth value of the first depth map; the second depth map generation means generates the second depth map based on the multi-view image and the label map by stereo matching assuming a local surface structure, refers to the label map of the multiple viewpoints, and if the number of labels indicating a surface structure-likeness at the second provisional depth of the second provisional depth map is greater than the number of labels indicating a line structure-likeness, corrects the value of the second provisional depth to a value indicating that it is invalid, and sets the corrected value as the depth value of the second depth map. A program according to one aspect of the present invention includes a label map generating means for generating a label map in which a label indicating a likelihood of a surface structure and a likelihood of a line structure is assigned to each pixel in each viewpoint image of a multi-viewpoint image, which is two or more images obtained by capturing an object from different viewpoints; a three-dimensional information map generating means for generating a three-dimensional information map that indicates three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map, the three-dimensional information map being a depth map including a first depth map in an area of ​​the surface structure and a second depth map in an area of ​​the line structure; and a three-dimensional information generating means for generating a three-dimensional information map that indicates three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the multi-viewpoint images and the label map. and a second depth map generation means for generating the second depth map by stereo matching assuming a local line structure, based on the multi-view image and the label map, wherein the first depth map generation means performs stereo matching using a weighting function that increases the weight of pixels in the label map that indicate a surface structure-likeness, and the second depth map generation means performs stereo matching using a weighting function that increases the weight of pixels in the label map that indicate a line structure-likeness. Effect of the Invention

[0012] According to the present invention, it is possible to three-dimensionally reconstruct with high accuracy the shape of an object having a mixture of planar and line structures. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of a configuration of a three-dimensional shape model generating device according to an embodiment of the present invention. [Diagram 2] 1A to 1C are diagrams illustrating an example of a multi-view image of an object according to an embodiment of the present invention. [Diagram 3] FIG. 13 is a diagram illustrating an example of a weighting function used in generating a label map according to the embodiment. [Figure 4] FIG. 13 is a diagram illustrating an example of a weighting function used in generating a label map according to the embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of a weighting function used in generating a label map according to the embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of a label map according to the embodiment. [Figure 7] 4 is a diagram showing an example of a surface structure depth map according to the embodiment; FIG. [Figure 8] FIG. 4 is a diagram showing an example of a line structure depth map according to the embodiment; [Figure 9] 1A to 1C are diagrams illustrating an example of a three-dimensional reconstruction result of a surface structure portion of an object according to the present embodiment. [Figure 10] 1A to 1C are diagrams illustrating an example of a three-dimensional reconstruction result of a line structure portion in an object according to the present embodiment. [Figure 11] 1A to 1C are diagrams illustrating an example of a result of 3D reconstruction of an entire object according to an embodiment of the present invention. [Figure 12] 5 is a flowchart showing an example of a processing flow in the three-dimensional shape model generating device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0015] <1. Configuration of the 3D shape model generating device> Fig. 1 is a block diagram showing an example of the configuration of a three-dimensional shape model generating device according to this embodiment. As shown in Fig. 1, the three-dimensional shape model generating device 100 includes, for example, a multi-view image input unit 110, a label map generating unit 120, a three-dimensional information map generating unit 130, a multi-view image storage unit 140, a label map storage unit 150, a depth map storage unit 160, and a three-dimensional model storage unit 170.

[0016] In this embodiment, an external device (not shown) captures images of an object to be restored as a three-dimensional shape model from multiple different viewpoints. A plurality of captured images of the object captured by the external device are supplied as data of viewpoint images. Each of the viewpoint images is a multi-view image captured from a different viewpoint.

[0017] (1) Multi-viewpoint image input unit 110 The multi-viewpoint image input unit 110 inputs data of a multi-viewpoint image, which is a plurality of images of an object captured by an imaging device (not shown) from a plurality of different viewpoints. The multi-viewpoint image input unit 110 assigns viewpoint identification information to the multi-viewpoint image. The multi-viewpoint image input unit 110 stores the multi-viewpoint image with the viewpoint identification information assigned in the multi-viewpoint image storage unit 140.

[0018] Here, a multi-view image of an object according to this embodiment will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of a multi-view image of an object according to this embodiment. A multi-view image 200 shown in FIG. 2 is one of a plurality of multi-view images captured by an imaging device (not shown). An object 201 is captured in the multi-view image 200. The object 201 is a mannequin 204 wearing a wig 202 and a shirt 203. The three-dimensional shape of the object 201 is composed of a surface structure and a line structure. The surface structure of the object 201 is the shirt 203 and the skin portion (face and neck) of the mannequin 204. The line structure of the object 201 is the wig 202.

[0019] (2) Label map generator 120 The label map generating unit 120 generates a label map to which a label indicating the likelihood of a surface structure and the likelihood of a line structure is assigned to each pixel in each viewpoint image of a multi-viewpoint image. The label map generating unit 120 stores the generated label map in the label map storage unit 150.

[0020] The label map is, for example, a two-channel map having two values ​​for each pixel. Of the two channels, the first channel has a value from 0 to 1 indicating the probability that the pixel has a plane structure, and the second channel has a value from 0 to 1 indicating the probability that the pixel has a line structure. In other words, the closer the value of the first channel of the label map is to 1, the higher the probability that the object has a plane structure, and the closer the value of the second channel of the label map is to 1, the higher the probability that the object has a line structure. The channel value of each pixel of the label map may be constrained so that the sum of the values ​​of the two channels is 1. The value of each channel of the label map may be a real number value from 0 to 1, or may be an integer value of 0 or 1. The label map may be configured as a two-channel map or a one-channel map. When the label map is configured as a one-channel map, for example, the value of each pixel indicates the probability that the object has a plane structure, and the value obtained by subtracting the value of each pixel from 1 indicates the probability that the object has a line structure.

[0021] The label map generating unit 120 generates a label map based on each image of the multi-view image. For example, the label map generating unit 120 generates a label map for each viewpoint image based on the luminance value and color information of each viewpoint image of the multi-view image. The label map generating unit 120 may generate a label map for each pixel of each image of the multi-view image using only the luminance value and color information of the pixel, or may generate a label map using the luminance value and color information of the pixel and surrounding pixels in combination.

[0022] The label map generating unit 120 may generate the label map based on the depth map. The depth map is map information indicating the distance in the depth direction from the imaging device that captured each viewpoint image to the target object. For example, the label map generating unit 120 may generate the label map using a surface structure depth map (an example of a first depth map) generated by a surface structure depth map generating unit 131 (an example of a first depth map generating unit) described later and a line structure depth map (an example of a second depth map) generated by a line structure depth map generating unit 132 (an example of a second depth map generating unit) described later. The surface structure depth map is a depth map generated by stereo matching assuming a local (for example, pixel-by-pixel) surface structure. The line structure depth map is a depth map generated by stereo matching assuming a local line structure. The details of the generation of each depth map by the surface structure depth map generating unit 131 and the line structure depth map generating unit 132 will be described later.

[0023] Furthermore, the label map generating unit 120 may generate a label map by using not only each depth map generated by the surface structure depth map generating unit 131 and the line structure depth map generating unit 132, but also a matching score map indicating a matching score calculated by stereo matching when generating the depth map. As an example, the label map generating unit 120 generates a label map by comparing a matching score map calculated when the surface structure depth map generating unit 131 generates the surface structure depth map with a matching score map calculated when the line structure depth map generating unit 132 generates the line structure depth map.

[0024] Specifically, the label map generating unit 120 generates a label map based on information obtained in generating a provisional surface structure depth map (an example of a first provisional depth map) generated as intermediate data by the surface structure depth map generating unit 131 and a provisional line structure depth map (an example of a second provisional depth map) generated as intermediate data by the line structure depth map generating unit 132. Note that the provisional depth maps generated by each of the surface structure depth map generating unit 131 and the line structure depth map generating unit 132 do not necessarily need to use a label map.

[0025] The label map generating unit 120 calculates a surface structure matching score (an example of a first matching score) which is a matching score of stereo matching for the provisional surface structure depth map, and a line structure matching score (an example of a second matching score) which is a matching score of stereo matching for the provisional line structure depth map. The label map generating unit 120 compares the surface structure matching score with the line structure matching score. If the surface structure matching score is higher than the line structure matching score, the label map generating unit 120 determines a label indicating a high surface structure likelihood. On the other hand, if the line structure matching score is higher than the surface structure matching score, the label map generating unit 120 determines a label indicating a high line structure likelihood.

[0026] More specifically, when each matching score is calculated based on the similarity in stereo matching, if the matching score for each pixel of the label map calculated by the surface structure depth map generation unit 131 is higher than the matching score for the same pixel calculated by the line structure depth map generation unit 132, the label map generation unit 120 sets the value of the first channel, which indicates the surface structure-like nature of the label map, to a high value.

[0027] Similarly, for each pixel of the label map, if the matching score for the same pixel calculated by the line structure depth map generation unit 132 is higher than the matching score for the same pixel calculated by the surface structure depth map generation unit 131, the label map generation unit 120 sets the value of the second channel, which indicates the line structure-likeliness of the label map, to a higher value.

[0028] However, if each matching score is calculated based on the degree of dissimilarity in stereo matching, the magnitude relationship between the matching score calculated by the surface structure depth map generation unit 131 and the matching score calculated by the line structure depth map generation unit 132 will be the opposite of when each matching score is calculated based on the degree of similarity.

[0029] The label map generating unit 120 may generate a label map based on a weight function. The weight function is, for example, a function based on a line structure, and is calculated based on a temporary line structure depth map. The label map generating unit 120 performs stereo matching on a local surface structure calculated from the temporary surface structure depth map using a weight function based on a line structure. The label map generating unit 120 calculates a matching score in the stereo matching. The label map generating unit 120 determines a label depending on the magnitude of the calculated matching score.

[0030] Here, an example of the weighting function according to this embodiment will be described with reference to Fig. 3 to Fig. 5. Fig. 3 to Fig. 5 are diagrams showing an example of the weighting function in generating a label map according to this embodiment. Fig. 3 shows an example of a matching window 210 and a line structure 211. The matching window 210 indicates a predetermined area (local area) in a multi-view image.

[0031] Figures 4 and 5 show an example of a weighting function based on the line structure 211 in the matching window 210 shown in Figure 3. The value of each pixel in the weighting function is a real number between 0 and 1. The closer the color of each pixel in each weighting function in Figures 4 and 5 is to white, the larger the weight of the pixel (closer to 1), and the closer the color is to black, the smaller the weight of the pixel (closer to 0).

[0032] For example, the weighting function 220 shown in Fig. 4 is a function that increases (weights) as the pixel values ​​(weights) move along the line structures 211 (color is white, value approaches 1). On the other hand, the weighting function 230 shown in Fig. 5 is a function that decreases (weights) as the pixel values ​​(weights) move along the line structures 211 (color is black, value approaches 0). The angle of the weighting function (line structures 211) follows the angle of the line structures calculated from the second depth map.

[0033] For example, the label map generating unit 120 performs stereo matching for a local surface structure calculated from the surface structure depth map using each of the weighting function 220 shown in Fig. 4 and the weighting function 230 shown in Fig. 5. The label map generating unit 120 calculates and compares matching scores in stereo matching using each weighting function.

[0034] Here, the label map generation unit 120 may assume a local surface structure from only the surface structure depth map generated by the surface structure depth map generation unit 131, or may assume a local surface structure using both the surface structure depth map and the normal vector map generated by the surface structure depth map generation unit 131.

[0035] In the calculation of the matching score using a weighting function, the higher the value of the weighting function, the stronger the influence of the pixel on the matching score. For example, when the object has a surface structure, the matching score in stereo matching using either weighting function 220 or weighting function 230 will be about the same. On the other hand, when the object has a line structure, the matching score (corresponding to similarity) will be high in stereo matching using a weighting function in which the values ​​of pixels along the line structure are high, such as weighting function 220. However, when the object has a line structure, the matching score in stereo matching using a weighting function in which the values ​​of pixels around the line structure are high, such as weighting function 230, will be low.

[0036] Thus, when the difference between the matching score calculated using the weighting function 220 and the matching score calculated using the weighting function 230 is small, the label map generating unit 120 sets a high value in the first channel, which indicates the likelihood of the label map having a surface structure. On the other hand, when the difference between the matching score calculated using the weighting function 220 and the matching score calculated using the weighting function 230 is large, the label map generating unit 120 sets a high value in the second channel, which indicates the likelihood of the label map having a line structure.

[0037] The label map generating unit 120 may generate the label map using, for example, a machine learning method. One of the machine learning methods called deep learning is a convolutional neural network (CNN). The label map generating unit 120 generates the label map using the CNN. The CNN is made to learn parameters using an input image and a label map corresponding to the input image as a learning data set. This makes it possible to construct a CNN that generates a label map. As an example, the CNN is made to learn using, as an input, information on each image of a multi-view image, a surface structure depth map and a line structure depth map, and each matching score map calculated when the surface structure depth map and the line structure depth map are generated, or a combination of each information.

[0038] In addition, the label map generating unit 120 may generate two or more types of label maps from among a label map generated based on a matching score map, a label map generated based on a difference in matching scores using a weighting function, and a label map generated by CNN, and combine them to create a single label map.

[0039] The label map generating unit 120 may repeatedly perform a process of generating a label map using the depth maps generated by each of the surface structure depth map generating unit 131 and the line structure depth map generating unit 132. For example, the label map generating unit 120 regenerates a label map using the depth maps generated by each of the surface structure depth map generating unit 131 and the line structure depth map generating unit 132 based on the generated label map. The label map generating unit 120 regenerates a label map using the depth maps generated by each of the surface structure depth map generating unit 131 and the line structure depth map generating unit 132 based on the regenerated label map. By repeating the generation of the label map in this manner, the label map generating unit 120 can improve the accuracy of the label map generation.

[0040] Here, an example of a label map according to this embodiment will be described with reference to Fig. 6. Fig. 5 is a diagram showing an example of a label map according to this embodiment. In the label map 240 shown in Fig. 6, the parts of colour C1 and colour C2 indicate a plane structure, and the part of colour C3 indicates a line structure. The part shown in colour C1 is specifically the skin part of the face of the mannequin 204. The part shown in colour C2 is specifically the skin part of the neck of the mannequin 204, the shirt 203 and the background part. The part shown in colour C3 is specifically the wig 202.

[0041] (3) Three-dimensional information map generator 130 The three-dimensional information map generating unit 130 generates a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of an object having a surface structure and a line structure based on the label map. In the following, in this embodiment, an example will be described in which the three-dimensional information map generated based on the label map is a depth map including a surface structure depth map in a surface structure area and a line structure depth map in a line structure area.

[0042] As shown in FIG. 1, the three-dimensional information map generation unit 130 includes a surface structure depth map generation unit 131 (an example of a first depth map generation unit), a line structure depth map generation unit 132 (an example of a second depth map generation unit), a surface structure three-dimensional model generation unit 133 (an example of a first three-dimensional model generation unit), and a line structure three-dimensional model generation unit 134 (an example of a second three-dimensional model generation unit).

[0043] (3-1) Surface structure depth map generating unit 131 The surface structure depth map generating unit 131 generates a surface structure depth map. For example, the surface structure depth map generating unit 131 generates a surface structure depth map by stereo matching assuming a local surface structure based on a multi-view image and a label map. Specifically, the surface structure depth map generating unit 131 generates a surface structure depth map corresponding to the multi-view image of each viewpoint using each of the multi-view image and the label map corresponding to each viewpoint. The surface structure depth map generating unit 131 stores the generated surface structure depth map in the depth map storage unit 160. The surface structure depth map is a map indicating depth information in three-dimensional coordinates for each pixel of the multi-view image of each viewpoint. The surface structure depth map generating unit 131 controls stereo matching using the label map generated by the label map generating unit 120, thereby making it possible to restore only the surface structure of the target.

[0044] The surface structure depth map generating unit 131 may store a matching score map indicating a matching score for each pixel calculated when generating the surface structure depth map together with the surface structure depth map in the depth map storage unit 160. The surface structure depth map generating unit 131 may also store a normal vector map for each pixel estimated simultaneously when generating the surface structure depth map together with the surface structure depth map in the depth map storage unit 160. Here, the normal vector map is a map indicating surface orientation information in three-dimensional coordinates for each pixel of the multi-viewpoint images from each viewpoint.

[0045] The surface structure depth map generating unit 131 performs matching between a reference viewpoint of a multi-viewpoint image and a group of neighboring viewpoints to be matched in stereo matching. At this time, the surface structure depth map generating unit 131 does not perform matching on the entire image, but performs matching between local regions (matching windows) of the image. For each pixel of the reference viewpoint, the surface structure depth map generating unit 131 sets a matching window around the pixel, and assumes a depth and a normal vector of a certain three-dimensional coordinate. This allows the surface structure depth map generating unit 131 to set a matching window for each viewpoint of the group of neighboring viewpoints. The surface structure depth map generating unit 131 calculates a similarity between the matching window of the reference viewpoint and the matching window of the neighboring viewpoint. Then, the surface structure depth map generating unit 131 sets the calculated similarity as a matching score.

[0046] In stereo matching, the surface structure depth map generator 131 searches for the depth and normal vector direction that maximizes the matching score for each pixel of the reference viewpoint. In addition, the surface structure depth map generator 131 assumes a local surface structure, and assumes that the three-dimensional shape of the target is a small plane defined by a depth and a normal vector. Therefore, the shape of a matching window set to, for example, a square at the reference viewpoint is set to a shape obtained by projective transformation of the square, not a square, at nearby viewpoints. As a matching score indicating the similarity between the matching windows, for example, normalized cross-correlation is used. As a method for searching for the depth and normal vector direction that maximizes the matching score, for example, a patch match stereo method or plane sweeping is used.

[0047] Here, the surface structure depth map generating unit 131 may generate a surface structure depth map based on a provisional surface structure depth map and a label map. For example, the surface structure depth map generating unit 131 generates a provisional surface structure depth map by stereo matching assuming a local surface structure based on a multi-view image without using a label map. That is, the provisional surface structure depth map is a surface structure depth map generated without using a label map.

[0048] The surface structure depth map generating unit 131 modifies the value of the provisional surface structure depth (an example of the first provisional depth) in the generated provisional surface structure depth map based on the label map, and sets the modified value as the depth value of the surface structure depth map. Specifically, the surface structure depth map generating unit 131 refers to the label maps of multiple viewpoints, and compares the labels in the label maps with the labels in the provisional surface structure depth map.

[0049] If the label indicating the surface structure resemblance in the provisional surface structure depth is greater than the label indicating the line structure resemblance, the surface structure depth map generating unit 131 does not modify the value of the provisional surface structure depth. Then, the surface structure depth map generating unit 131 sets the unmodified provisional surface structure depth value as the depth value of the surface structure depth map. On the other hand, if the label indicating the line structure resemblance in the provisional surface structure depth is greater than the label indicating the surface structure resemblance, the surface structure depth map generating unit 131 modifies the provisional surface structure depth value to a value indicating that it is invalid (hereinafter also referred to as an "invalid value"). Then, the surface structure depth map generating unit 131 sets the modified provisional surface structure depth value as the depth value of the surface structure depth map. The invalid value is, for example, a value such as -1, but is not limited to such an example.

[0050] That is, for each pixel of the reference viewpoint, the surface structure depth map generating unit 131 compares the values ​​of each channel of the label map corresponding to the same pixel of the same viewpoint. If the value of the first channel indicating the surface structure is greater than the value of the second channel indicating the line structure, the surface structure depth map generating unit 131 substitutes the provisional surface structure depth value as it is for the depth of the pixel. If the value of the second channel indicating the line structure is greater than the value of the first channel indicating the surface structure, the surface structure depth map generating unit 131 substitutes a modified value of the provisional surface structure depth for the depth of the pixel.

[0051] The surface structure depth map generating unit 131 may correct the value of the provisional surface structure depth using label maps of multiple viewpoints, such as a label map of the reference viewpoint and a group of neighboring viewpoints, a label map of all viewpoints, etc., instead of using a label map of only the reference viewpoint. In this case, the surface structure depth map generating unit 131 can obtain corresponding pixels for viewpoints other than the reference viewpoint by projecting three-dimensional coordinates calculated from the provisional surface structure depth onto each viewpoint.

[0052] Furthermore, the surface structure depth map generating unit 131 may be configured to determine visibility when determining pixels corresponding to each viewpoint, and not use viewpoints from which the three-dimensional coordinates calculated from the provisional surface structure depth are not visible in modifying the provisional surface structure depth. In this case, the surface structure depth map generating unit 131 can determine the provisional surface structure depth for all pixels of the multiple viewpoints, generate provisional surface structure depth maps for the multiple viewpoints, and compare the provisional surface structure depth maps to determine visibility while taking occlusion into consideration.

[0053] When using label maps of multiple viewpoints, the surface structure depth map generating unit 131 compares the value of the first channel indicating the surface structure and the value of the second channel indicating the line structure for pixels corresponding to each viewpoint. The surface structure depth map generating unit 131 counts the number of viewpoints in which the value of the first channel indicating the surface structure is greater than the value of the second channel indicating the line structure. Similarly, the surface structure depth map generating unit 131 counts the number of viewpoints in which the value of the second channel indicating the line structure is greater than the value of the first channel indicating the surface structure. As a result of the counting, it is assumed that the number of viewpoints in which the value of the first channel indicating the surface structure is greater than the value of the second channel indicating the line structure is smaller than the number of viewpoints in which the value of the second channel indicating the line structure is greater than the value of the first channel indicating the surface structure. In this case, the surface structure depth map generating unit 131 corrects the provisional surface structure depth to an incorrect value. In this way, the surface structure depth map generator 131 is able to accurately correct the depth of pixels that appear to be line structures by using multiple viewpoints, even if the label map at a certain viewpoint is inaccurate or the label at a certain pixel is inaccurate.

[0054] Furthermore, the surface structure depth map generating unit 131 may process the modification of the provisional surface structure depth by the label map with the depth value in the surface structure depth map, or may process the modification of the provisional surface structure depth with respect to the three-dimensional points calculated from the provisional surface structure depth. Considering that, for a certain viewpoint, the three-dimensional information represented by the surface structure depth map and the three-dimensional information represented by the three-dimensional point cloud calculated from the surface structure depth map are equivalent, modifying the provisional surface structure depth with the surface structure depth map is equivalent to modifying the provisional surface structure depth with the three-dimensional point cloud.

[0055] The method of stereo matching using the label map by the surface structure depth map generating unit 131 is not limited to the above example. For example, the surface structure depth map generating unit 131 may perform stereo matching by a method using a weighting function according to the label map. The weighting function according to the label map is, for example, a function that increases the weight of a pixel that indicates a surface structure in the label map.

[0056] For example, for each pixel in the matching window set at the reference viewpoint, the surface structure depth map generating unit 131 sets the value of the first channel indicating the surface structure of the label map corresponding to the same viewpoint as the weight of each pixel. When calculating the matching score, the surface structure depth map generating unit 131 calculates the matching score using the weight of each pixel. As a result, the influence of a pixel determined to be a surface structure in the label map on the matching score becomes large. Therefore, in the calculation of the depth of a pixel of a surface structure in a pixel where a surface structure and a line structure are adjacent to each other, the effect of the adjacent line structure pixel is reduced, and the accuracy of the surface structure depth map generated by the surface structure depth map generating unit 131 is improved.

[0057] Furthermore, the surface structure depth map generating unit 131 may perform stereo matching by a method of calculating depth only from pixels in the label map that indicate a surface structure. For example, the surface structure depth map generating unit 131 performs stereo matching only from pixels in the label map that have a high value in the first channel that indicates a surface structure. This allows depth calculation to be performed only from pixels in the first channel that indicate a surface structure. Therefore, depth calculation is not performed for pixels that have a line structure as an object. Thus, the surface structure depth map generating unit 131 can reduce calculation costs.

[0058] Here, an example of a surface structure depth map according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a surface structure depth map according to this embodiment. The colors in the surface structure depth map 250 shown in Fig. 7 are set so that the greater the distance in the depth direction (i.e., the further back the object is located), the closer the color becomes to white, and the smaller the distance in the depth direction (i.e., the closer the object is located), the closer the color becomes to black. Note that in Fig. 7, the background wall portion is located at the farthest back, but for convenience of illustration, the color is set to black.

[0059] (3-2) Linear structure depth map generating unit 132 The line structure depth map generating unit 132 generates a line structure depth map. For example, the line structure depth map generating unit 132 generates a line structure depth map by stereo matching assuming a local line structure based on a multi-view image and a label map. Specifically, the line structure depth map generating unit 132 generates a line structure depth map corresponding to the multi-view image of each viewpoint using each of the multi-view image and the label map corresponding to each viewpoint. The line structure depth map generating unit 132 stores the generated line structure depth map in the depth map storage unit 160. The line structure depth map is a map indicating depth information in three-dimensional coordinates for each pixel of the multi-view image of each viewpoint. The line structure depth map generating unit 132 controls stereo matching using the label map generated by the label map generating unit 120, thereby making it possible to restore only the line structure of the target.

[0060] The line structure depth map generating unit 132 may store a matching score map indicating a matching score for each pixel calculated when generating the line structure depth map together with the line structure depth map in the depth map storage unit 160. The line structure depth map generating unit 132 may store a line segment direction vector map for each pixel estimated at the same time when generating the line structure depth map together with the line structure depth map in the depth map storage unit 160. Here, the line segment direction vector map is a map indicating the direction information of the line structure in the three-dimensional coordinates for each pixel of the multi-viewpoint images of each viewpoint.

[0061] The line structure depth map generating unit 132 performs matching between the reference viewpoint of the multi-viewpoint image and a group of neighboring viewpoints to be matched in the stereo matching. At this time, the line structure depth map generating unit 132 performs matching between matching windows, similar to the stereo matching of the surface structure depth map generating unit 131. For each pixel of the reference viewpoint, the line structure depth map generating unit 132 sets a matching window around the pixel, and assumes a depth of a certain three-dimensional coordinate and a line segment direction vector. This allows the line structure depth map generating unit 132 to set a matching window for each viewpoint of the group of neighboring viewpoints. The line structure depth map generating unit 132 calculates a similarity between the matching window of the reference viewpoint and the matching window of the neighboring viewpoint. Then, the line structure depth map generating unit 132 sets the calculated similarity as a matching score.

[0062] In stereo matching, the line structure depth map generating unit 132 searches for the depth and the direction of the line segment direction vector that maximizes the matching score for each pixel of the reference viewpoint. In addition, the line structure depth map generating unit 132 assumes a local line structure, and assumes that the three-dimensional shape of the target is a small three-dimensional line segment defined by a depth and a line segment direction vector. Therefore, for example, the shape of the matching window for each viewpoint is set to a rectangle along a two-dimensional line segment obtained by projecting the assumed three-dimensional line segment onto each viewpoint.

[0063] For example, normalized cross-correlation is used as the matching score indicating the similarity between the matching windows. Note that, when calculating the matching score indicating the similarity between the matching windows, the line structure depth map generating unit 132 may calculate 2D-Orientation indicating the direction of the local texture of each image and calculate the matching score from the angle difference of the 2D-Orientation of the corresponding pixels of each viewpoint. As a method for searching for the depth and the direction of the line segment direction vector that maximizes the matching score, for example, a patch match stereo method is used.

[0064] Here, the line structure depth map generating unit 132 may generate the line structure depth map based on the provisional line structure depth map and the label map. For example, the line structure depth map generating unit 132 generates the provisional line structure depth map by stereo matching assuming a local line structure based on a multi-view image without using a label map. That is, the provisional line structure depth map is a line structure depth map generated without using a label map.

[0065] The linear structure depth map generating unit 132 modifies the value of the provisional linear structure depth (an example of the second provisional depth) in the provisional linear structure depth map based on the label map, and sets the modified value as the depth value of the linear structure depth map. Specifically, the linear structure depth map generating unit 132 refers to the label maps of multiple viewpoints and compares the labels in the label maps with the labels in the provisional linear structure depth map. If the label indicating the surface structure-likeness in the provisional lineation depth is greater than the label indicating the surface structure-likeness, the lineation depth map generating unit 132 does not modify the provisional lineation depth value. Then, the lineation depth map generating unit 132 sets the unmodified provisional lineation depth value as the depth value of the lineation depth map. On the other hand, if the label indicating the surface structure-likeness in the provisional lineation depth is greater than the label indicating the surface structure-likeness, the lineation depth map generating unit 132 modifies the provisional lineation depth value to an invalid value. Then, the lineation depth map generating unit 132 sets the modified provisional lineation depth value as the depth value of the lineation depth map. The invalid value is, for example, a value such as -1, but is not limited to such an example.

[0066] That is, for each pixel of the reference viewpoint, the line structure depth map generating unit 132 compares the values ​​of each channel of the label map corresponding to the same pixel of the same viewpoint. If the value of the second channel indicating the line structure-likeness is greater than the value of the first channel indicating the surface structure-likeness, the line structure depth map generating unit 132 substitutes the provisional line structure depth value as it is for the depth of the pixel. If the value of the first channel indicating the surface structure-likeness is greater than the value of the second channel indicating the line structure-likeness, the line structure depth map generating unit 132 substitutes a modified value of the provisional line structure depth for the depth of the pixel.

[0067] The lineation depth map generating unit 132 may correct the value of the provisional lineation depth using label maps of multiple viewpoints, such as a label map of the reference viewpoint and a group of neighboring viewpoints, a label map of all viewpoints, etc., instead of using a label map of only the reference viewpoint. In this case, the lineation depth map generating unit 132 can obtain corresponding pixels for viewpoints other than the reference viewpoint by projecting three-dimensional coordinates calculated from the provisional lineation depth to each viewpoint.

[0068] Also, the lineation depth map generating unit 132 may be configured to determine visibility when obtaining corresponding pixels of each viewpoint, and not use viewpoints that do not see the three-dimensional coordinates calculated from the provisional lineation depth for correcting the provisional lineation depth. In this case, the lineation depth map generating unit 132 can determine visibility while taking occlusion into consideration by obtaining provisional lineation depths for all pixels of the multiple viewpoints, generating provisional lineation depth maps for the multiple viewpoints, and comparing the provisional lineation depth maps with each other.

[0069] When using a label map of multiple viewpoints, the linear structure depth map generating unit 132 compares the value of the second channel indicating the linear structure-likeness with the value of the first channel indicating the planar structure-likeness for pixels corresponding to each viewpoint. The linear structure depth map generating unit 132 counts the number of viewpoints in which the value of the second channel indicating the linear structure-likeness is greater than the value of the first channel indicating the planar structure-likeness. Similarly, the linear structure depth map generating unit 132 counts the number of viewpoints in which the value of the first channel indicating the planar structure-likeness is greater than the value of the second channel indicating the linear structure-likeness. As a result of the counting, it is assumed that the number of viewpoints in which the value of the second channel indicating the linear structure-likeness is greater than the value of the first channel indicating the planar structure-likeness is smaller than the number of viewpoints in which the value of the first channel indicating the planar structure-likeness is greater than the value of the second channel indicating the linear structure-likeness. In this case, the linear structure depth map generating unit 132 corrects the provisional linear structure depth to an incorrect value. In this way, the line structure depth map generation unit 132 is able to accurately correct the depth of pixels that appear to have a surface structure by using multiple viewpoints, even if the label map at a certain viewpoint is inaccurate or the label at a certain pixel is inaccurate.

[0070] In addition, the lineation depth map generating unit 132 may process the correction of the provisional lineation depth by the label map with the depth value in the lineation depth map, or may process it with respect to the three-dimensional points calculated from the provisional lineation depth. Considering that the three-dimensional information represented by the lineation depth map and the three-dimensional information represented by the three-dimensional point cloud calculated from the lineation depth map are equivalent for a certain viewpoint, correcting the provisional lineation depth by the lineation depth map is equivalent to correcting the provisional lineation depth by the three-dimensional point cloud.

[0071] The method of stereo matching using the label map by the line structure depth map generating unit 132 is not limited to the above example. For example, the line structure depth map generating unit 132 may perform stereo matching by a method using a weight function according to the label map. The weight function according to the label map is, for example, a function that increases the weight of a pixel that indicates a line structure-likeness in the label map.

[0072] For example, for each pixel in the matching window set at the reference viewpoint, the line structure depth map generating unit 132 sets the value of the second channel indicating the line structure of the corresponding label map of the same viewpoint as the weight of each pixel. When calculating the matching score, the line structure depth map generating unit 132 calculates the matching score using the weight of each pixel. As a result, the influence of a pixel determined to be a line structure in the label map on the matching score becomes large. Therefore, when calculating the depth of a pixel of a line structure in a pixel where a surface structure and a line structure are adjacent to each other, the effect of the adjacent surface structure pixel is reduced, and the accuracy of the line structure depth map generated by the line structure depth map generating unit 132 is improved.

[0073] In addition, the line structure depth map generating unit 132 may perform stereo matching by a method of calculating depth only from pixels in the label map that indicate line structure resemblance. For example, the line structure depth map generating unit 132 performs stereo matching only from pixels in the label map that have a high value in the second channel that indicates line structure resemblance. This causes depth calculation to be performed only from pixels in the second channel that indicate line structure resemblance. Therefore, depth calculation is not performed for pixels that have a planar structure as the target. Therefore, the line structure depth map generating unit 132 can reduce calculation costs.

[0074] Here, an example of a lineation depth map according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of a lineation depth map according to this embodiment. The colors in the lineation depth map 260 shown in Fig. 8 are set to approach white as the distance in the depth direction increases (i.e., the further back the object is located), and to approach black as the distance in the depth direction decreases (i.e., the closer the object is located). Note that in Fig. 8, the background wall portion is located at the farthest back, but for convenience of illustration, the color is set to black.

[0075] (3-3) Planar structure three-dimensional model generation unit 133 The plane structure three-dimensional model generating unit 133 generates a three-dimensional mesh model related to the plane structure of the object. For example, the plane structure three-dimensional model generating unit 133 reads out a plane structure depth map from the depth map storage unit 160, and generates a plane structure three-dimensional point group (an example of a first three-dimensional point group) indicating the plane structure of the object from the plane structure depth map. The plane structure three-dimensional model generating unit 133 stores the generated plane structure three-dimensional point group in the three-dimensional model storage unit 170. In addition, the plane structure three-dimensional model generating unit 133 may further generate a three-dimensional mesh model indicating the plane structure of the object from the generated plane structure three-dimensional point group. When the three-dimensional mesh model is generated, the plane structure three-dimensional model generating unit 133 may store the generated three-dimensional mesh model in the three-dimensional model storage unit 170.

[0076] Here, an example of a three-dimensional reconstruction result of a surface structure portion of an object according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of a three-dimensional reconstruction result of a surface structure portion of an object according to this embodiment. A surface structure reconstruction result 270 shown in Fig. 9 shows a three-dimensional shape model of the skin portion (face and neck) and shirt 203 of a mannequin 204 reconstructed as a surface structure.

[0077] (3-4) Linear structure 3D model generation unit 134 The line structure three-dimensional model generating unit 134 generates a three-dimensional mesh model related to the line structure of the object. For example, the line structure three-dimensional model generating unit 134 reads out a line structure depth map from the depth map storage unit 160, and generates a line structure three-dimensional point cloud (an example of a second three-dimensional point cloud) indicating the line structure of the object from the line structure depth map. The line structure three-dimensional model generating unit 134 stores the generated line structure three-dimensional point cloud in the three-dimensional model storage unit 170. The line structure three-dimensional model generating unit 134 may further generate a three-dimensional broken line group indicating the line structure of the object from the generated line structure three-dimensional point cloud. When the three-dimensional broken line group is generated, the line structure three-dimensional model generating unit 134 may store the generated three-dimensional broken line group in the three-dimensional model storage unit 170. The line structure three-dimensional model generating unit 134 may also generate a three-dimensional mesh model by applying a thin cylindrical three-dimensional mesh model to the generated three-dimensional broken line group. When the three-dimensional mesh model is generated, the line structure three-dimensional model generation unit 134 may store the generated three-dimensional mesh model in the three-dimensional model storage unit 170 .

[0078] Here, an example of a three-dimensional reconstruction result of a line structure part in an object according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of a three-dimensional reconstruction result of a line structure part in an object according to this embodiment. A line structure reconstruction result 280 shown in Fig. 10 shows a three-dimensional shape model of a wig 202 worn by a mannequin 204, which is reconstructed as a line structure.

[0079] Here, an example of a result of three-dimensional reconstruction of the entire object according to this embodiment will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of a result of three-dimensional reconstruction of the entire object according to this embodiment. The object reconstruction result 290 shown in Fig. 11 shows a three-dimensional shape model of the skin part of the face and neck of the mannequin 204 and the shirt 203, which are reconstructed as a plane structure, and a three-dimensional shape model of the wig 202 worn by the mannequin 204, which is reconstructed as a line structure. Even if the three-dimensional shape of the object 201 contains a mixture of surface and line structures as shown in Figure 2, by applying the three-dimensional restoration method of this embodiment, the shape of the object 201 containing a mixture of surface and line structures can be three-dimensionally restored with high accuracy as shown in Figure 11.

[0080] (4) Multi-viewpoint image storage unit 140 Multi-viewpoint image storage section 140 stores the data of the multi-viewpoint image input from multi-viewpoint image input section 110 in association with the viewpoint identification information.

[0081] (5) Label map storage unit 150 The label map storage unit 150 stores the data of the label map generated by the label map generation unit 120.

[0082] (6) Depth map storage unit 160 The depth map storage unit 160 stores the surface structure depth map generated by the surface structure depth map generation unit 131 and the line structure depth map generated by the line structure depth map generation unit 132. The depth map storage unit 160 may also store the matching score map and the normal vector map generated by the surface structure depth map generation unit 131. The depth map storage unit 160 may also store the matching score map and the line segment direction vector map generated by the line structure depth map generation unit 132.

[0083] (7) Three-dimensional model storage unit 170 The three-dimensional model storage unit 170 stores the plane structure three-dimensional point cloud generated by the plane structure three-dimensional model generation unit 133 and the line structure three-dimensional point cloud generated by the line structure three-dimensional model generation unit 134. The three-dimensional model storage unit 170 may also store a three-dimensional mesh model generated by the plane structure three-dimensional model generation unit 133. The three-dimensional model storage unit 170 may also store a three-dimensional broken line group or a three-dimensional mesh model generated by the line structure three-dimensional model generation unit 134.

[0084] <2. Processing flow> The above describes the configuration of the three-dimensional shape model generating device 100. Next, the flow of processing in the three-dimensional shape model generating device 100 according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the flow of processing in the three-dimensional shape model generating device 100 according to this embodiment.

[0085] As shown in FIG. 12, first, the multi-viewpoint image input unit 110 of the three-dimensional shape model generating device 100 inputs data of multi-viewpoint images of an object captured by an imaging device from a plurality of different viewpoints (step S1).

[0086] Next, the label map generating unit 120 of the three-dimensional shape model generating device 100 generates a label map based on the data of the multi-viewpoint images input by the multi-viewpoint image input unit 110 (step S2).

[0087] Next, the surface structure depth map generation unit 131 of the three-dimensional shape model generating device 100 generates a surface structure depth map based on the multi-view image data input by the multi-view image input unit 110 and the label map generated by the label map generation unit 120 (step S3).

[0088] Next, the planar structure three-dimensional model generating unit 133 of the three-dimensional shape model generating device 100 generates a planar structure three-dimensional point cloud based on the planar structure depth map generated by the planar structure depth map generating unit 131 (step S4).

[0089] Next, the planar three-dimensional model generating unit 133 generates a planar three-dimensional mesh model based on the generated planar three-dimensional point group (step S5).

[0090] In addition, the line structure depth map generation unit 132 of the three-dimensional shape model generating device 100 generates a line structure depth map based on the multi-view image data input by the multi-view image input unit 110 and the label map generated by the label map generation unit 120 (step S6).

[0091] Next, the lineation three-dimensional model generating unit 134 of the three-dimensional shape model generating device 100 generates a lineation three-dimensional point cloud based on the lineation depth map generated by the lineation depth map generating unit 132 (step S7).

[0092] Next, the lineation three-dimensional model generating unit 134 generates a lineation three-dimensional broken line group based on the generated lineation three-dimensional point group (step S8).

[0093] Next, the line structure three-dimensional model generating unit 134 generates a line structure three-dimensional mesh model based on the generated line structure three-dimensional broken line group (step S9).

[0094] As described above, in the three-dimensional shape model generating device 100 according to this embodiment, the label map generating unit 120 generates a label map in which a label indicating the likelihood of a surface structure and a line structure is assigned to each pixel in each viewpoint image of a multi-view image of an object. The three-dimensional information map generating unit 130 generates a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of an object having a surface structure and a line structure, based on the label map generated by the label map generating unit 120.

[0095] With this configuration, the three-dimensional shape model generating device 100 of this embodiment can restore a three-dimensional shape model of the surface structure portion of the object based on information regarding the surface structure indicated by the three-dimensional information map, and can restore a three-dimensional shape model of the line structure portion of the object based on information regarding the line structure indicated by the three-dimensional information map.

[0096] Therefore, the three-dimensional shape model generating device 100 according to this embodiment can three-dimensionally reconstruct the shape of an object having a mixture of planar and line structures with high accuracy.

[0097] <3. Modifications> The above is a description of the process flow in the three-dimensional shape model generating device 100 according to this embodiment. Next, a modification of this embodiment will be described.

[0098] In this embodiment, the three-dimensional shape model generating device 100 may be configured to alternately repeat the label map generation process by the label map generating unit 120 and the depth map generation process by the surface structure depth map generating unit 131 and the line structure depth map generating unit 132. This improves the accuracy of the generation of the label map by the label map generating unit 120. The improvement in the accuracy of the label map generation improves the accuracy of the generation of the surface structure depth map and the line structure depth map by the surface structure depth map generating unit 131 and the line structure depth map generating unit 132. The improvement in the accuracy of the generation of the surface structure depth map and the line structure depth map improves the accuracy of the generation of the label map by the label map generating unit 120. In this way, by alternately repeating the label map generation process and the depth map generation process, it is possible to improve the accuracy of the generation of the label map and the depth map each time the process is repeated. In addition, in such an iterative process, the label map generating unit 120 may be configured to generate a label map without using a surface structure depth map and a line structure depth map in the first iteration, so that the label map generating unit 120 can generate a label map even in the first iteration in which the surface structure depth map and the line structure depth map have not yet been generated. Alternatively, the surface structure depth map generating unit 131 and the line structure depth map generating unit 132 may be configured to perform stereo matching without using a label map in the first iteration to generate a depth map.

[0099] In the above embodiment, an example has been described in which the three-dimensional information map generated based on the label map is a depth map including a surface structure depth map and a line structure depth map, but is not limited to such an example. For example, the three-dimensional information map generated based on the label map may be a three-dimensional point cloud including a surface structure three-dimensional point cloud indicating the surface structure of the object and a line structure three-dimensional point cloud indicating the line structure of the object. In this case, the plane structure depth map generation unit 131 generates a plane structure depth map based only on the multi-viewpoint images input by the multi-viewpoint image input unit 110. Then, the plane structure 3D model generation unit 133 generates a plane structure 3D point cloud by stereo matching assuming a local plane structure, based on the plane structure depth map generated only from the multi-viewpoint images and the label map generated by the label map generation unit 120. Moreover, the line structure depth map generating unit 132 generates a line structure depth map based only on the multi-viewpoint images input by the multi-viewpoint image input unit 110. Then, the line structure 3D model generating unit 134 generates a line structure 3D point cloud by stereo matching assuming a local line structure based on the line structure depth map generated only from the multi-viewpoint images and the label map generated by the label map generating unit 120.

[0100] The embodiment of the present invention has been described above. The functions of the three-dimensional shape model generating device 100 in the above-described embodiment may be realized by a computer. In this case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. The term "computer system" as used herein includes hardware such as an OS (Operating System) and peripheral devices. The term "computer system" also includes a WWW (World Wide Web) system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc - Read Only Memory), and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Furthermore, the above program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in a computer system, or may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0101] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0102] 100 3D shape model generation device 110 Multi-viewpoint image input unit 120 Label Map Generation Unit 130 Three-dimensional information map generation unit 131 Surface Structure Depth Map Generation Unit 132 Lineation depth map generator 133 Surface structure 3D model generation unit 134 Linear structure 3D model generation unit 140 Multi-viewpoint image storage unit 150 Label map storage unit 160 Depth map storage unit 170 3D model storage unit

Claims

1. a first depth map generator that generates a first provisional depth map by stereo matching assuming a local surface structure based on multi-view images, which are two or more images captured from different viewpoints of an object; a second depth map generator that generates a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image; a label map generating unit that generates a label map in which a label indicating a surface structure likelihood and a line structure likelihood is assigned to each pixel in each viewpoint image of the multi-view image based on information obtained in generating the first temporary depth map and the second temporary depth map; a three-dimensional information map generation unit that generates a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map; Equipped with the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; The second depth map generating unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; the label map generation unit calculates a first matching score which is a matching score of stereo matching for the first provisional depth map and a second matching score which is a matching score of stereo matching for the second provisional depth map, and determines a label indicating a high likelihood of a surface structure when the first matching score is higher than the second matching score, and determines a label indicating a high likelihood of a line structure when the second matching score is higher than the first matching score. 3D shape model generator.

2. A first depth map generating unit that generates a first provisional depth map by stereo matching assuming a local surface structure based on multi-view images, which are two or more images of an object captured from different viewpoints; a second depth map generator that generates a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image; a label map generating unit that generates a label map in which a label indicating a surface structure likelihood and a line structure likelihood is assigned to each pixel in each viewpoint image of the multi-view image based on information obtained in generating the first temporary depth map and the second temporary depth map; a three-dimensional information map generation unit that generates a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map; Equipped with the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; The second depth map generating unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; the label map generation unit calculates a matching score by performing stereo matching on a local surface structure calculated from the first provisional depth map using a weighting function based on a line structure calculated from the second provisional depth map, and determines a label depending on the magnitude of the matching score. 3D shape model generator.

3. A first depth map generating unit that generates a first provisional depth map by stereo matching assuming a local surface structure based on multi-view images, which are two or more images of an object captured from different viewpoints; a second depth map generator that generates a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image; a label map generating unit that generates a label map in which a label indicating a surface structure likelihood and a line structure likelihood is assigned to each pixel in each viewpoint image of the multi-view image based on information obtained in generating the first temporary depth map and the second temporary depth map; a three-dimensional information map generation unit that generates a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map; Equipped with the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; referring to the label maps of multiple viewpoints, if the number of labels indicating a line structure in the first provisional depth of the first provisional depth map is greater than the number of labels indicating a plane structure, correcting the value of the first provisional depth to a value indicating that the value is invalid, and setting the corrected value as the depth value of the first depth map; The second depth map generating unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; referring to the label maps of multiple viewpoints, if the number of labels indicating a surface structure in the second provisional depth of the second provisional depth map is greater than the number of labels indicating a line structure, correcting the value of the second provisional depth to a value indicating that the value is invalid, and setting the corrected value as the depth value of the second depth map; 3D shape model generator.

4. A label map generation unit that generates a label map in which a label indicating a surface structure-likeness and a line structure-likeness is assigned to each pixel in each viewpoint image of a multi-view image, which is two or more images captured of an object from different viewpoints; a three-dimensional information map generator that generates a three-dimensional information map that indicates three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map, the three-dimensional information map being a depth map including a first depth map in an area of ​​the surface structure and a second depth map in an area of ​​the line structure; a first depth map generator that generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; a second depth map generation unit that generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; Equipped with the first depth map generating unit performs stereo matching using a weighting function that increases weights of pixels in the label map that indicate a surface structure-like nature; The second depth map generating unit performs stereo matching using a weighting function that increases weights of pixels in the label map that indicate a line structure-like nature. 3D shape model generator.

5. The first depth map generation unit calculates a depth using only pixels in the label map that indicate a surface structure, The second depth map generation unit calculates a depth using only pixels in the label map that indicate a line structure-likeness. The three-dimensional shape model generating device according to any one of claims 1 to 4.

6. the three-dimensional information map is a three-dimensional point cloud including a first three-dimensional point cloud representing a surface structure of the object and a second three-dimensional point cloud representing a line structure of the object, a first 3D model generation unit that generates the first 3D point cloud by stereo matching assuming a local surface structure based on a first depth map generated from the multi-view image and the label map; a second 3D model generation unit that generates the second 3D point cloud by stereo matching assuming a local line structure based on a second depth map generated from the multi-view image and the label map; The three-dimensional shape model generating device according to claim 1 , further comprising:

7. The label map generation unit generates a label map based on each image of the multi-view image. The three-dimensional shape model generating device according to any one of claims 1 to 6.

8. The label map generation unit generates the label map using a Convolutional Neural Network (CNN). The three-dimensional shape model generating device according to any one of claims 1 to 7.

9. A process in which a first depth map generating unit generates a first provisional depth map by stereo matching assuming a local surface structure based on multi-view images, which are two or more images of an object captured from different viewpoints; a step of generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image by a second depth map generating unit; A label map generating unit generates a label map in which a label indicating a surface structure and a line structure is assigned to each pixel in each viewpoint image of the multi-view image based on information obtained in generating the first temporary depth map and the second temporary depth map; a step of generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map by a three-dimensional information map generating unit; Including, the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; The second depth map generating unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; the label map generation unit calculates a first matching score which is a matching score of stereo matching for the first provisional depth map and a second matching score which is a matching score of stereo matching for the second provisional depth map, and determines a label indicating a high likelihood of a surface structure when the first matching score is higher than the second matching score, and determines a label indicating a high likelihood of a line structure when the second matching score is higher than the first matching score. A method for generating three-dimensional shape models.

10. A process in which a first depth map generating unit generates a first provisional depth map by stereo matching assuming a local surface structure based on multi-view images, which are two or more images of an object captured from different viewpoints; a step of generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image by a second depth map generating unit; A label map generating unit generates a label map in which a label indicating a surface structure and a line structure is assigned to each pixel in each viewpoint image of the multi-view image based on information obtained in generating the first temporary depth map and the second temporary depth map; a step of generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map by a three-dimensional information map generating unit; Including, the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; The second depth map generating unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; the label map generation unit calculates a matching score by performing stereo matching on a local surface structure calculated from the first provisional depth map using a weighting function based on a line structure calculated from the second provisional depth map, and determines a label depending on the magnitude of the matching score. A method for generating three-dimensional shape models.

11. A process in which a first depth map generating unit generates a first provisional depth map by stereo matching assuming a local surface structure based on multi-view images, which are two or more images of an object captured from different viewpoints; a step of generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image by a second depth map generating unit; A label map generating unit generates a label map in which a label indicating a surface structure and a line structure is assigned to each pixel in each viewpoint image of the multi-view image based on information obtained in generating the first temporary depth map and the second temporary depth map; a step of generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map by a three-dimensional information map generating unit; Including, the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; referring to the label maps of multiple viewpoints, if the number of labels indicating a line structure in the first provisional depth of the first provisional depth map is greater than the number of labels indicating a plane structure, correcting the value of the first provisional depth to a value indicating that the value is invalid, and setting the corrected value as the depth value of the first depth map; The second depth map generating unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; referring to the label maps of multiple viewpoints, if the number of labels indicating a surface structure in the second provisional depth of the second provisional depth map is greater than the number of labels indicating a line structure, correcting the value of the second provisional depth to a value indicating that the value is invalid, and setting the corrected value as the depth value of the second depth map; A method for generating three-dimensional shape models.

12. A process in which a label map generating unit generates a label map in which a label indicating a surface structure-likeness and a line structure-likeness is assigned to each pixel in each viewpoint image of a multi-view image, which is two or more images captured of an object from different viewpoints; a step of generating a three-dimensional information map based on the label map, the three-dimensional information map being a depth map indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure, the three-dimensional information map including a first depth map in an area of ​​the surface structure and a second depth map in an area of ​​the line structure; a first depth map generation step in which a first depth map generation unit generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; a second depth map generation step in which a second depth map generation unit generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; Including, the first depth map generating unit performs stereo matching using a weighting function that increases weights of pixels in the label map that indicate a surface structure-like nature; The second depth map generating unit performs stereo matching using a weighting function that increases weights of pixels in the label map that indicate a line structure-like nature. A method for generating three-dimensional shape models.

13. Computer, a first depth map generating means for generating a first provisional depth map by stereo matching assuming a local surface structure based on multi-viewpoint images, which are two or more images obtained by capturing an object from different viewpoints; a second depth map generating means for generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image; a label map generating means for generating a label map in which a label indicating a likelihood of a surface structure and a likelihood of a line structure is assigned to each pixel in each viewpoint image of the multi-viewpoint image based on information obtained in generating the first provisional depth map and the second provisional depth map; a three-dimensional information map generating means for generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map; Function as a the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating means generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; The second depth map generating means generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; the label map generating means calculates a first matching score which is a matching score of stereo matching for the first provisional depth map and a second matching score which is a matching score of stereo matching for the second provisional depth map, and determines a label indicating a high likelihood of a surface structure when the first matching score is higher than the second matching score, and determines a label indicating a high likelihood of a line structure when the second matching score is higher than the first matching score; program.

14. A computer comprising: a first depth map generating means for generating a first provisional depth map by stereo matching assuming a local surface structure based on multi-viewpoint images, which are two or more images obtained by capturing an object from different viewpoints; a second depth map generating means for generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image; a label map generating means for generating a label map in which a label indicating a likelihood of a surface structure and a likelihood of a line structure is assigned to each pixel in each viewpoint image of the multi-viewpoint image based on information obtained in generating the first provisional depth map and the second provisional depth map; a three-dimensional information map generating means for generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map; Function as a the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating means generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; The second depth map generating means generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; the label map generating means calculates a matching score by performing stereo matching on a local surface structure calculated from the first provisional depth map using a weighting function based on a line structure calculated from the second provisional depth map, and determines a label depending on the magnitude of the matching score. program.

15. A computer comprising: a first depth map generating means for generating a first provisional depth map by stereo matching assuming a local surface structure based on multi-viewpoint images, which are two or more images obtained by capturing an object from different viewpoints; a second depth map generating means for generating a second provisional depth map by stereo matching assuming a local line structure based on the multi-view image; a label map generating means for generating a label map in which a label indicating a likelihood of a surface structure and a likelihood of a line structure is assigned to each pixel in each viewpoint image of the multi-viewpoint image based on information obtained in generating the first provisional depth map and the second provisional depth map; a three-dimensional information map generating means for generating a three-dimensional information map indicating three-dimensional information for generating a three-dimensional shape model of the object having a plane structure and a line structure based on the label map; Function as a the three-dimensional information map is a depth map including a first depth map in the area of ​​the surface structure and a second depth map in the area of ​​the line structure; The first depth map generating means generates the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; referring to the label maps of multiple viewpoints, if the number of labels indicating a line structure in the first provisional depth of the first provisional depth map is greater than the number of labels indicating a plane structure, correcting the value of the first provisional depth to a value indicating that the value is invalid, and setting the corrected value as the depth value of the first depth map; The second depth map generating means generates the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; referring to the label maps of multiple viewpoints, if the number of labels indicating a surface structure in the second provisional depth of the second provisional depth map is greater than the number of labels indicating a line structure, correcting the value of the second provisional depth to a value indicating that the value is invalid, and setting the corrected value as the depth value of the second depth map; program.

16. A computer comprising: A label map generating means for generating a label map in which a label indicating a likelihood of a surface structure and a likelihood of a line structure is assigned to each pixel in each viewpoint image of a multi-viewpoint image, which is two or more images obtained by capturing an object from different viewpoints; a three-dimensional information map generating means for generating a three-dimensional information map, which is a depth map indicating three-dimensional information for generating a three-dimensional shape model of the object having a surface structure and a line structure based on the label map and including a first depth map in the surface structure region and a second depth map in the line structure region; a first depth map generating means for generating the first depth map by stereo matching assuming a local surface structure based on the multi-view image and the label map; a second depth map generating means for generating the second depth map by stereo matching assuming a local line structure based on the multi-view image and the label map; Function as a the first depth map generating means performs stereo matching using a weighting function that increases weights of pixels in the label map that indicate a surface structure-like nature; the second depth map generating means performs stereo matching using a weighting function that increases weights of pixels in the label map that indicate a line structure-like nature; program.

Images