Depth map generating device, depth map generating method, and program

Abstract

To calculate a depth value accurately and suppress increase in the calculation time when a depth map is generated from a multi-viewpoint image using a hierarchical search method.SOLUTION: A depth map generation device comprises: a hierarchical image generation unit which generates a hierarchical image by reducing the multi-viewpoint image at a reduction rate corresponding to each hierarchy; a first decision unit which decides a nearby viewpoint image number being the number of nearby viewpoint image used for calculation of the depth value as a pair of stereo matching to a reference viewpoint image selected as a target for calculation of the depth value from the hierarchical image in each hierarchy as a value in which the nearby viewpoint image number in at least one hierarchy is different from the nearby viewpoint image number in another hierarchy; and a depth map generation unit which generates a depth map using the nearby viewpoint image according to the nearby viewpoint image number decided by the first decision unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a depth map generating device, a depth map generating method, and a program. [Background technology]

[0002] Conventionally, there is a method for generating a depth map of an object using multiple images (multi-view images) of the object captured from different viewpoints. This method takes advantage of the fact that the object looks different in each multi-view image, and calculates the depth value of each pixel in the image using the principle of stereo matching. This generates a depth map of the object.

[0003] Patent Document 1 discloses a technology for generating a depth map using a hierarchical search method. The hierarchical search method is a technique for calculating depth values ​​using images reduced according to the reduction ratios of each of multiple layers, and generating a depth map for each layer. By using the hierarchical search method, depth values ​​calculated in higher layers can be used to calculate depth values ​​in lower layers, making it possible to calculate depth values ​​with high accuracy.

[0004] To accurately calculate depth values, it is important to select appropriate nearby viewpoint images. A nearby viewpoint image is an image that is paired for stereo matching with an image (reference viewpoint image) selected from the multi-viewpoint images and whose depth values ​​are to be calculated as a depth map. If nearby viewpoint images are selected randomly without selecting appropriate nearby viewpoint images, for example, an image that does not capture the target for which depth values ​​are to be calculated may be selected as the nearby viewpoint image. In this case, an occlusion region, for which depth values ​​cannot be calculated, occurs in the depth map. One possible way to reduce the occurrence of occlusion regions is to increase the number of nearby viewpoint images, for example, to calculate depth values ​​for all images in the multi-viewpoint images that are different from the reference viewpoint image as nearby viewpoint images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5787398 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the calculation time for depth values ​​is roughly proportional to the number of nearby viewpoint images. When the number of nearby viewpoint images is large, the calculation time increases. When using the hierarchical search method, a depth map is generated at each layer, so the calculation time required to generate the final depth map becomes enormous, making it unrealistic.

[0007] The present invention has been made in consideration of this situation, and provides a depth map generation device, a depth map generation method, and a program that can accurately calculate depth values ​​and suppress increases in calculation time when generating a depth map from a multi-viewpoint image using a hierarchical search method. [Means for solving the problem]

[0008] The depth map generating device of the present invention is a depth map generating device that generates a depth map of an object from a plurality of multi-viewpoint images obtained by capturing the object from different viewpoints using a hierarchical search method, and includes: a hierarchical image generating unit that generates hierarchical images by reducing the multi-viewpoint images at a reduction ratio corresponding to each layer; a first determination unit that determines the number of neighboring viewpoint images, which is the number of neighboring viewpoint images to be used in calculating depth values ​​as a stereo matching pair for a reference viewpoint image selected as a target for calculating depth values ​​from the hierarchical images at each layer, so that the number of neighboring viewpoint images in at least one layer is a value different from the number of neighboring viewpoint images in other layers; and a depth map generating unit that generates a depth map using the neighboring viewpoint images according to the number of neighboring viewpoint images determined by the first determination unit.

[0009] In the depth map generating device of the present invention, the first determination unit sets the number of nearby viewpoint images in a higher layer to a value greater than the number of nearby viewpoint images in a lower layer.

[0010] In the depth map generating device of the present invention, the first determination unit sets the number of nearby viewpoint images in a first layer different from the top layer to a value smaller than the number of nearby viewpoint images in a second layer higher than the first layer.

[0011] The depth map generating device of the present invention further includes a second determination unit that determines the resolution of the depth map generated at each layer, wherein the first determination unit determines the number of nearby viewpoint images according to the resolution determined by the second determination unit, and the depth map generating unit generates a depth map according to the resolution determined by the second determination unit using the nearby viewpoint images according to the number determined by the first determination unit.

[0012] The depth map generating device of the present invention further includes a management unit that selects the neighboring viewpoint images for each of the reference viewpoint images or for each of multiple pixel groups in the reference viewpoint image, depending on the number of neighboring viewpoint images determined by the first determination unit.

[0013] In the depth map generating device of the present invention, the management unit uses a nearby viewpoint map showing candidate images that are candidates for the reference viewpoint image to select the nearby viewpoint image from the candidate images shown in the nearby viewpoint map according to the number of nearby viewpoint images, and the depth map generating unit generates a depth map using the reference viewpoint image selected by the management unit.

[0014] In the depth map generating device of the present invention, the nearby viewpoint map indicates an evaluation value indicating the degree to which the candidate image is appropriate as the nearby viewpoint image, and the management unit selects the nearby viewpoint image based on the evaluation value of the candidate image indicated in the nearby viewpoint map and the number of nearby viewpoint images.

[0015] In the depth map generating device of the present invention, if the number of nearby viewpoint images in a third layer different from the lowest layer is the same as or smaller than the number of nearby viewpoint images in a fourth layer below the third layer, the management unit selects all of the nearby viewpoint images selected in the third layer as nearby viewpoint images in the fourth layer, and if the number of nearby viewpoint images in the third layer is larger than the number of nearby viewpoint images in the fourth layer, the management unit selects the nearby viewpoint images in the fourth layer from the nearby viewpoint images selected in the third layer according to the evaluation values ​​of the nearby viewpoint images selected in the third layer.

[0016] The depth map generation method of the present invention is a depth map generation method performed by a depth map generation device that generates a depth map of an object from a plurality of multi-viewpoint images of the object captured from different viewpoints using a hierarchical search method, and includes: a hierarchical image generation step of generating hierarchical images by reducing the multi-viewpoint images at a reduction ratio corresponding to each layer; a first determination step of determining the number of neighboring viewpoint images, which is the number of neighboring viewpoint images to be used in calculating depth values ​​as a stereo matching pair for a reference viewpoint image selected as a target for calculating depth values ​​from the hierarchical images in each layer, so that the number of neighboring viewpoint images in at least one layer is a value different from the number of neighboring viewpoint images in other layers; and a depth map generation step of generating a depth map using the neighboring viewpoint images according to the number of neighboring viewpoint images determined by the first determination step.

[0017] The program of the present invention is a program that causes a computer to operate as a depth map generation device that generates a depth map of an object from a plurality of multi-viewpoint images of the object captured from different viewpoints using a hierarchical search method, and that operates as follows: hierarchical image generation means that generates hierarchical images by reducing the multi-viewpoint images at a reduction ratio corresponding to each layer; first determination means that determines the number of neighboring viewpoint images, which is the number of neighboring viewpoint images to be used in calculating depth values ​​as a stereo matching pair for a reference viewpoint image selected as a target for calculating depth values ​​from the hierarchical images in each layer, so that the number of neighboring viewpoint images in at least one layer is a value different from the number of neighboring viewpoint images in other layers; and depth map generation means that generates a depth map using the neighboring viewpoint images according to the number of neighboring viewpoint images determined by the first determination means. [Effects of the Invention]

[0018] According to the present invention, when a depth map is generated from multi-viewpoint images using a hierarchical search method, depth values ​​can be calculated with high accuracy and an increase in calculation time can be suppressed. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 10 is a diagram illustrating an example of the number of nearby viewpoint images in each layer according to the embodiment. [Figure 2] 1 is a block diagram showing an example of the configuration of a depth map generating device 1 according to an embodiment. [Figure 3] 2 is a diagram showing an example of information stored in a multi-viewpoint image information storage unit 107 according to the embodiment. FIG. [Figure 4] 3 is a diagram showing an example of information stored in a layered image information storage unit 108 according to the embodiment. FIG. [Figure 5] 10 is a diagram showing an example of information stored in a nearby viewpoint image number information storage unit 109 according to the embodiment. FIG. [Figure 6A] 3 is a diagram showing an example of information stored in a nearby viewpoint map information storage unit 110 according to the embodiment. FIG. [Figure 6B]3 is a diagram showing an example of information stored in a nearby viewpoint map information storage unit 110 according to the embodiment. FIG. [Figure 7] 3 is a diagram showing an example of information stored in a depth map information storage unit 111 according to the embodiment. FIG. [Figure 8] 3 is a flowchart showing the flow of processing performed by the depth map generating device 1 according to the embodiment. [Figure 9] FIG. 1 is a diagram illustrating a conventional hierarchical search method. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, a depth map generating device according to an embodiment will be described with reference to the drawings.

[0021] (Regarding conventional hierarchical search methods) First, a conventional hierarchical search method will be described with reference to Fig. 9. Fig. 9 is a diagram illustrating the conventional hierarchical search method. Fig. 9 shows four hierarchical images KG (hierarchical images KG1 to KG4) and four depth maps DM (depth maps DM1 to DM4).

[0022] As shown in FIG. 9, the hierarchical search method provides multiple layers LR (layers LR1 to LR4). While the example in this figure shows four layers, any number of layers may be used as long as the number of layers is two or greater. The layer images KG in each layer (layer images KG1 to KG4) are images obtained by reducing the multi-viewpoint image TG. For example, the layer image KG4 in the lowest layer LR4 is a multi-viewpoint image TG. The layer image KG3 in the layer LR3, which is one layer above the layer LR4, is an image obtained by reducing the multi-viewpoint image TG to 1 / 2 in both the vertical and horizontal directions. The layer image KG2 in the layer LR2, which is one layer above the layer LR3, is an image obtained by reducing the multi-viewpoint image TG to 1 / 4 in both the vertical and horizontal directions. The layer image KG1 in the layer LR1, which is one layer above the layer LR2 (i.e., the top layer), is an image obtained by reducing the multi-viewpoint image TG to 1 / 8 in both the vertical and horizontal directions.

[0023] In the hierarchical search method, depth values ​​are calculated for each pixel using the hierarchical images KG in each layer, from the top to the bottom. In the hierarchical search method, the depth values ​​calculated in the upper layers are used in the layers below. For example, they may be used as initial values ​​when calculating depth values, or they may be directly adopted as the depth values. Specifically, a depth map DM1 is first generated using the hierarchical image KG1 in the top layer LR1. Next, a depth map DM2 is generated using the hierarchical image KG2 in the layer LR2, which is one layer below LR1, and the depth map DM1. Next, a depth map DM3 is generated using the hierarchical image KG3 in the layer LR3, which is one layer below LR2, and the depth map DM2. Finally, a depth map DM4 is generated using the hierarchical image KG4 in the layer LR4, which is one layer below LR3, and the depth map DM3. The depth map DM4 is the depth map DM corresponding to the multi-view image TG, and is the final depth map to be output.

[0024] In the conventional hierarchical search method, if all images in the hierarchical image KG that are different from the reference viewpoint image are used as neighboring viewpoint images, the calculation of depth values ​​takes an enormous amount of time, which is not practical.

[0025] To address this issue, in this embodiment, the number of nearby viewpoint images used in calculating depth values ​​in each layer (hereinafter referred to as the number of nearby viewpoint images) is set individually for each layer. This allows for a creative combination of the number of nearby viewpoint images in each layer, making it possible to generate a depth map without requiring an enormous amount of calculation time or reducing the accuracy of depth value calculation.

[0026] (Overview of Depth Map Generator 1) An overview of the depth map generating device 1 will now be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the number of nearby viewpoint images in each layer according to an embodiment. In Fig. 1, layers LR are shown vertically, and the number of nearby viewpoint images used for calculating depth values ​​is shown horizontally. In the example shown in this diagram, an example is shown in which five layers are provided. Specifically, from top to bottom, a top layer, intermediate layer 1, intermediate layer 2, intermediate layer 3, and bottom layer are provided.

[0027] FIG. 1 shows four patterns, pattern 1 to pattern 4, as patterns of the number of nearby viewpoint images.

[0028] Pattern 1 is a pattern in which the number of nearby viewpoint images in the highest layer among the layers is set to the largest value, and the number of nearby viewpoint images in a lower layer is set to a smaller value than the number of nearby viewpoint images in the layer above that.

[0029] When generating a depth map DM at each layer, it is necessary to calculate depth values ​​for the number of pixels (i.e., resolution) in the depth map DM to be generated. The calculation time for calculating the depth values ​​is proportional to the number of nearby viewpoint images used in the calculation. In other words, the calculation time required to generate the depth map DM is approximately proportional to the resolution of the depth map DM and the number of nearby viewpoint images.

[0030] For example, consider generating a depth map DM at each layer with the same resolution as the layer images KG at that layer, and using all neighboring viewpoint images among the layer images KG at that layer that are different from the reference viewpoint image to calculate depth values. In this case, the calculation time required to generate the depth map DM is proportional to the resolution of the depth map DM and the number of neighboring viewpoint images, so the lower the layer, the longer the time required to generate the depth map DM.

[0031] Therefore, in pattern 1, the number of nearby viewpoint images in lower layers is set to a smaller value than in the lower layers. In the example of this figure, the number of nearby viewpoint images in the top layer is 64, the number of nearby viewpoint images in middle layer 1 is 32, the number of nearby viewpoint images in middle layer 2 is 16, the number of nearby viewpoint images in middle layer 3 is 8, and the number of nearby viewpoint images in the bottom layer is 4. As a result, the number of nearby viewpoint images can be set to a smaller value in lower layers where the resolution of the depth map DM is higher, and it is possible to suppress an increase in the time required to generate the depth map DM.

[0032] Pattern 2 is a pattern in which the number of nearby viewpoint images in a layer (referred to as the first layer) different from the top layer is set to a smaller value than in the layer (referred to as the second layer) above the first layer.

[0033] When generating a depth map DM using the hierarchical search method, in order to prevent an increase in calculation time, the number of pixels for which depth values ​​are calculated at each layer may be reduced, i.e., so-called "thinning out." For example, if the resolution of the hierarchical image KG at a certain layer is 640,000 pixels, the resolution of the depth map DM generated at that layer may be set to a value smaller than 640,000 pixels, for example, 80,000 pixels.

[0034] As a result of this thinning, for example, the resolution of the depth map DM generated in the first layer may be smaller than the resolution of the depth map DM generated in the second layer above the first layer. For example, the resolution of the depth map DM generated in the first layer may be 80,000 pixels, and the resolution of the depth map DM generated in the second layer may be 640,000 pixels. In such a case, the number of nearby viewpoint images in the first layer may be set to a smaller value than that in the second layer. For example, the number of nearby viewpoint images in the first layer may be set to 16, and the number of nearby viewpoint images in the second layer may be set to 32. This allows the number of nearby viewpoint images to be set to a smaller value for layers with higher resolution depth maps DM, regardless of whether they are higher layers, thereby suppressing an increase in the time required to generate the depth map DM.

[0035] Pattern 3 is a pattern in which the number of neighboring viewpoint images in layers other than the top layer is uniform. In the example of this figure, the number of neighboring viewpoint images in the top layer is 16, and the number of neighboring viewpoint images in layers other than the top layer is a uniform value of 32. For example, the above-mentioned thinning may be performed, and the resolution of the depth map DM generated in layers other than the top layer may be set uniformly. In such a case, as shown in pattern 3, the number of neighboring viewpoint images in layers other than the top layer is uniform. This allows the number of neighboring viewpoint images to be a uniform value, i.e., the same number, for layers with the same resolution of the depth map DM, regardless of whether the layer is higher or lower, and prevents an increase in the time required to generate the depth map DM.

[0036] Pattern 4 is a pattern in which the number of nearby viewpoint images in layers other than the lowest layer is uniform. In the example shown in this figure, the number of nearby viewpoint images in the lowest layer is 32, and the number of nearby viewpoint images in layers other than the lowest layer is a uniform value of 8.

[0037] For example, in a multi-viewpoint image TG, when there are many images capturing the same region, stereo matching pairs can be established even if the images have low resolution, i.e., even if the number of nearby viewpoint images in the higher-level hierarchical images KG is small. On the other hand, since the depth map DM generated in the lowest layer is the depth map DM that is generated finally, it is preferable to select nearby viewpoint images that are particularly good in order to establish stereo matching pairs. From this perspective, in pattern 4, the number of nearby viewpoint images in layers other than the lowest layer is set to a uniform value, and the number of nearby viewpoint images in the lowest layer is set to a larger value than the uniform number of nearby viewpoint images in the other layers.

[0038] (Configuration of depth map generating device 1) Here, the configuration of the depth map generation device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the depth map generation device 1 according to an embodiment. The depth map generation device 1 includes, for example, a multi-viewpoint image information acquisition unit 101, a hierarchical image generation unit 102, a depth map resolution determination unit 103, a neighboring viewpoint image number determination unit 104, a depth map generation unit 105, a neighboring viewpoint image management unit 106, a multi-viewpoint image information storage unit 107, a hierarchical image information storage unit 108, a neighboring viewpoint image number information storage unit 109, a neighboring viewpoint map information storage unit 110, and a depth map information storage unit 111.

[0039] The multi-viewpoint image information acquisition unit 101 acquires image information of the multi-viewpoint image TG from the multi-viewpoint image information storage unit 107. The multi-viewpoint image TG is a plurality of images of an object captured from different viewpoints. The object here is an object that can be captured and has any three-dimensional shape. The multi-viewpoint image information acquisition unit 101 outputs the acquired image information of the multi-viewpoint image TG to the hierarchical image generation unit 102.

[0040] The hierarchical image generation unit 102 generates hierarchical images KG. The hierarchical images KG are images obtained by reducing the multi-viewpoint images TG at a predetermined reduction rate. The hierarchical image generation unit 102 reduces the multi-viewpoint images TG according to a reduction rate predetermined for each layer, thereby generating the hierarchical images KG for that layer. The hierarchical image generation unit 102 stores image information of the hierarchical images KG in the hierarchical image information storage unit 108.

[0041] The depth map resolution determination unit 103 determines the resolution of the depth map DM to be generated in each layer. The depth map resolution determination unit 103 determines the resolution of the depth map DM to be generated in each layer depending on, for example, the calculation capacity of the depth map generation device 1, the number of multi-view images TG, the resolution of the depth map DM to be finally generated, etc.

[0042] The neighboring viewpoint image number determination unit 104 determines the number of neighboring viewpoint images in each layer. The neighboring viewpoint image number determination unit 104 selects, for example, one of the above-mentioned patterns 1 to 4, and determines the number of neighboring viewpoint images in each layer based on the selected pattern. The neighboring viewpoint image number determination unit 104 stores the determined number of neighboring viewpoint images in the neighboring viewpoint image number information storage unit 109.

[0043] Specifically, the neighboring viewpoint image number determination unit 104 sets the number of neighboring viewpoint images in a higher layer to a value greater than the number of neighboring viewpoint images in a lower layer, as shown in Pattern 1. Alternatively, the neighboring viewpoint image number determination unit 104 may set the number of neighboring viewpoint images in a first layer different from the top layer to a value smaller than the number of neighboring viewpoint images in a second layer higher than the first layer, as shown in Pattern 2. Alternatively, the neighboring viewpoint image number determination unit 104 may set the number of neighboring viewpoint images in each layer different from the top layer to a uniform value, as shown in Pattern 3. Alternatively, the neighboring viewpoint image number determination unit 104 may set the number of neighboring viewpoint images in each layer different from the bottom layer to a uniform value, as shown in Pattern 4.

[0044] Furthermore, the number of neighboring viewpoint images determination unit 104 may determine the number of neighboring viewpoint images in each layer according to the resolution of the depth map DM to be generated in each layer determined by the depth map resolution determination unit 103. For example, based on the resolution of the depth map DM in each layer, the number of neighboring viewpoint images determination unit 104 sets a small value for the number of neighboring viewpoint images in a layer where the resolution of the depth map DM is high, and sets a large value for the number of neighboring viewpoint images in a layer where the resolution of the depth map DM is low.

[0045] The neighboring viewpoint image management unit 106 selects the neighboring viewpoint images to be used for calculating the depth value in accordance with the number determined by the neighboring viewpoint image number determination unit 104 (the number of neighboring viewpoint images).

[0046] For example, the nearby viewpoint image management unit 106 generates a nearby viewpoint map and selects nearby viewpoint images to be used in calculating depth values ​​using the created nearby viewpoint map. The nearby viewpoint map is information about hierarchical images KG that can be nearby viewpoint images. Here, the information about the hierarchical images KG includes, for example, identification information of the hierarchical images KG, coordinate values ​​that serve as reference viewpoints in the hierarchical images KG, and evaluation values. The evaluation value indicates the degree of suitability of the hierarchical images KG as nearby viewpoint images.

[0047] For example, the nearby viewpoint image management unit 106 determines, among the hierarchical images KG, an image in which a region for which a depth value is to be calculated is captured as a hierarchical image KG that can be a nearby viewpoint image. In this case, the nearby viewpoint map is information about each of the images in which a region for which a depth value is to be calculated is captured.

[0048] The nearby viewpoint image management unit 106 may select a nearby viewpoint image for each predetermined region. The predetermined region here is a region in the reference viewpoint image, such as the reference viewpoint image, a group of pixels in the reference viewpoint image, or one pixel in the reference viewpoint image.

[0049] For example, when selecting a neighboring viewpoint image for each reference viewpoint image, the neighboring viewpoint image management unit 106 generates a neighboring viewpoint map (referred to as a first map) in which information about hierarchical images KG that can become neighboring viewpoint images is associated with each reference viewpoint image. When selecting a neighboring viewpoint image for each of a plurality of pixel groups in the reference viewpoint image, the neighboring viewpoint image management unit 106 generates a neighboring viewpoint map (referred to as a second map) in which information about hierarchical images KG that can become neighboring viewpoint images is associated with each of a plurality of pixel groups in the reference viewpoint image. Alternatively, when selecting a neighboring viewpoint image for each pixel in the reference viewpoint image, the neighboring viewpoint image management unit 106 generates a neighboring viewpoint map (referred to as a third map) in which information about hierarchical images KG that can become neighboring viewpoint images is associated with each pixel in the reference viewpoint image.

[0050] As described above, by generating a nearby viewpoint map according to a predetermined area for selecting nearby viewpoint images, it becomes possible to accommodate the memory capacity of the depth map generating device 1 or the accuracy of the depth map DM to be generated.

[0051] For example, in the case of the first map, each image is associated with information about hierarchical images KG that can be nearby viewpoint images. Therefore, the memory capacity of the first map can be reduced compared to the memory capacity of the second and third maps. On the other hand, when calculating depth values ​​using nearby viewpoint images selected using the first map, the same group of nearby viewpoint images must be used to calculate the depth value of one reference viewpoint image. Therefore, depending on the coordinates, a stereo matching pair may not be appropriate. In such cases, the accuracy of the depth value calculation may be reduced.

[0052] Alternatively, when the third map is generated, information about the hierarchical image KG that can be a nearby viewpoint image is associated with each pixel. The third map requires a larger memory capacity than the first map and the second map, but on the other hand, for each pixel for which a depth value is to be calculated, a nearby viewpoint image can be selected according to that pixel. Therefore, when the third map is used, an appropriate stereo matching pair can be generated for each pixel, making it possible to calculate the depth value with high accuracy.

[0053] The nearby viewpoint image management unit 106 calculates an evaluation value for each of the hierarchical images KG that can be nearby viewpoint images. As the evaluation value, for example, an image matching correlation value, an occlusion determination result, a texture determination result, a viewpoint distance, a viewpoint angle, a result of estimating a reliability value by machine learning, or a combination of these can be used.

[0054] The image matching correlation value is a score obtained by matching two images. For example, the nearby viewpoint image management unit 106 matches (aligns) the reference viewpoint image with a hierarchical image KG that can be a nearby viewpoint image. Matching can be performed using, for example, normalized cross-correlation in a local region (patch) of the image, SSD (Sum Squared Difference) in a local region of the image, or stereo matching such as phase-only correlation. The nearby viewpoint image management unit 106 matches the reference viewpoint image with a hierarchical image KG that can be a nearby viewpoint image, and sets, as an evaluation value, a value corresponding to the number of pixels that match in the two images with a matching score equal to or greater than a threshold, or a value corresponding to the stereo matching correlation value in that hierarchical image KG.

[0055] Occlusion determination is a determination of whether an object is occluded in an image. The nearby viewpoint image management unit 106 determines whether an object is occluded in a hierarchical image KG that can be a nearby viewpoint image. For example, the nearby viewpoint image management unit 106 extracts the contour of the object captured in the hierarchical image KG, compares the shape of the extracted contour with the shape of the contour of the object extracted from the reference viewpoint image, and quantifies the degree to which the object is occluded based on the degree of difference in shape. The nearby viewpoint image management unit 106 sets a value corresponding to the degree to which the object is occluded as an evaluation value.

[0056] Texture determination is a process of determining whether the textures of an object captured in both the reference viewpoint image and the hierarchical image KG that can be a nearby viewpoint image are similar. The nearby viewpoint image management unit 106 calculates the texture of the object captured in each of the two images. For example, the nearby viewpoint image management unit 106 quantifies the texture of the object based on the degree according to texture items of statistics such as the average and variance of pixel values ​​(RGB values) in a local region. For example, the nearby viewpoint image management unit 106 determines the distance between both images in a vector space with each texture item as an axis as the similarity of the texture of the local region in both images. The nearby viewpoint image management unit 106 determines a value according to the similarity of the texture of the object in both images as an evaluation value.

[0057] The inter-viewpoint distance is the distance between the viewpoints (imaging positions) of the reference viewpoint image and the hierarchical image KG that can be a nearby viewpoint image. The nearby viewpoint image management unit 106 calculates the inter-viewpoint distance using, for example, camera parameters. The camera parameters are variables (parameters) related to imaging when capturing an image. The camera parameters include, for example, the focal length, optical center, imaging position, and imaging direction of the camera. The camera parameters can be estimated using, for example, SfM (Structure from Motion). The nearby viewpoint image management unit 106 sets a value corresponding to the inter-viewpoint distance as an evaluation value.

[0058] The viewpoint angle is the angle formed by a line connecting the imaging position in the reference viewpoint image to a predetermined position (e.g., the center of gravity) on the object and a line connecting the imaging position in a hierarchical image KG that can be a nearby viewpoint image to the same predetermined position (e.g., the center of gravity) on the object. The nearby viewpoint image management unit 106 calculates the viewpoint angle using, for example, camera parameters. The nearby viewpoint image management unit 106 sets a value corresponding to the viewpoint angle as an evaluation value.

[0059] The estimation of the reliability value by machine learning is to estimate, by machine learning, a reliability value indicating whether or not a hierarchical image KG that can be a nearby viewpoint image is appropriate as a nearby viewpoint image. The nearby viewpoint image management unit 106 estimates the reliability value by using, for example, a trained model. The trained model here is a model that has learned the correspondence between images and reliability values ​​by performing machine learning using a training dataset. The training dataset is, for example, information in which training images (reference viewpoint image and nearby viewpoint image) are associated with reliability values ​​when depth values ​​are calculated using the training images.

[0060] The neighboring viewpoint image management unit 106 acquires the number of neighboring viewpoint images by referring to the neighboring viewpoint image number information storage unit 109. The neighboring viewpoint image management unit 106 refers to the neighboring viewpoint map information storage unit 110 and selects, as the neighboring viewpoint image to be used in calculating the depth value, the hierarchical image KG having a high evaluation value among the hierarchical images KG that can be the neighboring viewpoint image, extracted in accordance with the number of neighboring viewpoint images.

[0061] The nearby viewpoint image management unit 106 may select a nearby viewpoint image selected in a higher layer as a nearby viewpoint image in a lower layer.

[0062] For example, the nearby-viewpoint image management unit 106 compares the number of nearby viewpoint images in a higher layer (referred to as the third layer) with the number of nearby viewpoint images in a layer below the third layer (referred to as the fourth layer). If the numbers of nearby viewpoint images in the two layers are the same, the nearby-viewpoint image management unit 106 selects all of the nearby viewpoint images selected in the third layer as nearby viewpoint images in the fourth layer. Specifically, the nearby-viewpoint image management unit 106 selects layer images KG in the fourth layer corresponding to each of the nearby viewpoint images selected in the third layer as nearby viewpoint images in the fourth layer.

[0063] If the number of neighboring viewpoint images in the third layer is smaller than the number of neighboring viewpoint images in the fourth layer, the neighboring viewpoint image management unit 106 selects all of the neighboring viewpoint images selected in the third layer as neighboring viewpoint images in the fourth layer. Then, the neighboring viewpoint image management unit 106 further selects the shortage of neighboring viewpoint images in the fourth layer. For example, the neighboring viewpoint image management unit 106 selects layer images KGs in the neighboring viewpoint map that are different from the layer images KGs in the fourth layer that correspond to the neighboring viewpoint images selected in the third layer and have high evaluation values, depending on the shortage of the layer images. The neighboring viewpoint image management unit 106 sets the selected layer images KGs as neighboring viewpoint images to be added to the fourth layer for use.

[0064] When the number of neighboring viewpoint images in the third layer is greater than the number of neighboring viewpoint images in the fourth layer, the neighboring viewpoint image management unit 106 selects a neighboring viewpoint image in the fourth layer from the neighboring viewpoint images selected in the third layer. For example, the neighboring viewpoint image management unit 106 refers to the evaluation value of each neighboring viewpoint image selected in the third layer in the neighboring viewpoint map, and selects an image with a high evaluation value according to the number of neighboring viewpoint images in the fourth layer. The neighboring viewpoint image management unit 106 sets the layer image KG in the fourth layer corresponding to the selected neighboring viewpoint image as the neighboring viewpoint image in the fourth layer.

[0065] The depth map generation unit 105 generates a depth map DM by a hierarchical search method. The depth map generation unit 105 generates a depth map DM with the resolution determined by the depth map resolution determination unit 103. At this time, the depth map generation unit 105 performs calculations using depth values ​​using nearby viewpoint images selected by the nearby viewpoint image management unit 106. The depth map generation unit 105 stores the generated depth map DM in the depth map information storage unit 111.

[0066] FIG. 3 is a diagram showing an example of information stored in the multi-viewpoint image information storage unit 107 according to the embodiment. As shown in FIG. 3, the multi-viewpoint image information storage unit 107 stores, as multi-viewpoint image information, information corresponding to, for example, a multi-viewpoint image ID, image coordinates, and camera parameters. The multi-viewpoint image ID is identification information that uniquely identifies the multi-viewpoint image TG. The image information is image information in the multi-viewpoint image TG, and is, for example, information indicating pixel coordinate values ​​and RGB values. The camera parameters are information indicating the camera parameters in the multi-viewpoint image TG.

[0067] Fig. 4 is a diagram showing an example of information stored in the hierarchical image information storage unit 108 according to the embodiment. As shown in Fig. 4, the hierarchical image information storage unit 108 stores, as hierarchical image information, information corresponding to, for example, a hierarchical image ID, image coordinates, and camera parameters. The hierarchical image ID is identification information that uniquely identifies a hierarchical image KG. The image information is image information in the hierarchical image KG, and is, for example, information indicating pixel coordinate values ​​and RGB values. The camera parameters are information indicating the camera parameters in the hierarchical image KG.

[0068] 5 is a diagram showing an example of information stored in the neighboring viewpoint image number information storage unit 109 according to the embodiment. As shown in FIG. 5, the neighboring viewpoint image number information storage unit 109 stores, for example, information corresponding to a layer ID and the number of neighboring viewpoint images as neighboring viewpoint image number information. The layer ID is identification information that uniquely identifies each layer in the layer search method. The number of neighboring viewpoint images is information indicating the number of neighboring viewpoint images used to calculate depth values ​​in a layer.

[0069] 6 (FIGS. 6A and 6B) are diagrams illustrating an example of information stored in the nearby viewpoint map information storage unit 110 according to the embodiment. The nearby viewpoint map information storage unit 110 stores nearby viewpoint map information. The nearby viewpoint map information is a map in which nearby viewpoint images that are candidates for selecting a nearby viewpoint image for a reference viewpoint image are associated with each other. The nearby viewpoint map information is generated for each reference viewpoint image.

[0070] 6A shows an example of nearby viewpoint map information generated when a nearby viewpoint image is selected for each pixel in a reference viewpoint image. As shown in FIG. 6A, the nearby viewpoint map information includes, for example, an item of pixel (pixel 1, pixel 2, ...), and each pixel is associated with a plurality of candidates (candidate 1, candidate 2, ...). A candidate is a candidate to be selected as a nearby viewpoint image, and is associated with, for example, an ID (hierarchical image ID) and an evaluation value of the candidate image. The evaluation value indicates the degree of suitability as a nearby viewpoint image, and is a value calculated by the nearby viewpoint image management unit 106.

[0071] 6B shows an example of nearby viewpoint map information generated when a nearby viewpoint image is selected for each region (region consisting of a plurality of pixels) in a reference viewpoint image. As shown in FIG. 6B, the nearby viewpoint map information includes items for region (region 1, region 2, ...), and each region is associated with a plurality of candidates (candidate 1, candidate 2, ...). A candidate is a candidate to be selected as a nearby viewpoint image, and is associated with, for example, an ID (hierarchical image ID) and an evaluation value of the candidate image. The evaluation value indicates the degree of suitability as a nearby viewpoint image, and is a value calculated by the nearby viewpoint image management unit 106.

[0072] 7 is a diagram illustrating an example of information stored in the depth map information storage unit 111 according to the embodiment. As illustrated in FIG. 7, the depth map information storage unit 111 stores, as depth map information, information corresponding to each of a depth map ID and depth map information, for example. The depth map ID is identification information that uniquely identifies the depth map DM. The depth map information is information indicating, for example, pixel coordinate values ​​and depth values.

[0073] (Processing of depth map generating device 1) The processing performed by the depth map generating device 1 will now be described with reference to Fig. 8. Fig. 8 is a flowchart showing the flow of processing performed by the depth map generating device 1 according to the embodiment.

[0074] The depth map generating device 1 acquires a multi-viewpoint image TG (step S10). Next, the depth map generating device 1 generates hierarchical images KG for each layer using the multi-viewpoint image TG (step S11). Next, the depth map generating device 1 determines the resolution of the depth map DM to be generated for each layer (step S12). The depth map generating device 1 determines the number of nearby viewpoint images to be used for calculating depth values ​​for each layer (step S13). The depth map generating device 1 selects nearby viewpoint images to be used for calculating depth values ​​for the layer according to the number determined in step S13 (step S14). The depth map generating device 1 generates the depth map DM using a hierarchical search method (step S15). In this case, the depth map generating device 1 calculates depth values ​​using the nearby viewpoint images selected in step S14, A depth map DM is generated for each layer.

[0075] As described above, the depth map generation device 1 according to the embodiment uses a hierarchical search method to generate a depth map DM of an object from a plurality of multi-viewpoint images TG obtained by capturing the object from different viewpoints. The depth map generation device 1 includes a hierarchical image generation unit 102, a number of neighboring viewpoint images determination unit 104 (first determination unit), and a depth map generation unit 105. The hierarchical image generation unit 102 generates hierarchical images KG. The number of neighboring viewpoint images determination unit 104 determines the number of neighboring viewpoint images in at least one layer to be a value different from the number of neighboring viewpoint images in other layers. The number of neighboring viewpoint images is the number of neighboring viewpoint images used in calculating depth values ​​as a pair for stereo matching with respect to a reference viewpoint image selected as a target for calculating depth values ​​from the hierarchical images in the layer. The depth map generation unit 105 generates a depth map at the resolution determined by the determination unit for each layer.

[0076] As a result, the depth map generating device 1 according to the embodiment can set the number of nearby viewpoint images individually for each layer. Therefore, by devising a combination of the number of nearby viewpoint images in each layer, it is possible to generate a depth map without excessively long calculation times and without reducing the accuracy of depth value calculation. Therefore, when generating a depth map using multi-viewpoint images using the hierarchical search method, it is possible to calculate depth values ​​with high accuracy and suppress an increase in calculation time.

[0077] Furthermore, in the depth map generating device 1 of the embodiment, the neighboring viewpoint image number determination unit 104 sets the number of neighboring viewpoint images in a higher layer to a value greater than the number of neighboring viewpoint images in a lower layer. As a result, the depth map generating device 1 of the embodiment can calculate depth values ​​using many neighboring viewpoint images when using hierarchical images KG with a relatively low resolution in a higher layer. Therefore, depth values ​​can be calculated accurately, and since the resolution is relatively low, the number of depth values ​​to be calculated is small, thereby suppressing an increase in calculation time. On the other hand, when hierarchical images KG with a relatively high resolution in a lower layer are used, depth values ​​can be calculated using fewer neighboring viewpoint images. Therefore, when the resolution is relatively high and the number of depth values ​​to be calculated is large, the number of neighboring viewpoint images can be reduced, thereby suppressing an increase in calculation time. Furthermore, since the depth values ​​calculated in the higher layer can be used in the lower layer, it is also possible to prevent a decrease in the accuracy of the depth value calculation.

[0078] Furthermore, in the depth map generating device 1 of the embodiment, the neighboring viewpoint image number determination unit 104 may set the number of neighboring viewpoint images in a first layer different from the highest layer to a smaller value than the number of neighboring viewpoint images in a second layer higher than the first layer. In this way, the depth map generating device 1 of the embodiment can set the number of neighboring viewpoint images in an intermediate layer to a large value by thinning out the images when the resolution of the depth map DM generated in the intermediate layer is set to be small. Therefore, when the resolution of the depth map DM to be generated is relatively small, it is possible to improve the calculation accuracy of depth values ​​while suppressing an increase in calculation time, regardless of whether the depth map DM is a higher layer.

[0079] Moreover, the depth map generating device 1 of the embodiment further includes a depth map resolution determining unit 103 (second determining unit). The depth map resolution determining unit 103 determines the resolution of the depth map to be generated in each layer. The neighboring viewpoint image number determining unit 104 determines the number of neighboring viewpoint images according to the resolution determined by the depth map resolution determining unit 103. The depth map generating unit 105 generates a depth map according to the resolution determined by the depth map resolution determining unit 103, using neighboring viewpoint images according to the number determined by the neighboring viewpoint image number determining unit 104. In this way, the depth map generating device 1 of the embodiment can determine the number of neighboring viewpoint images according to the resolution of the depth map DM to be generated in each layer. Therefore, it is possible to determine neighboring viewpoint images in each layer that improve the calculation accuracy of depth values ​​while suppressing an increase in calculation time.

[0080] Moreover, the depth map generation device 1 of the embodiment further includes a neighboring viewpoint image management unit 106. The neighboring viewpoint image management unit 106 determines the number of neighboring viewpoint images for each reference viewpoint image or for each of a plurality of pixel groups in the reference viewpoint image, depending on the number of neighboring viewpoint images determined by the neighboring viewpoint image number determination unit 104. This allows the depth map generation device 1 of the embodiment to select neighboring viewpoint images for each image (reference viewpoint image) or for each pixel group, depending on the number of neighboring viewpoint images. Therefore, neighboring viewpoint images can be selected depending on the balance between the memory capacity of the depth map generation device 1 and the accuracy of the depth map DM to be generated.

[0081] Furthermore, in the depth map generating device 1 of the embodiment, the nearby viewpoint image management unit 106 selects nearby viewpoint images according to the number of nearby viewpoint images using a nearby viewpoint map. The nearby viewpoint map is information indicating hierarchical images KG (candidate images) that are candidates for reference viewpoint images. This allows the depth map generating device 1 of the embodiment to select nearby viewpoint images by a simple method of selecting from the candidates indicated in the nearby viewpoint map.

[0082] Furthermore, in the depth map generating device 1 of the embodiment, an evaluation value is indicated in the nearby viewpoint map. The evaluation value indicates the degree to which a candidate image is appropriate as a nearby viewpoint image. The nearby viewpoint image management unit 106 selects a nearby viewpoint image according to the evaluation value of the candidate image indicated in the nearby viewpoint map and the number of nearby viewpoint images. This makes it possible for the depth map generating device 1 of the embodiment to select an image that is more appropriate as a nearby viewpoint image as the nearby viewpoint image according to the evaluation value.

[0083] Furthermore, in the depth map generating device 1 of the embodiment, the neighboring viewpoint image management unit 106 compares the number of neighboring viewpoint images in the third layer, which is different from the lowest layer, with the number of neighboring viewpoint images in the fourth layer, which is below the third layer. If the number of neighboring viewpoint images in the third layer is equal to or smaller than the number of neighboring viewpoint images in the fourth layer, which is below the third layer, the neighboring viewpoint image management unit 106 selects all of the neighboring viewpoint images selected in the third layer as neighboring viewpoint images in the fourth layer. Furthermore, if the number of neighboring viewpoint images in the third layer is larger than the number of neighboring viewpoint images in the fourth layer, the neighboring viewpoint image management unit 106 selects neighboring viewpoint images in the fourth layer from the neighboring viewpoint images selected in the third layer according to the evaluation value. This allows the depth map generating device 1 of the embodiment to use neighboring viewpoint images used in higher layers in lower layers as well. Therefore, the processing load for selecting neighboring viewpoint images can be reduced. Moreover, neighboring viewpoint images to be used in lower layers can be selected using all or part of the images selected in higher layers. This makes it possible to select appropriate images. Furthermore, even when a nearby viewpoint image is selected for an image selected at a higher level, the nearby viewpoint image can be selected based on the evaluation value, making it possible to select an appropriate image.

[0084] All or part of the depth map generating device 1 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, or devices that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client. The program may be for implementing part of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0085] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0086] 1...Depth map generator 101...Multi-viewpoint image information acquisition unit 102...hierarchical image generation unit 103... Depth map resolution determination unit (second determination unit) 104...neighboring viewpoint image number determination unit (first determination unit) 105...Depth map generation unit 106...Nearby viewpoint image management unit (management unit) 110... Nearby viewpoint map information storage unit

Claims

1. A depth map generation device that generates a depth map of an object from a plurality of multi-viewpoint images of the object captured from mutually different viewpoints using a hierarchical search method, comprising: a hierarchical image generating unit that generates hierarchical images by reducing the multi-viewpoint images at reduction ratios corresponding to the respective layers; a first determination unit that determines a number of neighboring viewpoint images, which is the number of neighboring viewpoint images to be used in calculating depth values ​​as a pair of stereo matching for a reference viewpoint image selected as a target for calculating depth values ​​from the layer images in each layer, so that the number of neighboring viewpoint images in at least one layer is a value different from the number of neighboring viewpoint images in other layers; a depth map generation unit that generates a depth map using the neighboring viewpoint images according to the number of neighboring viewpoint images determined by the first determination unit; Equipped with Depth map generator.

2. the first determination unit sets the number of nearby viewpoint images in a higher layer to a value greater than the number of nearby viewpoint images in a lower layer. The depth map generating device according to claim 1 .

3. the first determination unit sets the number of nearby viewpoint images in a first layer different from the top layer to a value smaller than the number of nearby viewpoint images in a second layer higher than the first layer. The depth map generating device according to claim 1 .

4. a second determination unit that determines the resolution of a depth map generated in each layer; the first determination unit determines the number of nearby viewpoint images in accordance with the resolution determined by the second determination unit; the depth map generation unit generates a depth map according to the resolution determined by the second determination unit, using the nearby viewpoint images according to the number determined by the first determination unit. The depth map generating device according to any one of claims 1 to 3.

5. a management unit that selects the neighboring viewpoint images for each of the reference viewpoint images or for each of a plurality of pixel groups in the reference viewpoint image according to the number of neighboring viewpoint images determined by the first determination unit, The depth map generating device according to any one of claims 1 to 4.

6. the management unit uses a nearby viewpoint map in which candidate images that are candidates for the nearby viewpoint image are indicated, and selects the nearby viewpoint image from the candidate images indicated in the nearby viewpoint map in accordance with the number of nearby viewpoint images; the depth map generation unit generates a depth map using the nearby viewpoint image selected by the management unit. The depth map generating device according to claim 5 .

7. The nearby viewpoint map indicates an evaluation value indicating the degree to which the candidate image is appropriate as the nearby viewpoint image, the management unit selects the nearby viewpoint image according to the evaluation value of the candidate image shown in the nearby viewpoint map and the number of nearby viewpoint images. The depth map generating device according to claim 6 .

8. the management unit selects all of the neighboring viewpoint images selected in the third layer as the neighboring viewpoint images in the fourth layer when the number of neighboring viewpoint images in a third layer different from the lowest layer is the same as or smaller than the number of neighboring viewpoint images in a fourth layer below the third layer, and selects the neighboring viewpoint images in the fourth layer from the neighboring viewpoint images selected in the third layer according to the evaluation values ​​of the neighboring viewpoint images selected in the third layer when the number of neighboring viewpoint images in the third layer is larger than the number of neighboring viewpoint images in the fourth layer; The depth map generating device according to claim 7 .

9. A depth map generation method performed by a depth map generation device that generates a depth map of an object from a plurality of multi-viewpoint images of the object captured from mutually different viewpoints using a hierarchical search method, the method comprising: a hierarchical image generating step of generating hierarchical images by reducing the multi-viewpoint images at reduction ratios corresponding to the respective layers; a first determination step of determining the number of neighboring viewpoint images, which is the number of neighboring viewpoint images to be used in calculating depth values ​​as a pair of stereo matching for a reference viewpoint image selected as a target for calculating depth values ​​from the layer images in each layer, so that the number of neighboring viewpoint images in at least one layer is a value different from the number of neighboring viewpoint images in other layers; a depth map generating step of generating a depth map using the nearby viewpoint images according to the number of nearby viewpoint images determined in the first determining step; Including, Depth map generation method.

10. A program for causing a computer to operate as a depth map generating device that generates a depth map of an object from a plurality of multi-viewpoint images of the object captured from mutually different viewpoints using a hierarchical search method, the program comprising: a hierarchical image generating means for generating hierarchical images by reducing the multi-viewpoint images at reduction ratios corresponding to the respective layers; a first determination means for determining a number of neighboring viewpoint images, which is the number of neighboring viewpoint images to be used in calculating depth values ​​as a pair for stereo matching, for a reference viewpoint image selected as a target for calculating depth values ​​from the layer images in each layer, such that the number of neighboring viewpoint images in at least one layer is a value different from the number of neighboring viewpoint images in other layers; a depth map generating means for generating a depth map using the nearby viewpoint images according to the number of nearby viewpoint images determined by the first determining means; A program to run as.

Images