Depth map generation device, depth map generation method, and program

Abstract

To accurately calculate a depth value and prevent an increase of calculation time, in using a hierarchical search method to generate a depth map from a multi-viewpoint image.SOLUTION: A depth map generation device includes: a hierarchical image generation part configured to generate a hierarchical image obtained by reducing the multi-viewpoint image at a reduction rate corresponding to each hierarchy level; a determination part configured to determine a resolution of a depth map generated at each hierarchy level; and a depth map generation part configured to generate a depth map according to the resolution determined by the determination part. The determination part determines a resolution of a depth map generated at each hierarchy level so that a thinning-out rate being the resolution of the depth map to the resolution of the hierarchical image in at least one hierarchy level is of a value different from a thinning-out rate in another hierarchy level.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a depth map generation device, a depth map generation method, and a program.

Background Art

[0002] Conventionally, there is a technique for generating a depth map of an object using a plurality of images (multi-viewpoint images) obtained by imaging the object from different viewpoints. In this technique, by utilizing the fact that the appearance of the object varies for each multi-viewpoint image, the depth value (depth value) of each pixel in the image is calculated using the principle of stereo matching. Thereby, a depth map of the object is generated.

[0003] Patent Document 1 discloses a technique for generating a depth map using a hierarchical search method. The hierarchical search method is a technique for calculating depth values using images reduced according to the reduction ratio of each of a plurality of hierarchies, and generating depth maps for each hierarchy. By using the hierarchical search method, the depth values calculated in the upper layer can be used for calculating the depth values in the lower layer, and it becomes possible to calculate depth values with high accuracy.

[0004] Also, when using the hierarchical search method, if an attempt is made to calculate the depth value for all pixels in the image of each layer, the calculation time becomes enormous and is not practical. For this reason, so-called "subsampling" is performed to reduce the number of pixels for which the depth value is calculated in the image of each layer. When pixels in an image at a certain layer are subsampled to reduce the pixels for which the depth value is calculated, the resolution of the depth map generated from the image at that layer becomes the resolution according to the subsampling rate.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, thinning Number of pixels When the value is large, the calculation time is reduced, but the calculation accuracy deteriorates, so decimation Number of pixels When the value is small, the calculation accuracy improves, but the problem is that the calculation time becomes enormous.

[0007] The present invention has been made in view of the above circumstances, and provides a depth map generation device, a depth map generation method, and a program that can accurately calculate depth values ​​and suppress the increase in calculation time when generating a depth map from multi-view images using a hierarchical search method. [Means for solving the problem]

[0008] The depth map generation apparatus of the present invention is a depth map generation apparatus that generates a depth map of an object from a plurality of multi-view images of the object taken from different viewpoints using a hierarchical search method, comprising: a hierarchical image generation unit that generates hierarchical images by reducing the multi-view images at a reduction ratio corresponding to each hierarchical level; a determination unit that determines the resolution of the depth map generated at each hierarchical level; and a depth map generation unit that generates a depth map according to the resolution determined by the determination unit, wherein the determination unit Based on one of several predetermined patterns, each Decimation rate, which is the resolution of the depth map relative to the resolution of the hierarchical image in the hierarchy. or Determine the resolution of the depth map generated at each hierarchical level. Furthermore, each of the above-mentioned multiple patterns is a pattern in which the thinning rate in the top layer is set to be equal to or greater than the thinning rate in a different layer from the top layer. .

[0009] In the depth map generation apparatus of the present invention, the determination unit is: Among the aforementioned multiple patterns, The resolution of the depth map generated in the intermediate layer is determined by first determining the resolution of the depth map generated in the top layer and second determining the resolution of the depth map generated in the bottom layer, and then interpolating between the first and second resolutions. Based on the pattern, the thinning rate or depth map resolution at each hierarchical level is determined. .

[0010] In the depth map generation apparatus of the present invention, the determination unit is: Among the aforementioned multiple patterns, The third resolution, which is the resolution of the depth map generated in the top layer, is determined, and the decimation rate in layers other than the top layer is set to a uniform value. Based on the pattern, the thinning rate or depth map resolution at each hierarchical level is determined. .

[0011] In the depth map generation apparatus of the present invention, the determination unit is: Among the aforementioned multiple patterns, The decimation rate in each hierarchical level is set to a uniform value. If there is a hierarchical level in which the resolution of the depth map generated falls below a threshold, the decimation rate in that hierarchical level is changed so that the resolution of the depth map generated in that hierarchical level becomes equal to the threshold. Based on the pattern, the thinning rate or depth map resolution at each hierarchical level is determined. .

[0012] In the depth map generation apparatus of the present invention, the determination unit is: Among the aforementioned multiple patterns, The resolution of the depth map generated in at least one intermediate layer is determined such that the resolution of the depth map generated in that intermediate layer is smaller than the resolution of the depth map generated in the layer above that intermediate layer. Based on the pattern, the thinning rate or depth map resolution at each hierarchical level is determined. .

[0013] The depth map generation method of the present invention is a depth map generation method performed by a depth map generation device that generates a depth map of an object from a plurality of multi-view images of the object taken from different viewpoints using a hierarchical search method, and includes a hierarchical image generation step of generating hierarchical images by reducing the multi-view images by a reduction ratio corresponding to each hierarchical level, a determination step of determining the resolution of the depth map generated at each hierarchical level, and a depth map generation step of generating a depth map according to the resolution determined in the determination step, wherein in the determination step, Based on one of several predetermined patterns, each Decimation rate, which is the resolution of the depth map relative to the resolution of the hierarchical image in the hierarchy. or Determine the resolution of the depth map generated at each hierarchical level. Furthermore, each of the above-mentioned multiple patterns is a pattern in which the thinning rate in the top layer is set to be equal to or greater than the thinning rate in a different layer from the top layer. .

[0014] The program of the present invention is a program for operating a computer as a depth map generation device that generates a depth map of an object from a plurality of multi-viewpoint images obtained by imaging the object from different viewpoints using a hierarchical search method. The computer is caused to operate as hierarchical image generation means for generating hierarchical images obtained by reducing the multi-viewpoint images at reduction ratios corresponding to respective hierarchies, determination means for determining the resolution of the depth map generated in each hierarchy, and depth map generation means for generating a depth map according to the resolution determined by the determination means. In the determination means, Based on one of several predetermined patterns, each The decimation ratio, which is the resolution of the depth map with respect to the resolution of the hierarchical image in the hierarchy or Determine the resolution of the depth map generated in each hierarchy Furthermore, each of the above-mentioned multiple patterns is a pattern in which the thinning rate in the top layer is set to be equal to or greater than the thinning rate in a different layer from the top layer. is a program.

Advantages of the Invention

[0015] According to the present invention, when generating a depth map from multi-viewpoint images using a hierarchical search method, the depth value can be accurately calculated, and an increase in calculation time can be suppressed.

Brief Description of the Drawings

[0016] [Figure 1] It is a diagram showing an example of the relationship between the resolution of the hierarchical image and the depth map of each hierarchy according to the embodiment. [Figure 2] It is a block diagram showing a configuration example of the depth map generation device 1 according to the embodiment. [Figure 3] It is a flowchart showing the flow of processing performed by the depth map generation device 1 according to the embodiment. [Figure 4] It is a flowchart showing the flow of processing performed by the depth map generation device 1 according to the embodiment. [Figure 5] It is a diagram for explaining a conventional hierarchical search method.

Embodiments of the Invention

[0017] Hereinafter, the depth map generation device according to the embodiment will be described with reference to the drawings.

[0018] (Regarding conventional hierarchical search methods) First, we will explain the conventional hierarchical search method using Figure 5. Figure 5 is a diagram illustrating the conventional hierarchical search method. Figure 5 shows four hierarchical images KG (hierarchical images KG1 to KG4) and four depth maps DM (depth maps DM1 to DM4).

[0019] As shown in Figure 5, the hierarchical search method has multiple hierarchies LR (hierarchies LR1 to LR4). In this example, four hierarchies are shown, but the number of hierarchies can be any number as long as it is two or more. The hierarchical images KG (hierarchical images KG1 to KG4) in each hierarchy are images that have been reduced in size from the multi-view image TG. For example, the hierarchical image KG4 of the lowest hierarchy LR4 is the multi-view image TG. The hierarchical image KG3 of the hierarchy LR3, one level above LR4, is an image that has been reduced in both the width and height of the multi-view image TG. The hierarchical image KG2 of the hierarchy LR2, one level above LR3, is an image that has been reduced in both the width and height of the multi-view image TG. The hierarchical image KG1 of the hierarchy LR1, one level above LR2, i.e., the highest level, is an image that has been reduced in both the width and height of the multi-view image TG.

[0020] In the hierarchical search method, depth values ​​are calculated pixel by pixel using the hierarchical image KG at each hierarchical level, working sequentially from the top to the bottom. The depth values ​​calculated in the upper levels are then used in the lower levels. For example, they may be used as initial values ​​when calculating depth values, or they may be adopted directly as depth values. Specifically, first, depth map DM1 is generated using the hierarchical image KG1 of the top-level hierarchical level LR1. Next, depth map DM2 is generated using the hierarchical image KG2 of the hierarchical level LR2, one level below LR1, and depth map DM1. Next, depth map DM3 is generated using the hierarchical image KG3 of the hierarchical level LR3, one level below LR2, and depth map DM2. Finally, depth map DM4 is generated using the hierarchical image KG4 of the hierarchical level LR4, one level below LR3, and depth map DM3. Depth map DM4 is the depth map DM corresponding to the multi-view image TG, and is the final output depth map.

[0021] Conventional hierarchical search methods have the problem that calculating the depth value of all pixels in the hierarchical image KG at each hierarchical level results in an enormous computation time, and if too many pixels are omitted from the hierarchical image KG at each hierarchical level, the accuracy of the depth value calculation decreases.

[0022] To address these issues, this embodiment sets the thinning rate individually for each hierarchical level, rather than setting a uniform rate for each level. This allows for individual setting of the thinning rate for each level, and by devising combinations of thinning rates for each level, it becomes possible to generate a depth map without drastically increasing computation time and without reducing the accuracy of depth calculations.

[0023] (Overview of Depth Map Generator 1) Here, an overview of the depth map generation device 1 will be explained using Figure 1. Figure 1 is a diagram showing an example of the relationship between the resolution of the hierarchical image and the depth map for each hierarchical level according to the embodiment. In Figure 1, the resolutions of the hierarchical levels LR are shown vertically, and the resolutions of the hierarchical image KG and depth map DM are shown horizontally. The resolution of the depth map DM relative to the resolution of the hierarchical image KG is also shown as the decimation rate. In this example, an example with five hierarchical levels is shown. Specifically, from top to bottom, the top layer, intermediate layer 1, intermediate layer 2, intermediate layer 3, and the bottom layer are provided.

[0024] Figure 1 shows four patterns, Pattern 1 to Pattern 4, as patterns for thinning rates.

[0025] Pattern 1 is a pattern in which the thinning rate at each hierarchical level is determined individually. The thinning rate to be set for each hierarchical level may be determined by the designer or may be stored in advance. In the example in this diagram, the thinning rate at the top level is 1 / 2, the thinning rate at intermediate level 1 is 1 / 8, the thinning rate at intermediate level 2 is 1 / 32, the thinning rate at intermediate level 3 is 1 / 16, and the thinning rate at the bottom level is 1 / 64.

[0026] In Pattern 1, the thinning rate at each hierarchical level can be set individually, making it possible to set the optimal combination depending on factors such as the target object and the accuracy of the generated depth map.

[0027] In the above example, the case of determining the decimation rate was explained, but it is also possible to determine the resolution first and then set the decimation rate according to the determined resolution. The resolution here is the resolution of the depth map DM generated at each layer. Since the decimation rate is the resolution of the depth map DM relative to the resolution of the layer image KG, once the resolution of the depth map DM generated at each layer is determined, the decimation rate at each layer is determined. In the example in this figure, the resolution of the top layer is 80,000 pixels, the resolution of the intermediate layer 1 is 80,000 pixels, the resolution of the intermediate layer 2 is 80,000 pixels, the resolution of the intermediate layer 3 is 640,000 pixels, and the resolution of the lowest layer is 640,000 pixels.

[0028] Pattern 2 is a pattern in which the resolution of the depth map DM in each of the intermediate layers 1 to 3 is determined by interpolation based on the resolution of the top layer (referred to as the first resolution) and the resolution of the bottom layer (referred to as the second resolution).

[0029] When generating depth maps (DM) using a hierarchical search method, it is necessary to carefully generate the depth map DM at the top layer. This is because the top layer has no hierarchy above it and cannot utilize the depth map DM generated in the upper layers. For example, if the resolution at the top layer is small, that is, at the top layer... The number of pixels to be thinned Large To make it better If a certain decimation rate is set, the depth value cannot be calculated, and the generated depth map DM may contain pixels without depth values. In this case, the depth value for those pixels may not be calculated in the lower layers, and the final generated depth map DM may contain pixels without depth values. For this reason, it is preferable that the top layer be configured so that the resolution does not become too small, either by the designer setting it intentionally or by setting a pre-stored resolution.

[0030] Furthermore, the resolution of the lowest layer becomes the resolution of the final generated depth map (DM). For this reason, it is preferable that the resolution of the lowest layer be set intentionally by the designer, or that a pre-stored resolution be set. In contrast, for the intermediate layers, having the resolution set automatically reduces the burden on the designer compared to when the designer sets it. It also reduces the memory capacity compared to when each resolution of the intermediate layers is pre-stored.

[0031] From this perspective, in Pattern 2, the resolution of the depth map DM in each of the intermediate layers 1 to 3 is determined by interpolation based on the first and second resolutions. This figure shows an example where the degree of hierarchy in the intermediate layers is determined to change linearly. Specifically, the resolution of the lower layers is set to a value that is uniformly 320,000 pixels higher than that of the upper layers. More specifically, the resolution of the top layer is 160,000 pixels, the resolution of intermediate layer 1 is 480,000 pixels, the resolution of intermediate layer 2 is 800,000 pixels, the resolution of intermediate layer 3 is 1,120,000 pixels, and the resolution of the lowest layer is 1,440,000 pixels. In this case, the decimation rates for each layer are 1 / 1 for the top layer, 3 / 4 for intermediate layer 1, 5 / 16 for intermediate layer 2, 7 / 64 for intermediate layer 3, and 9 / 256 for the lowest layer.

[0032] In Pattern 2, the resolution of the top layer can be set to prevent it from becoming too small, making it easier to maintain accuracy. Furthermore, the resolution of the lowest layer can be set, allowing control over the final generated depth map (DM). Additionally, the resolution of the intermediate layers can be set automatically, reducing the burden on the designer and minimizing memory usage.

[0033] Although the above example illustrates linear interpolation, it is not limited to this, and interpolation can, of course, be performed using any function, such as a quadratic function, a cubic function, or an exponential function.

[0034] Pattern 3 is a pattern in which the resolution of the top layer is set, and the decimation rate in layers other than the top layer is uniform. In the example in this figure, the resolution of the top layer is set to 160,000 pixels, and the decimation rate in layers other than the top layer (intermediate layers 1 to 3, and the bottom layer) is uniformly 1 / 16.

[0035] In Pattern 3, the resolution of the top layer can be set so that it does not become too small, making it easier to maintain accuracy. In addition, the decimation rate can be set uniformly for layers other than the top layer, which improves computational efficiency.

[0036] Pattern 4 sets a uniform decimation rate for all layers, and in layers where the resolution corresponding to the set decimation rate falls below a threshold, it changes the decimation rate for that layer so that the resolution becomes the threshold. In the example in this figure, first, the decimation rate for all layers is set uniformly to 1 / 16. In this case, the resolution of the top layer is 10,000 pixels, the resolution of the first intermediate layer is 40,000 pixels, the resolution of the second intermediate layer is 320,000 pixels, the resolution of the third intermediate layer is 640,000 pixels, and the resolution of the lowest layer is 2,560,000 pixels.

[0037] Next, the resolution at each layer is compared to a threshold. The threshold can be set arbitrarily, but for example, it is 160,000 pixels. If the threshold is 160,000 pixels, the resolution of the top layer and intermediate layer 1 will be below the threshold. In this case, the resolution of the top layer and intermediate layer 1 will be changed so that they are at the threshold (160,000 pixels). As a result, the resolution of the top layer is 160,000 pixels, the resolution of intermediate layer 1 is 160,000 pixels, the resolution of intermediate layer 2 is 320,000 pixels, the resolution of intermediate layer 3 is 640,000 pixels, and the resolution of the lowest layer is 2,560,000 pixels. In this case, the decimation rates for each layer are 1 / 1 at the top layer, 1 / 4 at intermediate layer 1, 1 / 16 at intermediate layer 2, 1 / 16 at intermediate layer 3, and 1 / 16 at the lowest layer.

[0038] In Pattern 4, the depth map (DM) resolution can be set to prevent it from becoming too small, making it easier to maintain accuracy. Furthermore, a uniform decimation rate can be set for layers where the resolution exceeds a threshold, improving computational efficiency.

[0039] As an alternative to the above, the decimation rate in at least one intermediate layer may be determined such that the resolution of the depth map generated in that intermediate layer is smaller than the resolution of the depth map generated in the layer above that intermediate layer. Depth maps DM generated in intermediate layers are less prone to accuracy degradation even when their resolution is smaller compared to the top layer. Therefore, by setting a smaller resolution in intermediate layers where accuracy degradation is less likely, it is possible to improve computational efficiency while suppressing accuracy degradation.

[0040] (Configuration of Depth Map Generator 1) Here, the configuration of the depth map generation device 1 will be explained using Figure 2. Figure 2 is a block diagram showing an example of the configuration of the depth map generation device 1 according to an embodiment. The depth map generation device 1 includes, for example, a multi-view image information acquisition unit 101, a hierarchical image generation unit 102, a determination unit 103, a depth map generation unit 104, a multi-view image information storage unit 105, a hierarchical image information storage unit 106, a resolution information storage unit 107, and a depth map information storage unit 108.

[0041] The multi-view image information acquisition unit 101 acquires image information of the multi-view image TG from the multi-view image information storage unit 105. The multi-view image TG consists of multiple images of an object captured from different viewpoints. The object here is any object that can be captured and has an arbitrary three-dimensional shape. The multi-view image information acquisition unit 101 outputs the acquired image information of the multi-view image TG to the hierarchical image generation unit 102.

[0042] The hierarchical image generation unit 102 generates a hierarchical image KG. The hierarchical image KG is an image obtained by reducing a multi-view image TG by a predetermined reduction ratio. The hierarchical image generation unit 102 generates a hierarchical image KG for each hierarchical level by reducing the multi-view image TG according to a reduction ratio predetermined for each level. The hierarchical image generation unit 102 stores the image information of the hierarchical image KG in the hierarchical image information storage unit 106.

[0043] The determination unit 103 determines the decimation rate or resolution at each hierarchical level. The determination unit 103 determines the decimation rate or resolution at each hierarchical level based, for example, on one of the patterns 1 to 4 described above. If the determination unit 103 uses a pre-stored resolution as the resolution at a certain hierarchical level, for example, it uses a resolution obtained from the resolution information storage unit 107 as the resolution at that hierarchical level.

[0044] The depth map generation unit 104 generates a depth map DM using a hierarchical search method with the decimation rate set for each hierarchical level. The depth map generation unit 104 stores the generated depth map DM in the depth map information storage unit 108.

[0045] First, the depth map generation unit 104 generates a depth map DM in the top layer using the hierarchical image KG in the top layer. For example, the depth map generation unit 104 selects an image (reference viewpoint image) from the hierarchical image KG in the top layer for which to obtain the depth map DM. For each pixel in the selected reference viewpoint image, the depth map generation unit 104 selects an image (neighboring viewpoint image) to be used as a stereo matching pair. The depth map generation unit 104 calculates the depth value at a pixel by performing stereo matching with the neighboring viewpoint images selected according to the pixel in the reference viewpoint image. By performing this process for each pixel in the reference viewpoint image, the depth map generation unit 104 generates a depth map DM in the top layer.

[0046] Next, the depth map generation unit 104 generates a depth map DM in the intermediate layer 1 using the hierarchical image KG in the intermediate layer 1 and the depth map DM in the top layer. For example, the depth map generation unit 104 selects an image (reference viewpoint image) from the hierarchical image KG in the intermediate layer 1 for which to obtain a depth map DM. For example, for pixels in the reference viewpoint image whose depth value has already been calculated in the depth map DM in the top layer, the depth map generation unit 104 adopts the depth value from the depth map DM in the top layer as the depth value of that pixel without calculating the depth value in the intermediate layer 1. This omits the process of calculating the depth value and reduces the processing load.

[0047] Alternatively, the depth map generation unit 104 adopts the depth value from the top-level depth map DM as the initial value when calculating the depth value of a pixel in the intermediate layer 1, for pixels in the reference viewpoint image whose depth value has already been calculated in the top-level depth map DM. This reduces the processing load required to calculate the depth value.

[0048] The depth map generation unit 104 similarly generates a depth map DM for each layer below the intermediate layer 1 (intermediate layer 2, intermediate layer 3, and the lowest layer) using the layer image KG in that layer and the depth map DM in the layer above it.

[0049] (Processing by depth map generator 1) The processing performed by the depth map generation device 1 will now be explained using Figures 3 and 4. Figures 3 and 4 are flowcharts showing the processing flow performed by the depth map generation device 1 according to this embodiment.

[0050] Figure 3 shows the general flow of processing performed by the depth map generation device 1. The depth map generation device 1 acquires a multi-view image TG (step S10). Next, the depth map generation device 1 generates a hierarchical image KG for each hierarchical level using the multi-view image TG (step S11). Next, the depth map generation device 1 determines the resolution of the depth map DM to be generated at each hierarchical level (step S12). The depth map generation device 1 generates the depth map DM using a hierarchical search method (step S13). In this case, the depth map generation device 1 ensures that a depth map DM corresponding to the resolution at each hierarchical level determined by the determination unit 103 is generated at each hierarchical level.

[0051] Figure 4 shows a detailed flow of the process performed in step S12 of Figure 3. The depth map generation device 1 individually sets the decimation rate for each layer as pattern 1 (step S120). Alternatively, the depth map generating device 1 may, as pattern 2, determine the resolution of the topmost layer (step S121), the resolution of the bottommost layer (step S122), and determine the resolution of the intermediate layers so as to interpolate the resolution of the topmost layer and the resolution of the bottommost layer (step S123). Alternatively, the depth map generation device 1 may determine the resolution of the top layer as pattern 3 (step S124), and determine the resolution of the bottom layer and the intermediate layer so that the thinning rate of the bottom layer and the intermediate layer is uniform (step S125). Alternatively, the depth map generation device 1 may, as a provisional decision in pattern 4, set a uniform decimation rate for all layers (step S126), and if there is a layer whose resolution falls below the threshold, change the decimation rate in that layer so that the resolution of that layer becomes the threshold (step S127).

[0052] As described above, the depth map generation device 1 according to the embodiment generates a depth map DM of an object from a plurality of multi-view images TG of the object, which are captured from different viewpoints, using a hierarchical search method. The depth map generation device 1 comprises a hierarchical image generation unit 102, a determination unit 103, and a depth map generation unit 104. The hierarchical image generation unit 102 generates a hierarchical image KG. The determination unit 103 determines the resolution of the depth map DM to be generated using the hierarchical image KG in each hierarchical level. The depth map generation unit 104 generates a depth map with the resolution determined by the determination unit in each hierarchical level. The determination unit 103 sets the decimation rate in at least one hierarchical level to a different value from the decimation rate in other hierarchical levels (Pattern 1). The decimation rate is the resolution of the depth map DM relative to the resolution of the hierarchical image KG.

[0053] As a result, the depth map generation device 1 of the embodiment can individually set the resolution of the depth map DM for each hierarchical level. Therefore, by devising combinations of decimation rates for each hierarchical level, it is possible to generate a depth map without increasing the computation time and without reducing the accuracy of the depth value calculation. Consequently, when generating a depth map using multi-view images with a hierarchical search method, it is possible to calculate the depth value with high accuracy and suppress the increase in computation time.

[0054] Furthermore, in the depth map generation device 1 of this embodiment, the determination unit 103 determines the first resolution, which is the resolution of the depth map generated in the uppermost layer, and the second resolution, which is the resolution of the depth map generated in the lowest layer. By interpolating between the first and second resolutions, the resolution of the depth map generated in the intermediate layer is determined (Pattern 2). As a result, the depth map generation device 1 of this embodiment can maintain accuracy by setting the resolution in the uppermost layer so that it does not become too small. In addition, the resolution of the lowest layer can be set to control the resolution of the final generated depth map DM.

[0055] Furthermore, in the depth map generation device 1 of this embodiment, the determination unit 103 determines the third resolution, which is the resolution of the depth map generated in the top layer, and determines a uniform decimation rate for layers other than the top layer (Pattern 3). As a result, the depth map generation device 1 of this embodiment can be set so that the resolution in the top layer does not become too small, and accuracy can be maintained. In addition, since the decimation rate can be set uniformly for layers other than the top layer, computational efficiency can be improved.

[0056] Furthermore, in the depth map generation device 1 of the embodiment, the determination unit 103 sets the decimation rate in each hierarchical level to a uniform value, and if there is a hierarchical level in which the resolution of the depth map generated in that hierarchical level falls below a threshold, the decimation rate in that hierarchical level is changed so that the resolution of the depth map generated in that hierarchical level becomes the threshold (Pattern 4). As a result, the depth map generation device 1 of the embodiment can be set so that the depth map DM resolution does not become too small, thereby maintaining accuracy. In addition, since the decimation rate can be set uniformly in hierarchical levels in which the resolution exceeds the threshold, it is possible to improve computational efficiency.

[0057] Furthermore, in the depth map generation device 1 of the embodiment, the determination unit 103 determines the decimation rate in at least one intermediate layer such that the resolution of the depth map generated in that intermediate layer is smaller than the resolution of the depth map generated in the layer above that intermediate layer. As a result, the depth map generation device 1 of the embodiment can set a small resolution in the intermediate layer, which is less prone to accuracy degradation, thereby improving computational efficiency while suppressing accuracy degradation.

[0058] Furthermore, in the depth map generation device 1 of the embodiment, the determination unit 103 is located at the top layer. The number of pixels to thin out However, at a different tier than the top tier The number of pixels to thin out twist LargeThe decimation rate in the top layer is determined to achieve this. This allows the depth map generation device 1 of the embodiment to improve computational efficiency while preventing the resolution in the top layer, where accuracy tends to decrease, from becoming too small.

[0059] The depth map generation device 1 in the above-described embodiment may be implemented in whole or in part by a computer. In that case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such a case. Furthermore, the above-mentioned program may be for implementing a part of the above-mentioned function, or it may be a program that can implement the above-mentioned function in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA.

[0060] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of symbols]

[0061] 1…Depth map generation device 101...Multi-view image information acquisition unit 102...Hierarchical Image Generation Unit 103...Decision Section 104... Depth map generation unit

Claims

1. A depth map generation device that generates a depth map of an object from multiple multi-view images of the object taken from different viewpoints using a hierarchical search method, A hierarchical image generation unit generates hierarchical images by reducing the aforementioned multi-view images with a reduction ratio corresponding to each hierarchical level, A determination unit that determines the resolution of the depth map generated at each hierarchical level, A depth map generation unit generates a depth map corresponding to the resolution determined by the determination unit, Equipped with, The determination unit determines, based on one of a predetermined set of patterns, the decimation rate which is the resolution of the depth map relative to the resolution of the hierarchical image in each hierarchical level, or the resolution of the depth map generated in each hierarchical level. Each of the aforementioned multiple patterns is a pattern in which the thinning rate in the top layer is set to be equal to or greater than the thinning rate in a different layer from the top layer. Depth map generation device.

2. The determination unit determines a first resolution, which is the resolution of the depth map generated in the uppermost layer, and a second resolution, which is the resolution of the depth map generated in the lowest layer, from among the plurality of patterns, and determines the decimation rate or depth map resolution in each layer based on the pattern that determines the resolution of the depth map generated in the intermediate layer by interpolating between the first resolution and the second resolution. The depth map generation apparatus according to claim 1.

3. The determination unit determines a third resolution, which is the resolution of the depth map generated in the topmost layer, from among the plurality of patterns, and determines the decimation rate or the resolution of the depth map in each layer based on a pattern that determines the decimation rate in layers other than the topmost layer to a uniform value. The depth map generation apparatus according to claim 1.

4. The determination unit determines the decimation rate or depth map resolution for each layer based on a pattern in which, among the plurality of patterns, the decimation rate in each layer is set to a uniform value, and if there is a layer in which the resolution of the depth map generated in that layer falls below a threshold, the decimation rate in that layer is changed so that the resolution of the depth map generated in that layer becomes the threshold. The depth map generation apparatus according to claim 1.

5. The determination unit determines the decimation rate or depth map resolution in each layer based on a pattern for determining the resolution of the depth map generated in an intermediate layer, such that the resolution of the depth map generated in at least one intermediate layer is smaller than the resolution of the depth map generated in a layer higher than that intermediate layer. A depth map generating apparatus according to any one of claims 1 to 3.

6. A depth map generation method performed by a depth map generation device that generates a depth map of an object from multiple multi-view images of the object taken from different viewpoints using a hierarchical search method, A hierarchical image generation step generates a hierarchical image by reducing the aforementioned multi-view images with a reduction ratio corresponding to each hierarchical level, A decision process to determine the resolution of the depth map generated at each hierarchical level, A depth map generation step that generates a depth map corresponding to the resolution determined by the above determination step, Includes, In the aforementioned determination step, based on one of a predetermined set of patterns, the decimation rate, which is the resolution of the depth map relative to the resolution of the hierarchical image in each hierarchical level, or the resolution of the depth map generated in each hierarchical level is determined. Each of the aforementioned multiple patterns is a pattern in which the thinning rate in the top layer is set to be equal to or greater than the thinning rate in a different layer from the top layer. Method for generating depth maps.

7. A program that operates a computer as a depth map generation device that generates a depth map of an object from multiple multi-view images of the object taken from different viewpoints using a hierarchical search method, The aforementioned computer, A hierarchical image generation means that generates a hierarchical image by reducing the aforementioned multi-view image by a reduction ratio corresponding to each hierarchical level, A determination means for determining the resolution of the depth map generated at each hierarchical level. A depth map generation means that generates a depth map corresponding to the resolution determined by the determination means, To operate as, The determination means determines, based on one of a predetermined set of patterns, the decimation rate which is the resolution of the depth map relative to the resolution of the hierarchical image in each hierarchical level, or the resolution of the depth map generated in each hierarchical level. Each of the aforementioned multiple patterns is a pattern in which the thinning rate in the top layer is set to be equal to or greater than the thinning rate in a different layer from the top layer. program.

Images