3D model generation device, method and program

Abstract

To add reliability to each of voxels of a model generated by a visual hull method and correct a shape using depth information obtained by a different approach, thereby improving model quality especially of a concave part.SOLUTION: A 3D model generation apparatus is mainly configured by a 3D model generation unit 1 and a 3D model correction unit 2, and generates a 3D voxel model of an object using RGB images obtained from multiple cameras having different viewpoints and depth maps obtained from multiple depth cameras having different viewpoints. The 3D model generation unit 1 extracts a silhouette of the object for each camera by a background difference method or the like, and applies silhouette images to a visual hull method to generate the 3D voxel model of the object. The 3D model correction unit 2 performs a correction, during modeling by the visual hull method, to remove a part of the voxel model which cannot be accurately modeled due to noise or the like, and outputs the corrected voxel model as a qualified 3D voxel model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a 3D model generation device, method, and program for generating a 3D voxel model from an image, and in particular to a 3D model generation device, method, and program for performing modeling processing based on reliability taking into account color information and depth information of the position where a voxel is projected based on camera parameters. [Background technology]

[0002] There are known techniques for generating 3D models using camera images. For example, a 3D model can be generated from multiple images taken by multiple cameras arranged around a subject.

[0003] Non-Patent Document 1 discloses a volume intersection method that generates the shape of a 3D model from multiple silhouette images. Non-Patent Document 2 discloses a Space Carving method that generates the shape of an object, including recesses, based on the consistency of color information between cameras when points of a 3D model are projected onto each camera.

[0004] Patent Document 1 discloses a method for generating the shape of an object by considering the smoothness of the surface in addition to the consistency of color information. Patent Document 2 discloses a technique for restoring a point that has been deleted in a 3D space according to the density of points around the target point in the 3D space after deleting the point based on the consistency of color information.

[0005] All of the above prior art technologies generate the shape of a 3D model from images from multiple cameras, but non-patent document 3 discloses a technology that actively obtains the depth value of an object from a depth sensor and generates the shape of a 3D model. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2012-208759 A [Patent Document 2] JP 2018-32938 A [Non-patent literature]

[0007] [Non-Patent Document 1] A. LAURENTINI, "The Visual Hull Concept for Silhouette-Based Image Understanding", IEEE Transactions on Pattern Analysis and Machine Intelligence", 150-162, 1994 [Non-Patent Document 2] KNKUTULAKOS, SM SEITZ, "A Theory of Shape by Space Carving," International Journal of Computer Vision 38(3), 199-218, 2000 [Non-Patent Document 3] Mori et al., "Real-time 3D shape reconstruction of moving objects using a range image sensor," Information Processing Society of Japan Interaction, 2014 Summary of the Invention [Problem to be solved by the invention]

[0008] The volume intersection method in Non-Patent Document 1 can accurately restore a model of a convex shape, but cannot restore concave parts. In Non-Patent Document 2, it is possible to express a concave by removing the surface shape based on the consistency of color. However, since it relies only on the consistency of color to determine whether to remove the surface shape, depending on the threshold value, it is not possible to sufficiently express concaves, etc., which reduces the quality of the model. The same is true for Patent Documents 1 and 2.

[0009] In non-patent document 3, the depth values ​​obtained by the depth sensor may contain noise in some areas due to the influence of the reflective characteristics of the object and the accuracy of the hardware, and the quality of a model restored using only the depth values ​​will be reduced due to the influence of this noise.

[0010] The object of the present invention is to provide a 3D model generation device, method, and program that solves the above-mentioned technical problems, assigns a degree of reliability to each voxel of a model generated by the visual volume intersection method, and corrects the shape using depth information obtained by a different approach, thereby improving model quality, particularly in concave shaped parts. [Means for solving the problem]

[0011] In order to achieve the above object, the present invention is characterized in that a 3D model generation device that generates a 3D model of an object based on a camera image has the following configuration.

[0012] (1) The present invention comprises a means for generating a voxel model based on a plurality of camera images having different viewpoints, and a means for correcting the voxel model based on depth maps acquired from a plurality of depth cameras having different viewpoints, the correcting means comprising a means for calculating a reliability for at least each surface voxel of the voxel model, and a means for removing the voxel model on a voxel-by-voxel basis based on the reliability and the corresponding depth value of the depth map.

[0013] (2) The apparatus includes a means for generating a voxel model based on a plurality of camera images with different viewpoints, and a means for correcting the voxel model based on depth maps acquired from a plurality of depth cameras with different viewpoints, the means for correcting including: a means for associating, for each surface voxel of the voxel model, a surface voxel candidate that would become the surface voxel if the surface voxel were removed; a means for calculating the reliability of the surface voxel and the surface voxel candidate; and a means for comparing, for each surface voxel, the reliability TA with the reliability TB of the associated surface voxel candidate, and removing surface voxels whose TA does not exceed TB. Effect of the Invention

[0014] (1) It is possible to generate a 3D model that accurately represents the shape by evaluating each voxel of a 3D voxel model and eliminating noise in concave parts of objects that cannot be reproduced using the volume intersection method, as well as noise caused by an insufficient number of cameras or insufficient accuracy in silhouette extraction.

[0015] (2) The reliability of each surface voxel of the generated voxel model is compared with that of a surface voxel candidate. If the reliability of the surface voxel candidate is higher, the surface voxel is removed. Conversely, if the reliability of the surface voxel is higher, the surface voxel is maintained. This prevents excessive removal of voxels during correction and enables the generation of a more accurate 3D voxel model. [Brief description of the drawings]

[0016] [Figure 1] 1 is a functional block diagram showing a configuration of a main part of a 3D model generating device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating the operation of the first embodiment. [Diagram 3] 4A to 4C are diagrams illustrating a method of correcting a voxel model in the first embodiment. [Figure 4] FIG. 13 is a diagram illustrating a first calculation method of a voxel reliability. [Diagram 5] FIG. 13 is a diagram illustrating a second calculation method of a voxel reliability. [Figure 6] FIG. 13 is a diagram illustrating a third method for calculating a voxel reliability. [Figure 7] FIG. 13 is a diagram illustrating a fourth method for calculating a voxel reliability. [Figure 8] FIG. 13 is a diagram illustrating a fifth method for calculating a voxel reliability. [Figure 9] FIG. 13 is a diagram illustrating a sixth calculation method of a voxel reliability. [Figure 10] 1 is a flowchart (part 1) showing a procedure for correcting a voxel model. [Figure 11] 13 is a flowchart (part 2) showing a procedure for correcting a voxel model. [Figure 12] FIG. 11 is a functional block diagram showing a configuration of a main part of a 3D model generating device according to a second embodiment of the present invention. [Figure 13] FIG. 11 is a diagram illustrating the operation of the second embodiment. [Figure 14] 10 is a flowchart showing an outline of the operation of the second embodiment. [Figure 15] 13 is a flowchart (part 1) illustrating in detail the operation of the second embodiment. [Figure 16] 13 is a flowchart (part 2) illustrating in detail the operation of the second embodiment. [Figure 17] 13 is a flowchart (part 3) illustrating in detail the operation of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a functional block diagram showing the configuration of the main part of a 3D model generating device according to a first embodiment of the present invention, and Fig. 2 is a diagram showing a schematic diagram of the operation thereof.

[0018] The 3D model generating device of the present invention mainly comprises a 3D model generating unit 1 and a 3D model correcting unit 2, and generates a 3D voxel model of a subject (object) using RBG images acquired from multiple cameras c with different viewpoints. Furthermore, the 3D voxel model is corrected using depth maps acquired from multiple depth cameras d or depth sensors (hereinafter collectively referred to as depth cameras d) with different viewpoints. The cameras c and the depth cameras d may be integrated or may be separated as shown in FIG. 2(a), and the position and number of the viewpoints may be arbitrary.

[0019] In the 3D model generation unit 1, the silhouette image generation unit 101 extracts the silhouette of an object (foreground region) for each camera using a background subtraction method or the like to generate a silhouette image. The model generation unit 102 applies a volume intersection method to each silhouette image to generate a 3D voxel model vA of the object, as shown in FIG. 2(b).

[0020] In this embodiment, the 3D space is divided into voxels, and each voxel v is projected to each camera using its camera parameters by the volume intersection method. As a result, voxels v that are determined to be inside the silhouette by all cameras are considered to have an object (On), and voxels v that are determined to be outside the silhouette by one or more cameras are considered to have no object (Off). The same process is performed for all voxels v, and a group of voxels v that are determined to be On is recognized as a 3D voxel model.

[0021] As shown in Figures 2(b) and (c), the 3D model correction unit 2 performs a correction to remove, on a voxel basis, a portion of the voxel model vA, which was generated larger than the object because it was not possible to accurately model concave portions, particularly due to the influence of noise, during modeling using the visual volume intersection method, and uses the corrected voxel model vC as the output model.

[0022] In the 3D model correction unit 2, the first reliability calculation unit 201 judges occlusion from each camera viewpoint for each voxel (hereinafter, sometimes referred to as a surface voxel) exposed on the surface of the voxel model vA as shown in Fig. 3(a), and calculates a reliability TA for each voxel v that is judged not to be occluded by at least one camera. A voxel v that is judged to be occluded by all cameras is excluded from the reliability calculation as an inner voxel that is not exposed on the surface of the object. However, if the inner voxel is also to be corrected (removed), its reliability can be calculated in the same way.

[0023] 4 is a diagram showing a first calculation method of the voxel reliability TA. The first reliability calculation unit 201 projects each surface voxel v of the voxel model vA to each camera using the camera parameters, and calculates the reliability TA of each voxel v based on the statistical value of the color information (e.g., RGB value) of each pixel at the projection position as an index.

[0024] In this embodiment, for each voxel v, the Euclidean distance d between the RGB value of the projection position of each camera c and the origin of the RGB space is cThe average μ is calculated using the following equations (1) and (2), and the standard deviation σ is calculated for each camera c based on the following equation (3).

[0025]

number

[0026] Here, C is the number of cameras, p c (x,y) R ,p c (x,y) G ,p c (x,y) B is the RGB value of the projection position of camera c, and q R , q G , q B is the RGB value at the origin of the RGB space. The confidence level TA is the lower threshold TH L , the upper threshold is TH H The calculation is performed using the following equations (4), (5), and (6).

[0027]

number

[0028] In the example of Figure 4, the RGB values ​​of the projection positions of voxel v1 onto each camera are all the same as shown in Figure 4(a), whereas the RGB values ​​of the projection positions of voxel v2 onto each camera are not the same as shown in Figure 4(b). As a result, the reliability TA of voxels v1 and v2 are calculated as "1.0" and "0.3", respectively, as shown in Figure 4(c).

[0029] 5 is a diagram showing a schematic diagram of a second calculation method of the voxel reliability TA. The first reliability calculation unit 201 projects each surface voxel v of the voxel model vA to each camera using the camera parameters, and calculates the reliability of the voxel v based on a statistical value of whether the projection position is an edge region of a silhouette image or not.

[0030] A characteristic of silhouette images is that there is a high possibility that voxels near the edges of a silhouette will be mistakenly determined to be the foreground. Therefore, in this embodiment, the reliability TA of each voxel is calculated so that voxels projected near the edges have a low reliability and voxels projected other than near the edges have a high reliability.

[0031] In a silhouette image, if the pixel value is white, it is determined that the area is an area where an object exists (On), and if the pixel value is black, it is determined that the area is an area where no object exists (Off). For each pixel in the On area, in this embodiment, it is determined whether or not there is an Off area within N neighborhoods (e.g., 8 neighborhoods) around the pixel of interest. A pixel that does not have an Off area within N neighborhoods is determined to not be in an edge area, and a pixel that has an Off area is determined to be in an edge area.

[0032] When there are multiple depth maps to project, the reliability of the voxel can be determined by the average reliability calculated for each depth map, etc. The reliability TA of a voxel vi of interest can be expressed by the following equations (7), (8), and (9), where f is the determination equation for whether or not the pixel x, y at the position where the voxel vi is projected onto each camera c using the camera parameters is an edge, and C is the number of cameras.

[0033]

number

[0034] The reliability TA ranges from 0.0 to 1.0, but is subject to a certain threshold TH Bin If the value is larger, TA may be binarized as 1.0, and if the value is smaller, TA may be binarized as 0.0. Also, instead of the average value of each camera, a majority vote may be used. In the case of a tie, TA may be set to 1.0.

[0035] In the example of Fig. 5, as shown in Fig. 5(a), none of the projection positions of voxel v1 onto each camera are in the edge region, whereas all of the projection positions of voxel v2 are in the edge region, as shown in Fig. 5(b). As a result, the reliability TA of voxels v1 and v2 are calculated as "1.0" and "0.0", respectively, as shown in Fig. 5(c).

[0036] 6 is a diagram showing a schematic diagram of a third calculation method of the voxel reliability TA. The first reliability calculation unit 201 projects the surface voxel v of the voxel model vA to each depth camera using the camera parameters. In this embodiment, based on the statistical value for the set of depth values ​​at the projection positions of the depth map, if the variation of the depth values ​​is large, the reliability TA is determined to be low, and if the variation is small, the reliability TA is determined to be high.

[0037] The confidence level T of a voxel v (ranges from 0.0 to 1.0) is calculated by the number of depth cameras D projecting the voxel v and the depth value in the depth map p d (x, y), mean μ, standard deviation σ, lower threshold TH L , the upper threshold is TH H This can be expressed by the following equation:

[0038]

number

[0039] In the example of Fig. 6, as shown in Fig. 6(a), the depth value d of the projection position of voxel v1 onto each depth map is all "10", while the depth value d of the projection position of voxel v2 onto each depth map is different, as shown in Fig. 6(b), namely "1", "5", and "10". As a result, the reliability TA of voxels v1 and v2 are calculated as "1.0" and "0.0", respectively, as shown in Fig. 6(c).

[0040] 7 is a diagram showing a schematic diagram of a fourth calculation method of the voxel reliability TA. The first reliability calculation unit 201 projects the surface voxel v of the voxel model vA onto the depth map of each depth camera d using the camera parameters. In this embodiment, the reliability of the voxel v is calculated based on a statistical value of whether the projection position is an edge or not, so that the reliability is low near the edge and high outside the edge. This is because the depth value near the edge often fluctuates greatly as a characteristic of the depth map.

[0041] In this embodiment, a predetermined threshold value TH is used to determine an area in the depth map where an object exists. Foreground The depth value of each pixel in the depth map is calculated using TH Foreground The smaller range is defined as an area where an object exists (On), and the larger range is defined as an area where an object does not exist (Off).

[0042] Next, for each pixel in the On region, based on whether or not there is an Off region within N neighborhoods (e.g., 8 neighborhoods) around the pixel of interest, a pixel that does not have an Off region within N neighborhoods is determined to be not an edge region, and a pixel that has an Off region is determined to be an edge region. As a result, voxels whose projection position based on the camera parameters is an edge region are evaluated as having a low reliability TA, and voxels outside the edge region are evaluated as having a high reliability TA.

[0043] When there are multiple depth maps to project, the reliability of the voxel may be determined, for example, by the average value of the reliability calculated for each depth map.

[0044] The reliability TA of each voxel v can be expressed by the following equations (15), (16), and (17), where f is the decision equation that indicates whether the pixel (x, y) at the position where the voxel vi is projected onto each depth camera d using its camera parameters is an edge or not, and D is the number of cameras.

[0045]

number

[0046] T ranges from 0.0 to 1.0, but it is subject to a certain threshold TH Bin If the value is larger, TA may be binarized as 1.0, and if the value is smaller, TA may be binarized as 0.0. Also, instead of the average value of each camera, a majority vote may be used. In the case of a tie, TA may be set to 1.0.

[0047] In the example of Fig. 7, as shown in Fig. 7(a), none of the projection positions of voxel v1 onto each depth map are in the edge region, whereas all of the projection positions of voxel v2 onto each depth map are in the edge region, as shown in Fig. 7(b). As a result, the reliability TA of voxels v1 and v2 are calculated as "1.0" and "0.0", respectively, as shown in Fig. 7(c).

[0048] 8 is a diagram showing a schematic diagram of a fifth calculation method of the voxel reliability TA. The first reliability calculation unit 201 projects each voxel of the voxel model vA to each depth camera with its camera parameters, and evaluates the reliability TA of a voxel having a smaller temporal variation in the depth value of the projection position in the depth map.

[0049] Depth maps may have temporal fluctuations even when the subject is not moving, so the reliability of the fluctuating areas may be considered low. Temporal variations can be calculated within a range of a certain number of frames (frame width L) that is unlikely to change significantly due to the movement of the person. For example, if no significant movement occurs within three frames, the frame width L = 3, and the temporal variations can be calculated for each frame.

[0050] In this embodiment, a statistical value of the depth values ​​at the projection positions of each depth map for each voxel is calculated as, for example, a standard deviation, and a reliability TA is calculated based on whether the standard deviation is equal to or smaller than a predetermined threshold value.

[0051] The reliability TA (range 0.0-1.0) of a voxel v is calculated by the calculation range (frame width) of the depth value variation L, and the pixel value p of the position where the voxel v in time frame l is projected onto each depth camera d using the camera parameters. dThe camera average of (x, y, l) can be calculated by the following equations (18)-(23), where μ is the camera average and σ is the standard deviation. Here, the number of depth cameras to be projected is D, and the lower threshold is TH L , the upper threshold is TH H It is also possible to obtain an evaluation index σ by integrating the standard deviation as the camera average μ. The evaluation index σ may be the minimum or maximum value among the cameras.

[0052]

number

[0053] In the example of Fig. 8, as shown in Fig. 8(a), the depth value of the projection position in each depth map of voxel v1 is the same at all times, whereas the depth value of the projection position in each depth map of voxel v2 changes with time as shown in Fig. 8(b), indicating large temporal fluctuations. As a result, the reliability TA of voxels v1 and v2 is calculated as "1.0" and "0.0", respectively, as shown in Fig. 8(c).

[0054] At this time, a weight Wd is assigned to each camera, and the evaluation index σ is calculated for each camera as in the following equation (24). d It may be calculated as a weighted average of

[0055]

number

[0056] The weight Wd may be given in advance or may be determined automatically. In the case of automatically determining the weight Wd, for example, a still object is set as a subject in advance, a depth map of a predetermined number of frames M is obtained for each camera, and the average σ0 of the temporal variation of the depth value in pixel units over the entire depth map is calculated. d The weight Wd for each camera may be calculated from

[0057] If the pixel values ​​of pixels x and y in frame m are l(x, y, m), the average evaluation index σ0d for the entire depth map of camera d is calculated by the following equation (25).

[0058]

number

[0059] However, if σ0d = 0.0, σ0d = β (β = 0.0001, etc.) is used as a measure to deal with division exceptions. The weight Wd is calculated using the following equations (26) and (27).

[0060]

number

[0061] In the above first to fifth calculation methods, the reliability TA of each voxel is explained as being calculated based on one evaluation index, but the present invention is not limited to this. The reliability TA may also be calculated based on a weighted average of multiple evaluation indexes, as in the sixth calculation method shown in FIG. 9.

[0062] The weights can be set manually or automatically. When setting them automatically, the most similar parameters can be selected by comparing the restored image and the camera image when the generated 3D model is rendered at the camera position among several weight parameters. The reliability TA is calculated by the reliability TA estimated by method m as shown in the following equation (28). m The weight W of method m m It can be calculated as the sum of the values ​​multiplied by

[0063]

number

[0064] Returning to FIG. 1, as shown in FIG. 3(b), the correction unit 202 removes voxels with low reliability from the voxel model vA, so that the voxels hidden behind the removed voxels are exposed and a corrected voxel model vC is generated as new surface voxels, which is output as a normal 3D voxel model.

[0065] FIGS. 10 and 11 are flowcharts showing the correction procedure in the 3D model correction unit 2. In step S101, a corrected voxel model vC imitating the voxel model vA generated by the volume intersection method is generated in the first reliability calculation unit 201. In step S102, the voxel identifier i for identifying each voxel in the 3D voxel space is reset (=0). In step S103, the voxel vi with the identifier i is selected as the current target voxel.

[0066] In step S104, it is determined whether the target voxel vi is the surface voxel vAi of the voxel model vA. If it is the surface voxel vAi, the process proceeds to step S105. In step S105, it is determined whether there is a camera that does not cause occlusion for the target voxel vAi (=vi). If there is at least one such camera, the process proceeds to step S106.

[0067] In step S106, the reliability TAi of the target voxel vAi is calculated by any one of the first to sixth calculation methods based on the color information, depth value, or statistical value of the time series of the projection positions of each camera and the depth map.

[0068] In step S200, the reliability TAi of the target voxel vAi is compared with a predetermined threshold TAth. If TAi < TAth is not satisfied, the reliability of the target voxel vAi is sufficiently high and no correction is required, so the process proceeds to step S209. In step S209, it is determined whether the above processing has been completed for all voxels v. If not, the identifier i is updated and the process returns to step S103, and the target voxel vi is switched to the next voxel and the above respective processes are repeated.

[0069] On the other hand, if TAi < TAth, the process proceeds to step S201, and the depth camera identifier j for identifying the depth camera d is reset. In step S202, the depth camera dj corresponding to the identifier j is selected as the current target depth camera. In step S203, occlusion of the target voxel vAi is determined from the perspective of the target depth camera dj. If there is no occlusion, the process proceeds to step S204.

[0070] In step S204, the target voxel vAi is projected onto the depth map of the target depth camera dj to obtain the depth value dAij(x, y) at the projection position (x, y). In step S205, the distance δAij between the target voxel vAi and the target depth camera dj is obtained. In this embodiment, in the world coordinate system, the Euclidean distance between the three-dimensional position (x1, y1, z1) of the depth camera dj and the three-dimensional position (x2, y2, z2) of the target voxel vAi is calculated based on the following equation (29).

[0071]

Equation

[0072] In step S206, the depth value dAij(x, y) and the distance δAij are compared. In this embodiment, the accuracy of the depth camera di is prioritized. If dAij(x, y) ≤ δAij, it is determined that the modeling accuracy of the target voxel vAi is not low from the perspective of the target depth camera dj, and the process proceeds to step S207. In step S207, it is determined whether the above processing has been completed for all depth cameras d. If not, the identifier j is updated, and the process returns to step S202. The target depth camera dj is switched to the next depth camera, and the above respective processes are repeated.

[0073] On the other hand, if dAij(x, y) > δAij, it is determined that the modeling accuracy of the target voxel vAi is low, and the process proceeds to step S208, where the voxel vi(vCi) corresponding to the target voxel vAi is removed from the correction voxel model vC.

[0074] In step S209, it is determined whether the above processing has been completed for all voxels v. If not, the identifier i is updated, the process returns to step S103, the voxel of interest vi is switched to the next voxel, and the above processing is repeated. If the processing has been completed for all voxels v, the current correction voxel model vC is output as the regular 3D voxel model, and the processing ends.

[0075] According to this embodiment, each voxel of a 3D voxel model is evaluated and voxels estimated to have low reliability are deleted, making it possible to generate a 3D model that can accurately represent the shape by eliminating noise in recesses of objects that cannot be reproduced by the visual volume intersection method, and noise caused by an insufficient number of cameras or insufficient accuracy in silhouette extraction.

[0076] FIG. 12 is a functional block diagram showing the configuration of the main parts of a 3D model generation device according to a second embodiment of the present invention, and FIG. 13 is a diagram showing the operation of the device. Since the same reference numerals as those used above indicate the same or equivalent parts, their explanation will be omitted.

[0077] This embodiment is characterized in that, in order to avoid accidentally removing too many voxels when correcting voxels, if removing a voxel would improve reliability, it is removed, but if removing a voxel would actually decrease reliability, it is not removed, thereby preventing more voxels than necessary from being removed.

[0078] In the 3D model correction unit 2, the first reliability calculation unit 201 calculates a reliability TA for each voxel that is determined to be not an occlusion in relation to at least one camera, for the voxel model vA generated by the 3D model generation unit 1 as shown in Figure 13(a).

[0079] As shown in FIG. 13(b), the provisional correction unit 203 removes surface voxels from the voxel model vA, thereby generating a provisional corrected voxel model vB in which voxels hidden by the removed surface voxels are exposed instead and can become new surface voxels (surface voxel candidates).

[0080] As shown in FIG. 13(c), the second reliability calculation unit 204 calculates the reliability TB of the surface voxel candidate of the temporary correction voxel model vB by the same method as the reliability TA of the surface voxel.

[0081] As shown in FIG. 13(d), the voxel association unit 205 associates the surface voxel removed from the voxel model vA with the surface voxel candidate that will be exposed on the surface in place of the removed surface voxel in the temporary correction voxel model vB.

[0082] As shown in FIG. 13(e), the correction unit 206 compares the reliabilities TA and TB of the associated surface voxel and surface voxel candidate, and removes the surface voxel with TA < TB from the voxel model vA, thereby generating a corrected voxel model vC in which the surface voxel candidate has become a surface voxel.

[0083] As shown in FIG. 13(f), the convergence determination unit 207 compares the difference ΔN (= NvA - NvC) between the number of voxels NvA and NvC of the voxel models vA and vC before and after correction with the convergence determination threshold Nth. As a result, while ΔN > Nth, the corrected voxel model vC is regarded as a new voxel model vA and the above processes are repeated, and when ΔN ≤ Nth, the corrected voxel model vC at that time is output as a normal 3D voxel model.

[0084] FIG. 14 is a flowchart showing an outline of the operation of the 3D model correction unit 2 in the second embodiment, and FIGS. 15 to 17 are flowcharts showing the details thereof.

[0085] In step S1, in the first reliability calculation unit 201, the reliability TAi of the surface voxel vAi of the voxel model vA generated by the volume intersection method is calculated by any one of the first to sixth calculation methods.

[0086] In step S2, in the temporary correction unit 203, surface voxels of the voxel model that mimics the voxel model vA are removed, and instead, a temporary correction voxel model vB in which surface voxel candidates are exposed is generated.

[0087] In step S3, in the second reliability calculation unit 204, the reliability TBi of the surface voxel candidate vBi of the temporary correction voxel model vB is calculated by any one of the first to sixth calculation methods.

[0088] In step S4, in the voxel association unit 205, for each surface voxel vAi of the voxel model, a surface voxel candidate vBi that would be exposed on the surface if the surface voxel were removed is associated.

[0089] In step S5, in the correction unit 206, the reliabilities TA and TB of the associated surface voxels and surface voxel candidates are compared, and a correction voxel model vC is generated by removing the surface voxels where TA < TB from the voxel model that mimics the voxel model vA.

[0090] In step S6, in the convergence determination unit 207, the difference ΔN (= NvA - NvC) between the number of voxels NvA and NvC of the voxel models vA and vC before and after correction is compared with a predetermined convergence determination threshold Nth. If ΔN ≧ Nth, the correction voxel model vC is regarded as a new voxel model vA and the process returns to step S1, and the above processes are repeated. If ΔN < Nth, the process proceeds to step S7, and the current correction voxel model vC is output as a normal 3D voxel model.

[0091] 15 to 17, the processes in steps S101 to S106 correspond to the operation of step S1. The processes in steps S201 to S209 correspond to the operation of step S2. The processes in steps S301 to S305 correspond to the operation of step S3. The processes in steps S401 to S409 correspond to the operation of step S4. The processes in steps S501 to S513 correspond to the operation of step S5. The processes in steps S601 to S602 correspond to the operation of step S6.

[0092] In step S101, a provisionally corrected voxel model vB is generated that imitates a voxel model vA generated by the volume intersection method. In step S102, a voxel identifier i that identifies each voxel in the 3D voxel space is reset. In step S103, a voxel vi with identifier i is selected as the current voxel of interest.

[0093] In step S104, it is determined whether the voxel of interest vi is a surface voxel vAi of the voxel model vA, and if it is a surface voxel vAi, the process proceeds to step S105. In step S105, it is determined whether there is an RGB camera that does not occlude the voxel of interest vAi (=vi), and if there is even one such camera, the process proceeds to step S106.

[0094] In step S106, the reliability TAi of the target voxel vAi is calculated by any one of the first to sixth calculation methods based on the color information of each RGB camera or the projection position onto the depth map DM, the depth value, or the time series statistical value.

[0095] In step S201, a depth camera identifier j that identifies a depth camera d is reset. In step S202, a depth camera dj with identifier j is selected as the current target depth camera. In step S203, occlusion of a target voxel vAi is determined from the viewpoint of the target depth camera dj. If there is no occlusion, the process proceeds to step S204.

[0096] In step S204, the voxel of interest vAi is projected onto the depth map of the depth camera of interest dj to obtain a depth value dAij (x, y) at the projection position (x, y). In step S205, the distance δAij between the voxel of interest vAi and the depth camera of interest dj is obtained.

[0097] In step S206, the depth value dAij (x, y) is compared with the distance δAij, and if dAij (x, y)≦δAij, it is determined that the modeling accuracy of the target voxel vAi is not low from the viewpoint of the target depth camera dj, and correction is not required at this point, and the process proceeds to step S207. In step S207, it is determined whether the above process has been completed for all depth cameras d. If not, the identifier j is updated and the process returns to step S202, the target depth camera dj is switched to the next depth camera, and the above processes are repeated.

[0098] On the other hand, if dAij(x,y)≦δAij is not true, it is determined that the modeling accuracy of the voxel of interest vAi is low, and the process proceeds to step S208, where the surface voxel vi(vBi) corresponding to the voxel of interest vAi is removed from the provisional corrected voxel model vB.

[0099] In step S209, it is determined whether or not the above processing has been completed for all voxels v. If not, the identifier i is updated, the process returns to step S103, the voxel of interest vi is switched to the next voxel, and the above processing is repeated.

[0100] 16, in step S301, the voxel identifier i is reset. In step S302, a voxel of interest vi is selected again. In step S303, it is determined whether the voxel of interest vi is a surface voxel candidate vBi of the provisional corrected voxel model vB. If the voxel of interest vi is a surface voxel candidate vBi, the process proceeds to step S304.

[0101] In step S304, it is determined whether or not there is a depth camera d that does not occlude the voxel of interest vBi (=vi), and if there is at least one such depth camera, the process proceeds to step S305. In step S305, the reliability TBi of the voxel of interest vBi is calculated by any one of the first to sixth calculation methods based on the color information of each RGB camera or the projection position onto the depth map DM, the depth value, or the time series statistical value.

[0102] In step S401, the depth camera identifier j is reset. In step S402, the current depth camera dj of interest is selected based on the identifier j. In step S403, the occlusion of the voxel of interest vBi is determined from the viewpoint of the depth camera of interest dj. If there is no occlusion, the process proceeds to step S404.

[0103] In step S404, the voxel of interest vBi is projected onto the depth map of the depth camera of interest dj to obtain a depth value dBij(x, y) at the projection position (x, y). In step S405, the distance δBij between the voxel of interest vBi and the depth camera of interest dj is obtained.

[0104] In step S406, the depth value dBij(x,y) is compared with the distance δBij, and if dBij(x,y)≦δBij, it is determined that the modeling accuracy of the voxel vBi is not low from the viewpoint of the depth camera of interest dj, and the process proceeds to step S407. In step S407, it is determined whether the above process has been completed for all depth cameras d. If not, the identifier j is updated and the process returns to step S402, where the depth camera of interest dj is switched to the next depth camera, and the above processes are repeated.

[0105] On the other hand, if dBij(x,y)≦δBij is not satisfied, it is determined that the modeling accuracy of the voxel vBi is low at least from the viewpoint of the depth camera of interest dj, and the process proceeds to step S408, where the current voxel identifier i is registered in the voxel correspondence list Pj(x,y). That is, in the voxel correspondence list Pj(x,y), the correspondence between the identifier i of the voxel v whose modeling accuracy is determined to be low from the viewpoint of any depth camera and the projection position (x,y) of the voxel vi in ​​each depth map is registered.

[0106] In step S409, it is determined whether the above processing has been completed for all voxels. If not, the identifier i is updated, the process returns to step S302, the voxel of interest vi is switched to the next voxel, and the above processing is repeated.

[0107] 17, in step S501, a correction voxel model vC that imitates the voxel model vA is generated. In step S502, the voxel identifier i is reset. In step S503, the voxel vi with the identifier i is selected as the current voxel of interest.

[0108] In step S504, it is determined whether the voxel of interest vi is a surface voxel vAi of the voxel model vA, and if it is a surface voxel vAi, the process proceeds to step S505. In step S505, it is determined whether there is an RGB camera that does not occlude the voxel of interest vAi, and if there is even one such camera, the process proceeds to step S506.

[0109] In step S506, the depth camera identifier j is reset. In step S507, the depth camera dj with the identifier j is selected as the current target depth camera. In step S508, the target voxel vAi is projected onto the target depth camera dj to obtain the projection position (x, y).

[0110] In step S509, the reliability TBi of the surface voxel candidate vBi identified by the voxel identifier i registered in the correspondence list Pj(x, y) selected using the projection position (x, y) of the target depth camera dj as an index is obtained.

[0111] If multiple voxel identifiers i are registered in the correspondence list Pj(x,y), the average value TBave(x,y) of the reliability TB of all the corresponding voxels vBi is calculated. The reliability TBave(x,y) is the average value of the reliability of multiple voxels registered in the correspondence list Pj(x,y).

[0112] In step S510, the reliability TAi of the voxel of interest vAi of the voxel model vA is compared with the reliability TB or its representative value TB ave(x, y) of the surface voxel candidate vB that will be exposed on the surface of the provisionally corrected voxel model vB when the voxel of interest vAi is removed. If TAi>TB ave(x, y), the correction of the voxel of interest vi is not necessary, so the process proceeds to step S511.

[0113] In step S511, it is determined whether the above processes are completed for all depth cameras. If not, the identifier j is updated and the process returns to step S507, the target depth camera dj is switched to the next depth camera, and the above processes are repeated. On the other hand, if the reliability TA_i>TB ave(x, y) is not satisfied, the process proceeds to step S512, and the voxel vi(vCi) corresponding to the target voxel vAi is removed from the correction voxel model vC.

[0114] In step S513, it is determined whether the above processes have been completed for all voxels. If not, the identifier i is updated and the process returns to step S503, the voxel of interest is switched to the next voxel, and the above processes are repeated. When the above processes have been completed for all voxels, the process proceeds to step S601.

[0115] In step S601, the number of voxels NvA constituting the voxel model vA is compared with the number of voxels NvC constituting the correction voxel model vC. If the difference in the number of voxels ΔN (=NvA-NvC) before and after correction is not equal to or less than a predetermined convergence determination threshold Nth, the process proceeds to step S602, the correction voxel model vC is regarded as a new voxel model vA, and the process returns to step S102 to repeat the above processes.

[0116] If the voxel number difference ΔN is equal to or smaller than a predetermined convergence determination threshold Nth, the corrective voxel model vC is output as a regular 3D voxel model, and the process ends.

[0117] According to this embodiment, the reliability of each surface voxel of the generated voxel model is compared with that of a surface voxel candidate, and if the reliability of the surface voxel candidate is higher, the surface voxel is removed, whereas if the reliability of the surface voxel is higher, the surface voxel is maintained. This prevents excessive removal of voxels during correction and enables the generation of a more accurate 3D voxel model.

[0118] According to each of the above embodiments, a high-quality 3D voxel model can be generated and provided in real time via communication infrastructure, making it possible to provide a variety of entertainment to many people regardless of geographic or economic disparity. As a result, it is possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations. [Explanation of symbols]

[0119] 1...3D model generation unit, 2...3D model correction unit, 101...silhouette image generation, 102...model generation unit, 201...first reliability calculation unit, 202...correction unit, 203...temporary correction unit, 204...second reliability calculation unit, 205...voxel correspondence unit, 206...correction unit, 207...convergence determination unit

Claims

1. A 3D model generation device that generates a 3D model of an object based on a camera image, A means for generating a voxel model based on a plurality of camera images having different viewpoints; and a means for correcting the voxel model based on depth maps acquired from a plurality of depth cameras having different viewpoints. The means for correcting the means for computing a confidence for each voxel of the voxel model; and a means for removing the voxel model on a voxel-by-voxel basis based on the reliability and the corresponding depth value of the depth map.

2. The 3D model generating device according to claim 1, characterized in that the removing means, for each voxel whose reliability is below a predetermined threshold, compares the depth value on the depth map when projected onto each depth camera using its camera parameters with the distance to each depth camera, and removes voxels whose distance is below the depth value.

3. The means for calculating the reliability calculates a reliability for each surface voxel of the voxel model; 3. The 3D model generating device according to claim 2, wherein the removing means removes surface voxels whose reliability falls below a predetermined threshold and whose distance falls below a depth value.

4. A 3D model generation device that generates a 3D model of an object based on a camera image, A means for generating a voxel model based on a plurality of camera images having different viewpoints; and a means for correcting the voxel model based on depth maps acquired from a plurality of depth cameras having different viewpoints. The means for correcting the means for associating, for each surface voxel of the voxel model, a surface voxel candidate that would become a surface voxel instead if the surface voxel were removed; means for calculating the confidence of surface voxels and candidate surface voxels; and means for comparing, for each surface voxel, its reliability TA with the reliability TB of the associated surface voxel candidate, and removing surface voxels whose TA does not exceed TB.

5. The 3D model generating device according to claim 4, characterized in that the removing means compares, for each surface voxel of the voxel model, the depth value on the depth map when projected onto each depth camera using its camera parameters with the distance to the depth camera, and removes surface voxels whose distance is less than the depth value.

6. 6. The 3D model generating device according to claim 4 or 5, characterized in that the correction means compares the number of voxels between a voxel model after the surface voxels have been removed and a voxel model before the removal, and repeats the correction by regarding the voxel model after the surface voxels have been removed as the voxel model before the removal, until the difference in the number of voxels falls below a predetermined threshold.

7. 7. A 3D model generating device according to claim 1, wherein the reliability calculating means calculates the reliability based on pixel information of the projection position of each voxel, including at least a surface vector of the voxel model, onto at least one of each camera and each depth map.

8. The 3D model generating device according to claim 7, characterized in that the means for calculating the reliability calculates a higher reliability the smaller the variation between cameras in color information of each projection position when each voxel is projected onto each camera.

9. The 3D model generating device according to claim 7, characterized in that the means for calculating the reliability calculates the reliability to be higher the lower the rate at which the projection positions of each voxel when projected onto each camera are on the edge of the object silhouette.

10. The 3D model generating device according to claim 7, characterized in that the means for calculating the reliability calculates a higher reliability when the variation between cameras in the depth values ​​of the projection positions when each voxel is projected onto each depth map is smaller.

11. The 3D model generating device according to claim 7, characterized in that the means for calculating the reliability calculates the reliability to be higher the lower the proportion of projection positions of each voxel projected onto each depth map that are on the edge of an object region.

12. The 3D model generating device according to claim 7, characterized in that the means for calculating the reliability calculates a higher reliability when the variation between cameras in the depth values ​​at each time of the projection position when each voxel is projected onto each depth map captured by each depth camera at different times is smaller.

13. The means for calculating the reliability includes: a first calculation means for calculating a reliability using a variation between cameras in color information at each projection position when each voxel is projected onto each camera as an index; a second calculation means for calculating a reliability using a ratio of a projection position of each voxel projected onto each camera to an edge of an object silhouette as an index; a third calculation means for calculating a reliability using a variation between cameras in a depth value of a projection position when each voxel is projected onto each depth map as an index; a fourth calculation means for calculating a reliability using a ratio of a projection position of each voxel projected onto each depth map to an edge of an object region as an index; and and at least two fifth calculation means for calculating reliability based on a variation between cameras in a depth value at each time of a projection position when each voxel is projected onto each depth map captured by each camera at different times, The 3D model generating device according to claim 7, wherein the reliability of each voxel is calculated based on a plurality of index values.

14. 14. The 3D model generating device according to claim 13, wherein the reliability calculating means calculates the reliability of each voxel based on a weighted average of a plurality of index values.

15. A 3D model generation method in which a computer generates a 3D model of an object based on a camera image, comprising: A voxel model is generated based on multiple camera images with different viewpoints, Acquire depth maps from multiple depth cameras with different viewpoints, Calculate the confidence for each voxel of the voxel model, A method for generating a 3D model, comprising removing a voxel model on a voxel-by-voxel basis based on said confidence level and the corresponding depth value of the depth map.

16. A 3D model generation method in which a computer generates a 3D model of an object based on a camera image, comprising: A voxel model is generated based on multiple camera images with different viewpoints, Acquire depth maps from multiple depth cameras with different viewpoints, for each surface voxel of the voxel model, a surface voxel candidate is associated that would become a surface voxel if the surface voxel were removed; Calculating confidence scores for surface voxels and candidate surface voxels; A method for generating a 3D model, comprising: comparing, for each surface voxel, its reliability TA with the reliability TB of the associated surface voxel candidate; and removing from the voxel model any surface voxel whose TA does not exceed TB.

17. In a 3D model generation program that generates a 3D model of an object based on a camera image, A step of generating a voxel model based on a plurality of camera images having different viewpoints; A procedure for acquiring depth maps from multiple depth cameras with different viewpoints; A procedure for calculating a confidence score for each voxel of the voxel model; and removing the voxel model on a voxel-by-voxel basis based on the reliability and the corresponding depth value of the depth map.

18. In a 3D model generation program that generates a 3D model of an object based on a camera image, A step of generating a voxel model based on a plurality of camera images having different viewpoints; A procedure for acquiring depth maps from multiple depth cameras with different viewpoints; for each surface voxel of the voxel model, associating a surface voxel candidate that would replace the surface voxel if the surface voxel were removed; calculating the confidence of surface voxels and candidate surface voxels; a step of comparing, for each surface voxel, its reliability TA with the reliability TB of the associated surface voxel candidate, and removing from the voxel model any surface voxel whose TA does not exceed TB.

Images