Three-dimensional shape measuring device and program for the same

Abstract

To provide a three-dimensional measuring device that can accurately measure the three-dimensional shape of a subject.SOLUTION: A three-dimensional shape measuring device 1 comprises: visual hull determination means 11 that performs frustum inside / outside determination of determining whether spatial points projected on a silhouette image are located inside a subject, to determine a visual hull voting result through voting processing on the spatial points located inside the subject; and depth constraint determination means 12 that determines whether the spatial points projected on a depth image are located on the depth side of the subject, determines a depth constraint voting result through voting processing on the spatial points located on the depth side of the subject, and outputs the spatial points in which the visual hull voting result is equal to or more than a visual hull threshold and the depth constraint voting result is equal to or more than a depth constraint threshold, as the three-dimensional shape of the subject.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a three-dimensional shape measuring device and a program therefor. [Background technology]

[0002] The stereo method is known as a method for measuring the three-dimensional shape of a subject. This stereo method obtains parallax information by matching local patterns between images taken from different viewpoints based on images taken from multiple viewpoints. That is, the stereo method focuses on local image features and patterns on the subject, and measures the three-dimensional shape of the subject based on the principle of triangulation by matching each local area between images taken by cameras installed at different viewpoints.

[0003] Also, a known method for measuring the three-dimensional shape of a subject is the volume intersection method, which calculates cones (view frustums) that have the viewpoints as their apexes and pass through the inside of the silhouette based on the silhouette of the subject seen from multiple viewpoints, and measures the three-dimensional shape of the subject based on the intersection area of ​​all the view frustums.

[0004] Also, a technique is known that achieves both the robustness of the volume intersection method and the accuracy of the stereo method by weighting matching so that the closer to the surface of the volume determined by the volume intersection method the lower the cost when performing depth search in the stereo method (Non-Patent Document 1). With the method described in Non-Patent Document 1, even if there are multiple similar patterns on the subject, it is possible to match patterns that are closer to the surface of the volume, thereby reducing matching errors. Also, with the method described in Non-Patent Document 1, even if there is a concave portion on the subject, the shape of the concave portion can be correctly estimated if the matching is performed correctly in the depth search.

[0005] Furthermore, in an apparatus for generating three-dimensional shape data from multi-viewpoint images, a technique is known in which the three-dimensional shape data is generated based on a reliability determined on the basis of the spatial resolution of an object in the image (Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 7195785 [Non-patent literature]

[0007] [Non-Patent Document 1] Tomiyama, Yoshihiro; Katayama, Miwa; Iwadate, Yuichi; Imaizumi, Hiroyuki. "3D Dynamic Object Generation from Multi-Viewpoint Images Using Volume Intersection and Stereo Matching," Journal of the Institute of Image Information and Television Engineers, Vol. 58, No. 6, pp. 797-806 (2004) Summary of the Invention [Problem to be solved by the invention]

[0008] In the above-mentioned stereo method, if there is no distinctive pattern on the subject, or if there are multiple similar patterns on the subject, correct matching may not be performed, and as a result, the three-dimensional shape of the subject may not be measured correctly.

[0009] The volume intersection method does not include matching between images as in the stereo method, so it does not cause problems even if there is no distinctive pattern on the subject or if there are multiple similar patterns. However, in the volume intersection method, concave parts of the subject are not reflected in the viewing frustum, so there is a possibility that concave parts are not reflected correctly in the three-dimensional shape of the subject.

[0010] In the method described in Non-Patent Document 1, if there is no characteristic pattern on the subject, the depth position may be erroneously associated with a different depth position from the actual depth. In addition, in the method described in Non-Patent Document 1, it is possible to localize the association result to the view volume surface by weighting, but artifacts such as wavy surface shape may occur within a certain distance from the view volume surface due to noise in the image. Since this artifact is easily affected by noise in the image, when a time-series three-dimensional shape is acquired, there is a possibility that shape flicker will occur in which the shape is distorted in a time- and spatially wavy manner.

[0011] The technology in Patent Document 1 can take into account the reliability based on the resolution when projecting components of a three-dimensional shape onto an image, but does not take into account the reliability due to the presence or absence of texture in the subject, resulting in insufficient accuracy of the three-dimensional shape.

[0012] SUMMARY OF THE PRESENT EMBODIMENTS The present invention has an object to solve the above-mentioned problems and to provide a three-dimensional shape measuring device and a program therefor that can measure the three-dimensional shape of a subject with high accuracy. [Means for solving the problem]

[0013] In order to solve the above-mentioned problems, the three-dimensional shape measuring device of the present invention is a three-dimensional shape measuring device that measures the three-dimensional shape of a subject based on silhouette images from each viewpoint representing the contour of the subject, a first parameter that matches the three-dimensional coordinates of each point in space to the image coordinates of the silhouette image, one or more depth images that represent the depth value of the subject, and a second parameter that matches the three-dimensional coordinates of each point in space to the image coordinates of the depth image, and is configured to include a visual volume intersection determination means and a depth constraint determination means.

[0014] According to this configuration, the visual hull intersection determination means performs a visual frustum inside / outside determination to determine whether or not each spatial point projected onto each silhouette image using the first parameter is located inside the subject, and obtains a visual hull intersection voting result by a voting process for each spatial point located inside the subject.

[0015] The depth constraint determination means determines whether each spatial point projected onto the depth image using the second parameter is located on the back side of the subject, and obtains a depth constraint voting result by a voting process for each spatial point located on the back side of the subject.The depth constraint determination means then outputs each spatial point whose visual volume intersection voting result is equal to or greater than a preset visual volume intersection threshold and whose depth constraint voting result is equal to or greater than a preset depth constraint threshold as the three-dimensional shape of the subject.

[0016] That is, the three-dimensional shape measuring device globally combines the voting results of the volume intersection method and the depth search using the volume intersection threshold and the depth constraint threshold, thereby enabling the three-dimensional shape of the subject to be measured with high accuracy.

[0017] The present invention can also be realized by a program for causing a computer to function as the above-mentioned three-dimensional shape measuring device. Effect of the Invention

[0018] According to the present invention, the three-dimensional shape of a subject can be measured with high accuracy. [Brief description of the drawings]

[0019] [Figure 1] 1 is a block diagram showing a configuration of a three-dimensional shape measuring apparatus according to an embodiment. [Diagram 2] 5 is a flowchart showing the operation of a volume intersection determination means in the embodiment. [Figure 3A] 5 is a flowchart showing the operation of a depth constraint determination means in the embodiment. [Figure 3B] 5 is a flowchart showing the operation of a depth constraint determination means in the embodiment. [Figure 4] FIG. 13A is an explanatory diagram illustrating a visual volume intersection voting result, and FIG. 13B is an explanatory diagram illustrating a depth constrained voting result in an embodiment. [Diagram 5] FIG. 2 is an explanatory diagram for explaining block matching in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and descriptions thereof may be omitted.

[0021] [Configuration of 3D shape measurement device] The configuration of a three-dimensional shape measuring device 1 according to an embodiment will be described with reference to FIG. The three-dimensional shape measuring device 1 measures the three-dimensional shape of a subject based on a plurality of silhouette images, a first parameter, one or more depth images, and a second parameter. As shown in Fig. 1, the three-dimensional shape measuring device 1 includes a three-dimensional shape measuring unit 2, M silhouette extraction means 20, and N stereo means (depth image generation means) 30.

[0022] It should be noted that M is an integer representing the number of viewpoints, that is, the number of silhouette images, and is preset to a value equal to or greater than 2 (in the example of FIG. 1, M=4). Furthermore, N is an integer representing the number of depth images, and is preset to a value equal to or greater than 1 (in the example of FIG. 1, N=2).

[0023] <3D shape measurement section> First, the three-dimensional shape measuring unit 2 will be described, and then the silhouette extraction means 20 and the stereo means 30 will be described. The three-dimensional shape measurement unit 2 uses a volume intersection method and a depth search in combination, globally combining the voting results of both methods using a volume intersection threshold and a depth constraint threshold, and measures the three-dimensional shape of the subject. At this time, when the reliability of the depth search is low, the three-dimensional shape measurement unit 2 uses only the volume intersection method. As shown in FIG. 1, the three-dimensional shape measurement unit 2 includes a volume intersection determination means 11 and a depth constraint determination means 12.

[0024] <<Visual volume intersection determination means>> The visual hull intersection determination means 11 performs a visual frustum inside / outside determination to determine whether or not each spatial point projected onto each silhouette image using the first parameter is located inside the subject, and obtains a visual hull intersection voting result by a voting process for each spatial point located inside the subject.

[0025] A silhouette image is an image that represents the outline of a subject as seen from each viewpoint. A method for generating a silhouette image will be described later. The first parameters are functions or camera parameters that associate the three-dimensional coordinates of each point in space with the image coordinates of the silhouette image. The first parameters also correspond one-to-one to the silhouette image. Hereinafter, the first parameters may be simply referred to as parameters.

[0026] Here, the visual volume intersection determination means 11 receives M silhouette images from the silhouette extraction means 20 described below. Also, the visual volume intersection determination means 11 receives M parameters (first parameters) from the user of the three-dimensional shape measuring device 1.

[0027] Let m-th (m∈{1,2,…,M}) silhouette image be S m Image coordinates [x,y] T Silhouette image S m The pixel values ​​of S m Let (x,y) be the silhouette image S m Pixel value S m For (x, y), '1' is set for image coordinates within the subject, and '0' is set for other areas.

[0028] The first parameter is the three-dimensional coordinates of each point in space [X,Y,Z] T The silhouette image S m Image coordinates [x m ,y m ] T In the example of FIG. 1, the first parameters corresponding to the silhouette images S1 to S4 are illustrated as parameters 1 to 4. For example, the first parameters are expressed by the three-dimensional coordinates [X, Y, Z] as shown in the following formula (1).T Silhouette image S m Image coordinates [x m ,y m ] T A function F that shows the projection onto m It can be defined as:

[0029]

number

[0030] This function F m can be implemented with a look-up table (LUT), which returns an output value of two-dimensional coordinates (image coordinates) for an input value of three-dimensional coordinates. For example, this LUT can be generated by capturing a point whose three-dimensional coordinates are known with a camera and measuring the two-dimensional coordinates of that point. In this case, each point of the two-dimensional coordinates may be measured, or only some points may be measured and the remaining points may be interpolated.

[0031] In addition, the camera parameters θ m For example, the formula (1) may be calculated by perspective projection using the camera parameters θ m contains information on the camera position, camera attitude, lens angle of view or focal length, and the pixel size and resolution of the image sensor. m may contain information about lens distortion.

[0032] The visual hull intersection determination means 11 defines an array for voting in the voxel space as V vote The three-dimensional coordinate system [X,Y,Z] is the world coordinate system. T The votes for the array V vote (X,Y,Z). This array V vote represents the visual hull intersection voting result.

[0033] Here, the three-dimensional coordinates [X,Y,Z] T Each component of is assumed to be discretized. Also, the three-dimensional coordinates [X,Y,Z] T Each component is represented by an integer and exists at a grid point. Also, the three-dimensional coordinates [X,Y,Z]T Each component is assumed to be within the range expressed by the following formula (2): Three-dimensional coordinates [X0, Y0, Z0] T represents the starting point of the scan in the voxel space, and its three-dimensional coordinates are [X1, Y1, Z1] T represents the scan end point in the voxel space.

[0034]

number

[0035] The processing contents of the visual volume intersection determination means 11 will be described in detail with reference to FIG. As shown in FIG. 2, in step S100, the visual volume intersection determination means 11 calculates an array V vote Initialize (X,Y,Z) to 0. In step S101, the visual volume intersection determination means 11 calculates the silhouette image S m A variable m for selecting the viewpoint is initialized to 1.

[0036] In step S102, the visual volume intersection determination means 11 assigns Z0 to a variable Z' for scanning the voxel space. In step S103, the visual volume intersection determination means 11 assigns Y0 to a variable Y' for scanning the voxel space. In step S104, the visual volume intersection determination means 11 assigns X0 to a variable X' for scanning the voxel space.

[0037] In step S105, the visual volume intersection determination means 11 calculates the function F m to three-dimensional coordinates [X', Y', Z'] T By substituting the above, the image coordinates [x´ m ,y´ m ] T Calculate.

[0038]

number

[0039] In step S106, the visual volume intersection determination means 11 calculates the silhouette image S m Pixel value S m In step S105, it is determined whether (x, y)=1. In step S105, the three-dimensional coordinates [X', Y', Z'] in the voxel space are converted into a function F m Silhouette image S m When projected onto the object, it is determined whether the point is located inside the object (inside / outside view frustum determination).

[0040] Silhouette image S m Pixel value S m If (x, y)=1 (Yes in step S106), the visual volume intersection determination means 11 proceeds to the process of step S107. Silhouette image S m Pixel value S m If (x, y) is not 1 (No in step S106), the visual volume intersection determination means 11 proceeds to the process of step S108.

[0041] In step S107, the visual volume intersection determination means 11 calculates the array V vote The values ​​of (X', Y', Z') are incremented. In this step S107, a voting process is performed for the three-dimensional coordinates [X', Y', Z'] located inside the subject. In step S108, the visual volume intersection determination means 11 increments the variable X'.

[0042] In step S109, the visual volume intersection determination means 11 determines whether or not the variable X' exceeds the integer X1. If the variable X' exceeds the integer X1 (Yes in step S109), the visual volume intersection determination means 11 proceeds to the process of step S110. If the variable X' does not exceed the integer X1 (No in step S109), the visual volume intersection determination means 11 returns to the process of step S105.

[0043] In step S110, the visual volume intersection determination means 11 increments the variable Y'.

[0044] In step S111, the visual volume intersection determination means 11 determines whether or not the variable Y' exceeds the integer Y1. If the variable Y' exceeds the integer Y1 (Yes in step S111), the visual volume intersection determination means 11 proceeds to the process of step S112. If the variable Y' does not exceed the integer Y1 (No in step S111), the visual volume intersection determination means 11 returns to the process of step S104.

[0045] In step S112, the visual volume intersection determination means 11 increments the variable Z'.

[0046] In step S113, the visual volume intersection determination means 11 determines whether or not the variable Z' exceeds the integer Z1. If the variable Z' exceeds the integer Z1 (Yes in step S113), the visual volume intersection determination means 11 proceeds to the process of step S114. If the variable Z' does not exceed the integer Z1 (No in step S113), the visual volume intersection determination means 11 returns to the process of step S103.

[0047] In step S114, the visual volume intersection determination means 11 increments the variable m.

[0048] In step S115, the visual volume intersection determination means 11 determines whether the variable m exceeds an integer M or not. If the variable m exceeds the integer M (Yes in step S115), the visual volume intersection determination means 11 proceeds to the process of step S116. If the variable m does not exceed the integer M (No in step S115), the visual volume intersection determination means 11 returns to the process of step S102.

[0049] In step S116, the visual volume intersection determination means 11 calculates the array V vote is output to the depth constraint determination means 12 as the visual hull intersection voting result.

[0050] <<Depth constraint determination means>> Returning to FIG. 1, the description of the three-dimensional shape measuring unit 2 will continue. The depth constraint determination means 12 determines whether each spatial point projected onto the depth image using the second parameter is located on the back side of the subject, and obtains a depth constraint voting result by a voting process for each spatial point located on the back side of the subject.The depth constraint determination means 12 then outputs each spatial point whose visual volume intersection voting result V1 is equal to or greater than a preset visual volume intersection threshold and whose depth constraint voting result is equal to or greater than a preset depth constraint threshold as the three-dimensional shape of the subject.

[0051] In this embodiment, the depth constraint determination means 12 receives the reliability associated with each pixel of each depth image. In this case, the depth constraint determination means 12 determines whether each spatial point projected onto the depth image using the second parameter is located on the back side of the subject and the reliability of each spatial point projected onto the depth image is equal to or greater than a preset reliability threshold. Then, the depth constraint determination means 12 obtains a depth constraint voting result by a voting process for each spatial point located on the back side of the subject and having a reliability equal to or greater than the reliability threshold.

[0052] Here, the depth constraint determination means 12 receives N depth images and a reliability image from the stereo means 30 described below. In addition, the depth constraint determination means 12 receives N second parameters, a visual volume intersection threshold, a depth constraint threshold, and a reliability threshold from the user of the three-dimensional shape measuring device 1. Hereinafter, the second parameters may be simply referred to as parameters.

[0053] The depth image is an image that represents the depth value of a subject. The reliability image is an image that represents the reliability of the depth image. The method of generating the depth image and the reliability image will be described in detail later. The second parameters are functions or camera parameters that associate the three-dimensional coordinates of each point in space with the image coordinates of the depth image, and have a one-to-one correspondence with the depth image.

[0054] Let the nth (n∈{1,2,…,N}) depth image be D n Image coordinates [x,y] T Depth image D n Let the depth value (pixel value) of n Let (x, y) be the depth image D. n Depth value D n For (x, y), the optical axis direction component of the vector from viewpoint n to the subject part is set for image coordinates inside the subject, and infinity '∞' is set for other areas.

[0055] The nth confidence image is R n Image coordinates [x,y] T Confidence image R n The pixel value of R n Let (x, y) be the confidence image R n Pixel value R n (x,y) is the depth image D n Depth value D n The more reliable (x, y) is, the larger the value is set.

[0056] The second parameter is the three-dimensional coordinates of each point in space [X,Y,Z] T Depth image D n Image coordinates [ξ n ,η n ] T For example, the second parameter is a function or camera parameter that corresponds to the three-dimensional coordinates [X, Y, Z] as shown in the following formula (4). T Depth image D n Image coordinates [ξ n ,η n ] T A function G that shows the projection onto n It can be defined as:

[0057]

number

[0058] This function G n is the function F mAs with the previous example, it can be implemented with an LUT that returns output values ​​in two-dimensional coordinates (image coordinates) for input values ​​in three-dimensional coordinates. For example, this LUT can be generated by taking a picture of a point whose three-dimensional coordinates are known with a camera and measuring the two-dimensional coordinates of that point. In this case, each point in the two-dimensional coordinates may be measured, or only some points may be measured and the remaining points may be interpolated.

[0059] In addition, the camera parameter φ n For example, the equation (4) may be calculated by perspective projection using the camera parameters φ n contains information on the camera position, camera attitude, lens angle of view or focal length, and the pixel size and resolution of the image sensor. n may contain information about lens distortion.

[0060] Note that the first parameter, the function F m and the camera parameter θ m and the second parameter, the function G n and the camera parameter φ n Although the number M of silhouette images and the number N of depth images are different, they are defined by different letters, but the calculation content is the same. Therefore, in FIG. 1, parameters 1 to 4 are input to the visual volume intersection determination means 11 and the depth constraint determination means 12, respectively.

[0061] Depth image D n The three-dimensional coordinates of the viewpoint where the n, B n, C n ] T In addition, the depth image D n The optical axis direction is the unit vector L n In this case, the depth image D n From the viewpoint where the measurement was made, the three-dimensional coordinate [X , Y , Z] T Depth ζ when looking at n Function H for finding n is expressed by the following equation (5).

[0062]

number

[0063] The depth constraint determination means 12 defines an array for voting in the voxel space as V vote2 The three-dimensional coordinate system [X,Y,Z] is the world coordinate system. T The votes for the array V vote2 (X,Y,Z). This array V vote2 represents the depth-constrained voting result.

[0064] The processing contents of the depth constraint determination means 12 will be described in detail with reference to FIGS. 3A and 3B. As shown in FIG. 3A, in step S200, the depth constraint determination means 12 calculates an array V vote2 Initialize (X,Y,Z) to 0. In step S201, the depth constraint determination means 12 calculates the depth image D n A variable n for selecting the viewpoint is initialized to 1.

[0065] In step S202, the depth constraint determination means 12 assigns Z0 to a variable Z' for scanning the voxel space. In step S203, the depth constraint determination means 12 assigns Y0 to a variable Y' for scanning the voxel space. In step S204, the depth constraint determination means 12 assigns X0 to a variable X' for scanning the voxel space.

[0066] In step S205, the depth constraint determination means 12 calculates the function G n to three-dimensional coordinates [X', Y', Z'] T By substituting the above, the image coordinates [ξ´ n ,η´ n ] T Calculate.

[0067]

number

[0068] In step S206, the depth constraint determination means 12 calculates the function H n to three-dimensional coordinates [X', Y', Z'] T By substituting the above, the depth ζ' is calculated as shown in the following equation (7). n Calculate.

[0069]

number

[0070] In step S207, the depth constraint determination means 12 calculates the depth image D n Depth value D n (ξ´ n ,η´ n ) is the depth ζ n and the confidence image R n Pixel value R n (ξ´ n ,η´ n ) is equal to or greater than a reliability threshold value ρ.

[0071] In this step S207, the judgment condition in the first stage is that the three-dimensional coordinates [X', Y', Z'] in the voxel space are converted into a function G n Depth image D n When projected onto the object, it indicates whether the point is located on the back side of the object. In other words, in the solid model of the object (surface and interior), the depth ζ n Since the subject is present on the back side of the three-dimensional position indicated by and is not present on the front side of the three-dimensional position, the previous determination condition is met.

[0072] The latter judgment condition is to apply the three-dimensional coordinates [X', Y', Z'] of the voxel space to the function G n Depth image D n When projected onto the point, the reliability R n (ξ´ n ,η´ n ) is equal to or greater than the reliability threshold value ρ.

[0073] Depth value D n (ξ´ n ,η´ n ) is the depth ζ n Less than or equal to pixel value R n (ξ´ n ,η´ n If ρ is equal to or greater than the reliability threshold value ρ (Yes in step S207), the depth constraint determination means 12 proceeds to the process of step S208. Depth value D n (ξ´ n ,η´ n ) is the depth ζ n Less than or equal to pixel value R n (ξ´ n ,η´ n If ρ is not equal to or greater than the reliability threshold value ρ (No in step S207), the depth constraint determination means 12 proceeds to the process of step S209.

[0074] In step S208, the depth constraint determination means 12 calculates the array V vote2 In step S209, the values ​​of (X', Y', Z') are incremented. n (ζ´ n ,η´ n ) is greater than or equal to the reliability threshold ρ, and a voting process is performed for the three-dimensional coordinates [X', Y', Z']. In step S209, the depth constraint determination means 12 increments the variable X'.

[0075] In step S210, the depth constraint determination means 12 determines whether or not the variable X' exceeds the integer X1. If the variable X' exceeds the integer X1 (Yes in step S210), the depth constraint determination means 12 proceeds to the process of step S211. If the variable X' does not exceed the integer X1 (No in step S210), the depth constraint determination means 12 returns to the process of step S205.

[0076] As shown in FIG. 3B, in step S211, the depth constraint determination means 12 increments the variable Y'.

[0077] In step S212, the depth constraint determination means 12 determines whether or not the variable Y' exceeds the integer Y1. If the variable Y' exceeds the integer Y1 (Yes in step S212), the depth constraint determination means 12 proceeds to the process of step S213. If the variable Y' does not exceed the integer Y1 (No in step S212), the depth constraint determination means 12 returns to the process of step S204.

[0078] In step S213, the depth constraint determination means 12 increments the variable Z'.

[0079] In step S214, the depth constraint determination means 12 determines whether or not the variable Z' exceeds the integer Z1. If the variable Z' exceeds the integer Z1 (Yes in step S214), the depth constraint determination means 12 proceeds to the process of step S215. If the variable Z' does not exceed the integer Z1 (No in step S214), the depth constraint determination means 12 returns to the process of step S203.

[0080] In step S215, the depth constraint determination means 12 increments the variable n.

[0081] In step S216, the depth constraint determination means 12 determines whether the variable n exceeds the integer N or not. If the variable n exceeds the integer N (Yes in step S115), the depth constraint determination means 12 proceeds to the process of step S217. If the variable n does not exceed the integer N (No in step S115), the depth constraint determination means 12 returns to the process of step S202.

[0082] In step S217, the depth constraint determination means 12 assigns Z0 to a variable Z' for scanning the voxel space. In step S218, the depth constraint determination means 12 assigns Y0 to a variable Y' for scanning the voxel space. In step S219, the depth constraint determination means 12 assigns X0 to a variable X' for scanning the voxel space.

[0083] In step S220, the depth constraint determination means 12 calculates voxel data V(X', Y', Z') representing the three-dimensional shape of the object using the following formula (8). vote is greater than or equal to the hull intersection threshold (Md) and the depth-constrained voting result V vote2 If the depth constraint threshold (Ne) or more, the voxel data V(X', Y', Z') is set to '1', otherwise the voxel data V(X', Y', Z') is set to '0'. In this step S220, the visual volume intersection voting result V vote is greater than or equal to the hull intersection threshold (Md) and the depth-constrained voting result V vote2 3D coordinates [X´,Y´,Z´] where the depth constraint threshold (Ne) is exceeded T is being sought.

[0084]

number

[0085] In step S221, the depth constraint determination means 12 increments the variable X'.

[0086] In step S222, the depth constraint determination means 12 determines whether or not the variable X' exceeds the integer X1. If the variable X' exceeds the integer X1 (Yes in step S222), the depth constraint determination means 12 proceeds to the process of step S223. If the variable X' does not exceed the integer X1 (No in step S222), the depth constraint determination means 12 returns to the process of step S220.

[0087] In step S223, the depth constraint determination means 12 increments the variable Y'.

[0088] In step S224, the depth constraint determination means 12 determines whether or not the variable Y' exceeds the integer Y1. If the variable Y' exceeds the integer Y1 (Yes in step S224), the depth constraint determination means 12 proceeds to the process of step S225. If the variable Y' does not exceed the integer Y1 (No in step S224), the depth constraint determination means 12 returns to the process of step S219.

[0089] In step S225, the depth constraint determination means 12 increments the variable Z'.

[0090] In step S226, the depth constraint determination means 12 determines whether or not the variable Z' exceeds the integer Z1. If the variable Z' exceeds the integer Z1 (Yes in step S226), the depth constraint determination means 12 proceeds to the process of step S227. If the variable Z' does not exceed the integer Z1 (No in step S226), the depth constraint determination means 12 returns to the process of step S218.

[0091] In step S227, the depth constraint determination means 12 outputs the voxel data V to the outside as the three-dimensional shape of the subject.

[0092] As described above, the depth constraint determination means 12 determines the three-dimensional shape of the subject (set ν) based on the visual volume intersection voting result V vote The 3D shape (set ν1) represented by and the depth-constrained voting result V vote2 The set product ν=ν1∩ν2 is calculated with the three-dimensional shape (set ν2) represented by

[0093] As shown in FIG. 4(a), a set ν1 is a set of M silhouette images S m The view frustum A formed by m In the example of FIG. 4(a), since M=4, cameras C1 to C4 and view frustums A1 to A4 corresponding to four viewpoints are illustrated.

[0094] Therefore, when d=0, the set ν1 represents the three-dimensional shape obtained by the volume intersection method. In addition, the larger the value of d, the more robust it becomes against defects caused by noise, etc., but the larger the area J becomes, and the larger the error becomes. Taking such properties into consideration, the volume intersection threshold (Md) can be set. For example, when M=26, the volume intersection threshold (Md) is set (d=1) so that the area where 25 or more of the 26 view frustums overlap is voted for.

[0095] As shown in FIG. 4(b), the set ν2 is a set of N depth images D n The region E that satisfies the constraints based on n In the example of FIG. 4(b), since N=2, cameras C5 and C6 and areas E1 and E2 corresponding to two viewpoints are illustrated.

[0096] Therefore, when e = 0, the set ν2 is the set of all depth images D n It represents a three-dimensional shape to which a constraint based on N is applied. In addition, as the value of e increases, the robustness against defects caused by errors in depth measurement increases, but the area F becomes larger and the error increases. Taking such characteristics into consideration, the depth constraint threshold (Ne) can be set. For example, when N=9, the depth constraint threshold (Ne) is set so that the area where five or more of the nine depth maps overlap is voted for (e=4).

[0097] In addition, in step S220 of FIG. 3B, the depth constraint determination means 12 performs threshold determination using a visual volume intersection threshold (Md) and a depth constraint threshold (Ne) so that areas that are not included in the logical product of area J in FIG. 4(a) and area K in FIG. 4(b) are also accepted as measurement results of the three-dimensional shape. <Silhouette extraction method> Returning to FIG. 1, the description of the three-dimensional shape measuring unit 2 will continue. The silhouette extraction means 20 generates a silhouette image by performing a silhouette extraction process on viewpoint images obtained by photographing a subject from each viewpoint. In the example of Fig. 1, since M=4, the silhouette extraction means 20 is composed of a first silhouette extraction means 21 to a fourth silhouette extraction means 24.

[0098] Moreover, a multi-viewpoint image is input to the silhouette extraction means 20 by a user of the three-dimensional shape measuring device 1. Specifically, a viewpoint 1 image to a viewpoint 4 image are input to the first silhouette extraction means 21 to the fourth silhouette extraction means 24, respectively. Hereinafter, the viewpoint 1 image to the viewpoint 4 image will be referred to as I1 to I4. It should be noted that a multi-viewpoint image is an image obtained by photographing a subject from each of the viewpoints, and a silhouette image is an image that represents the outline of a subject as viewed from each of the viewpoints.

[0099] The first silhouette extraction means 21 performs a silhouette extraction process on the input viewpoint 1 image I1 to identify the subject or background of the viewpoint 1 image I1, and generates a first silhouette image in which the subject and background are distinguished for each pixel.

[0100] For example, the first silhouette extraction means 21 performs chromakey processing as the silhouette extraction processing. This chromakey processing generates a first silhouette image by identifying a specific color (for example, green or blue) as a background region. The first silhouette extraction means 21 may use a background subtraction method as the silhouette extraction process. This background subtraction method evaluates the difference between a pixel of a pre-generated background image and the first viewpoint image, and identifies pixels with a large difference as a subject area to generate the first silhouette image. Furthermore, the first silhouette extraction means 21 may generate the first silhouette image by using a subject region recognition process by a neural network as the silhouette extraction process.

[0101] The second silhouette extraction means 22 performs a silhouette extraction process on the input viewpoint 2 image I2 to generate a second silhouette image. The third silhouette extraction means 23 performs a silhouette extraction process on the input viewpoint 3 image I3 to generate a third silhouette image. The fourth silhouette extraction means 24 performs silhouette extraction processing on the input viewpoint 4 image I4 to generate a fourth silhouette image.

[0102] The second silhouette extraction means 22 to the fourth silhouette extraction means 24 are similar to the first silhouette extraction means 21 except for the viewpoint images to be processed, and therefore a detailed description thereof will be omitted.

[0103] The silhouette extraction means 20 outputs the generated M silhouette images (first silhouette image I1 to fourth silhouette image I4) to the visual volume intersection determination means 11.

[0104] <Stereo Means> The stereo means 30 generates a depth image by matching local regions of viewpoint images. The stereo means 30 also calculates the reliability based on the variance of pixel values ​​in the viewpoint images. In the example of FIG. 1, since N=2, the stereo means 30 is composed of a first stereo means 31 and a second stereo means 32.

[0105] Moreover, the stereo means 30 receives multi-viewpoint images and parameters from the user of the three-dimensional shape measuring device 1. Specifically, the first stereo means 31 receives a viewpoint 1 image, a viewpoint 2 image, parameters 1 and 2. Furthermore, the second stereo means 32 receives a viewpoint 3 image, a viewpoint 4 image, parameters 3 and 4.

[0106] The first stereo means 31 evaluates the similarity of patterns of each local region between the viewpoint 1 image and the viewpoint 2 image to associate the local regions with each other. Then, the first stereo means 31 generates a viewpoint 1 depth image and a viewpoint 1 reliability image based on the result of the association.

[0107] Note that the viewpoint 1 depth image is a depth image corresponding to the viewpoint 1 image, that is, an image that represents the depth value of each pixel in the viewpoint 1 image. Moreover, the viewpoint 1 reliability image is a reliability image corresponding to the viewpoint 1 depth image, that is, an image representing the reliability of each pixel of the viewpoint 1 depth image.

[0108] First, the generation of the viewpoint 1 depth image will be described. For example, the first stereo means 31 uses block matching to associate local regions with each other. FIG. 5 illustrates an overview of block matching. Three-dimensional coordinates [X, Y, Z] T From viewpoint n image I n Image coordinates [ξ n ,η n ] T The projection onto the function G' is given by the following equation (9): n is defined as:

[0109]

number

[0110] The first stereo means 31 detects the image coordinates [ξ n ,η n ] T The depth value D1(ξ, η) at the image coordinate [ξ n ,η n ] T The set of points S1(ξ,η) in the three-dimensional space projected onto is defined by the following equation (10) using the function G'1.

[0111]

number

[0112] As shown in Fig. 5, the position vector of a point on set S1(ξ,η) and having a depth value of d from viewpoint 1 is defined as P1(ξ,η;d)∈S1(ξ,η). In other words, the point from viewpoint 1 that passes through the image coordinates (ξ,η) of viewpoint 1 image I1 and touches subject 9 is point P. Note that viewpoints 1 and 2 are illustrated as V1 and V2, respectively, in Fig. 5.

[0113] Next, as shown in the following equation (11), the point P1(ξ,η;d) is projected onto the viewpoint 2 image I2 using the function G'2, and its image coordinates are set as (u2(ξ,η;d),v2(ξ,η;d)).

[0114]

number

[0115] Furthermore, the image coordinates of the viewpoint 1 image I1 (ξ n ,η n ) and the image coordinates (u2(ξ,η;d),v2(ξ,η;d)) of the viewpoint 2 image I2 are calculated as an error U 1,2 (ξ,η;d). For example, the error U 1,2 (ξ, η; d) can be evaluated as the sum of absolute error, as shown in the following equation (12).

[0116]

number

[0117] Here, the region B represents a neighborhood region around the origin. For example, the region B is a region that is a predetermined non-negative constant b x ,b y For this, we can make the phrasal domain defined by the following equation (13).

[0118]

number

[0119] Next, the similarity is maximized (i.e., the error U 1,2When the depth value d is changed so that the depth d becomes the minimum, the depth value D1(ξ, η) at the image coordinates (ξ, η) of the viewpoint 1 depth image D1 is expressed by the following equation (14).

[0120]

number

[0121] Here, arg min is a function that returns the parameter (d in equation (14)) that gives the minimum value. In equation (14), the sum of absolute error is used as the error, but this is not limiting. For example, the sum of squared error may be used as the error.

[0122] For example, in matching local regions in block matching, the cross-correlation of luminance or color vectors within a block may be calculated, and the larger the cross-correlation, the better the matching may be evaluated.

[0123] Next, the generation of the viewpoint 1 reliability image R1 will be described. The pixel value R1(ξ, η) of the viewpoint 1 reliability image R1 is calculated based on the evaluation value when the local regions are associated with each other. For example, the pixel value R1(ξ, η) of the viewpoint 1 reliability image R1 is calculated based on the evaluation value when the local regions are associated with each other, based on the minimum error value U 1,2 It can be calculated based on (ξ, η; D1).

[0124]

number

[0125] The function f is a monotonically decreasing function. For example, the function f may be the function shown in the following formula (16) or formula (17).

[0126]

number

[0127]

number

[0128] In addition, when calculating the pixel value R1(ξ,η) of the viewpoint 1 reliability image R1, the variance Φ(ξ,η) in the vicinity of the image coordinate (ξ,η) of the viewpoint 1 image I1 may be taken into consideration. For example, as shown in the following formula (18), the pixel value R1(ξ,η) of the viewpoint 1 reliability image R1 can be calculated using a function g.

[0129]

number

[0130] It should be noted that the variance Φ in equation (18) represents the case where the pixel values ​​of the viewpoint 1 image I1 are scalar values ​​(that is, the viewpoint 1 image I1 is a grayscale image). If the viewpoint 1 image I1 is a color image, the variance may be calculated for each color component, and the sum of the variances of the color components may be used as the variance Φ. If the viewpoint 1 image I1 is a color image, the variance for a specific color component may be used as the variance Φ. If the viewpoint 1 image I1 is a color image, the color components may be weighted with a predetermined weighting to obtain a monochromatic image, and the variance of the monochromatic image may be used as the variance Φ.

[0131] When images are flat, that is, when the variance Φ is close to 0, it is difficult to accurately match images. Therefore, by using the following equation (19) as the function g in equation (18), the pixel value R1(ξ, η) of the viewpoint 1 reliability image R1 can be made small and its reliability can be lowered. For example, in the case of a subject without texture, the variance Φ in equation (18) approaches 0 infinitesimally, so the pixel value R1(ξ, η) of the viewpoint 1 reliability image R1 becomes small.

[0132]

number

[0133] The second stereo means 32 evaluates the similarity of the patterns of each local region between the viewpoint 3 image I3 and the viewpoint 4 image I4, thereby associating the local regions with each other. Then, the second stereo means 32 generates a viewpoint 3 depth image D3 and a viewpoint 3 reliability image R3 based on the result of the association. The second stereo means 32 is similar to the first stereo means 31 except for the viewpoint image to be processed, and therefore a detailed description thereof will be omitted.

[0134] [Actions and Effects] As described above, the three-dimensional shape measuring device 1 according to the embodiment globally combines the voting results of the volume intersection method and the depth search using the volume intersection threshold and the depth constraint threshold. Furthermore, the three-dimensional shape measuring device 1 locally adjusts the data combination so as not to use the voting result when the reliability of the depth search is low. Therefore, the three-dimensional shape measuring device 1 can measure the three-dimensional shape of the subject with high accuracy.

[0135] Furthermore, the three-dimensional shape measuring device 1 can measure the three-dimensional shape of a subject from multi-viewpoint images, thereby improving convenience. Furthermore, the three-dimensional shape measuring device 1 calculates the reliability based on the variance of pixel values ​​in the viewpoint images. As a result, when the multi-viewpoint images are locally flat and featureless, the three-dimensional shape measuring device 1 lowers the reliability of the corresponding part in the depth image, so that the voting result of that part is not used, and thus it is possible to suppress instability of the three-dimensional shape of the subject.

[0136] (Modification) Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and includes design modifications and the like within the scope of the present invention.

[0137] In the above embodiment, the three-dimensional shape measuring device calculates the reliability, but this is not limiting. In other words, the three-dimensional shape measuring device does not need to calculate the reliability. In this case, the stereo means does not need to generate a reliability image, and the depth constraint determination means does not need to perform a threshold determination of the reliability. More specifically, in step S207 of FIG. 3A, the determination condition is set to ζ n ≧D n (ζ´ n ,η´ n ) can be used.

[0138] In the above embodiment, the three-dimensional shape measuring device is described as having a silhouette extraction means and a stereo means, but it is not necessary to have both means. In other words, the three-dimensional shape measuring unit 2 in FIG. 1 may be an independent device.

[0139] In the above embodiment, the three-dimensional shape measuring device is described as an independent piece of hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program for causing hardware resources such as a CPU, memory, and hard disk of a computer to function as the above-mentioned three-dimensional shape measuring device. This program may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed. [Explanation of symbols]

[0140] 1. Three-dimensional shape measurement device 2. Three-dimensional shape measurement section 11 Visual volume intersection determination method 12 Depth constraint determination means 20 Silhouette extraction method 21 First silhouette extraction means 22 Second silhouette extraction means 23 Third silhouette extraction means 24 Fourth silhouette extraction means 30 Stereo means (depth image generating means) 31 first stereo means 32 second stereo means

Claims

1. 1. A three-dimensional shape measuring device that measures a three-dimensional shape of a subject based on silhouette images of each viewpoint that represent an outline of the subject, a first parameter that associates three-dimensional coordinates of each point in space with image coordinates of the silhouette images, one or more depth images that represent a depth value of the subject, and a second parameter that associates the three-dimensional coordinates of each point in space with image coordinates of the depth images, a volume intersection determination means for determining whether each spatial point projected onto each silhouette image by the first parameter is located inside the subject or not, and for determining a volume intersection voting result by a voting process for each spatial point located inside the subject; a depth constraint determination means for determining whether each spatial point projected onto the depth image using the second parameter is located on the back side of the subject, determining a depth constrained voting result by a voting process for each spatial point located on the back side of the subject, and outputting each spatial point for which the visual hull intersection voting result is equal to or greater than a preset visual hull intersection threshold and the depth constrained voting result is equal to or greater than a preset depth constraint threshold as the three-dimensional shape of the subject; A three-dimensional shape measuring device comprising:

2. The depth constraint determination means A reliability value associated with each pixel of each depth image is input; determining whether each spatial point projected onto the depth image using the second parameter is located on the back side of the subject and whether the reliability of each spatial point projected onto the depth image is equal to or greater than a preset reliability threshold; 2. The three-dimensional shape measuring apparatus according to claim 1, wherein the depth-constrained voting result is obtained by a voting process for each spatial point that is located on the back side of the subject and whose reliability is equal to or greater than the reliability threshold.

3. a silhouette extraction means for generating the silhouette image by performing a silhouette extraction process on viewpoint images obtained by photographing the subject from each viewpoint; a depth image generating means for generating the depth image by associating local regions of the viewpoint images; The volume intersection determination means receives the silhouette image from the silhouette extraction means, 3. The three-dimensional shape measuring apparatus according to claim 2, wherein the depth constraint determining means receives the depth image from the depth image generating means.

4. the depth image generating means calculates the reliability based on a variance of pixel values ​​in the viewpoint images; 4. The three-dimensional shape measuring apparatus according to claim 3, wherein the depth constraint determining means receives the reliability from the depth image generating means.

5. A program for causing a computer to function as the three-dimensional shape measuring device according to any one of claims 1 to 4.

Images