Three-dimensional shape correction device and program therefor

Abstract

To provide a three-dimensional shape correction device capable of correcting a shape error of three-dimensional data of a subject with accuracy.SOLUTION: A three-dimensional shape correction device 3 comprises: a depth map creation part 21 creating depth maps; a subject data creation part 22 for coupling a subject image and the depth maps in a channel direction; a subject data division reduction part 23 for gradually dividing the subject data and reducing the same into regions of an equal size for regularizing the regions; a correction network applying part 24 for applying a depth map correction network to each region; a depth map coupling expansion part 31 for coupling and expanding the depth maps; a channel coupling part 32 coupling the depth maps in the channel direction; a coupling network applying part 33 for applying a depth map coupling network to the depth maps; and a three-dimensional data restoration part 40 for restoring the three-dimensional data from the depth maps.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a three-dimensional shape correcting device and a program therefor that correct shape errors in three-dimensional data of a subject. [Background technology]

[0002] In recent years, three-dimensional data of real-world subjects is increasingly being used in games and entertainment. Three-dimensional data can be generated by sensing using a sensor or by three-dimensional reconstruction of the subject. As shown in FIG. 11, three-dimensional reconstruction is a method of restoring the three-dimensional shape of a subject 90 from subject images G obtained by photographing the subject 90 using multiple cameras C, such as in a volumetric studio. Note that subject 90A is the subject 90 restored by three-dimensional reconstruction.

[0003] Various methods have been proposed for 3D reconstruction, such as volume intersection and stereo matching. In any method, shape errors occur in the reconstructed 3D data due to factors such as a calibration error of camera C, or a brightness or color shift error of camera C. Shape errors refer to differences in the shape of an object reconstructed from 3D data compared to an actual object.

[0004] Here, we consider stereo matching, a commonly used conventional method. Stereo matching is a method for searching for pixel positions that match between multiple cameras C. For this reason, in stereo matching, matching may occur at points different from the points on the surface of the subject 90 due to errors caused by the calibration or color adjustment of the camera C, resulting in shape errors in the three-dimensional data of the subject 90A. In stereo matching, the solution is the point where the lines of sight extended from multiple cameras C toward the subject 90 intersect in three-dimensional space, so the true point and the mismatched point exist on the same line of sight. In other words, when a mismatch occurs in a certain camera C, the true point is reached by moving forward or back along that line of sight.

[0005] FIG. 12 shows a schematic diagram of the effect of noise generation on the three-dimensional shape of an object 90A, taking the case where the three-dimensional data is in a point cloud format as an example. In FIG. 12, three-dimensional points 91 where no shape error occurs are shown as black circles. When no shape error occurs, the surface of the object 90A is smooth, as shown in FIG. 12(a). In contrast, when a shape error occurs, as shown in FIG. 12(b), three-dimensional points 91F protruding outward from the surface of the object 90A and three-dimensional points 91B receding inward exist, and the surface of the object 90A has irregularities. In this case, it is difficult to determine in advance how much the three-dimensional points 91F and 91B will deviate. Therefore, a conventional technique has been proposed in which three-dimensional data in a point cloud format is input to a neural network to correct shape errors in the three-dimensional data (Non-Patent Document 1).

[0006] In this conventional technology, 3D data including shape errors is input as input data, and 3D data without shape errors is input as training data to a neural network, and 3D data with corrected shape errors is output. In addition, in the conventional method described above, the neural network is trained in advance by minimizing the error between the output data and the training data. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] M. Rakotosaona, VL Barbera, P. Guerrero, NJ Mitra, M. Ovsjanikov: “POINTCLEANNET: Learning to denoise and remove outliers from dense point clouds”, Computer Graphics Forum, 39, 1, pp.185-203 (Jun. 2019) Summary of the Invention [Problem to be solved by the invention]

[0008] In the conventional technology described above, as shown in Fig. 13(a), it is possible to correct shape errors caused by noise 92 occurring in the normal direction of the surface of the object 90. However, in this conventional technology, as shown in Fig. 13(b), it is difficult to correct shape errors caused by noise 93 occurring in the tangential direction of the surface of the object 90.

[0009] With reference to Fig. 14, a shape error caused by noise will be explained in more detail. As shown in Fig. 14, light emitted from the object 90 reaches the imaging plate of the camera C through a search line L, so that a true point P on the surface of the object 90 exists on the search line L. Here, if no noise occurs, the depth value of the true point P is stored in the depth map. On the other hand, if noise occurs, the depth value of another point P' on the same search line L is stored in the depth map. The distance z between the points P and P' in the optical axis direction becomes an error, causing a shape error.

[0010] In view of the above problems, an object of the present invention is to provide a three-dimensional shape correcting device and a program therefor that can accurately correct shape errors in three-dimensional data of a subject. [Means for solving the problem]

[0011] In order to solve the above problems, the three-dimensional shape correction device of the present invention is a three-dimensional shape correction device that corrects shape errors in three-dimensional data obtained by 3D reconstructing a subject, and is configured to include a depth map generation unit, a subject data generation unit, a subject data division and reduction unit, a correction network application unit, a depth map combination and expansion unit, a channel combination unit, a combination network application unit, and a three-dimensional data restoration unit.

[0012] According to this configuration, the depth map generator receives three-dimensional data and generates a first depth map representing the depth value of each pixel in each object image from the input three-dimensional data. The object data generating unit generates object data by combining each object image with the first depth map in the channel direction. The object data division and reduction unit divides the object data into different sizes in stages, reduces the object data before division and the object data divided into different sizes into regions of the same size, and generates a second depth map by normalizing the first depth map of each region with the depth range of the object contained in each region. That is, each region is finally divided and reduced by combining each object image and the second depth map in the channel direction. Note that each region may be reduced by changing the aspect ratio of the original object image in order to make the processing in the depth map correction network in the subsequent stage more efficient.

[0013] The correction network application unit inputs each region including the normalized second depth map to a pre-trained depth map correction network, and outputs a second depth map with corrected shape errors for each region. The depth map combining and enlarging unit combines and enlarges the second depth map for each region so that it is the same size as the first depth map, thereby generating a third depth map that is the same size as the first depth map and has shape errors corrected. The channel combiner combines the third depth maps in the channel direction. The combination network application unit inputs the third depth map combined in the channel direction to a pre-trained depth map combination network, and outputs a fourth depth map in which shape errors have been corrected for each subject image. The three-dimensional data restoration unit restores three-dimensional data from the fourth depth map for each object image.

[0014] In this way, the 3D shape correction device standardizes the size of all regions and normalizes the depth map with the depth range of the subject contained in the region. Then, the 3D shape correction device inputs the depth map to the depth map correction network. In general, even if noise occurs in the tangential direction of the subject surface, if the directions of the search lines of the camera and the virtual camera are close, the shape error that causes it can be corrected with high accuracy. Therefore, when a depth map is generated with a virtual camera having the same position and orientation as the camera, the depth value can be corrected by changing the magnitude of the one-dimensional real value stored in each pixel of the depth map. In the case of an algorithm for matching between multiple cameras such as stereo matching, the search line of one camera is projected onto the screen of the other camera to search for corresponding points, so the true value can be reached simply by changing the value of the depth map of the camera that has the search line.

[0015] The present invention can also be realized by a program for causing a computer to function as the above-mentioned three-dimensional shape correction device. Effect of the Invention

[0016] According to the present invention, shape errors in three-dimensional data of a subject can be corrected with high accuracy. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a configuration of a 3D reconstruction system according to an embodiment. [Diagram 2] FIG. 4 is an explanatory diagram for explaining generation of a depth map in the embodiment. [Diagram 3] FIG. 4 is an explanatory diagram for explaining correction of a shape error in the embodiment. [Figure 4] FIG. 2 is an explanatory diagram for explaining an overview of processing of the 3D reconstruction system according to the embodiment. [Diagram 5] In an embodiment, (a) is an explanatory diagram explaining the depth range, (b) is an explanatory diagram explaining normalization in depth range A, and (c) is an explanatory diagram explaining normalization in depth range B. [Figure 6] FIG. 4 is an explanatory diagram for explaining division and reduction of object data in the embodiment. [Figure 7] FIG. 11 is an explanatory diagram illustrating combining and expanding depth maps in an embodiment. [Figure 8] FIG. 11 is an explanatory diagram illustrating application of a depth map combining network in an embodiment. [Figure 9] FIG. 2 is an explanatory diagram for explaining restoration of three-dimensional data in the embodiment. [Figure 10] 4 is a flowchart showing the operation of the three-dimensional reconstruction system according to the embodiment. [Figure 11] FIG. 1 is an explanatory diagram for explaining an outline of conventional three-dimensional reconstruction processing. [Figure 12] 1A and 1B are diagrams illustrating the effect that the generation of noise has on the three-dimensional shape of a subject in the conventional technology. [Figure 13] FIG. 1A is an explanatory diagram illustrating noise occurring in the normal direction of a subject's surface, and FIG. 1B is an explanatory diagram illustrating noise occurring in the tangential direction of the subject's surface, in the prior art. [Figure 14] FIG. 1 is an explanatory diagram illustrating a shape error caused by noise in the prior art. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and descriptions thereof may be omitted.

[0019] [Overview of 3D reconstruction system] An overview of a 3D reconstruction system 1 according to an embodiment will be described with reference to FIG. The 3D reconstruction system 1 is a system that performs 3D reconstruction to restore the 3D shape of a subject 90 (FIG. 4) from a subject image G captured by using multiple cameras C. As shown in FIG. 1, the 3D reconstruction system 1 includes the cameras C, a 3D data reconstruction device 2, and a 3D shape correction device 3.

[0020] The camera C is a general photographing camera that photographs the subject 90 from a predetermined viewpoint position. The camera C outputs a subject image G, which is an image obtained by photographing the subject 90, to the three-dimensional shape correction device 3. Note that, in FIG. 1, only one camera C is illustrated for ease of viewing the drawing, but in reality, K cameras C are arranged at predetermined viewpoint positions (where K is an integer that satisfies 2).

[0021] The three-dimensional data reconstruction device 2 performs three-dimensional reconstruction to restore the three-dimensional shape of the subject 90 from subject images G captured by a plurality of cameras C. In this embodiment, the three-dimensional data reconstruction device 2 generates three-dimensional data in which the subject 90 is three-dimensionally reconstructed from the subject images G stored in the three-dimensional shape correction device 3. For example, the three-dimensional data reconstruction device 2 performs three-dimensional reconstruction using a volumetric studio to generate three-dimensional data.

[0022] As described above, shape errors are included in the three-dimensional data of the three-dimensionally reconstructed object 90. Therefore, the three-dimensional data reconstruction device 2 outputs the three-dimensional data of the object 90 to the three-dimensional shape correction device 3 in order to correct the shape errors.

[0023] The three-dimensional shape correcting device 3 corrects shape errors in three-dimensional data obtained by three-dimensionally reconstructing the subject 90. In this embodiment, the three-dimensional shape correcting device 3 corrects shape errors in the three-dimensional data input from the three-dimensional data reconstruction device 2. Then, the three-dimensional shape correcting device 3 outputs the three-dimensional data in which the shape errors have been corrected to an external device (for example, a general CG video production device).

[0024] In this embodiment, the three-dimensional data is represented in a point cloud format. The three-dimensional shape correction device 3 uses a depth map format that can be handled by a general image conversion neural network as input and output to the image conversion neural network. As shown in FIG. 2, the depth map is a two-dimensional image in which the pixel value of each pixel is the distance (depth value d) from the camera C to the subject 90 in the optical axis cx direction. Note that the image conversion neural network is a neural network that converts a certain image (e.g., a subject image) into another image (e.g., a depth map, a contour line, or a segmentation).

[0025] That is, the three-dimensional shape correction device 3 enables correction of the depth map using a general image conversion neural network by converting the point cloud into a depth map as viewed from a virtual camera having the same position and orientation as each camera C. As shown in Fig. 3, even if the virtual cameras VC1 and VC2 search on the search lines L1 and L2, the true point P is not reached, but when the correction points P1 and P2 are obtained on the surface of the subject 90 by optimization, the error d between the true point P and the correction points P1 and P2 is proportional to the angle between the search line L and the search lines L1 and L2.

[0026] Here, if the depth map is optimized on a search line close to the search line L during 3D reconstruction, an optimal solution close to the true value can be obtained. The angles between the search line L of camera C and the search lines L1 and L2 of virtual cameras VC1 and VC2 are θ1 and θ2, respectively, and the surface of the subject 90 can be locally considered to be flat. In this case, the error d after correction with respect to the true value is proportional to tan θ (d=z*tan θ). At this time, if the positions and orientations of the virtual cameras VC1 and VC2 and camera C are close to each other, the formed θ becomes small, and when the positions and orientations of the virtual cameras VC1 and VC2 and camera C are the same, the formed θ becomes 0, so the shape error can be corrected.

[0027] FIG. 4 illustrates an overview of the processing of the three-dimensional reconstruction system 1 (K=3). The procedure up to generating three-dimensional data is the same as that in FIG. 11, and therefore the description will be omitted. As shown in FIG. 4, the three-dimensional shape correction device 3 captures an object 90A represented by three-dimensional data with a virtual camera VC, and generates a depth map M. Next, the three-dimensional shape correction device 3 combines the depth map M with a shape error and the object image G in the channel direction, and generates object data H (FIG. 6). At this time, the three-dimensional shape correction device 3 normalizes the depth map M after dividing and reducing it in multiple stages so that it can be efficiently calculated by the depth map correction network N1.

[0028] Next, the three-dimensional shape correction device 3 inputs the object data H divided and reduced in multiple stages to a depth map correction network N1 to generate a depth map D (FIG. 7) without shape errors. This process means correcting the shape errors of the depth map M by using the features of the object image G. Next, the three-dimensional shape correction device 3 inputs the depth maps D combined in the channel direction to a depth map combination network N2 to generate one depth map F for each camera C. Furthermore, the three-dimensional shape correction device 3 restores the object 90B represented by the point cloud from the depth map F without shape errors.

[0029] [Configuration of 3D shape correction device] Returning to FIG. 1, the configuration of the three-dimensional shape correcting device 3 according to the embodiment will be specifically described. As shown in FIG. 1, the three-dimensional shape correction device 3 includes a storage unit 10, a data correction unit 20, a depth map combination unit 30, and a three-dimensional data restoration unit 40.

[0030] The storage unit 10 is a storage device such as a memory or an HDD (Hard Disk Drive) that stores information necessary for various calculations of the three-dimensional shape correction device 3. The storage unit 10 also includes a subject image storage unit 11, a correction network storage unit 12, and a connection network storage unit 13.

[0031] The object image storage unit 11 stores the object image G captured by the camera C. The object image G stored in the object image storage unit 11 is read out by the three-dimensional data reconstruction device 2 and the object data generation unit 22. Thereafter, the i-th camera C is referred to as the camera C. i (where 1≦i≦K). Also, camera C i The subject image G taken by i It is written as follows.

[0032] The correction network storage unit 12 stores the depth map correction network N1. The depth map correction network N1 is referenced by a correction network application unit 24, which will be described later.

[0033] The combination network storage unit 13 stores the depth map combination network N2. The depth map combination network N2 is referenced by a combination network application unit 33, which will be described later.

[0034] In FIG. 1, for ease of understanding, the subject image storage unit 11, the correction network storage unit 12, and the connection network storage unit 13 are illustrated as separate components, but in the present invention, these may be integrated.

[0035] The data correction unit 20 performs processing related to correction of shape errors. As shown in Fig. 1, the data correction unit 20 includes a depth map generation unit 21, an object data generation unit 22, an object data division and reduction unit 23, and a correction network application unit 24.

[0036] The depth map generating unit 21 receives three-dimensional data and generates each object image G from the input three-dimensional data. i A depth map (first depth map) M representing the depth value of each pixel of i That is, the depth map generating unit 21 converts the three-dimensional data input from the three-dimensional data reconstruction device 2 into the object image G i Camera C took the photo i The depth map M seen from a virtual camera VC with the same position and orientation asi Then, the depth map generating unit 21 converts the generated depth map M i to the subject data generating unit 22.

[0037] The depth map generating unit 21 generates the depth map M by a general method without depending on the input from the three-dimensional data reconstruction device 2. i For example, the depth map generating unit 21 may obtain the parallax between the subject images G captured by two cameras C separated in the horizontal direction, determine a depth value according to the magnitude of the parallax, and generate the depth map M.

[0038] The subject data generating unit 22 generates each subject image G i and depth map M i The subject data H is a combination of i For example, the object image G i represents three colors of RGB values, so there are three channels. Depth map M i In this case, the object data generating unit 22 generates a three-channel object image G i and a 1-channel depth map M i and combine them to produce 4-channel subject data H i In other words, the object data H i Each pixel in the image is four-dimensional data. The object data generating unit 22 generates the object data H i to the object data division / reduction unit 23.

[0039] The object data division and reduction unit 23 divides the object data H i is divided into different sizes in stages, and the subject data before division H i , and the subject data divided into different sizes H i Each of these is an area of ​​equal size I i and each area I i In the depth range of the subject 90 included in each area I i Depth map M i This normalizes the

[0040] Point cloud to camera-view depth map M i When converting the image containing the object region into a depth map M i This directly affects the accuracy of the depth map M i It is preferable to normalize the depth values ​​in [0,1].

[0041] As shown in FIG. 5(a), the depth range from the maximum depth value A to the minimum depth value A is wider than the depth range from the maximum depth value B to the minimum depth value B. As shown in FIG. 5(b), when normalization is performed in the depth range from the maximum depth value A to the minimum depth value A, the ratio of the depth range in which the subject 90 does not exist becomes high relative to the depth range [0,1]. In this case, the depth resolution decreases, and it becomes difficult to distinguish between noise and correct data, and there is a possibility that shape errors are not sufficiently corrected. On the other hand, as shown in FIG. 5(c), when normalization is performed in the depth range from the maximum depth value B to the minimum depth value B, the ratio of the depth range in which the subject 90 does not exist becomes low relative to the depth range [0,1], and the depth resolution becomes high.

[0042] In order to increase the ratio of the depth range in which the subject 90 exists, a depth map M i A method of dividing the area into smaller areas and normalizing each area separately will be considered. If the area of ​​the area is small, the ratio of the depth range in which the subject 90 exists will be high, and the depth resolution will be high.

[0043] However, when the area of ​​the regions is reduced, there is a high possibility that inconsistencies will occur on the boundaries between the regions. Even if overlapping regions are cut out, depth-corrected, and then averaged together, the overlapping parts may not necessarily match because the image conversion neural network performs nonlinear conversion. Conversely, when the area of ​​the regions is increased, the depth resolution decreases, but the image conversion neural network refers to the overall features, so that inconsistencies on the boundaries are mitigated.

[0044] In order to solve the above-mentioned conflicting problems, the three-dimensional shape correction device 3 uses the subject data H i The area I in which the entire subject 90 is captured is divided into the following areas: i Area I i0 Then, in region I i0 m-th stage of domain I i1 ,…,I im Each of the regions is divided in a different way. i Hereafter, the jth division stage of the region I i Area I ij (where 0≦j≦m). The number of divisions in the jth stage is N j There are 10 pieces.

[0045] Region I obtained by division ij Let I be the set of j ={I ij 1 ,…,I ij Nj} In the 0th stage, no division is performed. In the 1st stage, the area I i0 Region I divided into N1 i1 Let I be the set of i1 ={I i1 1 ,…,I i1 N1} Similarly, in the second stage, region I i0 Region I divided into N2 i2 Let I be the set of i2 ={I i2 1 ,…,I i2 N2}. If N2>N1, the area I i0 This stepwise division is repeated m times to obtain the region I i0 ,…,I im is obtained.

[0046] Here, Region I i0 ,…,I im Since the sizes are different, we will continue with region I. i0,…,I im When the depth map correction network N1 learns the feature vector, it is necessary to change the size of the convolution kernel or increase or decrease the number of convolution layers each time in order to effectively extract the feature vector, which increases the learning cost. Therefore, the object data division and reduction unit 23 divides all the regions I i0 ,…,I im All regions I i0 ,…,I im In this case, the object data division and reduction unit 23 divides the area I i0 ,…,I im The smallest region I im so that the area I is the same size as i0 ,…,I im-1 The object data division and reduction unit 23 can reduce the image by using a known method such as a bilinear method or a bicubic method. The division at each stage j is performed so that all the regions after division have the same area. Region I ij Area S ij and Area I im Area S im Then, in region I ij The reduction rate is S im / S ij It becomes.

[0047] In the three-dimensional shape correction device 3, there is no limit to the number of vertical and horizontal divisions and the aspect ratio of each region, and the region can be divided arbitrarily as long as the area of ​​the region after division is equal. In the following, for the sake of simplicity, it is assumed that the number of vertical and horizontal divisions is equal and region I ij In other words, the object data division and reduction unit 23 divides the object data H i is divided stepwise into rectangular regions of different sizes.

[0048] Area I i0 ,…,I im The number of vertical and horizontal divisions of m =2 m In the case where the number of divisions is N m =2 2m =4 mFIG. 6 shows an example of division and reduction in two-stage division (m=2). In FIG. 6, the subject image G ij and Area I ij In order to make it easier to understand the correspondence with area I ij The letters are placed on the subject image G. ij Although it is illustrated inside of area I, ij The letters are subject image G ij From now on, the reduced area I will be referred to as area I. ~ It is written as follows.

[0049] As shown in FIG. 6, the object data division and reduction unit 23 first divides the object data H i0 Area I, which will be described later i2 By reducing the size to the same size as area I ~ i0 Next, the object data division and reduction unit 23 generates the object data H i1 4 regions I i1 ={I i1 1 ,…,I i1 4 Then, the object data division and reduction unit 23 divides the area I i1 ={I i1 1 ,…,I i1 4} into area I i2 By reducing the size to the same size as (half the width and height), the area I ~ i1 ={I ~ i1 1 ,…,I ~ i1 4 Next, the object data division and reduction unit 23 generates the object data H i2 16 regions I i2 ={I i2 1 ,…,I i2 16}.

[0050] That is, Area Ii0 represents the entire region, so in this region I i0 Subject data H containing i0 There is no need to split it, just reduce it. Region I i0 ~I i2 Of these, Area I i2 Since is the minimum size, this area I i2 Subject data H containing i2 There is no need to reduce it, just split it. Region I i1 By only dividing or reducing, the area I i2 Since the resulting image will not be the same size as the original image, both division and reduction are performed.

[0051] Then, the object data division and reduction unit 23 divides the area I ~ i0, I ~ i1, I i2 In the depth range of the subject 90 included in each of the areas I ~ i0, I ~ i1, I2 Each depth map M ~ i0, M ~ i1, M i2 (See FIG. 5(c)). Then, the object data division and reduction unit 23 normalizes the normalized depth map M ~ i0, M ~ i1, M i2 Region I including ~ i0, I ~ i1, I i2 to the correction network application unit 24. In the example of FIG. ~ i0 ,…,I i2 16} is the input to the depth map correction network N1.

[0052] Returning to FIG. 1, the configuration of the three-dimensional shape correcting device 3 will be further described. The correction network application unit 24 applies the normalized depth map M ij Each area including I ij By inputting the above, the shape error is corrected to create a depth map (second depth map) D ij Area I ij That is, the correction network application unit 24 outputs the region I ij is input to the depth map correction network N1, and the shape error-corrected depth map D ij get.

[0053] Then, the correction network application unit 24 generates a depth map D ij The depth map combining and enlarging unit 31 outputs the above-mentioned

[0054] The depth map combiner 30 combines the divided and reduced depth map D ij As shown in FIG 1, the depth map combining unit 30 includes a depth map combining and expanding unit 31, a channel combining unit 32, and a combining network application unit 33.

[0055] The depth map combining and expanding unit 31 is ij Depth map D ij Depth map M i By combining and enlarging the depth map M i A depth map F with the same size and corrected for shape errors. i In other words, the depth map combining and expanding unit 31 generates the depth map D ij to the original depth map M i and combine them horizontally and vertically according to the division step.

[0056] FIG. 7 illustrates an example of merging and expanding in a two-stage division (m=2) (where 0≦j≦m). In the example of FIG. 7, if the output of the depth map correction network N1 is defined as J, the data set {Ji0 ,…,J i2 16} is the output from the depth map correction network N1. Note that in FIG. 7, the depth map D ij and output J ij In order to make it easier to understand the correspondence with output J ij Depth map D ij The figure shows the actual output J ij The letter D stands for Depth Map. ij It is not displayed in the

[0057] As shown in FIG. 7, the depth map combining and expanding unit 31 first receives the output J i0 to the original depth map M i By enlarging it to the same size as the depth map D i0 Next, the depth map combining and expanding unit 31 generates the output J i1 For output J i1 Each element of J i1 1 ,…,J i1 4 The depth map combining and enlarging unit 31 enlarges each of the enlarged elements J i1 1 ,…,J i1 4 By combining the horizontal and vertical directions, the original depth map M i A depth map D of the same size as i1 Next, the depth map combining and expanding unit 31 generates an output J i2 For output J i2 Each element of J i2 1 ,…,J i2 16 By combining the horizontal and vertical directions, the original depth map M i A depth map D of the same size as i2 Generate.

[0058] That is, the area J i0 is the area I that has been reduced without being divided i0Since this corresponds to (Figure 6), there is no need to combine them and only enlarge them. i1 is the region I that has been both divided and reduced. i1 (Figure 6), so both the merge and the expansion are performed. i2 is the area I where only division was performed without shrinking. i2 (Figure 6), so there is no need to enlarge it, just combine it.

[0059] Then, the depth map combining and expanding unit 31 generates the combined and expanded depth map D ij to the channel combining unit 32.

[0060] The channel combiner 32 combines the depth map D ij In the example of Fig. 7, the depth map D i0 ,D i1 ,D i2 The channel combining unit 32 combines the depth map D i0 ,D i1 ,D i2 Combine them in the channel direction to get the depth map D i Generate.

[0061] Then, the channel combiner 32 generates a combined depth map D i to the combination network application unit 33. In other words, in the example of FIG. 7, the depth map D i is the input to the depth map combination network N2.

[0062] The combination network application unit 33 generates a depth map (third depth map) D i By inputting the above, the shape error is corrected to create a depth map (fourth depth map) F i The subject image G i It is output every time.

[0063] m+1 depth maps D m The area I is the area I that is combined in the channel direction and then input to the depth map combination network N2. m The purpose of the depth map D is to correct the inconsistency at the boundary between the m In other words, the depth map combination network N2 combines the overall depth features with the depth maps D m The local depth features are extracted from the depth map D m It is expected that the data will be extracted and optimized from the above.

[0064] For example, in the case of two-stage division (m=2), as shown in FIG. 8, the depth map combination network N2 generates a three-channel depth map D i0 ,D i1 ,D i2 The combined depth map D i The input is the depth map F i This depth map F i is Camera C i It represents the depth value without shape error in the object image G i Generated every 10 seconds.

[0065] Here, if the depth map correction network N1 and the depth map combination network N2 have an image conversion function, ij Since the depth map correction network N1 and the depth map combination network N2 can realize the correction, there is no limitation. In addition, the depth map correction network N1 and the depth map combination network N2 may use image conversion neural networks of the same architecture, and may be trained in advance using a depth map without shape errors as training data. For example, the depth map correction network N1 and the depth map combination network N2 may be pix2pixHD (Reference 1).

[0066] Reference 1: T. Wang, M. Liu, J. Zhu, A. Tao, J. Kautz, B. Catanzaro: “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs”, Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp.8798-8807 (Jun. 2018)

[0067] Then, the combination network application unit 33 calculates the depth map F i is output to the three-dimensional data reconstruction unit 40.

[0068] The three-dimensional data restoration unit 40 extracts the subject image G i Depth map F i Theoretically, as shown in FIG. 9, the subject 90 exists in a range behind the depth values ​​d1 and d2 on the nearest side as seen from the cameras C1 and C2. Therefore, the three-dimensional data restoration unit 40 restores three-dimensional data from each subject image G i Depth map F i The 3D data is restored by calculating the intersection of the inner region in voxel space.

[0069] 9 illustrates frustum regions R1 and R2 that represent the shooting ranges of cameras C1 and C2. In addition, the area of ​​frustum region R1 that is a blind spot of subject 90 when viewed from camera C1, and the area of ​​frustum region R2 that is a blind spot of subject 90 when viewed from camera C2 are illustrated with dots.

[0070] A known method can be used to convert the voxel space into three-dimensional data (for example, a point cloud). By using the voxel space, the depth map F i It can be determined whether a point in the three-dimensional space obtained from exists inside or outside of the subject 90. In this way, the three-dimensional data restoration unit 40 can reconstruct the three-dimensional data of the subject 90 without causing inconsistencies at the boundaries of the regions.

[0071] Thereafter, three-dimensional data restoration section 40 outputs the restored three-dimensional data of subject 90 to the outside.

[0072] [Operation of 3D reconstruction system] The operation of the three-dimensional reconstruction system 1 will be described with reference to FIG. 10, in step S1, object images G captured by a plurality of cameras C are input to the three-dimensional shape correction device 3. The object images G are stored in the object image storage unit 11. In step S2, the three-dimensional data reconstruction device 2 generates three-dimensional data from all of the object images G stored in the object image storage unit 11.

[0073] The three-dimensional shape correction device 3 repeats the processes of steps S3 to S25 while the initial value of the counter i=1, the increment value of the counter i=1, and the counter i≦the number of cameras K are satisfied. i The processes in steps S3 to S25 are repeated each time.

[0074] In step S4, the depth map generating unit 21 i Image of the subject taken with G i is read out from the subject image storage unit 11. In step S5, the depth map generating unit 21 i Depth map M when subject 90 is photographed with the same position and orientation as i Generate. In step S6, the subject data generating unit 22 generates each subject image G i and depth map M i The subject data H is a combination of i Generate. In step S7, the object data division and reduction unit 23 divides the object data into a minimum size area I m Area S m Calculate.

[0075] The three-dimensional shape correcting device 3 repeats the processes of steps S8 to S13 while the initial value of counter j=0, the increment value of counter j=1, and counter j≦division stage m are satisfied. In other words, the three-dimensional shape correcting device 3 repeats the processes of steps S8 to S13 for each division stage j.

[0076] In step S9, if division is necessary, the object data division and reduction unit 23 divides the object data H i N j Divide into pieces, area I ij Generate. In step S10, the object data division and reduction unit 23 divides the area I ij Depth map M ij Normalize. In step S11, the object data division and reduction unit 23 divides the area I j Area S j If reduction is required, area I ij S m / S j Reduce it by 2x. In step S12, the object data division and reduction unit 23 divides the reduced area I ij Add to the first data list.

[0077] In step S14, the correction network application unit 24 calculates the area I ij is input to the depth map correction network N1.

[0078] The three-dimensional shape correcting device 3 repeats the processes of steps S15 to S20 while the initial value of counter j=0, the increment value of counter j=1, and counter j≦division stage m are satisfied. In other words, the three-dimensional shape correcting device 3 repeats the processes of steps S15 to S20 for each division stage j.

[0079] In step S16, the depth map combining and enlarging unit 31 selects the region I ij Take out. In step S17, if enlargement is necessary, the depth map combining and enlarging unit 31 ij Depth map D ij Sj / S m Enlarge by double. In step S18, if combination is necessary, the depth map combining and enlarging unit 31 combines the enlarged depth map D ij are combined horizontally and vertically. In step S19, the depth map combining and expanding unit 31 generates the expanded and combined depth map D ij to the second data list.

[0080] In step S21, the channel combiner 32 selects a depth map D i0 ,…,D im Take out. In step S22, the channel combining unit 32 combines the depth map D i0 ,…,D im are combined in the channel direction. In step S23, the combination network application unit 33 applies the depth map D i Enter. In step S24, the combination network application unit 33 applies the depth map F output from the depth map combination network N2 to the depth map combination network N3. i to the depth map list.

[0081] In step S26, the three-dimensional data reconstruction unit 40 extracts all of the depth maps F1, . . . , F K Recover 3D data from the image. In step S27, the three-dimensional data restoration unit 40 outputs the restored three-dimensional data to the outside.

[0082] The inference process in the three-dimensional shape correction device 3 has been described above, so the learning process will also be briefly described. In the learning process, supervised learning is applied to the depth map correction network N1 and the depth map combination network N2. At this time, the teaching data is a true-value depth map (a depth map without shape errors). This true-value depth map can be generated from a CG model. In this case, a shape error is added intentionally to the input depth map. The true-value depth map may also be obtained by measuring the depth value of an actual subject using a sensor (for example, a laser range finder).

[0083] [Actions and Effects] As described above, the three-dimensional shape correction device 3 standardizes the size of all the regions I and normalizes the depth map with the depth range of the subject 90A included in the regions. As a result, the three-dimensional shape correction device 3 appropriately reflects the local depth characteristics and the global depth characteristics in the depth map D. Then, the three-dimensional shape correction device 3 inputs the depth map D to the depth map correction network N1, so that shape errors caused by noise generated in the tangential direction of the surface of the subject 90A can also be accurately corrected.

[0084] For example, in the field of content production such as games and videos, when generating three-dimensional data of an actual subject 90, the three-dimensional shape correcting device 3 can improve the quality of the three-dimensional data.

[0085] Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and includes design modifications and the like within the scope of the present invention.

[0086] In the above embodiment, the three-dimensional data is described as being in a point cloud format, but the present invention is not limited to this. For example, the three-dimensional data may be a polygon, a mesh, or a signed distance field in addition to a point cloud.

[0087] In the above embodiment, the three-dimensional shape correction device is described as an independent hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program for causing hardware resources such as a CPU, memory, and hard disk of a computer to function as the above-mentioned three-dimensional shape correction device. This program may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed. [Explanation of symbols]

[0088] 1 3D reconstruction system 2. 3D data reconstruction device 3 3D shape correction device 10 Storage section 11 Subject image storage unit 12 Correction network memory section 13. Network Memory 20 Data correction section 21 Depth map generator 22 Object data generation unit 23. Object data division and reduction section 24 Correction network application section 30 Depth Map Joint 31 Depth map combination enlargement section 32 Channel Junction 33 Connection network application section 40 3D Data Reconstruction Section C Camera N1 Depth Map Correction Network N2 depth map combination network

Claims

1. A three-dimensional shape correction device that corrects shape errors in three-dimensional data obtained by three-dimensionally reconstructing an object, comprising: a depth map generator that receives the three-dimensional data and generates a first depth map representing a depth value of each pixel of each object image from the received three-dimensional data; a subject data generating unit that generates subject data by combining each of the subject images and the first depth map in a channel direction; a subject data division and reduction unit that divides the subject data stepwise into different sizes, reduces the subject data before division and the subject data divided into different sizes into regions of the same size, and generates a second depth map by normalizing a first depth map of each region with a depth range of the subject included in each region; a correction network application unit that inputs each region including a normalized second depth map to a pre-trained depth map correction network, and outputs the second depth map in which a shape error has been corrected for each region; a depth map combining and enlarging unit that combines and enlarges the second depth map for each region to have the same size as the first depth map, thereby generating a third depth map that has the same size as the first depth map and has been corrected for shape errors; a channel combiner that combines the third depth maps in the channel direction; a combination network application unit that outputs a fourth depth map in which a shape error is corrected for each of the object images by inputting a third depth map combined in the channel direction to a pre-trained depth map combination network; a three-dimensional data restoration unit that restores the three-dimensional data from a fourth depth map for each of the object images; A three-dimensional shape correction device comprising:

2. The three-dimensional shape correction device according to claim 1, characterized in that the depth map correction network and the depth map combination network are image conversion neural networks of the same architecture, and are pre-trained using a depth map without shape errors as training data.

3. 2. The three-dimensional shape correcting device according to claim 1, wherein the object data dividing and reducing unit divides the object data into rectangular regions of different sizes in a stepwise manner.

4. The three-dimensional shape correction device according to claim 1 , wherein the three-dimensional data is represented in a point cloud format.

5. The three-dimensional shape correction device according to claim 1, characterized in that the three-dimensional data restoration unit restores the three-dimensional data by calculating the intersection of the deep regions in the fourth depth map of each of the subject images in voxel space.

6. A program for causing a computer to function as the three-dimensional shape correction device according to any one of claims 1 to 5.

Images