A three-dimensional reconstruction method and system

Abstract

The present invention provides a three-dimensional reconstruction method and system, the method comprising: obtaining distortion coefficients and camera intrinsic parameters using a first picture set of an object to be reconstructed acquired by a camera; performing distortion correction on a first picture included in the first picture set using the distortion coefficient to obtain a second picture set; calculating a sparse point cloud, and calculating camera extrinsic parameters and a first depth map of a second picture included in the second picture set; removing outliers from the first depth map; inputting the first depth map with outliers removed into a depth completion network model for depth completion to obtain an optimized second depth map; constructing a final three-dimensional model of the object to be reconstructed based on the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters, and sparse point clouds. This solution obtains a more complete second depth map through outlier removal operations and a depth completion network model, and then uses the second depth map to construct a final three-dimensional model, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a three-dimensional reconstruction method and system. Background Art

[0002] 3D reconstruction technology is used to restore the structure of a 3D scene from a 2D image. 3D reconstruction technology has broad application prospects in game development, film production, and cultural relics protection.

[0003] The existing 3D reconstruction technology is to calculate the depth map based on the sparse point cloud and RGB image obtained by sparse reconstruction, and then use the depth map for dense reconstruction to obtain a dense point cloud. However, in the above 3D reconstruction process, the depth map often has quality problems such as large-area missing, which will affect the accuracy of dense reconstruction and the 3D reconstruction effect is poor. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a 3D reconstruction method and system to solve the problems of poor 3D reconstruction effect in the existing 3D reconstruction technology.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a three-dimensional reconstruction method, the method comprising:

[0007] Using a first picture set of the object to be reconstructed acquired by a camera, a distortion coefficient and a camera intrinsic parameter of the camera are obtained;

[0008] Performing distortion correction on the first picture included in the first picture set by using the distortion coefficient to obtain a second picture set;

[0009] Calculating a sparse point cloud, and calculating a camera extrinsic parameter and a first depth map of a second picture included in the second picture set;

[0010] Performing outlier removal on the first depth map;

[0011] Inputting the first depth map with outliers removed into a pre-trained depth completion network model to perform depth completion, thereby obtaining an optimized second depth map;

[0012] A final three-dimensional model of the object to be reconstructed is constructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0013] Preferably, calculating the sparse point cloud, and calculating the camera extrinsic parameters and the first depth map of the second picture included in the second picture set, comprises:

[0014] Dividing the second pictures included in the second picture set into a plurality of picture groups;

[0015] Using the similarity between the feature points of each of the second pictures in the picture group, a plurality of groups of successfully matched feature points are obtained;

[0016] Calculate a sparse point cloud and camera extrinsic parameters of the second picture based on the camera intrinsic parameters, the second picture set, the picture group, and the multiple groups of successfully matched feature points;

[0017] A first depth map corresponding to the second image is calculated according to the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0018] Preferably, constructing a final three-dimensional model of the object to be reconstructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud includes:

[0019] Calculate a dense point cloud according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud;

[0020] Meshing the dense point cloud to obtain an initial three-dimensional model of the object to be reconstructed;

[0021] Mapping the RGB pixels of the second image to a texture space by using a mapping relationship between points on the second image and points in space, to obtain a texture map of the initial three-dimensional model;

[0022] The texture map is mapped onto the initial three-dimensional model to obtain a final three-dimensional model.

[0023] Preferably, removing outliers from the first depth map includes:

[0024] identifying valid points in the first depth map;

[0025] Determine the valid points that meet the preset abnormal conditions as outliers;

[0026] The outlier points in the first depth map are removed.

[0027] Preferably, obtaining the distortion coefficient and the camera intrinsic parameter of the camera by using the first picture set of the object to be reconstructed acquired by the camera includes:

[0028] Reading picture metadata of a first picture included in a first picture set of the object to be reconstructed acquired by a camera;

[0029] The distortion coefficient and the camera intrinsic parameter of the camera are obtained from the picture metadata.

[0030] Preferably, the deep completion network model includes at least: an encoder composed of a deep residual network, a decoder composed of multiple upsampling modules, a local network composed of multiple continuous convolutional layers, and a refinement network.

[0031] A second aspect of an embodiment of the present invention discloses a three-dimensional reconstruction system, the system comprising:

[0032] An acquisition unit, configured to acquire distortion coefficients and camera intrinsic parameters of the camera by using a first picture set of the object to be reconstructed acquired by the camera;

[0033] a correction unit, configured to perform distortion correction on the first picture included in the first picture set by using the distortion coefficient to obtain a second picture set;

[0034] A computing unit, configured to compute a sparse point cloud, and to compute a camera extrinsic parameter and a first depth map of a second picture included in the second picture set;

[0035] A removal unit, configured to remove outliers from the first depth map;

[0036] A depth completion unit, configured to input the first depth map from which outliers are removed into a pre-trained depth completion network model for depth completion, so as to obtain an optimized second depth map;

[0037] A construction unit is used to construct a final three-dimensional model of the object to be reconstructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0038] Preferably, the calculation unit comprises:

[0039] a dividing subunit, configured to divide the second pictures included in the second picture set into a plurality of picture groups;

[0040] An acquisition subunit, configured to acquire multiple groups of successfully matched feature points by using the similarity between the feature points of each of the second images in the image group;

[0041] A first calculation subunit, configured to calculate a sparse point cloud and camera extrinsic parameters of the second picture based on the camera intrinsic parameters, the second picture set, the picture group, and the multiple groups of successfully matched feature points;

[0042] The second calculation subunit is used to calculate a first depth map corresponding to the second image according to the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0043] Preferably, the building block comprises:

[0044] A calculation subunit, configured to calculate a dense point cloud according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud;

[0045] A gridding subunit, used for gridding the dense point cloud to obtain an initial three-dimensional model of the object to be reconstructed;

[0046] A first mapping subunit is used to map RGB pixels of the second image to a texture space by using a mapping relationship between points on the second image and points in space, so as to obtain a texture map of the initial three-dimensional model;

[0047] The second mapping subunit is used to map the texture map onto the initial three-dimensional model to obtain a final three-dimensional model.

[0048] Preferably, the removal unit comprises:

[0049] an identification subunit, configured to identify valid points in the first depth map;

[0050] A determination subunit, used to determine that the valid points satisfying the preset abnormal condition are outliers;

[0051] A removal subunit is used to remove the outliers in the first depth map.

[0052] Based on the above-mentioned embodiment of the present invention, a three-dimensional reconstruction method and system are provided. The method is as follows: using the first picture set of the object to be reconstructed acquired by the camera, the distortion coefficient and the camera intrinsic parameter are obtained; the first picture contained in the first picture set is distorted by the distortion coefficient to obtain the second picture set; the sparse point cloud is calculated, and the camera extrinsic parameter and the first depth map of the second picture contained in the second picture set are calculated; the first depth map is removed from the outliers and input into the depth completion network model for depth completion to obtain an optimized second depth map; the final three-dimensional model of the object to be reconstructed is constructed according to the depth information of the second depth map, the camera intrinsic parameter, the camera extrinsic parameter and the sparse point cloud. In this scheme, after the first depth map is calculated, the outliers of the first depth map are first removed, and then the depth completion network model is used to perform depth completion on the first depth map with the outliers removed to obtain a more complete second depth map, and then the second depth map is used to construct the final three-dimensional model of the object to be reconstructed, so as to improve the accuracy of dense reconstruction and the effect of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0054] Figure 1 A flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention;

[0055] Figure 2 A structural example diagram of a depth completion network model provided by an embodiment of the present invention;

[0056] Figure 3 A flowchart of calculating a sparse point cloud and a first depth map provided by an embodiment of the present invention;

[0057] Figure 4 A flowchart of constructing a final three-dimensional model provided by an embodiment of the present invention;

[0058] Figure 5 A structural block diagram of a three-dimensional reconstruction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0061] 3D reconstruction technology is used to restore the structure of a 3D scene from a 2D image. 3D reconstruction technology has broad application prospects in game development, film production, virtual reality, augmented reality, cultural relics protection, and medical imaging.

[0062] The existing 3D reconstruction technology is to calculate the depth map based on the sparse point cloud and RGB image obtained by sparse reconstruction, and then use the depth map for dense reconstruction to obtain a dense point cloud. However, in the above 3D reconstruction process, the depth map often has quality problems such as large-area missing, which will affect the accuracy of dense reconstruction and the 3D reconstruction effect is poor.

[0063] To solve the above problems, an embodiment of the present invention provides a three-dimensional reconstruction method and system. After calculating the first depth map, the outliers of the first depth map are first removed, and then a depth completion network model is used to perform depth completion on the first depth map with the outliers removed to obtain a more complete and higher quality second depth map. The second depth map is then used to construct the final three-dimensional model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.

[0064] In practical applications, this solution can be applied to 3D reconstruction in cultural relics protection, cultural heritage research, and industrial design.

[0065] See also Figure 1 , shows a flow chart of a 3D reconstruction method provided by an embodiment of the present invention, the 3D reconstruction method comprising:

[0066] Step S101: using a first picture set of the object to be reconstructed acquired by a camera, obtaining distortion coefficients and camera intrinsic parameters.

[0067] It should be noted that image metadata includes but is not limited to: shooting date, camera model, shooting location, aperture value, shutter speed, resolution, file format, sensitivity, camera intrinsic parameters (focal length and principal point coordinates), distortion coefficient (radial distortion and tangential distortion), etc.

[0068] In the specific implementation of step S101, the object to be reconstructed (such as a cultural relic) is placed at a specified position, and then a camera (such as a digital camera) is used to photograph the object to be reconstructed from multiple angles to obtain multiple first images, which constitute a first image set.

[0069] The picture metadata of the first picture contained in the first picture set of the object to be reconstructed acquired by the camera is read, and the distortion coefficient and the camera intrinsic parameter of the camera are obtained from the picture metadata.

[0070] For example, the object to be reconstructed is placed in a collection box, and photographed from three angles: looking down, looking straight, and looking up, with the pitch angles all between 30 and 45 degrees. The turntable is turned on, and 16 to 24 first images are taken at each angle to obtain a first image set.

[0071] Step S102: performing distortion correction on the first picture included in the first picture set by using the distortion coefficient to obtain a second picture set.

[0072] In the specific implementation of step S102, the first picture included in the first picture set is subjected to distortion correction using the obtained distortion coefficient to obtain a second picture set, wherein the second picture set includes a plurality of second pictures, and the second picture is the first picture after distortion correction.

[0073] Step S103: Calculate the sparse point cloud, and calculate the camera extrinsic parameters and the first depth map of the second picture included in the second picture set.

[0074] In the specific implementation of step S103, sparse reconstruction is performed using camera intrinsic parameters and the second picture set to obtain a sparse point cloud, camera extrinsic parameters of the second picture, and a first depth map corresponding to the second picture.

[0075] It should be noted that the camera extrinsic parameters of the second image specifically refer to: the extrinsic parameters of the camera when the second image is obtained by taking the camera.

[0076] Step S104: removing outliers from the first depth map.

[0077] It should be noted that the first depth map often has some outliers due to calculation errors. These outliers will interfere with the depth completion operation on the first depth map. Therefore, the outliers in the first depth map need to be removed.

[0078] In the specific implementation of step S104, valid points in the first depth map are identified, valid points that meet preset abnormal conditions are determined to be outliers, and outliers in the first depth map are removed.

[0079] Among them, the preset abnormal condition is: the value exceeds the mean by 3 standard deviations; the outlier point is: the valid point whose value exceeds the mean by 3 standard deviations.

[0080] Specifically, for the values ​​of all valid points in the first depth map, the values ​​of these valid points are recorded as x1 to x n ; If the value x of one of the valid points i If (i is greater than or equal to 1 and less than or equal to n) satisfies the preset abnormal condition shown in formula (1), the valid point is determined to be an outlier and needs to be removed.

[0081] (1)

[0082] In formula (1), is the mean of all valid points, s is the variance; the mean is calculated by formula (2): , and the variance s is calculated using formula (3).

[0083] (2)

[0084] (3)

[0085] Step S105: Inputting the first depth map with outliers removed into a pre-trained depth completion network model to perform depth completion, thereby obtaining an optimized second depth map.

[0086] In the specific implementation of step S105, after the outliers in the first depth map are removed, the first depth map without the outliers is input into a pre-trained depth completion network model for depth completion to obtain an optimized second depth map (equivalent to the optimized first depth map).

[0087] That is to say, a depth completion network model is pre-trained, and the depth completion network model is used to perform depth completion on the first depth map with outliers removed, so as to obtain an optimized second depth map.

[0088] In some embodiments, the deep completion network model includes at least: an encoder composed of a deep residual network, a decoder composed of multiple upsampling modules, a local network composed of multiple consecutive convolutional layers, and a refinement network.

[0089] More specifically, the structure of the deep completion network model is as follows Figure 2 As shown, the deep completion network model includes: concat, Encoder, Decoder, Local Network, Global Depth, Local Depth, fusion, RefinedNetwork, and Refined Depth.

[0090] Among them, Encoder is an encoder, which uses a deep residual network (such as ResNet34); Decoder is a decoder, which consists of 5 (just for example) upsampling modules, each of which consists of convolution, pixel shuffle (pixel reorganization) and channel attention branches;

[0091] The Local Network is a local network, which consists of 3 (just for example) consecutive convolutional layers; the Refined Network is a refined network, which uses a cross-stage partial network (Cross Stage Partial Network, CSPNet).

[0092] also, Figure 2The RGB Image is the input color image, Sparse Depth is the input sparse depth map, concat is the channel splicing operation, Global Depth is the globally optimized depth map, Local Depth is the locally optimized depth map, fusion is the fusion operation, and Refined Depth is the refined depth map.

[0093] In the process of training the depth completion network model, the loss function used consists of three parts: L1 loss (absolute loss), RMAE loss (root mean square absolute error loss) and gradient loss; in order to ensure the training stability of the depth completion network model, the L1 loss of the global depth, local depth and re-refined depth are calculated simultaneously during training; after the training of the depth completion network model is stable, only the L1 loss of the refined depth is calculated to obtain the optimal result.

[0094] Assuming that in the true depth map B, all points greater than 0.001 are considered valid points, and these valid points constitute a point set P, which contains N valid points. Then the calculation formulas for the L1 loss, RMAE loss, and gradient loss of the predicted depth map A and the true depth map B are as shown in Formula (4) to Formula (6).

[0095] (4)

[0096] (5)

[0097] (6)

[0098] In formula (4)-formula (6), L1 loss (A, B) is L1 loss, RMAE loss (A, B) is RMAE loss, Grandient loss (A, B) is the gradient loss, i and j are the coordinates of the pixel points, A i,j To predict the value of point (i, j) on the depth map A, B i,j is the value of the point (i, j) on the real depth map B;

[0099] Conv represents the convolution calculation, and kx represents the convolution for calculating the gradient in the x-axis direction. For example, kx is specified as , ky represents the convolution for calculating the gradient in the y-axis direction. For example, ky is specified as .

[0100] Based on the above formulas (4) to (6), the total training loss LOSS during the training of the deep completion network model is calculated by formula (7).

[0101] (7)

[0102] In formula (7), epoch is the current training round number, Epochs is the total number of training rounds, and Depth is the number of training rounds. Global is the global depth map, Depth Local is the local depth map, Depth Refined To refine the depth map (i.e., predict the depth map), Depth Truth The real depth map.

[0103] The above is an explanation of the deep completion network model.

[0104] Step S106: constructing a final three-dimensional model of the object to be reconstructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0105] In the specific implementation of step S106, after the second depth map is obtained through the depth completion network model, the final three-dimensional model of the object to be reconstructed is constructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters of the second image and the sparse point cloud.

[0106] In an embodiment of the present invention, after calculating the first depth map, the outliers of the first depth map are first removed, and then the depth completion network model is used to perform depth completion on the first depth map with the outliers removed to obtain a more complete second depth map, and then the second depth map is used to construct the final three-dimensional model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.

[0107] For the above-mentioned embodiments of the present invention Figure 1 For details on the calculation of the sparse point cloud and the first depth map involved in step S103, see Figure 3 , shows a flowchart of calculating a sparse point cloud and a first depth map provided by an embodiment of the present invention, including the following steps:

[0108] Step S301: Divide the second pictures included in the second picture set into a plurality of picture groups.

[0109] In the process of specifically implementing step S301, a feature extraction operator is used to extract picture features of the second picture included in the second picture set, and the feature extraction operator used is any one of the following: Scale-Invariant Feature Transform (SIFT), Scale-Invariant Feature Transform of Domain-size pooling inlocal descriptors (DSPSIFT for short), and Accelerated-KAZE algorithm (Accelerated-KAZE); wherein KAZE can be called Kaze Features.

[0110] Matching the second picture according to the extracted picture features of the second picture, so as to divide the second picture into a plurality of picture groups, that is, dividing the second picture set into a plurality of picture groups;

[0111] Specifically, an image matching algorithm is used to match the second image, so as to divide the second image into multiple image groups. The image matching algorithm used is any one of the following: vocabulary tree matching method (vocabularytree), sequential matching method (sequential), exhaustive matching method (exhaustive) and frustum matching method (frustum).

[0112] Step S302: using the similarity between the feature points of each second picture in the picture group, a plurality of groups of successfully matched feature points are obtained.

[0113] In the specific implementation process of step S302, the similarity between the feature points of each second image in the image group is calculated, and then multiple groups of successfully matched feature points are obtained according to the similarity between the feature points, wherein "successfully matched feature points" refer to feature points whose distance in the feature vector space is less than a threshold.

[0114] Specifically, the similarity between the feature points of each second image in the image group is calculated by vector dot multiplication, and then the RANSAC algorithm (random sample consensus algorithm) is used to filter out the mismatched feature points, thereby obtaining multiple groups of successfully matched feature points.

[0115] Step S303: Calculate the sparse point cloud and the camera extrinsic parameters of the second picture based on the camera intrinsic parameters, the second picture set, the picture group and the multiple groups of successfully matched feature points.

[0116] In the specific implementation of step S303, sparse reconstruction is performed through the SFM algorithm (or the Global SFM algorithm) based on the camera intrinsic parameters, the second picture set, the picture group and multiple groups of successfully matched feature points, so as to calculate the sparse point cloud and the camera extrinsic parameters of the second picture.

[0117] Among them, the camera extrinsic parameters include but are not limited to the camera's posture (including translation parameters and rotation parameters), the SFM algorithm is a structure-from-motion algorithm, and the Global SFM algorithm is a global structure-from-motion algorithm.

[0118] It should be noted that in the process of sparse reconstruction using the SFM algorithm, the camera intrinsic parameters will be combined to determine the position of the feature points in the three-dimensional space. The SFM algorithm will use the bundle adjustment method to optimize the camera's external parameters (position and posture) and optimize the position of the three-dimensional points. In this optimization process, the camera intrinsic parameters are involved as known constraints. Accurate camera intrinsic parameters can help the optimization algorithm converge to a better solution more quickly, thereby improving the efficiency and accuracy of the sparse reconstruction process.

[0119] Step S304: Calculate a first depth map corresponding to the second image according to the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0120] In the process of specifically implementing step S304, according to the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud, a semi-global stereo matching algorithm (SGM algorithm) or a local stereo matching algorithm (SAD algorithm) is used to calculate the first depth map corresponding to the second image, and a corresponding first depth map is calculated for each second image.

[0121] The above is an explanation of the calculation of the sparse point cloud and the first depth map.

[0122] For the above-mentioned embodiments of the present invention Figure 1 For details on the construction of the final three-dimensional model of the object to be reconstructed in step S106, see Figure 4 , shows a flow chart of constructing a final three-dimensional model provided by an embodiment of the present invention, including the following steps:

[0123] Step S401: Calculate a dense point cloud according to the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters and the sparse point cloud.

[0124] It should be noted that the depth information of the second depth map is the distance between the pixel point and the camera in the physical space.

[0125] In the specific implementation of step S401, dense reconstruction is performed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud to calculate and obtain a dense point cloud.

[0126] Step S402: meshing the dense point cloud to obtain an initial three-dimensional model of the object to be reconstructed.

[0127] In the specific implementation of step S402, the calculated dense point cloud is meshed to obtain an initial three-dimensional model (mesh model) of the object to be reconstructed;

[0128] Specifically, the dense point cloud is meshed using algorithms such as Poisson reconstruction, growth method reconstruction, or greedy projection triangulation to obtain an initial three-dimensional model of the object to be reconstructed.

[0129] Step S403: using the mapping relationship between points on the second image and points in space, the RGB pixels of the second image are mapped to the texture space to obtain a texture map of the initial three-dimensional model.

[0130] It should be noted that the initial three-dimensional model of the object to be reconstructed is a three-dimensional model without color, so it is necessary to generate a texture map in combination with the second image to perform texture mapping on the initial three-dimensional model.

[0131] In the specific implementation of step S403, the RGB pixels of the second image are mapped to the texture space by using the mapping relationship between points on the second image and points in the space, thereby obtaining a texture map of the initial three-dimensional model.

[0132] Step S404: Map the texture image onto the initial three-dimensional model to obtain a final three-dimensional model.

[0133] In the specific implementation process of step S404, after obtaining the texture map of the initial three-dimensional model, the texture map is mapped to the initial three-dimensional model to color the initial three-dimensional model (equivalent to texture mapping the initial three-dimensional model), thereby obtaining the final three-dimensional model of the object to be reconstructed (a three-dimensional model with color).

[0134] The above is an explanation of how to construct the final three-dimensional model of the object to be reconstructed.

[0135] It can be seen from the contents of the above embodiments that the present solution reduces the interference of outliers on depth completion and dense reconstruction by removing outliers in the depth map, performs depth completion on the depth map, thereby improving the integrity and accuracy of the depth map, and further ensuring the accuracy and stability of the three-dimensional reconstruction results, making the final reconstructed three-dimensional model more accurate, and providing a high-quality three-dimensional reconstruction solution for practical applications.

[0136] Corresponding to the three-dimensional reconstruction method provided by the above embodiment of the present invention, see Figure 5 , an embodiment of the present invention further provides a structural block diagram of a three-dimensional reconstruction system, the three-dimensional reconstruction system comprising: an acquisition unit 100, a correction unit 200, a calculation unit 300, a removal unit 400, a depth completion unit 500 and a construction unit 600;

[0137] The acquisition unit 100 is used to acquire distortion coefficients and camera intrinsic parameters of the camera by using a first picture set of the object to be reconstructed acquired by a camera.

[0138] In a specific implementation, the acquisition unit 100 is specifically used to: read the picture metadata of the first picture contained in the first picture set of the object to be reconstructed acquired by the camera; and acquire the distortion coefficient and the camera intrinsic parameter of the camera from the picture metadata.

[0139] The correction unit 200 is used to perform distortion correction on the first picture included in the first picture set by using the distortion coefficient to obtain the second picture set.

[0140] The computing unit 300 is configured to compute a sparse point cloud, and to compute camera extrinsic parameters and a first depth map of a second image included in the second image set.

[0141] The removal unit 400 is used to remove outliers from the first depth map.

[0142] The depth completion unit 500 is used to input the first depth map with outliers removed into a pre-trained depth completion network model to perform depth completion and obtain an optimized second depth map.

[0143] In some embodiments, the deep completion network model includes at least: an encoder composed of a deep residual network, a decoder composed of multiple upsampling modules, a local network composed of multiple consecutive convolutional layers, and a refinement network.

[0144] The construction unit 600 is used to construct a final three-dimensional model of the object to be reconstructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0145] In an embodiment of the present invention, after calculating the first depth map, the outliers of the first depth map are first removed, and then the depth completion network model is used to perform depth completion on the first depth map with the outliers removed to obtain a more complete second depth map, and then the second depth map is used to construct the final three-dimensional model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.

[0146] Preferably, combined Figure 5 As shown in the figure, the calculation unit 300 includes a division subunit, an acquisition subunit, a first calculation subunit and a second calculation subunit. The execution principle of each subunit is as follows:

[0147] The division subunit is used to divide the second pictures included in the second picture set into multiple picture groups.

[0148] The acquisition subunit is used to obtain multiple groups of successfully matched feature points by using the similarity between the feature points of each second picture in the picture group.

[0149] The first calculation subunit is used to calculate the sparse point cloud and the camera extrinsic parameters of the second picture based on the camera intrinsic parameters, the second picture set, the picture group and the multiple groups of successfully matched feature points.

[0150] The second computing subunit is used to compute a first depth map corresponding to the second image according to the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0151] Preferably, combined Figure 5 As shown in the figure, the construction unit 600 includes a calculation subunit, a gridding subunit, a first mapping subunit and a second mapping subunit. The execution principle of each subunit is as follows:

[0152] The computing subunit is used to compute a dense point cloud according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

[0153] The gridding subunit is used to grid the dense point cloud to obtain an initial three-dimensional model of the object to be reconstructed.

[0154] The first mapping subunit is used to map the RGB pixels of the second image to the texture space by using the mapping relationship between the points on the second image and the points in the space, so as to obtain the texture map of the initial three-dimensional model.

[0155] The second mapping subunit is used to map the texture map onto the initial three-dimensional model to obtain a final three-dimensional model.

[0156] Preferably, combined Figure 5 As shown in the figure, the removal unit 400 includes an identification subunit, a determination subunit and a removal subunit, and the execution principle of each subunit is as follows:

[0157] The identification subunit is used to identify valid points in the first depth map.

[0158] The determination subunit is used to determine valid points that meet preset abnormal conditions as outliers.

[0159] The elimination subunit is used to eliminate outliers in the first depth map.

[0160] To summarize, an embodiment of the present invention provides a three-dimensional reconstruction method and system. After calculating a first depth map, the outliers of the first depth map are first removed, and then a depth completion network model is used to perform depth completion on the first depth map with the outliers removed to obtain a more complete second depth map. The second depth map is then used to construct a final three-dimensional model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.

[0161] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0162] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0163] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Using a first picture set of the object to be reconstructed acquired by a camera, a distortion coefficient and a camera intrinsic parameter of the camera are obtained; Performing distortion correction on the first picture included in the first picture set by using the distortion coefficient to obtain a second picture set; Calculating a sparse point cloud, and calculating a camera extrinsic parameter and a first depth map of a second picture included in the second picture set; Performing outlier removal on the first depth map includes: identifying valid points in the first depth map; determining the valid points that meet preset abnormal conditions as outliers; and removing the outliers in the first depth map; Inputting the first depth map with outliers removed into a pre-trained depth completion network model to perform depth completion, thereby obtaining an optimized second depth map; A final three-dimensional model of the object to be reconstructed is constructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

2. The method according to claim 1, characterized in that Calculating a sparse point cloud, and calculating a camera extrinsic parameter and a first depth map of a second picture included in the second picture set, including: Dividing the second pictures included in the second picture set into a plurality of picture groups; Using the similarity between the feature points of each of the second pictures in the picture group, a plurality of groups of successfully matched feature points are obtained; Calculate a sparse point cloud and camera extrinsic parameters of the second picture based on the camera intrinsic parameters, the second picture set, the picture group, and the multiple groups of successfully matched feature points; A first depth map corresponding to the second image is calculated according to the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

3. The method according to claim 1, characterized in that Constructing a final three-dimensional model of the object to be reconstructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud, including: Calculate a dense point cloud according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud; Meshing the dense point cloud to obtain an initial three-dimensional model of the object to be reconstructed; Mapping the RGB pixels of the second image to a texture space by using a mapping relationship between points on the second image and points in space, to obtain a texture map of the initial three-dimensional model; The texture map is mapped onto the initial three-dimensional model to obtain a final three-dimensional model.

4. The method according to claim 1, characterized in that: Using a first picture set of the object to be reconstructed acquired by a camera, obtaining a distortion coefficient and a camera intrinsic parameter of the camera, including: Reading picture metadata of a first picture included in a first picture set of the object to be reconstructed acquired by a camera; The distortion coefficient and the camera intrinsic parameter of the camera are obtained from the picture metadata.

5. The method according to any one of claims 1 to 4, characterized in that: The deep completion network model at least includes: an encoder composed of a deep residual network, a decoder composed of multiple upsampling modules, a local network composed of multiple continuous convolutional layers, and a refinement network.

6. A three-dimensional reconstruction system, characterized in that: The system comprises: An acquisition unit, configured to acquire distortion coefficients and camera intrinsic parameters of the camera by using a first picture set of the object to be reconstructed acquired by the camera; a correction unit, configured to perform distortion correction on the first picture included in the first picture set by using the distortion coefficient to obtain a second picture set; A computing unit, configured to compute a sparse point cloud, and to compute a camera extrinsic parameter and a first depth map of a second picture included in the second picture set; A removal unit, configured to remove outliers from the first depth map; The removal unit includes: an identification subunit, a determination subunit and a removal subunit; The identification subunit is used to identify valid points in the first depth map; The determination subunit is used to determine that the valid points satisfying the preset abnormal condition are outliers; The elimination subunit is used to eliminate the outliers in the first depth map; A depth completion unit, configured to input the first depth map with outliers removed into a pre-trained depth completion network model for depth completion to obtain an optimized second depth map; A construction unit is used to construct a final three-dimensional model of the object to be reconstructed according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

7. The system according to claim 6, characterized in that The computing unit comprises: a dividing subunit, configured to divide the second pictures included in the second picture set into a plurality of picture groups; An acquisition subunit, configured to acquire multiple groups of successfully matched feature points by using the similarity between the feature points of each of the second pictures in the picture group; A first calculation subunit, configured to calculate a sparse point cloud and camera extrinsic parameters of the second picture based on the camera intrinsic parameters, the second picture set, the picture group, and the multiple groups of successfully matched feature points; The second calculation subunit is used to calculate a first depth map corresponding to the second image according to the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud.

8. The system according to claim 6, characterized in that The building block comprises: A calculation subunit, configured to calculate a dense point cloud according to the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters and the sparse point cloud; A gridding subunit, used for gridding the dense point cloud to obtain an initial three-dimensional model of the object to be reconstructed; A first mapping subunit is used to map RGB pixels of the second image to a texture space by using a mapping relationship between points on the second image and points in space, so as to obtain a texture map of the initial three-dimensional model; The second mapping subunit is used to map the texture map onto the initial three-dimensional model to obtain a final three-dimensional model.

Images