Three-dimensional reconstruction method of uncalibrated monocular vision system

Abstract

The application provides a three-dimensional reconstruction method of an uncalibrated monocular vision system, and relates to the technical field of three-dimensional reconstruction.The application detects blurred images caused by movement during shooting by using Harr wavelet transform, thereby reducing the mismatch problem caused by blurred images during feature point matching; secondly, for the data set after removing the blurred images, a key frame screening algorithm is used to screen out images with large base lines between adjacent images in the data set, so as to reduce the calculation amount and improve the subsequent feature matching accuracy. Secondly, point cloud reconstruction is carried out, first, the internal parameters of the camera are obtained from the image by using a camera self-calibration equation, secondly, the sparse point cloud structure information of the target object is calculated from the image sequence by using an incremental motion recovery structure method, and finally, the dense point cloud structure information of the target object is obtained through optimization of the sparse point cloud and a region growing algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional reconstruction technology, and particularly relates to a three-dimensional reconstruction method of an uncalibrated monocular vision system. BACKGROUND

[0002] Three-dimensional reconstruction technology is a popular research direction of computer vision, and is widely applied in augmented reality, archaeology, biomedicine and other fields. Three-dimensional reconstruction technology converts two-dimensional planar image information containing a target object into three-dimensional structure information of the corresponding target object, and presents it to people in a more intuitive way, so as to facilitate people to better carry out follow-up work such as cultural relic restoration and postoperative rehabilitation. With the rapid popularization of monocular shooting devices such as mobile phones and digital cameras, a large amount of high-resolution image and video data can be used to reconstruct the three-dimensional model of the object. However, for these massive image data, it is also one of the research contents of three-dimensional reconstruction technology to restore the three-dimensional structure information of the object under the condition that the camera parameters are unknown. SUMMARY

[0003] In view of the deficiencies of the prior art, the present application provides a three-dimensional reconstruction method of an uncalibrated monocular vision system, which pre-processes the obtained data set, and restores the three-dimensional point cloud structure information of the object by using a motion recovery structure algorithm and a region growing algorithm.

[0004] A three-dimensional reconstruction method of an uncalibrated monocular vision system, comprising the following steps:

[0005] Step 1: video data acquisition, using a monocular camera to perform 360° surround shooting around a target object to be reconstructed;

[0006] Step 2: pre-processing the video data;

[0007] Step 2.1: frame the video to obtain N images, obtain the height H and the width W of the image, that is, HxW, and establish an image data set;

[0008] Step 2.2: traverse the image data set, detect a blurred image by using a Harr wavelet transform, and remove the blurred image from the image data set;

[0009] The detection of the blurred image is that, for each pixel of the image, a transformation operation is performed on the horizontal vertical diagonal direction of the image to construct an edge map E i (x,y), as shown in formula (1):

[0010]

[0011] According to the G-Step and Roof edge type structure in the edge image, two-dimensional G-Step, Roof vector |E da |, G-Step, Roof fuzzy vector |E|, if the relationship shown in formula (2) is satisfied, the current image is marked as a fuzzy image, and is removed from the data set; if not, the current image is marked as a clear image, and the next image is judged until all data is traversed.

[0012]

[0013] Wherein, B t is the image blur threshold.

[0014] Step 2.3: Feature point matching is performed on the clear images screened out in step 2.2, the fundamental matrix F and the homography matrix H are calculated by using the epipolar geometric constraint relationship, and then the wide baseline image is screened out from the screened clear images by using the geometric robustness GRIC criterion, as shown in formula (3):

[0015]

[0016]

[0017]

[0018] Wherein, is the residual term about the feature point position, k is the number of parameters required by the fitting model H and F, λ1dn is the penalty term of the model dimension, and λ2k is the penalty term of the number of model parameters; according to formulas (3), (4) and (5), the scores GRIC(H) and GRIR(F) of the points in the last image and the current image under the constraint relationship of the F and H models are calculated and compared, if formula (6) is satisfied, the current image is marked as a key frame, if formula (6) is not satisfied, the current image is marked as a non-key frame, and the repeated operation is continued on the next image until all data in the data set is traversed;

[0019] GRIC(H)<GRIC(F) (6)

[0020] Step 3: Feature point detection and matching are performed;

[0021] Step 3.1: The scale-invariant feature transform detection algorithm SIFT is used to extract the features in the image, and the corresponding key points and descriptors are obtained;

[0022] Step 3.2: The line segment detection algorithm LSD is used to detect the line features in the image by calculating the horizontal line field formed by the horizontal line angles of each pixel;

[0023] Step 3.3: The similarity of the key points in the two images is determined by using the Euclidean distance between the key point feature vectors, and finally the feature point matching algorithm FLANN is used for feature point matching to obtain the feature point matching pairs;

[0024] Step 4: Removing the mis-matched feature points; the best homography matrix H is solved by using the random sample consensus method for the feature point matching pairs obtained in step 3, so as to remove the mis-matched feature point pairs and obtain the correct matching point pairs;

[0025] Step 5: Self-calibration of the camera to obtain the camera internal parameters; the center point coordinates of the image are calculated by using the H and W of the image according to the image obtained in step 2.1. The fundamental matrix F and the extreme point are calculated by using the normalized eight-point method according to the correct matching point pairs obtained in step 4, and the internal parameters of the camera, the focal length f, are solved by formula (7) Kruppa self-calibration equation:

[0026]

[0027] Wherein, p0=(u0, v0, 1), p1=(u1, v1, 1) are the center point coordinates of the two images to be matched, I=diag(1, 1, 0). Thus the camera internal parameter matrix K is obtained as formula (8), wherein the tilt factor s=0.

[0028]

[0029] Step 6: The first two images are taken out from the clear and large baseline image data set obtained in step 2, and the feature matching point pairs are obtained by repeating steps 3 and 4, and the three-dimensional point X is calculated according to the relationship between the two-dimensional point x in the image and the three-dimensional point X in space as formula (9).

[0030] x=K[R|t]X (9)

[0031] Wherein, K is the internal parameter matrix, R and t are the rotation matrix and translation vector of the camera.

[0032] Step 7: Sparse point cloud structure generation; the three-dimensional point calculation is performed on the clear and large baseline image data set by using the incremental motion recovery structure method, so as to recover the sparse point cloud structure information of the target object;

[0033] Step 8: Sparse point cloud optimization. The global optimization of the sparse point cloud structure of the target object is performed by using the bundle adjustment method, as shown in formula (10),

[0034]

[0035] The camera pose and three-dimensional structure obtained by the incremental motion recovery structure are optimized by minimizing the re-projection point P(X i ,R jt j and the distance d i,j between the real point x error to obtain the optimal camera pose, i.e., the rotation matrix R and the translation vector t, and the 3D coordinates X of the spatial three-dimensional points. Wherein, P is a projection matrix, m and n are the number of feature points;

[0036] Step 9: Dense point cloud structure generation. Project the three-dimensional points optimized in step 8 into the corresponding two-dimensional image to obtain the projection point x. Calculate the correlation coefficient NCC of the gray values of the 4 points, 8 points and 24 points around the projection point x, as shown in formula (11), and compare NCC with the set threshold B. If the set condition is met, the 4 points, 8 points and 24 points in the neighborhood of x are included in the subsequent calculation process of the three-dimensional points. If not, the neighborhood points of the next projection point are judged, and the sparse three-dimensional points p i (i = 1, 2, 3, … n) are traversed, and finally the candidate points are obtained. The candidate points are subjected to the dense point cloud structure obtained in step 7, so as to complete the three-dimensional reconstruction of the uncalibrated monocular vision system.

[0037]

[0038] The beneficial effects produced by the above technical solutions are as follows:

[0039] The present application provides a three-dimensional reconstruction method of an uncalibrated monocular vision system, which has the following beneficial effects:

[0040] 1. The present application uses a fuzzy-key frame screening algorithm to pre-process data, which can effectively solve the problems of few detected feature points in fuzzy images and false matching in feature point matching. At the same time, the wide baseline image screened out can reduce the uncertainty of three-dimensional point calculation. The method provides a reliable data preprocessing method for three-dimensional reconstruction based on images;

[0041] 2. The present application realizes a convenient three-dimensional reconstruction step and complete point cloud structure information. The three-dimensional point cloud structure information of the target object can be recovered using image information under the condition that the camera is not calibrated, and the proposed point cloud dense scheme makes the point cloud more complete, providing a scheme for three-dimensional reconstruction under the uncalibrated condition. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The present application provides a three-dimensional reconstruction method of an uncalibrated monocular vision system, which has the following beneficial effects:

[0043] Figure 2 The present application provides a three-dimensional reconstruction method of an uncalibrated monocular vision system, which has the following beneficial effects:

[0044] Figure 3 The present application provides a three-dimensional reconstruction method of an uncalibrated monocular vision system, which has the following beneficial effects: DETAILED DESCRIPTION

[0045] The specific embodiments of the present application are described in further detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0046] A three-dimensional reconstruction method of an uncalibrated monocular vision system overcomes the mismatch problem caused by blurred images in feature matching and the point cloud density problem. As shown in Figure 1 , the method comprises the following steps:

[0047] Step 1: Video data acquisition, using a monocular camera to take 360° surround shots around the target object to be reconstructed;

[0048] Step 2: Preprocessing of video data;

[0049] Step 2.1: Frame the video to obtain N images, obtain the height H and width W of the image, i.e. HxW, and establish an image data set;

[0050] Step 2.2: Traverse the image data set, detect blurred images using Harr wavelet transform, and remove the blurred images from the image data set, as shown in Figure 2 ;

[0051] The detection of blurred images is to traverse each pixel of the image, and perform transformation operations in the horizontal vertical diagonal directions of the image to construct an edge map E i (x,y), as shown in equation (1):

[0052]

[0053] According to the G-Step and Roof edge type structure in the edge image, obtain the two-dimensional G-Step, Roof vector |E da |, G-Step, Roof blur vector |E|, if the relationship shown in equation (2) is satisfied, mark the current image as a blurred image and remove it from the data set; if not, mark the current image as a clear image and judge the next one, until all data are traversed.

[0054]

[0055] Where B t is the image blur threshold.

[0056] Step 2.3: Feature point matching is performed on the clear images screened in step 2.2, a fundamental matrix F (Fundamental matrix) and a homography matrix H (Homography Matrix) are calculated using the epipolar constraint relationship, and a wide-baseline image, i.e., an image with a significant difference between adjacent images, is screened from the screened clear images through a geometric robustness criterion GRIC. The following formula is shown:

[0057]

[0058]

[0059]

[0060] wherein, is a residual term about the position of the feature points, k is the number of parameters required by the fitting model H and F, λ1dn is a penalty term of the model dimension, and λ2k is a penalty term of the number of model parameters; the values of the scores GRIC(H) and GRIR(F) of the points in the previous frame image and the current image under the constraint relationship of the two models F and H are calculated according to formulas (3), (4), and (5), and a comparison is made, if formula (6) is satisfied, the current image is marked as a key frame, if formula (6) is not satisfied, the current image is marked as a non-key frame, and the repeating operation continues for the next frame until all data in the data set are traversed;

[0061] GRIC(H) < GRIC(F) (17)

[0062] Step 3: Feature point detection and matching are performed;

[0063] Step 3.1: A scale-invariant feature transform detection algorithm SIFT is used to extract features in the image, and corresponding key points and descriptors are obtained;

[0064] Step 3.2: A line segment detection algorithm LSD is used to detect line features in the image by calculating a horizontal line field formed by the horizontal line angles of each pixel;

[0065] Step 3.3: The similarity of the key points in the two images is determined by using the Euclidean distance between the key point feature vectors, and finally a feature point matching algorithm FLANN is used for feature point matching to obtain a feature point matching pair;

[0066] Step 4: Remove the mis-matched feature points; the feature point matching pair obtained in step 3 is used to solve the best homography matrix H using the random sample consensus method, so as to remove the mis-matched feature point pairs and obtain the correct matching point pairs;

[0067] Step 5: Perform self-calibration on the camera to obtain its intrinsic parameters; based on the image obtained in Step 2.1, calculate the coordinates of the center point of the image using the H and W coordinates. Based on the correctly matched point pairs obtained in Step 4, calculate the fundamental matrix F and the poles using the normalized eight-point method, and solve for the camera's intrinsic parameters, focal length f, using equation (7) of the Kruppa self-calibration equation:

[0068]

[0069] Where p0 = (u0, v0, 1) and p1 = (u1, v1, 1) are the coordinates of the center points of the two images to be matched, and I = diag(1, 1, 0). Thus, the camera intrinsic parameter matrix K is obtained as shown in equation (8), where the tilt factor s = 0.

[0070]

[0071] Step 6: Extract the first two images from the clear image dataset with a large baseline obtained in Step 2, and repeat Steps 3 and 4 to obtain feature matching point pairs. Based on the relationship between the two-dimensional point x in the image and the three-dimensional point X in space, as shown in Equation (9), calculate the three-dimensional point.

[0072] x=K[R|t]X (20)

[0073] Where K is the intrinsic parameter matrix, and R, t is the camera's rotation matrix and translation vector.

[0074] Step 7: Sparse point cloud structure generation; Using the incremental motion recovery structure method, three-dimensional points are calculated on a clear image dataset with a large baseline, thereby recovering the sparse point cloud structure information of the target object.

[0075] Step 8: Sparse Point Cloud Optimization. The sparse point cloud structure of the target object is globally optimized using the bundle adjustment method, as shown in Equation (10).

[0076]

[0077] The camera pose and 3D structure obtained from incremental motion reconstruction are optimized by minimizing the reprojection point P(X) on the 2D image. i ,R j ,t j ) and the real point x i,j The distance d between error To obtain the optimal camera pose, i.e., the rotation matrix R and translation vector t, and the 3D coordinates X of the three-dimensional point in space. Here, P is the projection matrix, and m and n are the number of feature points;

[0078] Step 9: Dense point cloud structure generation. Project the three-dimensional points optimized in step 8 into the corresponding two-dimensional image to obtain the projection point x. Calculate the correlation coefficient NCC of the gray values of the 4 points, 8 points, and 24 points around the projection point x, and compare the NCC with the set threshold B. If the set condition is met, the 4 points, 8 points, and 24 points in the neighborhood of x are included in the calculation process of the subsequent three-dimensional points. If not, the neighborhood points of the next projection point are judged, and the operation is performed on the sparse points in turn. Finally, the candidate points are obtained, and the dense point cloud structure is obtained through step 7, and the three-dimensional reconstruction of the uncalibrated monocular vision system is completed.

[0079] As shown in Figure 3 , the obtained sparse point cloud is used as a seed point to project reversely onto the image, and the NCC value obtained by calculating the gray values of the 4, 8, and 24 pixel points around the projection point x is calculated according to the photo-consistency constraint. As shown in formula 10:

[0080]

[0081] If the NCC is greater than the set threshold B, the eight points around x are included in the secondary three-dimensional point calculation range, and the sparse three-dimensional points p i (i = 1, 2, 3, … n) are traversed. Then, the corresponding three-dimensional points are calculated based on the traversed two-dimensional points, so as to complete the densification of the sparse point cloud.

[0082] The above description is only the preferred embodiments of the present disclosure and the explanation of the applied technical principles. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions to form technical solutions.

Claims

1. A method for three-dimensional reconstruction of an uncalibrated monocular vision system, characterized in that, The method comprises the following steps: Step 1: video data acquisition, using a monocular camera to take 360° surround shots around the reconstructed target object; Step 2: pre-processing of the video data; Step 3: feature point detection and matching, to obtain a feature point matching pair; Step 4: removing mis-matched feature points; using a random sample consensus method to solve the best homography matrix H from the feature point matching pair obtained in step 3, so as to remove mis-matched feature point pairs and obtain correct matching point pairs; Step 5: self-calibration of the camera, to obtain the internal parameters of the camera; using the high H and wide W of the image to calculate the center point coordinates of the image, and using the correct matching point pairs obtained in step 4 to calculate the fundamental matrix F and the pole point by using the normalized eight-point method, and solving the internal parameters of the camera, the focal length f, by using Kruppa self-calibration equation (1): （1）； wherein, , are the center point coordinates of the two images to be matched, so that the camera intrinsic matrix K is obtained as in equation (2), wherein the skew factor s = 0; （2）； Step 6: taking the first two images from the clear and large-baseline image data set obtained in step 2, repeating steps 3 and 4 to obtain feature matching point pairs, and calculating three-dimensional points according to the relationship between the two-dimensional points x in the image and the three-dimensional points X in space, as shown in equation (3): （3）； Wherein, K is the internal parameter matrix, R and t are the rotation matrix and translation vector of the camera; Step 7: sparse point cloud structure generation; using the incremental motion recovery structure method, calculating the three-dimensional points of the clear and large-baseline image data set, so as to recover the sparse point cloud structure information of the target object; Step 8: sparse point cloud optimization; using the bundle adjustment method to globally optimize the sparse point cloud structure of the target object, as shown in equation (4): （4）； Optimizing the camera poses and 3D structure computed from structure from motion by minimizing the distance between the re-projection points and the real points in the 2D images to obtain the best camera pose, i.e. rotation matrix R and translation vector t, and 3D coordinates X of the spatial 3D points, where P is the projection matrix, m, n are the number of feature points. Step 9: dense point cloud structure generation; projecting the three-dimensional points optimized in step 8 into the corresponding two-dimensional image to obtain the projection point x, calculating the correlation coefficient NCC of the gray values of the four points, eight points and 24 points around the projection point x respectively, and comparing the NCC with the set threshold B, if the set condition is met, the four points, eight points and 24 points in the neighborhood of X are included in the subsequent three-dimensional point calculation process, if not, the neighborhood points of the next projection point are judged, and the sparse point cloud is operated in turn, and finally the candidate points are obtained, the dense point cloud structure is obtained by steps 7 and 8, and the three-dimensional reconstruction of the uncalibrated monocular vision system is completed; The step 2 specifically comprises the following steps: Step 2.1: Frame the video to get N images, get the height H and width W of the image, that is , and establish an image dataset; Step 2.2: traversing the image data set, detecting the blurred image by using the Harr wavelet transform, and removing the blurred image from the image data set; The detecting the blurred image is, traversing each pixel of the image, transforming in horizontal , vertical , diagonal directions of the image, constructing an edge map , as shown in equation (5): （5）； According to the G-Step and Roof edge type structure in the edge map, two-dimensional G-Step and Roof vectors are obtained , G-Step and Roof fuzzy vectors If a relationship shown in equation (6) is satisfied, the current image is marked as a fuzzy image and is removed from the dataset; if not, the current frame is marked as a clear image and the next image is judged until all data are traversed. （6）； wherein is an image blur threshold value; Step 2.3: performing feature point matching on the clear image selected in step 2.2, calculating the fundamental matrix F and the homography matrix H by using the epipolar geometric constraint relationship, and selecting the wide-baseline image from the selected clear image by using the geometric robustness criterion GRIC criterion, as shown in equation (7): （7）； （8）； （9）； wherein, is a residual term about the feature point position, k is the number of parameters required by the fitting model H and F, is a penalty term of model dimension, is a penalty term of model parameter number; according to formulas (7), (8) and (9), the values of the scores GRIC(H) and GRIR(F) of the points in the previous image and the current image under the constraint relationship of the two models F and H are calculated and compared, if formula (10) is satisfied, the current image is marked as a key frame, if formula (10) is not satisfied, the current image is marked as a non-key frame, and the repeated operation is continued on the next image until all data in the data set are traversed. （10）； The step 3 specifically comprises the following steps: Step 3.1: using the scale-invariant feature transform detection algorithm SIFT to extract the features in the image, to obtain the corresponding key points and descriptors; Step 3.2: using the line segment detection algorithm LSD to detect the line features in the image by calculating the horizontal line field formed by the horizontal line angles of each pixel; Step 3.3: The Euclidean distance between the key point feature vectors is used to determine the similarity of the key points in the two images, and finally the feature point matching algorithm FLANN is used for feature point matching to obtain the feature point matching pairs; The projection in step 9 is that the obtained sparse point cloud is reversely projected onto the image as a seed point, and the NCC value obtained according to the geometric consistency constraint calculation of the gray value size of 4, 8, 24 pixel points around the projection point x is as shown in formula (11): （11）； If the NCC is greater than a set threshold B, then 4, 8, 24 points around x are included in the three-dimensional point calculation range, and the sparse three-dimensional points are traversed; the obtained two-dimensional points are calculated to obtain corresponding three-dimensional points, thereby completing the densification of the sparse point cloud.

Images