Three-dimensional reconstruction method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application provide a three-dimensional reconstruction method, device, electronic equipment and computer readable storage medium, by extracting the weak texture region in each to-be-processed image corresponding to the target scene, and performing mask processing on the weak texture region, so as to avoid three-dimensional reconstruction of the weak texture region in the three-dimensional reconstruction process, after obtaining the dense point cloud of the region except the weak texture region in the target scene, according to the boundary line of the weak texture region and the dense point cloud, the point cloud of the weak texture region is filled, that is, the three-dimensional reconstruction of the weak texture region is not directly performed, but the point cloud data related to the boundary line of the weak texture region is used to fill the point cloud of the weak texture region, so as to complete the three-dimensional reconstruction process of the whole target scene, effectively solve the problem that the reconstruction effect of the weak texture region is not ideal in the prior art, avoid the phenomenon that there are holes and a large number of noise points in the three-dimensional reconstruction result, and realize good hole removal effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular, to a three-dimensional reconstruction method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] The three-dimensional reconstruction technology based on vision is to obtain two-dimensional image data information of an object through a related instrument, then analyze and process the obtained data information, and finally reconstruct the contour information of the object surface in the real environment by using the related theory of three-dimensional reconstruction. The three-dimensional reconstruction technology based on vision is to calculate the pose information of a camera by image feature matching and extract sparse point cloud, but it is difficult to extract feature information for special areas such as weak texture areas and non-texture areas, which causes the appearance of holes in dense reconstruction.

[0003] At present, most of the solutions are to optimize the three-dimensional reconstruction algorithm based on vision to improve the feature information extraction of weak texture areas, but this way is not ideal for the reconstruction effect of special areas, and there will still be holes in the three-dimensional reconstruction result. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a three-dimensional reconstruction method, device, electronic equipment and computer readable storage medium to solve the problem of unsatisfactory reconstruction effect of weak texture areas and easy appearance of holes in the existing three-dimensional reconstruction technology.

[0005] In order to achieve the above purpose, the technical scheme adopted by the embodiments of the present application is as follows:

[0006] In a first aspect, the present application provides a three-dimensional reconstruction method, which comprises:

[0007] extracting weak texture areas in each to-be-processed image corresponding to a target scene;

[0008] masking processing the weak texture areas in each to-be-processed image to obtain each target image;

[0009] performing three-dimensional reconstruction on a target area according to each target image to obtain a dense point cloud of the target area; the target area is an area in the target scene except the weak texture areas;

[0010] filling the point cloud of the weak texture areas according to the boundary line of the weak texture areas and the dense point cloud.

[0011] In an optional embodiment, the extracting weak texture areas in each to-be-processed image corresponding to a target scene comprises:

[0012] The weak texture region in each image to be processed is extracted by using pixel values in each image to be processed or a pre-trained detection model, or the weak texture region in each image to be processed corresponding to the target scene is selected by a user.

[0013] In an optional implementation, the weak texture region in each image to be processed is extracted by using pixel values in each image to be processed or a pre-trained detection model, and the method comprises the following steps.

[0014] A first weak texture region in each image to be processed is determined according to pixel values in each image to be processed.

[0015] Each image to be processed is input into a pre-trained detection model to obtain a second weak texture region in each image to be processed; wherein the weak texture region comprises the first weak texture region and the second weak texture region, and a reflectivity of the first weak texture region is higher than that of the second weak texture region.

[0016] In an optional implementation, the weak texture region is filled with point cloud data according to the boundary line of the weak texture region and the dense point cloud, and the method comprises the following steps.

[0017] The boundary line of the weak texture region is superimposed with the dense point cloud to obtain point cloud data at the boundary line of the weak texture region.

[0018] The point cloud data at the boundary line of the weak texture region is statistically distributed, and point cloud data within a preset range is selected according to a statistical result.

[0019] The weak texture region is filled with point cloud data according to the point cloud data within the preset range.

[0020] In an optional implementation, the weak texture region is filled with point cloud data according to the point cloud data within the preset range, and the method comprises the following steps.

[0021] When the weak texture region is a water area, an average value of the point cloud data within the preset range is taken as an elevation value of the water area, and the water area is filled with point cloud data.

[0022] In an optional implementation, the weak texture region is filled with point cloud data according to the point cloud data within the preset range, and the method comprises the following steps.

[0023] When the weak texture region is a non-water region, a normal vector of each point cloud in the point cloud data within the preset range is calculated, an average value of the normal vectors of the point clouds is taken as a vertical direction of the non-water region, and the non-water region is filled with point cloud data according to the point cloud data within the preset range and the vertical direction of the non-water region.

[0024] In an optional implementation, the step of performing three-dimensional reconstruction of the target region based on each target image to obtain a dense point cloud of the target region includes:

[0025] Feature points are extracted and matched for each target image to obtain matching point pairs between the target images.

[0026] Calculate the camera parameters and sparse point cloud corresponding to each target image based on the matching point pairs;

[0027] A dense point cloud is generated based on the camera parameters and the sparse point cloud.

[0028] In a second aspect, the present invention provides a three-dimensional reconstruction apparatus, the apparatus comprising:

[0029] The weak texture region extraction module is used to extract weak texture regions in each image to be processed corresponding to the target scene;

[0030] The masking module is used to perform masking on the weak texture regions in each image to be processed, so as to obtain each target image.

[0031] The 3D reconstruction module is used to perform 3D reconstruction of the target region based on the target images to obtain a dense point cloud of the target region; the target region is the region in the target scene other than the weak texture region.

[0032] The point cloud filling module is used to fill the weak texture region with point cloud based on the boundary line of the weak texture region and the dense point cloud.

[0033] Thirdly, the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the three-dimensional reconstruction method as described in any of the foregoing embodiments.

[0034] Fourthly, the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the three-dimensional reconstruction method as described in any of the foregoing embodiments.

[0035] The three-dimensional reconstruction method, device, electronic device and computer readable storage medium provided by the embodiment of the present application avoid three-dimensional reconstruction of the weak texture region in the three-dimensional reconstruction process by extracting the weak texture region in each to-be-processed image corresponding to the target scene and performing mask processing on the weak texture region, after obtaining the dense point cloud of the region except the weak texture region in the target scene, the weak texture region is filled with point cloud according to the boundary line of the weak texture region and the dense point cloud, that is, the three-dimensional reconstruction of the weak texture region is not directly performed, but the weak texture region is filled with point cloud by using the point cloud data related to the boundary line of the weak texture region, so that the three-dimensional reconstruction process of the entire target scene is completed, the problem that the reconstruction effect of the weak texture region is not ideal in the prior art is effectively solved, the phenomenon that holes and a large number of noise points appear in the three-dimensional reconstruction result is avoided, and good hole removal effect is achieved.

[0036] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following will describe a preferred embodiment in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0038] Figure 1 A schematic diagram of a point cloud hole is shown;

[0039] Figure 2 A flowchart of the three-dimensional reconstruction method provided by the embodiment of the present application is shown;

[0040] Figure 3 A schematic diagram of a weak texture region which is easy to cause a hole is shown;

[0041] Figure 4 Another flowchart of the three-dimensional reconstruction method provided by the embodiment of the present application is shown;

[0042] Figure 5 A schematic diagram of an original image of feature point matching is shown;

[0043] Figure 6 A result schematic diagram of feature point matching is shown;

[0044] Figure 7 A schematic diagram of epipolar geometry constraint is shown;

[0045] Figure 8Another flow diagram of the three-dimensional reconstruction method provided by the embodiment of the present application is shown;

[0046] Figure 9 A three-dimensional reconstruction effect diagram for removing holes is shown;

[0047] Figure 10 A functional module diagram of the three-dimensional reconstruction device provided by the embodiment of the present application is shown;

[0048] Figure 11 A block diagram of the electronic device provided by the embodiment of the present application is shown.

[0049] Icon: 500-three-dimensional reconstruction device; 700-electronic device; 510-weak texture area extraction module; 520-mask processing module; 530-three-dimensional reconstruction module; 540-point cloud filling module; 710-memory; 720-processor; 730-communication module. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0051] Therefore, the detailed description of the embodiments of the present application provided in the accompanying drawings below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0052] It should be noted that the relational terms such as "first" and "second" and the like are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0053] For the problem of easy to appear cavity phenomenon in three-dimensional reconstruction, most of the current solutions are to optimize the three-dimensional reconstruction algorithm based on vision to improve the feature information extraction of weak texture area, so as to achieve the purpose of cavity elimination, but the improved algorithm does not achieve the ideal effect, and the three-dimensional reconstruction result of the weak texture area will appear cavity and a large number of noise phenomena (such as Figure 1 In addition to the three-dimensional reconstruction based on vision, there is also three-dimensional reconstruction based on laser radar. Laser radar is a light sensor technology that scans laser beams to illuminate the object, and measures the distance and position from the object by measuring the time of the laser beam reflected by the object.

[0054] Since the method of improving the reconstruction of weak texture area by optimizing the three-dimensional reconstruction algorithm based on vision has not achieved very ideal effect. And the price of laser radar device is much higher than that of three-dimensional reconstruction device based on vision, and the accuracy of point position will decrease with the distance for large scene. Therefore, the embodiment of the present application provides a three-dimensional reconstruction method, device, electronic device and computer readable storage medium, which extracts the weak texture area in each to-be-processed image corresponding to the target scene, avoids the three-dimensional reconstruction of the weak texture area by performing mask processing on the weak texture area in the three-dimensional reconstruction process, fills the point cloud of the weak texture area according to the boundary line of the weak texture area and the dense point cloud after obtaining the dense point cloud of the area except the weak texture area in the target scene, that is, the three-dimensional reconstruction of the weak texture area is not directly performed, but the point cloud data related to the boundary line of the weak texture area is used to fill the point cloud of the weak texture area, so as to complete the three-dimensional reconstruction process of the whole target scene, effectively solve the problem of unsatisfactory reconstruction effect of weak texture area in the prior art, avoid the phenomenon of cavity and a large number of noise in the three-dimensional reconstruction result, and achieve good cavity removal effect. Since the weak texture area is subjected to mask processing in the three-dimensional reconstruction process, a large number of noise will not appear in the obtained three-dimensional reconstruction result, and the accuracy of the point cloud data after three-dimensional reconstruction is effectively improved.

[0055] Next, the three-dimensional reconstruction method provided by the embodiment of the present application will be described in detail.

[0056] Please refer to Figure 2 , a flowchart of the three-dimensional reconstruction method provided by the embodiment of the present application. It should be noted that the three-dimensional reconstruction method of the embodiment of the present application is not limited to the specific order of Figure 2 and the following, it should be understood that in other embodiments, the order of some steps of the three-dimensional reconstruction method of the embodiment of the present application can be exchanged according to actual needs, or some steps can be omitted or deleted. The three-dimensional reconstruction method can be applied in electronic devices, and the following will be described in detail. Figure 2The specific process shown is described in detail.

[0057] In step S201, weak texture regions in each image to be processed corresponding to the target scene are extracted.

[0058] In this embodiment, the target scene is a scene to be reconstructed, which can be an outdoor scene or an indoor scene, and no limitation is made thereto. For example, a flight path of a UAV can be planned, and the target scene is photographed by a camera carried by the UAV during flight of the UAV, and the photographed images are sent to the electronic device. In this way, the electronic device can obtain each image to be processed corresponding to the target scene.

[0059] In this embodiment, each image to be processed corresponding to the target scene can include images of the target scene at different viewing angles taken by one or more cameras.

[0060] After the electronic device obtains each image to be processed corresponding to the target scene, the electronic device can extract weak texture regions in each image to be processed by using a deep learning framework and in combination with ground object spectral information, so as to obtain weak texture regions in each image to be processed corresponding to the target scene. The weak texture region can be understood as a region with only a small amount of texture or no texture.

[0061] In step S202, the weak texture regions in each image to be processed are subjected to mask processing to obtain each target image.

[0062] In this embodiment, considering that it is difficult to extract feature information for the weak texture region in the process of three-dimensional reconstruction, the weak texture region in each image to be processed is subjected to mask processing before three-dimensional reconstruction, that is, the weak texture region on the image to be processed is shielded, so that the weak texture region does not participate in the subsequent three-dimensional reconstruction process. It can be understood that the target image is an image obtained by shielding the weak texture region in the image to be processed.

[0063] In step S203, the target region is subjected to three-dimensional reconstruction according to each target image to obtain a dense point cloud of the target region; the target region is a region in the target scene except the weak texture region.

[0064] In this embodiment, the target region can be understood as a non-weak texture region. Since the target image is an image obtained by shielding the weak texture region in the image to be processed, three-dimensional reconstruction of the non-weak texture region in the target scene can be realized based on each target image, so as to obtain a dense point cloud of the non-weak texture region.

[0065] In step S204, the weak texture region is subjected to point cloud filling according to the boundary line of the weak texture region and the dense point cloud.

[0066] In the embodiment, after the three-dimensional reconstruction of the non-weak texture region in the target scene is completed, for the weak texture region in the target scene, the boundary line related point cloud data of the weak texture region can be obtained by using the boundary line and the dense point cloud of the weak texture region, and then the point cloud filling of the weak texture region is completed according to the boundary line related point cloud data of the weak texture region. In this way, the three-dimensional reconstruction result of the entire target scene can be obtained.

[0067] It can be seen that the three-dimensional reconstruction method provided by the embodiment avoids the three-dimensional reconstruction of the weak texture region in the three-dimensional reconstruction process, and after the dense point cloud of the region except the weak texture region in the target scene is obtained, the point cloud filling of the weak texture region is performed according to the boundary line and the dense point cloud of the weak texture region, that is, the three-dimensional reconstruction of the weak texture region is not directly performed, but the point cloud filling of the weak texture region is performed by using the boundary line related point cloud data of the weak texture region, so that the three-dimensional reconstruction process of the entire target scene is completed, the problem of unsatisfactory reconstruction effect of the weak texture region in the prior art is effectively solved, the phenomenon of holes and a large number of noise points in the three-dimensional reconstruction result is avoided, and a good hole removal effect is achieved. Since the weak texture region is masked in the three-dimensional reconstruction process, a large number of noise points will not appear in the obtained three-dimensional reconstruction result, and the accuracy of the point cloud data after three-dimensional reconstruction is effectively improved.

[0068] In actual application, the weak texture region can be extracted based on different methods. That is, the step S201 can include: extracting the weak texture region in each to-be-processed image by using the pixel value in each to-be-processed image or a pre-trained detection model, or selecting the weak texture region in each to-be-processed image corresponding to the target scene by a user.

[0069] That is, after the electronic device obtains each to-be-processed image corresponding to the target scene, the user can select the weak texture region in each to-be-processed image, or the electronic device can extract the weak texture region in each to-be-processed image by using the pixel value in each to-be-processed image or a pre-trained detection model.

[0070] When the pixel value in the to-be-processed image is used to extract the weak texture region, the pixels in the to-be-processed image are traversed, and when the current pixel value is greater than a preset value, the region growing method is used to judge the pixel value of the neighborhood pixel. If the pixel value of the neighborhood pixel is greater than the preset value, the diffusion is continued outward until the pixel value is less than the preset value. In this way, the weak texture region in the to-be-processed image can be obtained. The preset value can be set according to actual needs, for example, 240.

[0071] The pre-trained detection model can be a deep learning convolutional neural network. When extracting weak texture regions using the pre-trained detection model, each image to be processed can be input into the pre-trained detection model to obtain the weak texture regions in each image. The specific method is as follows:

[0072] (1) Use historical UAV images to annotate weak texture areas, and use unannotated areas as background.

[0073] (2) Perform data augmentation on the original data and label data to increase the amount of data for model training; augmentation methods include cropping, rotation, flipping, scaling and adding noise, etc.

[0074] (3) Divide the prepared dataset into three parts according to a certain ratio, such as 6:2:2: training set, validation set, and test set. First, feed the training set data into a convolutional neural network. A convolutional neural network generally includes an input layer, convolutional layer, pooling layer, and fully connected layer. The convolutional neural network assigns an initial class label to each pixel. The convolutional layer can effectively capture local features in the image and nest many such modules together in a hierarchical manner. After the detection model is trained, evaluate the detection model using the validation set and select the best-performing set of hyperparameters for testing on the test set. Continuously adjust the hyperparameters to make the detection model perform optimally on the test set.

[0075] (4) Use the optimal detection model to classify the weak texture regions in each image to be processed, and obtain the weak texture regions and their corresponding categories in each image to be processed. The boundary lines of the weak texture regions can be obtained by vectorizing the classification results.

[0076] In practical applications, weakly textured areas that are prone to creating voids can be of various types (such as...). Figure 3 As shown, by utilizing the different methods described above for extracting weak texture regions, different types of weak texture regions can be extracted from the image to be processed. That is, the steps described above, which use pixel values ​​from each image to be processed or a pre-trained detection model to extract weak texture regions from each image to be processed, can include:

[0077] The first weak texture region in each image to be processed is determined based on the pixel values ​​in each image to be processed; each image to be processed is input into a pre-trained detection model to obtain the second weak texture region in each image to be processed; wherein, the weak texture region includes the first weak texture region and the second weak texture region, and the reflectivity of the first weak texture region is higher than that of the second weak texture region.

[0078] In the embodiment, the first weak-texture region is a strong-reflective region with high reflectivity, such as a strong-reflective road surface. The second weak-texture region is a region prone to cavities other than the strong-reflective region, which has lower reflectivity than the first weak-texture region, such as a water area, a roof, a solar reflector, and the like.

[0079] For the weak-texture region such as the strong-reflective region, the pixel intensity value can be used for determination. For the weak-texture region such as the water area, the roof, the solar reflector, and the like, the convolutional neural network (detection model) based on deep learning is used to complete the extraction, so that different methods are used to extract different types of weak-texture regions.

[0080] It can be seen that, according to the embodiment, different methods are used to extract weak-texture regions according to different types of weak-texture regions prone to cavities. For the strong-reflective region, the pixel intensity value is combined with the region growing method for determination. For the weak-texture region such as the water area, the roof, and the solar reflector, the pre-trained detection model based on the deep learning framework is used to extract the weak-texture region. In this way, the weak-texture region prone to cavities is accurately extracted, so that the three-dimensional reconstruction result does not have cavities and a large number of noise points, and the like.

[0081] In the embodiment, the process of performing three-dimensional reconstruction on the target region according to each target image can include feature point extraction and matching, sparse reconstruction, and dense reconstruction, and finally obtain the dense point cloud of the target region. Based on this, please refer to Figure 4 The step S203 can include the following sub-steps:

[0082] In the sub-step S2031, feature point extraction and matching are performed on each target image to obtain the matching point pairs between the target images.

[0083] The commonly used feature point detection includes SIFT (Scale Invariant Feature Transform), SURF (Speed Up Robust Feature), ORB (Oriented FAST and Rotated BRIEF), Harris, and the like. For example, the SIFT algorithm can be used for feature point extraction in the embodiment. The SIFT feature point is very stable and has invariance to rotation, scale, brightness, and the like. After the SIFT feature point is extracted, feature point matching is performed to obtain the matching point pairs between the target images.

[0084] In the sub-step S2032, the camera parameters and the sparse point cloud corresponding to each target image are calculated according to the matching point pairs.

[0085] In this embodiment, after the matching point pairs between each target image are obtained, sparse reconstruction can be performed through the Sfm (Structure from motion) technology, so as to estimate the camera parameters corresponding to each target image, and obtain a sparse point cloud according to triangulation. The Sfm is an algorithm for three-dimensional reconstruction, which can recover the camera pose and reconstruct the sparse three-dimensional coordinate points through two or more scenes (pictures).

[0086] In sub-step S2033, a dense point cloud is generated according to the camera parameters and the sparse point cloud.

[0087] In this embodiment, the MVS (Multi View Stereo) can be used to complete the dense reconstruction, that is, on the premise that the camera parameters corresponding to each target image and the sparse point cloud are known, the dense matching between the target images can be realized through a photo consistency function, such as the SSD (Sum of Squared Difference), the SAD (Sum of Absolute Differences) and the NCC (Normalized Cross Correlation), and then the dense point cloud corresponding to the target region in the target scene can be reconstructed.

[0088] For example, the dense matching based on depth map fusion can include the following processes: (1) selecting a neighborhood image set (candidate set) for each target image to form a stereo image group. In the process of selecting the neighborhood image set, a sufficient baseline between images should be considered. (2) calculating the depth of each pixel of the reference image, the classic framework: calculating the matching cost, cost aggregation, calculating the depth value, and finally depth optimization, generating the depth map and the normal map. Using the gray absolute value difference (AD, Absolute Differences), gray absolute value difference sum, normalized correlation coefficient, mutual information (MI, Mutual Information) method or Census transform (CT) method as the matching cost calculation method of two pixels. (3) optimizing the cost matrix according to certain criteria, such as adjacent pixels should have continuous depth values. The depth map of each target image is calculated by using information transmission strategies, including spatial propagation, view propagation, and time propagation. (4) removing the false depth caused by occlusion and noise by using left-right consistency checking algorithm, removing isolated abnormal points by using small connected domain algorithm, and smoothing the depth map by using median filtering, bilateral filtering and other smoothing algorithms to improve the quality of the depth map. (5) integrating multiple depth maps into a unified and enhanced scene representation while reducing left-right inconsistency, completing the fusion of the depth map, and finally obtaining the dense point cloud.

[0089] In an embodiment, the above-mentioned sub-step S2032 can specifically include: determining two initial target images from the target images, calculating camera parameters corresponding to the two initial target images according to matching point pairs between the two initial target images, and calculating sparse point clouds corresponding to the two initial target images according to the camera parameters corresponding to the two initial target images; continuously adding new target images, calculating camera parameters and sparse point clouds corresponding to the new target images according to the matching relationship between the currently generated sparse point clouds and feature points in the new target images, and optimizing the camera parameters and the sparse point clouds corresponding to the new target images, until camera parameters and sparse point clouds corresponding to all target images are calculated; and adjusting the camera parameters and the sparse point clouds corresponding to the target images, so that the sum of re-projection errors of all homonymous points projected onto corresponding target images is minimized.

[0090] That is to say, the sparse reconstruction in the three-dimensional reconstruction process can include three parts: initial reconstruction, incremental reconstruction, and global optimization.

[0091] In the initial reconstruction process, two-view images (two initial target images, as shown in Figure 5 ) with more visible regions between cameras are selected for initialization. According to matching point pairs (as shown in Figure 6 ) between the two-view images, a more accurate point pair can be obtained by using a fast approximate nearest neighbor algorithm. Next, epipolar geometry constraints (as shown in Figure 7shown) to complete the initial camera pose estimation: let P be any point in space, p1, p2 be a pair of matched points in left and right images, O1, O2 be the camera optical center positions of left and right images, R, t be the rotation matrix and translation matrix of camera motion (i.e. camera extrinsic parameters). According to the pinhole camera model, the following can be obtained:

[0092] s1p1=KP, s2p2=K(RP+t) (1)

[0093] where s1, s2 represent the depth information corresponding to the two matched points, and K is the camera intrinsic matrix (i.e. camera intrinsic parameters). Using the form of homogeneous coordinates, since s1, s2 are non-zero values, the following can be obtained:

[0094] p1=KP, p2=K(RP+t) (2)

[0095] The point coordinates of p1, p2 on the normalized plane are denoted as x1, x2, and the above equation (2) can be rearranged as:

[0096] x2=Rx1+t (3)

[0097] Multiplying t on both sides of equation (3) ∧ and multiplying x2 on the left T , the following can be obtained:

[0098] x2 T t ∧ Rx1=0 (4)

[0099] where t ∧ represents the skew-symmetric matrix of the translation matrix, and by introducing the skew-symmetric matrix, the cross product of the matrix can be changed to the dot product, i.e. linear transformation, which is beneficial for calculation; x2 T represents the transpose of the coordinate vector x2.

[0100] Substituting p1, p2 back into equation (4), the following can be obtained:

[0101] p2 T K -T t ∧ RK -1 p1=0 (5)

[0102] From equation (5), the essential matrix E = t ∧ R and the fundamental matrix F = K -T EK -1 can be obtained.

[0103] The homography matrix describes the mapping relationship between two planes, and is usually used for the transformation relationship of points on the same plane. Let the plane equation be:

[0104]

[0105] where n represents the normal vector of the plane, d represents the distance from the origin to the plane, and P represents a spatial point.

[0106] According to the coplanar relationship of the feature matching points, the following equation can be obtained:

[0107]

[0108] According to equation (7), the homography matrix H can be obtained.

[0109] Using the obtained matching point pairs, the estimation of the fundamental matrix F, the essential matrix E, and the homography matrix H can be completed according to the classical eight-point method; then, the camera motion R and t information can be recovered using singular value decomposition (SVD).

[0110] According to Figure 7 , the motion equation between the point pairs can be obtained:

[0111] s1x1=s2Rx2+t (8)

[0112] where s1 and s2 are the depth values of the feature points. Multiplying equation (8) by a (x1 is the inverse matrix of x1), the following equation can be obtained:

[0113] s1x1 ∧ x1=0=s2x1 ∧ Rx2+x1 ∧ t (9)

[0114] In equation (9), the left side is 0, and the right side can be regarded as an equation of s2. Since R and t are already known, the depth value s2 can be solved using the least square method. According to the solved s2, s1 can also be solved. In this way, the depth values of the feature points on the two initial target images can be obtained, and the spatial coordinates of each feature point, i.e., the sparse point cloud, can also be obtained.

[0115] Incremental reconstruction is to reconstruct other images one by one on the basis of the initial reconstruction, and the process is carried out in a loop iteration manner. The main processes inside the loop include three sub-processes: camera localization based on perspective-n-point (PnP), scene expansion based on triangulation, and optimization of camera parameters and scene point cloud based on BA (Bundle Adjustment). The iteration process is repeated until all cameras are successfully localized or no camera can continue to be localized. (1) PnP is a method for solving 3D to 2D point pair motion, which describes how to estimate the pose of the camera when n 3D space points and their projection positions are known. In the initial reconstruction process, we get the 3D coordinates of the feature points in the seed image, and in the incremental reconstruction process, we construct the 3D to 2D point pair motion problem by adding images. There are many methods to solve the PnP problem, such as P3P, which estimates the pose with 3 pairs of points, DLT, EPnP, UPnP, etc.; it can also use nonlinear optimization methods to construct a least squares problem and iteratively solve it. (2) Scene expansion based on triangulation, taking the third target image as an example, according to the 3D points generated by the initial reconstruction and the 2D matching points of the third image, the PnP problem is constructed, and the camera pose corresponding to the third target image is estimated by using P3P / EPnP method, then more 3D coordinates of points are obtained by triangulation, and the step is repeated, finally the camera pose and 3D points corresponding to all target images are calculated. (3) Optimization of camera parameters and scene point cloud based on BA, in the process of solving camera parameters, the camera pose is first estimated by using P3P / EPnP method, and then the estimated value is adjusted by constructing a least squares optimization problem (i.e. BA).

[0116] Global optimization is to further optimize the camera pose and sparse point cloud by bundle adjustment algorithm after solving the camera parameters and sparse point cloud corresponding to all target images, so that the sum of the reprojection errors of all homonymous points projected onto the corresponding target images is minimized, and finally the optimized camera parameters and sparse point cloud corresponding to all target images are obtained. Among them, the homonymous points are the homonymous image points, that is, the image points formed by the same object point in different images.

[0117] p2=H 21 p1,p1=H 12 p2 (10)

[0118] The reprojection error is represented as:

[0119]

[0120] wherein is the completely matched point pair in the two target images estimated, is the estimated value in the reprojection error process.

[0121] In an embodiment, the point cloud filling of the weak texture region can be performed according to the point cloud data at the boundary line of the weak texture region. Please refer to Figure 8 The step S204 can include the following sub-steps:

[0122] Sub-step S2041, superimpose the boundary line of the weak texture region with the dense point cloud to obtain the point cloud data at the boundary line of the weak texture region.

[0123] Sub-step S2042, perform distribution statistics on the point cloud data at the boundary line of the weak texture region, and select the point cloud data within a preset range according to the statistical result.

[0124] For example, the normal distribution statistics can be performed on the point cloud data at the boundary line of the weak texture region, and the point cloud data within the two standard deviation ranges (or other number of standard deviation ranges) is extracted according to the statistical result.

[0125] Sub-step S2043, perform point cloud filling on the weak texture region according to the point cloud data within the preset range.

[0126] In the embodiment, since the point cloud data within the preset range belongs to the point cloud data at the boundary line of the weak texture region, the point cloud filling on the weak texture region according to the point cloud data within the preset range can effectively ensure the three-dimensional reconstruction effect.

[0127] In an embodiment, the sub-step S2043 can include: when the weak texture region is a water area, taking the average value of the elevation of the point cloud data within the preset range as the elevation value of the water area, and performing point cloud filling on the water area.

[0128] In another embodiment, the sub-step S2043 can include: when the weak texture region is a non-water region, calculating the normal vector of each point cloud in the point cloud data within the preset range, taking the average value of the normal vector of each point cloud as the vertical direction of the non-water region, and performing point cloud filling on the non-water region by using the point cloud data within the preset range and the vertical direction of the non-water region. The color of the point cloud filled in the non-water region is the color of the position of the non-water region in the image to be processed.

[0129] That is to say, in the point cloud filling of the weak texture region, the weak texture region can be processed respectively according to the type (water area, non-water area) of the weak texture region. For the water area, the average value of the elevation of the point cloud data in the preset range is counted as the elevation value of the water area, and the point cloud filling of the water area is completed; for the non-water area, the normal vector of each point cloud in the point cloud data in the preset range is calculated, the average value of the normal vector of each point cloud is obtained as the vertical direction of the non-water area, and then the plane fitting is performed by using the point cloud data in the preset range and the average value of the normal vector, so as to complete the point cloud filling of the non-water area. The color of the filled point cloud is the color of the same position area in the image. Figure 9 As shown in FIG. 10, it is a schematic diagram of the three-dimensional reconstruction effect of removing the holes. In this way, the occurrence of the holes and a large number of noise points in the three-dimensional reconstruction result is effectively avoided, and a better hole removal effect and three-dimensional reconstruction effect of the weak texture region are achieved.

[0130] In order to perform the corresponding steps in the above-mentioned embodiments and various possible manners, an implementation manner of a three-dimensional reconstruction device is given below. Please refer to Figure 10 FIG. 11 is a functional module diagram of the three-dimensional reconstruction device 500 provided by the embodiment of the present application. It should be noted that the three-dimensional reconstruction device 500 provided by the embodiment has the same basic principle and technical effects as the above-mentioned embodiments, and for brief description, the part not mentioned in the embodiment can refer to the corresponding content in the above-mentioned embodiments. The three-dimensional reconstruction device 500 includes a weak texture region extraction module 510, a mask processing module 520, a three-dimensional reconstruction module 530 and a point cloud filling module 540.

[0131] The weak texture region extraction module 510 is configured to extract the weak texture region in each to-be-processed image corresponding to the target scene.

[0132] It can be understood that the weak texture region extraction module 510 can perform the above-mentioned step S201.

[0133] The mask processing module 520 is configured to perform mask processing on the weak texture region in each to-be-processed image to obtain each target image.

[0134] It can be understood that the mask processing module 520 can perform the above-mentioned step S202.

[0135] The three-dimensional reconstruction module 530 is configured to perform three-dimensional reconstruction on the target region according to each target image to obtain the dense point cloud of the target region; the target region is a region in the target scene except the weak texture region.

[0136] It can be understood that the three-dimensional reconstruction module 530 can perform the above-mentioned step S203.

[0137] The point cloud filling module 540 is configured to fill the point cloud in the weak texture region according to the boundary line of the weak texture region and the dense point cloud.

[0138] It can be understood that the point cloud filling module 540 can perform the step S204 described above.

[0139] Optionally, the weak texture region extraction module 510 is specifically configured to extract the weak texture region in each to-be-processed image by using a pixel value in each to-be-processed image or a pre-trained detection model, or by selecting the weak texture region in each to-be-processed image corresponding to a target scene by a user.

[0140] The weak texture region extraction module 510 is configured to determine a first weak texture region in each to-be-processed image according to a pixel value in each to-be-processed image, and input each to-be-processed image into a pre-trained detection model to obtain a second weak texture region in each to-be-processed image, wherein the weak texture region includes the first weak texture region and the second weak texture region, and a reflectivity of the first weak texture region is higher than that of the second weak texture region.

[0141] Optionally, the three-dimensional reconstruction module 530 is specifically configured to perform feature point extraction and feature point matching on each target image to obtain a matching point pair between the target images, calculate camera parameters corresponding to each target image and a sparse point cloud according to the matching point pair, and generate a dense point cloud according to the camera parameters and the sparse point cloud.

[0142] The three-dimensional reconstruction module 530 is configured to determine two initial target images from the target images, calculate camera parameters corresponding to the two initial target images according to a matching point pair between the two initial target images, and calculate a sparse point cloud corresponding to the two initial target images according to the camera parameters corresponding to the two initial target images, constantly add a new target image, calculate camera parameters corresponding to the new target image and a sparse point cloud corresponding to the new target image according to a matching relationship between feature points in the new target image and the sparse point cloud that has been generated, and optimize the camera parameters corresponding to the new target image and the sparse point cloud corresponding to the new target image until camera parameters and sparse point clouds corresponding to all target images are calculated, and adjust the camera parameters and the sparse point clouds corresponding to the target images so that a sum of re-projection errors of all homonymous points projected onto corresponding target images is minimized.

[0143] It can be understood that the three-dimensional reconstruction module 530 can also perform the sub-steps S2031-S2033 described above.

[0144] Optionally, the point cloud filling module 540 is specifically configured to superimpose the boundary line of the weak texture region on the dense point cloud to obtain point cloud data at the boundary line of the weak texture region; perform distribution statistics on the point cloud data at the boundary line of the weak texture region, and select point cloud data within a preset range according to the statistical result; and perform point cloud filling on the weak texture region according to the point cloud data within the preset range.

[0145] In the process, when the weak texture region is a water region, the point cloud filling module 540 is configured to take the average height of the point cloud data within the preset range as the height value of the water region, and perform point cloud filling on the water region. When the weak texture region is a non-water region, the point cloud filling module 540 is configured to calculate the normal vector of each point cloud in the point cloud data within the preset range, take the average value of the normal vectors of the point clouds as the vertical direction of the non-water region, and perform point cloud filling on the non-water region by using the point cloud data within the preset range and the vertical direction of the non-water region. In this case, the color of the point cloud filled in the non-water region is the color of the position of the non-water region in the image to be processed.

[0146] It can be understood that the point cloud filling module 540 can also perform the above-mentioned sub-steps S2041-S2043.

[0147] It can be seen that the three-dimensional reconstruction device provided by the embodiment of the present application extracts the weak texture region in each image to be processed corresponding to the target scene by the weak texture region extraction module, performs mask processing on the weak texture region in each image to be processed by the mask processing module to obtain each target image, performs three-dimensional reconstruction on the target region according to each target image by the three-dimensional reconstruction module to obtain the dense point cloud of the target region, and performs point cloud filling on the weak texture region according to the boundary line of the weak texture region and the dense point cloud by the point cloud filling module. In this way, by extracting the weak texture region in each image to be processed corresponding to the target scene and performing mask processing on the weak texture region, the three-dimensional reconstruction of the weak texture region is avoided in the three-dimensional reconstruction process. After obtaining the dense point cloud of the region except the weak texture region in the target scene, the point cloud filling is performed on the weak texture region according to the boundary line of the weak texture region and the dense point cloud, that is, the three-dimensional reconstruction is not directly performed on the weak texture region, but the point cloud filling is performed on the weak texture region by using the point cloud data related to the boundary line of the weak texture region, so that the three-dimensional reconstruction process of the entire target scene is completed, the problem of unsatisfactory reconstruction effect of the weak texture region in the prior art is effectively solved, the phenomenon of holes and a large number of noise points in the three-dimensional reconstruction result is avoided, and a good hole removal effect is achieved. Since the weak texture region is subjected to mask processing in the three-dimensional reconstruction process, a large number of noise points will not appear in the obtained three-dimensional reconstruction result, and the accuracy of the point cloud data after three-dimensional reconstruction is effectively improved.

[0148] Please refer to Figure 11Fig. 7 shows a block diagram of an electronic device 700 according to an embodiment of the present application. The electronic device 700 includes a memory 710, a processor 720, and a communication module 730. The memory 710, the processor 720, and the communication module 730 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, the memory 710, the processor 720, and the communication module 730 can be electrically connected to each other through one or more communication buses or signal lines.

[0149] The memory 710 is configured to store programs or data. The memory 710 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or the like.

[0150] The processor 720 is configured to read / write the data or programs stored in the memory 710 and perform corresponding functions. For example, when the computer program stored in the memory 710 is executed by the processor 720, the three-dimensional reconstruction method disclosed in the above embodiments can be realized.

[0151] The communication module 730 is configured to establish a communication connection between the electronic device 700 and other communication terminals through a network and to receive / transmit data through the network.

[0152] It should be understood that, Figure 11 The structure shown in Fig. 7 is only a structural schematic diagram of the electronic device 700. The electronic device 700 can further include more or less components than those shown in Fig. 7 or have a different configuration from that shown in Fig. 7. Figure 11 The components shown in Fig. 7 can be realized in hardware, software, or a combination thereof. Figure 11 Figure 11 The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by the processor 720 to realize the three-dimensional reconstruction method disclosed in the above embodiments.

[0153] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by the processor 720 to realize the three-dimensional reconstruction method disclosed in the above embodiments.

[0154] ​In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0155] In addition, the function modules in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0156] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0157] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Extract the weak texture regions from each image to be processed corresponding to the target scene; The weak texture regions in each image to be processed are masked to obtain each target image; Based on the target images, a 3D reconstruction of the target region is performed to obtain a dense point cloud of the target region; the target region is the region in the target scene excluding the weak texture region. The boundary line of the weak texture region is superimposed on the dense point cloud to obtain the point cloud data at the boundary line of the weak texture region; the weak texture region is then filled with point cloud data based on the point cloud data at the boundary line of the weak texture region.

2. The method according to claim 1, characterized in that, The extraction of weak texture regions from each image to be processed corresponding to the target scene includes: Weak texture regions in each image to be processed can be extracted using pixel values ​​or a pre-trained detection model, or by having the user select weak texture regions in each image to be processed corresponding to the target scene.

3. The method according to claim 2, characterized in that, The step of extracting weak texture regions from each image to be processed using pixel values ​​or a pre-trained detection model includes: The first weak texture region in each image to be processed is determined based on the pixel values ​​in each image to be processed. Each image to be processed is input into a pre-trained detection model to obtain a second weak texture region in each image to be processed; wherein, the weak texture region includes a first weak texture region and a second weak texture region, and the reflectivity of the first weak texture region is higher than that of the second weak texture region.

4. The method according to claim 1, characterized in that, The step of filling the weak texture region with point cloud data based on the point cloud data at the boundary line of the weak texture region includes: The distribution statistics of point cloud data at the boundary line of the weak texture region are performed, and point cloud data within a preset range are selected based on the statistical results. The weak texture region is filled with point cloud data within the preset range.

5. The method according to claim 4, characterized in that, The step of filling the weak texture region with point cloud data within the preset range includes: When the weak texture region is a body of water, the average elevation of the point cloud data within the preset range is used as the elevation value of the body of water, and the point cloud is filled into the body of water.

6. The method according to claim 4, characterized in that, The step of filling the weak texture region with point cloud data within the preset range includes: When the weak texture region is a non-water region, the normal vector of each point cloud in the point cloud data within the preset range is calculated, the average value of the normal vectors of each point cloud is taken as the vertical direction of the non-water region, and the point cloud is filled in the non-water region using the point cloud data within the preset range and the vertical direction of the non-water region.

7. The method according to claim 1, characterized in that, The step of performing three-dimensional reconstruction of the target region based on the target images to obtain a dense point cloud of the target region includes: Feature points are extracted and matched for each target image to obtain matching point pairs between the target images. Calculate the camera parameters and sparse point cloud corresponding to each target image based on the matching point pairs; A dense point cloud is generated based on the camera parameters and the sparse point cloud.

8. A three-dimensional reconstruction device, characterized in that, The device includes: The weak texture region extraction module is used to extract weak texture regions in each image to be processed corresponding to the target scene; The masking module is used to perform masking on the weak texture regions in each image to be processed, so as to obtain each target image. The 3D reconstruction module is used to perform 3D reconstruction of the target region based on the target images to obtain a dense point cloud of the target region; the target region is the region in the target scene other than the weak texture region. The point cloud filling module is used to overlay the boundary line of the weak texture region with the dense point cloud to obtain the point cloud data at the boundary line of the weak texture region; and to fill the weak texture region with point cloud data based on the point cloud data at the boundary line of the weak texture region.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the three-dimensional reconstruction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the three-dimensional reconstruction method as described in any one of claims 1-7.

Images