Plane prior optimization dense matching system and method considering texture and structural features

Abstract

The present invention discloses a plane prior optimization dense matching system and method taking into account texture and structural features, belonging to the field of image dense matching technology. The present invention includes the following steps: scene texture feature extraction; scene structure feature extraction; multi-view dense matching of plane prior optimization taking into account texture features and structural features; multi-feature constrained depth information block fusion to generate the final dense point cloud. The present invention not only improves the quality of dense point clouds in weak texture areas, but also can realize dense reconstruction of large scenes by scene segmentation, which can reduce computer resources and speed up dense reconstruction. The plane prior based on scene texture features and structural features is used to assist in optimizing the depth information of weak texture areas, so as to achieve accurate calculation of the depth information of weak texture areas and improve the fusion quality of dense point clouds, thereby obtaining more accurate and reliable dense point clouds.

Description

Technical Field

[0001] The invention relates to a plane priori optimized dense matching system and method taking into account texture and structural features, belonging to the technical field of image dense matching. Background Art

[0002] 3D Real Scene (3DRS) is a digital virtual space that reflects and expresses human production, life and ecological space in a real, three-dimensional and time-sequential manner. It is a new type of basic surveying and mapping standardized product, providing a unified spatial foundation for economic and social development and the informatization of various departments. With the rapid development of technologies such as multi-view image matching in the field of computer vision, it has become a reality to reproduce the three-dimensional model of the real world with computers. Therefore, three-dimensional model reconstruction technology has emerged.

[0003] As an important part of the construction of geographical scenes under the "Realistic 3D China Construction" system, the oblique photography 3D model can intuitively and realistically restore the city's appearance and bring great convenience to urban planning and construction. In recent years, with the rapid development of technologies such as unmanned aerial vehicles (UAVs) and multi-view image matching of computer vision, 3D modeling technology based on oblique photography has become the most promising automated 3D modeling method. Oblique photography 3D reconstruction technology repeatedly collects images of the same area from different angles in a large-scale, high-precision, and high-definition manner. It has high-performance collaborative parallel processing capabilities. After the collected images are processed through overall aerial triangulation, multi-view dense matching, mesh reconstruction, and texture reconstruction, high-precision, high-realism, and measurable oblique photography 3D models can be obtained. At present, this technology has been widely used in many fields such as cultural relics protection, disaster relief, national security, urban management, surveying and mapping. However, the dense matching method of the existing method still has problems such as reconstruction difficulties or incomplete reconstruction results in some areas (such as areas with weak textures, high reflectivity, and repeated textures), which reduces the quality of the model and affects the visualization effect of the 3D model. It cannot fully meet the needs of applications such as high-precision real-life 3D model reconstruction and smart cities. For example, the patent publication number is: CN117557617A, which discloses a multi-view dense matching method, system and device based on plane prior optimization. Based on the ACMP algorithm, a multi-view dense matching method based on plane prior assisted optimization with scene line feature constraints is designed. However, this method introduces a line segment detector (LSD) algorithm for feature line extraction, and only considers local gradient features, resulting in too much plane prior information in planar areas similar to chessboard features; in addition, during depth propagation, only the depth information of pixels with better matching costs is selected to update the candidate hypothesis of the current pixel, without considering whether the two pixels represent the same object, resulting in erroneous propagation of depth information. Based on the problems existing in the above-mentioned prior art, a robust, accurate, multi-feature dense matching method has become an urgent need. Summary of the invention

[0004] In view of the deficiencies of the prior art, the purpose of the present invention is to provide a plane priori optimized dense matching system and method taking into account texture and structural features, thereby solving the problems arising in the prior art.

[0005] The plane prior optimization dense matching method considering texture and structural features of the present invention comprises the following steps:

[0006] S1: Scene texture feature extraction: Based on the grayscale image of the reference image, the texture features corresponding to the reference image are extracted by using the weighted value of the sum of the absolute values ​​of the gradients in the horizontal and vertical directions and the standard deviation in the local regular window;

[0007] S2: scene structure feature extraction: adaptive binarization is performed based on the texture features extracted in step S1 to generate accurate scene structure features for the reference image;

[0008] S3: Multi-view dense matching with plane prior optimization under texture feature perception propagation: Matching points with different matching costs are selected as anchor points based on the scene structure features of the reference image, a Delaunay triangulation is constructed and plane prior information is generated. The depth map estimation of the weak texture area is optimized based on the plane prior information under texture feature perception propagation, and accurate depth information is generated for each image.

[0009] S4: Multi-feature constrained depth information block fusion: Multiple features are used as fusion constraint factors to fuse redundant depth information to generate object cube 3D points and generate the final dense point cloud.

[0010] Furthermore, the scene texture feature extraction in step S1 specifically includes the following sub-steps:

[0011] S11: Convert the color image of the reference image into a grayscale image based on a color weighted average method, wherein the weighted average method calculation formula is as follows:

[0012] (1)

[0013] In the formula, , and The red, green and blue bands of the reference image are in row and column coordinates The pixel value of .

[0014] S12: Based on the grayscale image, the scene texture feature is extracted by using the weighted value of the sum of the absolute values ​​of the gradients in the horizontal and vertical directions in the local regular window and the standard deviation. The weighted value calculation formula is as follows:

[0015] (2)

[0016] In the formula, , and They are the texture information, gradient and standard deviation of the corresponding pixels respectively; are the weights of the two, and both weight values ​​are taken as 0.5. The calculation formulas for the gradient and standard deviation are as follows:

[0017] (3)

[0018] (4)

[0019] In the formula, It is a rectangular window of 11*11 that is consistent with the NCC cost; is the pixel value within the rectangular window; is the mean gray value of the window pixels; is the total number of pixels in the rectangular window.

[0020] S13: Repeat the above calculation pixel by pixel to obtain the texture information of all pixels of the image and realize the scene texture feature extraction of the image.

[0021] Furthermore, the scene structure feature extraction in step 2 specifically includes the following sub-steps:

[0022] S21: Based on the scene texture features obtained in step 1, a Gaussian kernel function is used Perform local neighborhood Gaussian convolution operation to obtain the texture result after Gaussian filtering. ;

[0023] (5)

[0024] In the formula, is the Gaussian kernel function; is the Gaussian convolution operator.

[0025] S22: Based on the above texture results , using adaptive binarization processing method to obtain the scene structure feature attribute label corresponding to the reference image , 0 is the non-scene structure area, and 255 is the scene structure area. Its mathematical expression is:

[0026] (6)

[0027] S23: Repeat the above calculation pixel by pixel to obtain the structural feature attribute labels of all pixels of the image, thereby realizing the scene structure feature extraction of the image.

[0028] Furthermore, the multi-view dense matching of plane prior optimization under texture feature perception propagation in step S3 specifically includes the following sub-steps:

[0029] S31: Use ACMH algorithm for initial matching, and select matching cost in The matching points in the interval are taken as candidate structure points; then the corresponding scene structure attribute labels are obtained based on the pixel coordinates of the candidate structure points;

[0030] S32: Use the Delaunay algorithm to construct the network and generate triangular primitives. For each triangular primitive, use the three vertex coordinates of the primitive to calculate the plane structure information in the reference image camera coordinate system, that is, the mathematical expression of the plane equation , to extract the plane prior information, that is, the depth information of the plane structure .

[0031] (7)

[0032] (8)

[0033] In the formula, is the normal vector Length.

[0034] S33: According to the distance of the propagation path, the neighborhood pixels are divided into multiple layers, and two pixels with similar texture features and the lowest matching cost are selected in each layer, that is, pixels with similar texture features and reliable depth information, to complete the depth propagation at the pixel level. The depth information of the above reliable pixels is used As the candidate hypothesis of the current pixel to be updated, the matching cost of the candidate hypothesis is calculated according to the optical metric function, and the calculation formula is as follows:

[0035] (9)

[0036] in: is the visibility weight of the source image; For the current pixel In-depth information The photometric consistency of is the NCC matching cost;

[0037] S34: Based on the matching cost results of the above candidate hypotheses, texture features are used for constraints to optimize the matching with the current pixel The matching cost of consistent texture features is calculated as follows under texture feature perception propagation:

[0038] (10)

[0039] in: is the texture feature similarity measurement parameter; It is a numerical constant, and its value is 1.5; and are the texture feature values ​​of the reference image and the source image respectively. The second term on the right side of the equation is the matching cost, corresponding to the current pixel Depth information The first term is the regular term of the plane prior. If the texture features of the source image are consistent with those of the reference image, a reward is given, otherwise a penalty is given to encourage Propagation of depth information with similar texture features.

[0040] S35: Based on the maximum a posteriori probability estimation formula, a matching cost probability function is constructed, and its calculation formula is as follows:

[0041] (11)

[0042] S36: Use plane structure information to assist in optimizing dense matching and construct a plane optimization probability function. The calculation formula is as follows:

[0043] (12)

[0044] S37: By aggregating the above two probability functions, a plane prior optimization cost function under texture feature perception is constructed, and its calculation formula is as follows:

[0045] (13)

[0046] in: is the depth information of the current pixel, For the current pixel Corresponding to the depth information of the planar structure, , the first term on the right side of the equation is the matching cost under texture feature perception propagation, corresponding to the current pixel Depth information The photometric consistency of the second term is the regularization term of the plane prior. If the current pixel If the depth information of the weak texture area is consistent with the depth information of the planar structure, a reward is given, otherwise a penalty is given, thereby promoting the depth information of the weak texture area to tend to the depth information of the planar structure.

[0047] Furthermore, in step S4, a multi-feature constraint factor is used in the multi-feature constrained depth information block fusion to fuse the redundant depth information to generate the object cube three-dimensional point and generate the final dense point cloud, which specifically includes the following sub-steps:

[0048] S41: Based on the given reconstruction range, block size and block overlap range, a regular grid block strategy is used to divide the reconstruction range into multiple sub-blocks to achieve large-scale scene segmentation;

[0049] S42: Based on the above block results, first use the elevation of the sparse point cloud within the sub-block range to predict the surface undulation within the sub-block range and find the minimum elevation within the sub-block range. and maximum elevation ;

[0050] S43: Calculate reference image The intersection of the principal ray with the minimum and maximum elevation surfaces and , project the two intersection points to the matching images with the longest baseline distance in the adjacent image set of the reference image , and obtain the corresponding projection points and , calculate the ratio of the elevation difference to the Euclidean distance of the projection point as a reference Elevation step distance;

[0051] S44: Calculate the average elevation step of all images as the elevation step of the voxel grid ; Finally, the spatial resolution of the image at the principal point is calculated respectively, so as to calculate the average spatial resolution of all images as the horizontal step size of the voxel grid .

[0052] Furthermore, step S43 specifically includes the following sub-steps:

[0053] S431: Reference image Pixels Project to object space to get a 3D point If it is outside the reconstruction range, the pixel is not fused, otherwise the 3D point is calculated. The incident light With normal Angle ,like , then the pixel is not fused;

[0054] S432: 3D points Project to adjacent image sets respectively superior , and calculate its projection depth ;

[0055] S433: Obtaining adjacent images at the projection point Depth , and projected to the object space to obtain a three-dimensional point , compare the depth and object distance difference between the reference image and the adjacent image, if the depth difference exceeds the threshold Or the object distance difference exceeds , then this point does not participate in the fusion;

[0056] S434: Count the number of fusion points ,like , and get the final three-dimensional fusion point , the fusion formula is as follows:

[0057] (14)

[0058] (15)

[0059] in: is the Euclidean distance in object space, where ,also, Adaptively take twice the spatial resolution corresponding to the current pixel, For adjacent images at the projection point The photometric consistency at , i.e., the NCC matching cost.

[0060] Furthermore, in step S4, the multi-feature constraint factors include sub-block range, angle, object space distance and number of fusion points.

[0061] The plane prior optimization multi-view dense matching system taking into account scene texture features and structural features of the present invention comprises:

[0062] A scene texture feature extraction module is used to extract scene texture features and generate texture feature information for each pixel;

[0063] The scene structure feature extraction module is used to extract structural texture features and generate stable scene structure information for each image;

[0064] A multi-view dense matching module with plane prior optimization under texture feature-aware propagation to generate accurate depth information for each image;

[0065] The depth information block fusion module is used to fuse the redundant depth information to generate the object's three-dimensional points and generate the final dense point cloud.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The plane prior optimization dense matching system and method taking into account texture and structural features described in the present invention not only improves the quality of dense point clouds in weak texture areas, but also can use scene segmentation to achieve dense reconstruction of large scenes, which can reduce computer resources and speed up dense reconstruction. It can use scene texture features to select pixels with similar texture features and the lowest matching cost as candidate hypotheses for the current pixels to be updated, and use plane priors based on scene structural information to assist in optimizing the depth information of weak texture areas, so as to achieve accurate calculation of the depth information of weak texture areas, and complete the fusion of depth maps based on a multi-feature constraint fusion algorithm, thereby improving the fusion quality of dense point clouds and obtaining more accurate and reliable dense point clouds. The problems in the prior art are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is an overall flow chart of an embodiment of the present invention;

[0069] Figure 2 Schematic diagram of scene texture feature and structure information extraction in an embodiment of the present invention; Figure 2 (a) is the result of the original image extraction; Figure 2 (b) is the result image of scene texture feature extraction; Figure 2 (c) extracting a structural graph for structural features;

[0070] Figure 3 Schematic diagram of chessboard propagation in an embodiment of the present invention; Figure 3(a) is the distribution diagram of the chessboard, Figure 3 (b) is the pixel propagation path, Figure 3 (c) is the domain pixel map;

[0071] Figure 4 A schematic diagram of scene segmentation in an embodiment of the present invention;

[0072] Figure 5 Schematic diagram of voxel grid step calculation in an embodiment of the present invention; Figure 5 (a) is the horizontal step calculation process diagram. Figure 5 (b) is the elevation step calculation process diagram;

[0073] Figure 6 This is a schematic diagram of depth information fusion in an embodiment of the present invention;

[0074] Figure 7 is a result diagram of plane structure information extraction in an embodiment of the present invention; wherein Figure 7 (a) is the plane structure information diagram of the ACMP method. Figure 7 (b) is a planar structure information diagram of the method of the present invention;

[0075] Figure 8 is a result diagram of depth information calculation in an embodiment of the present invention; wherein Figure 8 (a) is the original image, Figure 8 (b) is the image processed by openMVS method. Figure 8 (c) is the image processed by ACMP method, Figure 8 (d) processing an image by the method of the present invention;

[0076] Fig. 9 is a diagram showing the depth information fusion result in an embodiment of the present invention; wherein Fig. 9 (a) is the dense point cloud fusion result of the openMVS method. Fig. 9 (b) is the result of ACMP method dense point cloud fusion. Fig. 9 (c) is a dense point cloud fusion structure diagram of the method of the present invention;

[0077] Fig.10 is a result diagram of the vertical direction of the dense point cloud in an embodiment of the present invention; wherein Fig.10 (a) is the dense point cloud elevation map of the openMVS method. Fig.10 (b) is the dense point cloud elevation map of the ACMP method. Fig.10 (c) is the elevation view of the dense point cloud obtained by the method of the present invention. DETAILED DESCRIPTION

[0078] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0079] Embodiment 1:

[0080] like Figure 1 As shown, the plane prior optimization dense matching method considering texture and structural features of the present invention comprises the following steps:

[0081] S1: Scene texture feature extraction: Based on the grayscale image of the reference image, the texture features corresponding to the reference image are extracted by using the weighted value of the sum of the absolute values ​​of the gradients in the horizontal and vertical directions and the standard deviation in the local regular window;

[0082] S2: scene structure feature extraction: adaptive binarization is performed based on the texture features extracted in step S1 to generate accurate scene structure features for the reference image;

[0083] S3: Multi-view dense matching with plane prior optimization under texture feature perception propagation: Based on the scene structure features of the reference image, stable matching points with different matching costs are selected as anchor points, a Delaunay triangulation is constructed and plane prior information is generated. The depth map estimation of the weak texture area is optimized based on the plane prior information under texture feature perception propagation, and accurate depth information is generated for each image.

[0084] S4: Multi-feature constrained depth information block fusion: Multiple features are used as fusion constraint factors to fuse redundant depth information to generate object cube 3D points and generate the final dense point cloud.

[0085] The method of this embodiment not only improves the quality of dense point clouds in weak texture areas, but also uses scene segmentation to achieve dense reconstruction of large scenes, which can reduce computer resources and speed up dense reconstruction. It can use scene texture features to select pixels with similar texture features and the lowest matching cost as candidate hypotheses for the current pixels to be updated, and use plane priors based on scene structure information to assist in optimizing the depth information of weak texture areas, so as to achieve accurate calculation of the depth information of weak texture areas, and complete the fusion of depth maps based on multi-feature constraint fusion algorithms, thereby improving the fusion quality of dense point clouds and obtaining more accurate and reliable dense point clouds.

[0086] In this embodiment, the specific steps of scene texture features are as follows:

[0087] First, the color image of the reference image is converted into a grayscale image based on the color weighted average method; then the scene texture feature is extracted by using the weighted value of the sum of the absolute values ​​of the gradients in the horizontal and vertical directions in the local regular window and the standard deviation to achieve the extraction of scene texture features. The scene texture feature calculation formula is as follows:

[0088] (2)

[0089] (3)

[0090] (4)

[0091] In the formula, are the weights of the two, and both weight values ​​are taken as 0.5; It is a rectangular window of 11*11 that is consistent with the NCC cost; is the pixel value within the rectangular window; is the mean gray value of the window pixels; is the total number of pixels in the rectangular window; , and They are the texture information, gradient and standard deviation of the corresponding pixels respectively. The scene texture feature extraction results are as follows Figure 2 (b) as shown.

[0092] In this embodiment, the specific steps of scene structure feature extraction are as follows:

[0093] Based on the above scene texture feature extraction results, the Gaussian kernel function is first used Perform local neighborhood Gaussian convolution operation to obtain the texture result after Gaussian filtering. ; Then, the adaptive binarization processing function is used to obtain the scene structure features to achieve the extraction of scene structure features. The scene structure feature calculation formula is as follows:

[0094] (5)

[0095] (6)

[0096] In the formula, is the Gaussian kernel function; is the Gaussian convolution operator; The adaptive binarization method is used to obtain the scene structure feature attribute label corresponding to the reference image. 0 represents the non-scene structure area and 255 represents the scene structure area. The scene structure feature extraction result is as follows: Figure 2 (c) as shown.

[0097] In this embodiment, the specific steps of multi-view dense matching of plane prior optimization under texture feature perception propagation are as follows:

[0098] Select the matching cost at The matching points are taken as candidate structure points; then based on the pixel coordinates of the candidate structure points Get the corresponding scene structure attribute label. If it is a non-scene structure area, select the matching cost as , improve the quality of structure points and reduce the number of structure points. If the label is the scene structure area, the matching cost is , while maintaining the quality, the number of structural points is increased; finally, the Delaunay algorithm is used to construct the network and generate triangular primitives. For each triangular primitive, the three vertex coordinates of the primitive are used to calculate the plane structure information in the reference image camera coordinate system, that is, the plane equation, to realize the extraction of plane prior information.

[0099] Figure 3 (a) is the distribution diagram of the chessboard. According to the distance of the propagation path, the propagation path is as follows Figure 3 As shown in (b), the neighborhood pixels are divided into multiple layers. In each layer, two pixels with similar texture features and the lowest matching cost are selected, that is, pixels with similar texture features and reliable depth information, to complete the depth propagation at the pixel level. The depth propagation under texture feature perception is shown in the figure below. Figure 3 (c) shows that the depth information of the above reliable pixels is used. As the candidate hypothesis of the current pixel to be updated, the matching cost of the candidate hypothesis is calculated according to the optical metric function, and the calculation formula is as follows:

[0100] (9)

[0101] Where: is the visibility weight of the source image; For the current pixel In-depth information The photometric consistency of is the NCC matching cost.

[0102] Based on the matching cost results of the above candidate hypotheses, texture features are used for constraints to optimize the matching with the current pixel. The matching cost of consistent texture features is calculated as follows under texture feature perception propagation:

[0103] (10)

[0104] in: is the texture feature similarity measurement parameter; It is a numerical constant, and its value is 1.5; and are the texture feature values ​​of the reference image and the source image respectively. The second term on the right side of the equation is the matching cost, corresponding to the current pixel Depth information The first term is the regular term of the plane prior. If the texture features of the source image are consistent with those of the reference image, a reward is given, otherwise a penalty is given to encourage Propagation of depth information with similar texture features.

[0105] Based on the maximum a posteriori probability estimation formula, a matching cost probability function is constructed, and its calculation formula is as follows:

[0106] (11)

[0107] Plane structure information is used to assist in optimizing dense matching and construct a plane optimization probability function. The calculation formula is as follows:

[0108] (12)

[0109] By aggregating the above two probability functions, a plane prior optimization cost function under texture feature perception is constructed, and its calculation formula is as follows:

[0110] (13)

[0111] in: is the depth information of the current pixel, For the current pixel Corresponding to the depth information of the planar structure, The first term on the right side of the equation is the matching cost under texture feature perception propagation, corresponding to the current pixel Depth information The photometric consistency of the second term is the regularization term of the plane prior. If the current pixel If the depth information of the weak texture area is consistent with the depth information of the planar structure, a reward is given, otherwise a penalty is given, thereby promoting the depth information of the weak texture area to tend to the depth information of the planar structure.

[0112] In this embodiment, the specific steps of multi-feature constrained depth information block fusion are as follows:

[0113] Using multiple feature constraint factors such as "sub-block range", "angle", "object space distance" and "number of fusion points", the redundant depth information is fused to generate object space 3D points and generate the final dense point cloud. The implementation method is as follows:

[0114] Based on the given reconstruction range, block size and block overlap range, a regular grid block strategy is used to divide the reconstruction range into multiple sub-blocks to achieve large-scale scene segmentation. The segmentation process is as follows: Figure 4 As shown in the figure. The black line is the reconstruction range line, and the gray line is the scene segmentation line;

[0115] Based on the above block results, the elevation of the sparse point cloud within the sub-block range is first used to predict the surface undulation within the sub-block range, and the minimum elevation within the sub-block range is found. and maximum elevation ; then calculate the reference image The intersection of the principal ray with the minimum and maximum elevation surfaces and , project the two intersection points to the matching images with the longest baseline distance in the adjacent image set of the reference image , and obtain the corresponding projection points and , calculate the ratio of the elevation difference to the Euclidean distance of the projection point as a reference The height step distance is calculated as follows: Figure 5 (b) is shown. The average elevation step of all images is calculated as the elevation step of the voxel grid. ; Finally, the spatial resolution of the image at the principal point is calculated respectively, so as to calculate the average spatial resolution of all images as the horizontal step size of the voxel grid , the calculation process is as follows Figure 5 (a) shown.

[0116] Point cloud fusion refers to the use of multi-feature constraints to fuse redundant depth information to generate three-dimensional points. The fusion process is as follows: Figure 6 As shown, first the reference image Pixels Project to object space to get a 3D point If it is outside the reconstruction range, the pixel is not fused, otherwise the 3D point is calculated. The incident light With normal Angle ,like , then the pixel is not fused; then the 3D point Project to adjacent image sets respectively superior , and calculate its projection depth ; Then get the adjacent image at the projection point Depth , and projected to the object space to obtain a three-dimensional point , according to formula (11), the difference in depth and object distance between the reference image and the adjacent image is compared. If the depth difference exceeds the threshold Or the object distance difference exceeds , then the point does not participate in the fusion; finally, the number of fusion points is counted ,like , and get the final three-dimensional fusion point The fusion formula is as follows:

[0117] (14)

[0118] (15)

[0119] In the formula, is the Euclidean distance in object space, where ,also, Adaptively take twice the spatial resolution corresponding to the current pixel. For adjacent images at the projection point The photometric consistency at , i.e., the NCC matching cost.

[0120] In order to verify the effectiveness and superiority of the algorithm of the present invention, multi-view oblique image data with different camera resolutions are used. The present invention uses the results of sparse reconstruction (internal and external parameters and sparse point clouds) as the input data of the present method, and compares them with open source frameworks such as OpenMVS and ACMP. OpenMVS is a method proposed by Shen in 2013, which is an efficient multi-view dense matching method based on PatchMatch using CPU multi-threading; ACMP is a plane prior optimization method proposed by Xu in 2020, which is a multi-view dense matching method based on PatchMatch using GPU acceleration. The present method is improved on the basis of ACMP, and the experimental results of the present method are compared and analyzed with those of OpenMVS and ACMP methods. The experimental content includes four parts: the results of plane prior information extraction, the results of depth information matching, the results of depth information fusion, and the accuracy comparison and analysis of dense point clouds.

[0121] 1. Plane prior information extraction results and analysis

[0122] Plane prior information refers to selecting matching points with good confidence as anchor points, using the Delaunay algorithm to construct a network and generate plane structure information, and optimizing the depth information of weak texture areas. Based on the existing method ACMP, the present invention has made improvements and optimized the extraction of plane prior information. The building area with more complex reconstruction scenes is selected for experiments and analysis. The plane prior information extraction results of different methods are as follows: Figure 7 shown.

[0123] from Figure 7 As can be seen from (a), in the ACMP method, in strong texture areas, the triangle primitives are relatively small, which can better maintain the scene structure; in weak texture areas, the triangle primitives are usually relatively large, which can express the scene structure of the plane area. However, in some strong texture plane areas, such as Figure 7 (a) In the black rectangular area, the triangular primitives of the roof and the sports field are too fragmented, which will weaken the flatness of the dense point cloud in these areas. Based on the previous method, this method introduces scene structure information, which can not only maintain the triangular primitives in the area with sudden depth value changes in the scene (such as eaves, etc.), but also Figure 7 (b) The black circle area can also reduce the number of triangle primitives in the scene where the depth value difference is not large (such as the roof, etc.). Figure 7 (b) Black rectangular area. This can optimize the flatness of the dense point cloud in the plane area.

[0124] 2. Depth information calculation results and analysis

[0125] PatchMatch is a multi-view dense matching method that can generate more accurate depth information. Based on the existing method ACMP, this paper makes improvements and optimizes the dense matching of weak texture areas. The building area with more complex reconstruction scenes is selected for experiment and analysis. The number of adjacent images is 6 and the number of iterations is 3. The depth information calculation results of different methods are shown in the following figure. Figure 8 shown.

[0126] from Figure 8 It can be seen that the three different methods can obtain good depth information results in strong texture areas, such as building areas, with little difference; but in weak texture areas, such as the red circle area, the depth information calculated by OpenMVS and ACMP methods has more noise, while the depth information calculated by this paper is smoother. The OpenMVS method only relies on the photometric consistency of the image to evaluate the quality of the match between images. In weak texture areas, photometric consistency is no longer reliable, resulting in more abnormal depth information, such as Figure 8 (b) As shown. The ACMP method introduces plane prior information based on photometric consistency and optimizes the depth information calculation of weak texture areas; however, the number of plane prior information in this method is large, and it only relies on photometric consistency without considering the scene structure information, so it can partially alleviate the abnormal depth information in weak texture areas, such as Figure 8 (c) As shown. Based on ACMP, this method introduces the initialization of depth information based on sparse point clouds and the construction of plane prior information taking into account scene structure information, which can better improve the problem of inaccurate depth information calculation in weak texture areas, such as Figure 8 (d) as shown.

[0127] 3. Deep information fusion results and analysis

[0128] Depth information fusion can fuse redundant depth information to generate a complete and unified scene, thereby obtaining a dense point cloud of the entire scene. Based on the existing method OpenMVS, the present invention has made improvements and optimized the fusion process of depth information. Similarly, a building area with a relatively complex reconstruction scene is selected for experiment and analysis. The depth information fusion results of different methods are shown in Figure 2. Fig. 9 shown.

[0129] from Fig. 9 It can be seen that the three different methods can obtain point clouds with high density. In the building area, the fusion algorithm of OpenMVS only considers the depth difference as a constraint to fuse the depth information, which leads to the divergence of dense point clouds, more noise points, and poor preservation of building angular structures, such as Fig. 9(a) Black circle area; ACMP and the proposed method optimize the fusion algorithm by introducing multiple feature constraints such as angle, object distance and number of fusion points. Therefore, the dense point cloud converges, the noise is less, and the angular structure of the building remains intact, such as Fig. 9 (b) and 9(c) black circle areas. In addition, in the plane area, the OpenMVS fused dense point cloud diverges, has more noise, and the plane structure is not well maintained. 9(a) black rectangle and square plane 1 area; because the plane prior information of ACMP is more fragmented in the plane area with strong texture, the dense point cloud is missing, such as Fig. 9 (b) Black rectangular area; The proposed method optimizes the plane prior information by introducing scene structure information, and the dense point cloud in the weak texture area is reconstructed completely, such as Fig. 9 (c) Black rectangle and square plane 1 regions.

[0130] from Fig.10 It can be seen that the dense point cloud fused by the OpenMVS method has more noise points, and the dense point cloud in the plane area is very uneven, such as Fig.10 (a) Black circle area; ACMP and the proposed method use the same optimized fusion method, the dense point cloud generated by fusion has fewer noise points, and the dense point cloud in the plane area is relatively smooth, such as Fig.10 (b) and 10(c) black circle areas. In addition, in the small parts area, since ACMP uses randomly initialized depth information, the reconstruction of small parts is incomplete, e.g. Fig.10 (b) White circle area; The proposed method uses sparse point cloud to initialize depth information, which accelerates the convergence of depth information, so that small parts can also maintain relatively complete structural information, such as Fig.10 (c) White circle area.

[0131] 4. Quantitative analysis of dense point clouds

[0132] (1) Quantitative analysis of point cloud accuracy

[0133] The dense point cloud results of the roof and ground are shown in Figure 9-10 As can be seen from the figure, the results of the point cloud generated by the fusion of different matching methods in the weak texture area are different. The ACMP method only uses points with relatively good matching costs to form a plane prior to optimize the multi-view dense matching. The number of plane priors in the weak texture area is large, resulting in the generated dense point cloud being not smooth enough. On this basis, the method in this paper adopts a multi-view dense matching method that takes into account the plane prior optimization of the scene structure information, optimizes the number of plane priors in the weak texture area, and can improve the flatness of the generated dense point cloud. In order to more effectively verify the above analysis results, we use local plane accuracy to evaluate the impact of plane prior on dense matching results. Local plane accuracy refers to the root mean square error (RMSE) of the distance from the dense point cloud to the local fitting plane, that is, flatness.

[0134] The two methods calculate the distance to the fitting plane in different plane areas, and the distance and flatness statistics are shown in Table 1.

[0135] Table 1. Comparison of flatness of dense point clouds generated by two methods

[0136]

[0137] Note: The unit of flatness of statistical dense point cloud is m.

[0138] It can be found from Table 1 that in different test areas, the flatness of the dense point cloud generated by the multi-view dense matching fusion of the plane prior optimization of the proposed method is about 26.98% higher than that of the ACMP method on average. In addition, in the building plane area, due to the relatively flat scene structure without large fluctuations, the flatness of the ACMP method is slightly greater than one pixel resolution, while the flatness of the dense point cloud of the proposed method is slightly less than one pixel resolution. This is mainly because the proposed method introduces depth information initialization and scene structure information. Under the same iteration conditions, the proposed method can converge faster and have higher flatness, which is about 30.95%. In the building facade area, due to the serious deformation of the oblique image and the influence of factors such as windows in the scene, there are a few fluctuations. The flatness of the ACMP method and the proposed method is greater than that of the plane area, which is about twice the flatness of the plane area. However, the flatness of the proposed method is still better than that of the ACMP method, with an increase of about 23.02%. Therefore, the proposed method can improve the dense matching accuracy of different reconstruction areas, reflecting the effectiveness and superiority of the proposed method.

[0139] (2) Quantitative analysis of point cloud density

[0140] The dense point cloud results of the roof and ground are shown in Figure 9-10 As can be seen from the figure, different fusion methods generate different point cloud densities in weak texture areas. The fusion algorithm used by OpenMVS only uses depth consistency and the number of visible images as constraints, and the generated dense point cloud has a lot of redundancy and noise; however, the method in this paper uses a deep information fusion method of voxel grid screening with multi-feature constraints, which can greatly reduce the number of dense point clouds, thereby greatly improving the efficiency of mesh reconstruction.

[0141] The statistics of the number of dense point clouds in different block areas of the two methods are shown in Table 2.

[0142] Table 2 Comparison of the number of dense point clouds generated by the two methods

[0143]

[0144] Note: The number of dense point clouds is counted.

[0145] It can be found from Table 2 that in different block test areas, the number of dense point clouds generated by the fusion of this method is reduced by an average of 39.3%, which can reduce the redundancy and noise problems of dense point clouds. In addition, the number of dense point clouds in the edge area has decreased, with an average reduction of 18.7%, mainly because the sub-block is located in the edge area of ​​the entire reconstructed scene, where the number of redundant images is small. The dense point cloud in the central area has increased significantly, with an average reduction of 56.3%, mainly because the sub-block is located in the central area of ​​the entire reconstructed scene, where the number of redundant images is large. Therefore, the method in this paper can be applied to the screening of dense point clouds in different reconstruction areas, reflecting the effectiveness and superiority of the method in this paper.

[0146] Embodiment 2:

[0147] The plane prior optimization dense matching system considering texture and structural features of the present invention comprises:

[0148] A scene texture feature extraction module is used to extract scene texture features and generate texture feature information for each pixel;

[0149] The scene structure feature extraction module is used to extract structural texture features and generate stable scene structure information for each image;

[0150] A multi-view dense matching module with plane prior optimization under texture feature-aware propagation to generate accurate depth information for each image;

[0151] The depth information block fusion module is used to fuse the redundant depth information to generate the object's three-dimensional points and generate the final dense point cloud.

[0152] Based on Example 1, the system of this embodiment can use scene texture features to select pixels with similar texture features and the lowest matching cost as candidate hypotheses for the current pixels to be updated, and use plane priors based on scene structure information to assist in optimizing the depth information of weak texture areas, thereby achieving accurate calculation of the depth information of weak texture areas, and completing the fusion of depth maps based on a multi-feature constraint fusion algorithm, thereby improving the fusion quality of dense point clouds and obtaining more accurate and reliable dense point clouds.

[0153] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A plane prior optimization dense matching method taking into account texture and structural features, characterized by: The following steps are involved: S1: Scene texture feature extraction: Based on the grayscale image of the reference image, the texture features corresponding to the reference image are extracted by using the weighted value of the sum of the absolute values ​​of the gradients in the horizontal and vertical directions and the standard deviation in the local regular window; S2: scene structure feature extraction: adaptive binarization is performed based on the texture features extracted in step S1 to generate accurate scene structure features for the reference image; S3: Multi-view dense matching with plane prior optimization under texture feature perception propagation: Matching points with different matching costs are selected as anchor points based on the scene structure features of the reference image, a Delaunay triangulation is constructed and plane prior information is generated. The depth map estimation of the weak texture area is optimized based on the plane prior information under texture feature perception propagation, and accurate depth information is generated for each image. S4: Multi-feature constrained depth information block fusion: multiple features are used as fusion constraint factors to fuse redundant depth information to generate object cube 3D points and generate the final dense point cloud; The scene texture feature extraction in step S1 specifically includes the following sub-steps: S11: Convert the color image of the reference image into a grayscale image based on a color weighted average method, wherein the weighted average method calculation formula is as follows: (1) In the formula, , and The red, green and blue bands of the reference image are in row and column coordinates The pixel value of S12: Based on the grayscale image, the scene texture feature is extracted by using the weighted value of the sum of the absolute values ​​of the gradients in the horizontal and vertical directions in the local regular window and the standard deviation. The weighted value calculation formula is as follows: (2) In the formula, , and They are the texture information, gradient and standard deviation of the corresponding pixels respectively; are the weights of the two, and both weight values ​​are taken as 0.

5. The calculation formulas for the gradient and standard deviation are as follows: (3) (4) In the formula, To be consistent with NCC price A rectangular window; is the pixel value within the rectangular window; is the mean gray value of the window pixels; is the total number of pixels in the rectangular window; S13: Repeat the above calculation pixel by pixel, so as to obtain the texture information of all pixels of the image and realize the scene texture feature extraction of the image; The scene structure feature extraction in step 2 specifically includes the following sub-steps: S21: Based on the scene texture features obtained in step 1, a Gaussian kernel function is used Perform local neighborhood Gaussian convolution operation to obtain the texture result after Gaussian filtering. ; (5) In the formula, is the Gaussian kernel function; is the Gaussian convolution operator; S22: Based on the above texture results , using adaptive binarization processing method to obtain the scene structure feature attribute label corresponding to the reference image , 0 is the non-scene structure area, 255 is the scene structure area, and its mathematical expression is: (6) S23: Repeat the above calculation pixel by pixel, so as to obtain the structural feature attribute labels of all pixels of the image, and realize the scene structure feature extraction of the image; The multi-view dense matching of plane prior optimization under texture feature perception propagation in step S3 specifically includes the following sub-steps: S31: Use ACMH algorithm for initial matching, and select matching cost in The matching points in the interval are taken as candidate structure points; then the corresponding scene structure attribute labels are obtained based on the pixel coordinates of the candidate structure points; S32: Use the Delaunay algorithm to construct the network and generate triangular primitives. For each triangular primitive, use the three vertex coordinates of the primitive to calculate the plane structure information in the reference image camera coordinate system, that is, the mathematical expression of the plane equation , to extract the plane prior information, that is, the depth information of the plane structure ; (7) (8) In the formula, is the normal vector Length; The step S32 further includes the following steps: S33: According to the distance of the propagation path, the neighborhood pixels are divided into multiple layers, and two pixels with similar texture features and the lowest matching cost are selected in each layer, that is, pixels with similar texture features and reliable depth information, to complete the depth propagation at the pixel level, using the depth information of the above reliable pixels As the candidate hypothesis of the current pixel to be updated, the matching cost of the candidate hypothesis is calculated according to the optical metric function, and the calculation formula is as follows: (9) in: is the visibility weight of the source image; For the current pixel In-depth information The photometric consistency of is the NCC matching cost; S34: Based on the matching cost results of the above candidate hypotheses, texture features are used for constraints to optimize the matching with the current pixel The matching cost of consistent texture features is calculated as follows under texture feature perception propagation: (10) in: is the texture feature similarity measurement parameter; It is a numerical constant, and its value is 1.5; and are the texture feature values ​​of the reference image and the source image respectively. The second term on the right side of the equation is the matching cost, corresponding to the current pixel Depth information The first term is the regular term of the plane prior. If the texture features of the source image are consistent with those of the reference image, a reward is given, otherwise a penalty is given to encourage Propagation of depth information with similar texture features; S35: Based on the maximum a posteriori probability estimation formula, a matching cost probability function is constructed, and its calculation formula is as follows: (11) S36: Use plane structure information to assist in optimizing dense matching and construct a plane optimization probability function. The calculation formula is as follows: (12) S37: By aggregating the above two probability functions, a plane prior optimization cost function under texture feature perception is constructed, and its calculation formula is as follows: (13) in: is the depth information of the current pixel, For the current pixel Corresponding to the depth information of the planar structure, , is the matching cost under texture feature perception propagation, corresponding to the current pixel Depth information Photometric consistency, is the regular term of the plane prior, if the current pixel If the depth information of the weak texture area is consistent with the depth information of the planar structure, a reward is given, otherwise a penalty is given, thereby promoting the depth information of the weak texture area to tend to the depth information of the planar structure; In the step S4, the multi-feature constrained depth information block fusion adopts a multi-feature constraint factor to fuse the redundant depth information to generate the object cube three-dimensional point and generate the final dense point cloud, which specifically includes the following sub-steps: S41: Based on the given reconstruction range, block size and block overlap range, a regular grid block strategy is used to divide the reconstruction range into multiple sub-blocks to achieve large-scale scene segmentation; S42: Based on the above block results, first use the elevation of the sparse point cloud within the sub-block range to predict the surface undulation within the sub-block range and find the minimum elevation within the sub-block range. and maximum elevation ; S43: Calculate reference image The intersection of the principal ray with the minimum and maximum elevation surfaces and , project the two intersection points to the matching images with the longest baseline distance in the adjacent image set of the reference image , and obtain the corresponding projection points and , calculate the ratio of the elevation difference to the Euclidean distance of the projection point as a reference Elevation step distance; S44: Calculate the average elevation step of all images as the elevation step of the voxel grid ; Finally, the spatial resolution of the image at the principal point is calculated respectively, so as to calculate the average spatial resolution of all images as the horizontal step size of the voxel grid ; The step S43 specifically includes the following sub-steps: S431: Reference image Pixels Project to object space to get a 3D point If it is outside the reconstruction range, the pixel is not fused, otherwise the 3D point is calculated. The incident light With Normal Angle ,like , then the pixel is not fused; S432: 3D points Project to adjacent image sets respectively superior , and calculate its projection depth ; S433: Obtaining adjacent images at the projection point Depth , and projected to the object space to obtain a three-dimensional point , compare the depth and object distance difference between the reference image and the adjacent image, if the depth difference exceeds the threshold Or the object distance difference exceeds , then this point does not participate in the fusion; S434: Count the number of fusion points ,like , and get the final three-dimensional fusion point , the fusion formula is as follows: (14) (15) in: is the Euclidean distance in object space, where ,also, Adaptively take twice the spatial resolution corresponding to the current pixel, For adjacent images at the projection point The photometric consistency at , i.e., the NCC matching cost.

2. The method for plane prior optimization dense matching taking into account texture and structural features according to claim 1, characterized in that: The multi-feature constraint factors in step S4 include sub-block range, angle, object space distance and number of fusion points.

3. A plane a priori optimized dense matching system taking into account texture and structural features, based on the plane a priori optimized dense matching method taking into account texture and structural features according to any one of claims 1-2, characterized in that: include: A scene texture feature extraction module is used to extract scene texture features and generate texture feature value information for each pixel; The scene structure feature extraction module is used to extract structural texture features and generate stable scene structure information for each image; A multi-view dense matching module with plane prior optimization under texture feature-aware propagation to generate accurate depth information for each image; The depth information block fusion module is used to fuse the redundant depth information to generate the object's three-dimensional points and generate the final dense point cloud.

Images