Building three-dimensional model reconstruction method based on improved MVSNet and reconstruction system thereof

Abstract

The invention belongs to the technical field of civil engineering structure health monitoring, and particularly relates to a building three-dimensional model reconstruction method based on an improved MVSNet and a reconstruction system thereof. Expanding a sparse depth map data set and an edge map data set corresponding to the picture set by using the DTU data set; training the proposed network ISENet to obtain an optimal weight; building appearance pictures of different visual angles are obtained to obtain a multi-visual-angle picture set; calculating and optimizing camera parameters and sparse depth maps corresponding to each picture in the multi-view picture set based on known camera internal reference and an SfM algorithm, and calculating an edge map corresponding to each picture; obtaining a corresponding depth map through an ISENet model by using the weight obtained by training; on the basis of a multi-view image set, a depth map and camera parameters, a dense building three-dimensional point cloud model is obtained based on a depth map synthesis point cloud algorithm. The method is used for solving the problems of incomplete active method model, inaccurate passive method model, inconsistent fusion method output and the like in building three-dimensional reconstruction in the prior art, and overcomes the problem of large building edge modeling error of a deep learning MVS method.

Description

Technical Field

[0001] The invention belongs to the technical field of civil engineering structure health monitoring, and in particular relates to a building three-dimensional model reconstruction method based on improved MVSNet and a reconstruction system thereof. Background Art

[0002] With the advancement of urbanization, a large number of housing construction and infrastructure construction have become necessary conditions to meet the functional needs of cities. However, civil engineering building structures often have a service life of several decades. How to accurately evaluate their safety to guide operation and maintenance management, and then build a digital urban management system has become a major challenge for future urban development. Traditional building structure operation and maintenance methods often make it difficult to timely discover and repair problems such as surface damage or settlement of buildings, which not only affects the service life of buildings, but also increases maintenance costs. In recent years, with the rapid development of big data, artificial intelligence and three-dimensional vision, digital twin technology has received widespread attention. By building a virtual model of physical entities, it realizes the mapping and interaction between the physical world and the digital world, providing a new way to solve the above challenges. The application of digital twin technology can effectively monitor the real-time status of buildings, discover potential problems in a timely manner, and support more accurate maintenance decisions, thereby extending the service life of buildings and reducing operation and maintenance costs. The primary problem faced by the effective implementation of digital twins is how to restore an accurate and complete three-dimensional geometric model of buildings.

[0003] The current three-dimensional model reconstruction methods are mainly divided into two categories: active method and passive method. The active method relies on laser radar, structured light transmitter, Tof camera and other equipment to actively transmit different types of electromagnetic waves to the surface of the object, and determines the position information of the object in space by calculating and measuring the flight time and related parameters of the electromagnetic waves. This type of method can quickly and accurately obtain the depth information of the object, but it is difficult to collect complete large-scene data due to the inconvenience of equipment use, and is not suitable for the reconstruction of architectural three-dimensional models. The passive method relies on the camera to capture the image of the object to be measured, establish the relative position relationship between the object to be measured and the camera, and thus obtain the three-dimensional information of the surface of the object to be measured. Therefore, the passive method has been widely used in many fields due to its advantages such as weak equipment dependence, low equipment cost, and convenient data acquisition.

[0004] Passive methods can be divided into monocular 3D reconstruction, binocular 3D reconstruction and multi-eye 3D reconstruction according to the number of views required. Figure 2The mapping from D to 3D is ambiguous, so monocular 3D reconstruction can only restore the depth information of a specific scene based on the prior information of the scene, and its application scope is limited. Stereoscopic 3D reconstruction restores scene depth information based on the inverse proportion of disparity to depth, but it is limited by the number of views and faces problems such as occlusion, perspective foreshortening, and repetitive patterns, resulting in large reconstruction errors and poor robustness. Multi-eye 3D reconstruction uses the features of multiple views and minimizes the reprojection error through the bundle adjustment method to overcome the problems of large errors and poor robustness of stereoscopic 3D reconstruction. It has the widest application scope. This method is mainly divided into two steps: Structure from Motion (SfM) and Multi View Stereo (MVS). The former uses the extraction and matching of the same key points in different views to restore camera parameters and sparse point clouds, and the latter uses camera parameters to perform stereo matching on each pair of images to restore dense point clouds. The advent of Colmap means that the development of SfM has entered a mature stage, but MVS still has many unresolved issues. Traditional MVS methods often fail in depth matching on mirrors, textureless and repeated textures due to the poor robustness of artificially designed features. Traditional MVS methods are not effective in 3D reconstruction of buildings due to the lack of artificially designed and constructed facade texture features.

[0005] In order to overcome the shortcomings of active and passive methods, many works focus on data fusion and combine active and passive methods. For example, the point cloud of the ground laser scanner is fused with the point cloud generated by the MVS method, and a data fusion algorithm is developed to achieve the reconstruction of the building 3D model, but the modeling efficiency is low. At present, most data fusion algorithms are based on "terminal fusion", that is, the point clouds obtained by multiple methods are simply registered to obtain the final model, and the in-depth full-process fusion is not carried out in the process of each method, and the characteristics of each method are not fully utilized, resulting in inconsistent color and uneven density of the point cloud model.

[0006] At present, the MVS method based on deep learning, represented by MVSNet, is developing rapidly. It relies on high-dimensional features extracted by deep neural networks to be more robust and has better performance in the model integrity of building 3D reconstruction. Compared with other learning-based methods, the multi-stage cascade network has better accuracy and integrity. Its coarse-to-fine estimation of the depth map makes the pixels at the edge of the rough depth map represent both the front and back of the edge, and the depth shows a bimodal distribution. However, this method assumes that the depth of each pixel is an unbiased unimodal distribution, and the depth estimation is the expectation of the depth division sample, which makes the depth estimation from coarse to fine appear at the edge. There are estimation errors and error accumulation. Due to the architectural functional requirements of building shape, lighting and ventilation, there are a large number of windows and areas with discontinuous depth on the facade of the building, which makes the above method produce large errors when applied to the 3D reconstruction of the building.

[0007] In fact, multi-view 3D reconstruction not only restores the camera parameters but also the accurate sparse point clouds in the SfM step. These sparse point clouds are used for rendering and new perspective synthesis by methods such as Nerf and 3D Gaussian sputtering, achieving high-fidelity visual quality. However, the above methods do not generate more depth information, which means that the models generated by these methods cannot be used for actual measurement. The current MVS method based on deep learning does not fully utilize the prior information of the above high-quality sparse point clouds. It only uses the minimum depth and maximum depth as a priori guidance for the depth division based on deep learning MVS, which makes the depth estimation inefficient.

[0008] In summary, in the 3D reconstruction of buildings, the active method model is incomplete, the passive method model is not accurate, and the existing fusion method does not fully integrate the characteristics of the two. However, although the MVS method based on deep learning has improved the modeling completeness, the modeling error at the edge of the building is large. Summary of the invention

[0009] The present invention provides a building three-dimensional model reconstruction method and a reconstruction system based on improved MVSNet, which are used to solve the problems of incomplete active method model, inaccurate passive method model, inconsistent fusion method output, etc. in the prior art in building three-dimensional reconstruction, and overcome the problem of large modeling errors in building edges of deep learning MVS method.

[0010] The present invention is achieved through the following technical solutions:

[0011] A method for reconstructing a three-dimensional building model based on an improved MVSNet, the method comprising the following steps:

[0012] Step 1: Use the DTU dataset to expand the sparse depth map dataset and edge map dataset corresponding to the image set;

[0013] Step 2: Use the expanded DTU dataset in step 1 to train the proposed network ISENet to obtain the optimal weight;

[0014] Step 3: Use a drone equipped with a camera to fly around the building, and control the camera to take pictures of the building's appearance from different perspectives to obtain a multi-perspective picture set;

[0015] Step 4: Based on the known camera intrinsic parameters and SfM algorithm, calculate and optimize the camera parameters and sparse depth map corresponding to each image in the multi-view image set obtained in step 3, and use the Canny operator to calculate the edge map corresponding to each image;

[0016] Step 5: Using the multi-view sample set obtained in step 3 and step 4, the corresponding depth map is obtained through the ISENet model using the trained weights, wherein the multi-view sample set includes an RGB image of the building, a sparse depth map, and an edge map;

[0017] Step 6: Based on the multi-view image set in step 3, the depth map obtained in step 5 and the camera parameters, a dense 3D point cloud model of the building is obtained based on the depth map synthesis point cloud algorithm.

[0018] Furthermore, the step 1 is specifically:

[0019] Step 1.1: According to the 49 camera parameter projections given in the dataset, the coordinates of each sparse point cloud are converted from the world coordinate system to the pixel coordinate system to obtain 49 sparse depth maps corresponding to each scene, and a total of 128 sparse depth map datasets corresponding to the scenes can be obtained;

[0020] Step 1.2: Perform Gaussian filtering on all the images in the DTU dataset to remove noise and smooth the images;

[0021] Step 1.3: Using the Sobel operator, calculate the gradient strength and direction of each pixel in the image:

[0022] Step 1.4: Use non-maximum suppression to remove points with weak gradient strength and only retain local maximum points;

[0023] Step 1.5: Set the low threshold to 150 and the high threshold to 200, so that points with gradient strength less than 150 are marked as non-edge points, and points with gradient strength greater than 200 are marked as edge points; when a pixel gradient is between 150 and 200, it is marked as an edge point only when it is adjacent to a pixel with gradient strength greater than 200;

[0024] Step 1.6: For all pixels marked as edge points, if their neighboring pixels are also edge points, they are connected; thus, the edge map dataset of the DTU dataset is obtained.

[0025] Furthermore, the step 2 specifically includes the following steps:

[0026] Step 2.1: Input the sparse depth map dataset, edge map dataset, and original image dataset obtained in step 1 into the feature pyramid network in a multi-view manner to obtain multi-view multi-stage features corresponding to the three datasets;

[0027] Step 2.2: Use the sparse depth map to guide the depth division to form a projection volume for subsequent projection;

[0028] Step 2.3: For the reference view of the smallest coarse feature obtained in step 2.1, copy its features on each assumed surface of the projection body obtained in step 2.2, and project the coarse features of the source view onto the projection body according to the homography transformation to form a feature body;

[0029] Step 2.4: Aggregate the features of different perspectives using variance to obtain the cost volume;

[0030] Step 2.5: Use 3D U-Net to regularize the cost volume obtained in step 2.4 and fuse the features of the three modalities; after multiple convolutional layers and pooling layers, set the last layer of the model to the Softmax layer to obtain the probability volume P;

[0031] Step 2.6: The depth map estimated in the first stage is obtained by weighted summation of the depth probabilities of each pixel, and back-propagation is performed with the true depth map using L1 loss;

[0032] Step 2.7: Calculate the gradients of the depth map estimated in the first stage in the u and v directions respectively, then synthesize them into the total gradient and perform logarithmic normalization:

[0033] Set the threshold to 0.15 so that G t Pixels with a value greater than 0.15 indicate that the position is a depth edge, otherwise it is not a depth edge, thus obtaining a depth edge mask; at the same time, the edge atlas obtained in step 2.7 is intersected with the corresponding depth edge mask to obtain the final edge mask; points with an edge mask of 0 indicate that the pixel position is not an edge, and points with a value of 1 indicate that the pixel position belongs to an edge;

[0034] Step 2.8: Take the probability volume of step 2.5 and the edge mask of step 2.7 and find the two depth hypothesis values ​​d with the highest probability at the edge top2 ;

[0035] Step 2.9: Upsample the depth map and edge mask obtained in step 2.7 for the second stage of depth segmentation, where the depth map adopts trilinear interpolation and the edge mask adopts nearest neighbor interpolation; take the upsampled depth map as the reference, take the depth of each point as the center, divide 32 depth assumption planes along the depth direction, and set the depth interval to 5; in addition, for the edge position corresponding to the edge mask, take the d obtained in step 2.8 as the reference, top2 16 depth assumption planes are re-divided as the center of the two depth values, and the divisions at the original edge positions are replaced to obtain the projection body of the second stage;

[0036] Step 2.10: Project the second-stage features obtained in step 2.1 in the same manner as step 2.3 to obtain the feature body of the second stage, then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the second stage, and continue to repeat steps 2.7, 2.8, and 2.9 to obtain the projection body of the third stage. At this time, the number of depth assumption planes is 8 and the depth interval is 2.5;

[0037] Step 2.11: Project the third-stage features obtained in step 2.1 in the same manner as step 2.3 to obtain the feature body of the third stage, and then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the third stage; if it is the fourth stage, continue to repeat steps 2.7, 2.8, and 2.9 to obtain the fourth-stage projection body. At this time, the number of depth assumption planes is 8 and the depth interval is 1.25;

[0038] Step 2.12: If it is four stages, project the fourth stage features obtained in step 2.1 according to the method of step 2.3 to obtain the feature body of the fourth stage, and then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the fourth stage;

[0039] Step 2.13: The total loss considering the losses at each stage is given by the following formula:

[0040]

[0041] In the formula, Loss total is the total loss; s represents the number of stages; p valid is a valid set of true value pixels, which means that only the subject within the mask is calculated, and the background outside the mask is not calculated; λ s is the weight of the loss at each stage. If it is a three-stage model, the weights of the first, second, and third stage losses are 0.5, 1.0, and 2.0 respectively. If it is a four-stage model, the weights of the first, second, third, and fourth stage losses are 0.5, 1.0, 2.0, and 4.0 respectively. d s (p) represents the true value of each stage, Indicates the estimated value of each pixel at each stage;

[0042] Step 2.14: Follow the above steps to train ISENet for 16 rounds to obtain the final training weights, and select the optimal weights as model parameters; the initial learning rate is 0.001, and the learning rate is halved in the 10th and 12th rounds of training, and the optimizer is Adam.

[0043] Furthermore, the step 3 specifically includes selecting clear weather with high visibility, controlling the drone equipped with a high-definition camera to fly multiple circles around the target building at different altitudes, and controlling the camera to shoot from multiple perspectives;

[0044] The distance between the drone and the building surface was kept at about 4 meters, and the photo content overlap rate was controlled at about 50%. Based on the built-in fixed-point cruise algorithm of the drone and manual camera posture adjustment, the original multi-view picture set S0{X i |i=1,...,N0}, a total of N0 pictures.

[0045] Furthermore, the step 4 specifically includes the following steps:

[0046] Step 4.1: Preprocess the images obtained from the multi-view sample set S0 to filter out meaningless scene images that do not contain the main building, and obtain the processed sample set S{X i |i=1,...,N}, a total of N pictures.

[0047] Step 4.2: Obtain the equivalent focal length f and image resolution from the camera manufacturer's manual as the known initial parameters of the camera intrinsic parameters.

[0048] Step 4.3: Calculate camera parameters and sparse point cloud based on SfM algorithm.

[0049] Step 4.4: Restore the sparse depth map corresponding to each image based on the correspondence between the sparse point cloud and the image, specifically according to step 2.3.

[0050] Step 4.5: Obtain the edge map of each image based on the Canny operator, specifically according to steps 2.4, 2.5, 2.6, and 2.7.

[0051] Furthermore, the step 5 specifically includes the following steps:

[0052] Step 5.1: Preprocess the image and use the bilinear interpolation method to downsample the width and height of the image by 1 / 2;

[0053] Step 5.2: Select the scale-invariant feature transformation algorithm to extract feature points;

[0054] Step 5.3: The size and direction of the gradient in the 16×16 region near the feature point are formed into a 128-dimensional vector according to the histogram statistics method, and it is normalized to obtain the feature point descriptor used for matching.

[0055] Step 5.4: Calculate the camera parameters based on the binocular stereo geometry method.

[0056] Step 5.5: Use multiple views to construct an over-constrained system and use the least squares method to solve X:

[0057] Step 5.6: Use the bundle adjustment algorithm to minimize reprojection, optimize the position information of the sparse point cloud, and finally output the sparse point cloud.

[0058] Furthermore, the step 6 specifically includes the following steps:

[0059] Step 6.1: Fill and average pool the probability volume P obtained in step 3.5 corresponding to each depth map, calculate the depth regression index and output the photometric consistency confidence map to measure the similarity of color information at the same point from different perspectives;

[0060] Step 6.2: Perform geometric consistency check on the obtained depth map on no less than 4 views. Specifically, any point on the reference view depth map is projected into the world coordinate system, then projected from the world coordinate system to the source view, and then reversely projected back to the reference view, and the degree of deviation of the pixel from the original pixel before and after the projection is calculated. A small deviation indicates good geometric consistency;

[0061] Step 6.3: Filter the depth map using the masks obtained in step 6.1 and step 6.2 to obtain a filtered depth map;

[0062] Step 6.4: Using the filtered depth map obtained in step 6.3 and its corresponding camera parameters, project the points on the depth map into the world coordinate system, and assign the color information of the RGB image corresponding to each point to the point cloud to obtain the final dense building point cloud model.

[0063] A building 3D model reconstruction system based on improved MVSNet, the system uses the building 3D model reconstruction method based on improved MVSNet as described above, the system comprises:

[0064] The dataset expansion module uses the DTU dataset to expand the sparse depth map dataset and edge map dataset corresponding to the image set;

[0065] The optimal weight calculation module uses the expanded DTU dataset to train the proposed network ISENet to obtain the optimal weight;

[0066] A multi-view image acquisition module acquires the building appearance from different viewpoints to obtain a multi-view image set;

[0067] The sparse depth map and edge map calculation module calculates and optimizes the camera parameters and sparse depth map corresponding to each image in the multi-view image set obtained by the multi-view image acquisition module based on the known camera intrinsic parameters and the SfM algorithm, and uses the Canny operator to calculate the edge map corresponding to each image;

[0068] A depth map calculation module uses the multi-view sample set obtained by the multi-view image acquisition module to obtain a corresponding depth map through the ISENet model using the trained weights, wherein the multi-view sample set includes an RGB image of the building, a sparse depth map, and an edge map;

[0069] The dense building 3D point cloud model module is based on the multi-view image set obtained by the multi-view image acquisition module, the depth map calculated by the depth map calculation module and the camera parameters, and obtains a dense building 3D point cloud model based on the depth map synthesis point cloud algorithm.

[0070] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above method is implemented.

[0071] A computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0072] The beneficial effects of the present invention are:

[0073] The present invention can make full use of the advantages of neural networks in high-dimensional data modeling, and use sparse depth prior and edge position prior to fully learn the complex structure and depth discontinuity of general building structures, to achieve accurate reconstruction from multi-view images to complete point cloud models, reducing manpower while ensuring the authenticity and reliability of the model.

[0074] The problem of poor modeling accuracy and integrity is caused by factors such as huge building scale, lack of texture, and complex modeling. Existing methods mostly rely on high-precision laser radar scanning to generate point clouds or rely on drones to collect photos and use multi-view stereo methods to generate point clouds. These methods require a lot of manual intervention, and there are problems such as large workload, low efficiency, and low model accuracy or integrity. In order to accurately and efficiently reconstruct the three-dimensional model of the building structure, especially the modeling accuracy in the depth discontinuous area, the present invention is proposed.

[0075] Aiming at the problem of building 3D model reconstruction, the present invention realizes the full automatic processing of the whole process from image processing, camera pose acquisition, model training, depth map generation to point cloud model generation. The whole process is convenient and fast, does not rely on manual parameter adjustment, and improves the efficiency and accuracy of building 3D model reconstruction, especially the accuracy of depth discontinuous areas.

[0076] It has a high tolerance for errors in camera pose estimation. Based on deep prior information and the powerful feature extraction capability of deep learning, it effectively alleviates the problem of point cloud model inability to reconstruct due to large errors in camera parameter estimation caused by image quality and environmental factors.

[0077] The accuracy and completeness of building model reconstruction are improved. The proposed ISENet introduces the high-precision point cloud prior information generated by SfM, utilizes the sparse depth prior in feature extraction, and uses the depth prior to guide the depth division to form a projection volume that is more in line with the actual depth distribution, thereby achieving a model reconstruction effect with better accuracy and completeness.

[0078] The modeling error is smaller in edge areas such as depth discontinuity. This is because the proposed ISENet introduces an edge mask, inputs the edge map into the feature extraction network to enhance the network's perception of edge parts, and performs depth re-division based on the edge mask during depth division, which makes the method pay more attention to the edge, thereby reducing the error in the depth discontinuity area.

[0079] Depth maps and point cloud models are generated fully automatically without human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flow chart of the method of the present invention.

[0081] Figure 2 It is a multi-view sample set of the present invention; from left to right, it is an RGB image, a sparse depth map, and an edge map.

[0082] Figure 3 It is a schematic diagram of a feature extraction module of the present invention.

[0083] Figure 4 Schematic diagram of adaptive depth division of the present invention; the left figure is the first stage of depth division, and the right figure is the subsequent stage of depth division.

[0084] Figure 5 It is the loss curve of the ISENet network training process of the present invention.

[0085] Figure 6 This is the depth map error curve of the present invention compared with the best method of the same type, MVSFormer++; the left figure is the depth Figure 2 mm error curve, the right picture is depth Figure 8 mm error curve.

[0086] Figure 7 It is the input picture and the output depth map of the present invention.

[0087] Figure 8 This is the reconstruction result of the three-dimensional point cloud model of an example of the present invention; the left picture is the reconstruction result of the present invention, and the right picture is the point cloud true value of the data set. DETAILED DESCRIPTION

[0088] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0089] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0090] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0091] The following is attached to this application specification Figure 1-8 , the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0092] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0093] Implementation Method 1

[0094] The embodiment of the present invention provides a method for reconstructing a three-dimensional building model based on an improved MVSNet, the method comprising the following steps:

[0095] Step 1: Use the DTU dataset to expand the sparse depth map dataset and edge map dataset corresponding to the image set;

[0096] Step 2: Use the expanded DTU dataset in step 1 to train the proposed network ISENet to obtain the optimal weight;

[0097] Step 3: Use a drone equipped with a camera to fly around the building, and control the camera to take pictures of the building's appearance from different perspectives to obtain a multi-perspective picture set;

[0098] Step 4: Based on the known camera intrinsic parameters and SfM (Structure from Motion) algorithm, calculate and optimize the camera parameters and sparse depth map corresponding to each image in the multi-view image set obtained in step 3, and use the Canny operator to calculate the edge map corresponding to each image;

[0099] Step 5: Use the multi-view sample set in step 3 and the trained weights to obtain the corresponding depth map through the ISENet model, wherein the multi-view sample set includes an RGB image of the building, a sparse depth map, and an edge map;

[0100] Step 6: Based on the multi-view image set in step 3, the depth map obtained in step 5 and the camera parameters, a dense 3D point cloud model of the building is obtained based on the depth map synthesis point cloud algorithm.

[0101] Furthermore, the step 1 is specifically as follows: the DTU dataset consists of 128 scenes including buildings; the image dataset and the depth map dataset are used for training, and the point cloud file is used for testing; the point cloud file is randomly downsampled at a 0.1 times downsampling rate to obtain a sparse point cloud corresponding to each scene;

[0102] Step 1.1: According to the 49 camera parameter projections given in the data set, the coordinates of each sparse point cloud are converted from the world coordinate system to the pixel coordinate system to obtain 49 sparse depth maps corresponding to each scene. A total of 128 sparse depth map data sets corresponding to the scenes can be obtained; the specific formula is as follows:

[0103]

[0104] In the formula, is the homogeneous coordinate; K is the camera intrinsic matrix; R is the rotation matrix; T is the translation matrix; x i ,y i , z i is the point cloud coordinate in the world coordinate system; p i′ is the normalized homogeneous coordinate, D(u i ,v i ) corresponds to (u i ,v i )The depth value of the pixel;

[0105] Step 1.2: For all the pictures in the DTU dataset, use the following formula to perform Gaussian filtering on the images to remove noise and smooth the images;

[0106] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0107]

[0108] Where (x, y) is the coordinate of the image pixel coordinate system; L(x, y, σ) is the Gaussian blurred image with σ as the parameter;

[0109] Step 1.3: Using the Sobel operator, calculate the gradient strength and direction of each pixel in the image using the following formula:

[0110]

[0111] Where G is the gradient strength, indicating the edge strength; G x and G y are the gradients along the x and y directions respectively; θ is the gradient direction indicating the edge direction;

[0112] Step 1.4: Use non-maximum suppression to remove points with weak gradient strength and only retain local maximum points; specifically, for each pixel p, according to its gradient direction θ p Find two neighboring pixels q and r. If the gradient strength of p is greater than the gradient strength of q and r, keep p; otherwise, set the gradient strength of p to zero.

[0113] Step 1.5: Set the low threshold to 150 and the high threshold to 200, so that points with gradient strength less than 150 are marked as non-edge points, and points with gradient strength greater than 200 are marked as edge points; when a pixel gradient is between 150 and 200, it is marked as an edge point only when it is adjacent to a pixel with gradient strength greater than 200;

[0114] Step 1.6: For all pixels marked as edge points, if their neighboring pixels are also edge points, they are connected; thus, the edge map dataset of the DTU dataset is obtained.

[0115] Furthermore, the step 2 specifically includes the following steps:

[0116] Step 2.1: Input the sparse depth map dataset, edge map dataset and original image dataset obtained in step 1 into the Feature Pyramid Network (FNP) in a multi-view manner to obtain multi-view multi-stage features corresponding to the three datasets; the number of stages can be set to three or four stages, and the number of upsampling times corresponding to different numbers of stages is different. If it is three stages, the FPN network is upsampled twice, and if it is four stages, the FPN network is upsampled three times. The following introduction will be based on the default of three stages;

[0117] Step 2.2: Use the sparse depth map to guide the depth division to form a projection body for subsequent projection; specifically, first, 48 assumed planes are evenly divided along the depth direction between the minimum depth 425 and the maximum depth 935 (if it is four stages, it corresponds to 64 assumed planes); for points with depth values ​​in the pixel coordinate system of the sparse depth map, the depth is used as the center to re-divide 48 assumed planes in the depth direction, and the depth interval is consistent with the previous one; for coordinates without depth values, the original division is implemented, and the projection body is obtained;

[0118] Step 2.3: For the reference view of the smallest coarse feature obtained in step 2.1, copy its features on each hypothetical surface of the projection body obtained in step 2.2, and project the coarse features of the source view onto the projection body according to the homography transformation using the following formula to form a feature body;

[0119]

[0120] In the formula, H i (d) represents the homography between the feature map of the i-th view and the reference feature map at depth d; K i ,R i ,t i They represent the camera intrinsic parameters, rotation and translation matrices of the i-th view respectively; n1 represents the principal axis of the reference camera;

[0121] Step 2.4: Aggregate the features of different perspectives using variance according to the following formula to obtain the cost volume;

[0122]

[0123] In the formula, C represents the cost body; Represents mapping; V i is a characteristic body; is the average value of the feature body;

[0124] Step 2.5: Use 3D U-Net to regularize the cost volume obtained in step 2.4 and fuse the features of the three modalities; after multiple convolutional layers and pooling layers, set the last layer of the model to the Softmax layer to obtain the probability volume P, which represents the probability of the corresponding depth of each pixel point on each depth hypothesis layer. The formula is:

[0125]

[0126] Step 2.6: The depth probabilities of each pixel are weighted and summed according to the following formula to obtain the estimated depth map of the first stage, and the L1 loss is used for back propagation with the true depth map;

[0127]

[0128] Step 2.7: Calculate the gradients of the depth map estimated in the first stage in the u and v directions respectively, then synthesize them into a total gradient and perform logarithmic normalization using the following formula:

[0129]

[0130] Set the threshold to 0.15 so that G t Pixels with a value greater than 0.15 indicate that the position is a depth edge, otherwise it is not a depth edge, thereby obtaining a depth edge mask; at the same time, the edge atlas obtained in step 2.7 is intersected with the corresponding depth edge mask to obtain the final edge mask, thereby eliminating the problem that the former will increase the background error edge and the latter will increase the texture error edge; a point with an edge mask of 0 indicates that the pixel position is not an edge, and a point with an edge mask of 1 indicates that the pixel position belongs to an edge;

[0131] Step 2.8: Take the probability volume of step 2.5 and the edge mask of step 2.7 and find the two depth hypothesis values ​​d with the highest probability at the edge top2 ;

[0132] Step 2.9: Upsample the depth map and edge mask obtained in step 2.7 for the second stage of depth segmentation, where the depth map adopts trilinear interpolation and the edge mask adopts nearest neighbor interpolation; take the upsampled depth map as the reference, take the depth of each point as the center, divide 32 depth assumption planes along the depth direction, and set the depth interval to 5; in addition, for the edge position corresponding to the edge mask, take the d obtained in step 2.8 as the reference, top2 16 depth assumption planes are re-divided as the center of the two depth values, and the divisions at the original edge positions are replaced to obtain the projection body of the second stage;

[0133] Step 2.10: Project the second-stage features obtained in step 2.1 in the same manner as step 2.3 to obtain the feature body of the second stage, then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the second stage, and continue to repeat steps 2.7, 2.8, and 2.9 to obtain the projection body of the third stage. At this time, the number of depth assumption planes is 8 and the depth interval is 2.5 (if it is four stages, the corresponding number of depth assumption planes is 16);

[0134] Step 2.11: Project the third-stage features obtained in step 2.1 in the same manner as step 2.3 to obtain the feature body of the third stage, and then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the third stage; if it is the fourth stage, continue to repeat steps 2.7, 2.8, and 2.9 to obtain the fourth-stage projection body. At this time, the number of depth assumption planes is 8 and the depth interval is 1.25;

[0135] Step 2.12: If it is four stages, project the fourth stage features obtained in step 2.1 according to the method of step 2.3 to obtain the feature body of the fourth stage, and then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the fourth stage;

[0136] Step 2.13: The total loss considering the losses at each stage is given by the following formula:

[0137]

[0138] In the formula, Loss total is the total loss; s represents the number of stages; p valid is a valid set of true value pixels, which means that only the subject within the mask is calculated, and the background outside the mask is not calculated; λ s is the weight of the loss at each stage. If it is a three-stage model, the weights of the first, second, and third stage losses are 0.5, 1.0, and 2.0 respectively. If it is a four-stage model, the weights of the first, second, third, and fourth stage losses are 0.5, 1.0, 2.0, and 4.0 respectively. d s (p) represents the true value of each stage, Indicates the estimated value of each pixel at each stage;

[0139] Step 2.14: Follow the above steps to train ISENet for 16 rounds to obtain the final training weights, and select the optimal weights as model parameters; the initial learning rate is 0.001, and the learning rate is halved in the 10th and 12th rounds of training, and the optimizer is Adam.

[0140] Furthermore, the step 3 specifically includes selecting clear weather with high visibility, controlling the drone equipped with a high-definition camera to fly multiple circles around the target building at different altitudes, and controlling the camera to shoot from multiple perspectives;

[0141] The distance between the drone and the building surface was kept at about 4 meters, and the photo content overlap rate was controlled at about 50%. Based on the built-in fixed-point cruise algorithm of the drone and manual camera posture adjustment, the original multi-view picture set S0{X i |i=1,...,N0}, a total of N0 pictures.

[0142] Furthermore, the step 4 specifically includes the following steps:

[0143] Step 4.1: Preprocess the images obtained from the multi-view sample set S0 to filter out meaningless scene images that do not contain the main building, and obtain the processed sample set S{X i |i=1,...,N}, a total of N pictures.

[0144] Step 4.2: Obtain the equivalent focal length f and image resolution from the camera manufacturer's manual as the known initial parameters of the camera intrinsic parameters.

[0145] Step 4.3: Calculate camera parameters and sparse point cloud based on SfM algorithm.

[0146] Step 4.4: Restore the sparse depth map corresponding to each image based on the correspondence between the sparse point cloud and the image, specifically according to step 2.3.

[0147] Step 4.5: Obtain the edge map of each image based on the Canny operator, specifically according to steps 2.4, 2.5, 2.6, and 2.7.

[0148] Furthermore, the step 5 specifically includes the following steps:

[0149] Step 5.1: Preprocess the image and use the bilinear interpolation method to downsample the width and height of the image by 1 / 2;

[0150] Step 5.2: Select the Scale-Invariant Feature Transform (SIFT) algorithm to extract feature points. The extraction formula is:

[0151] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0152]

[0153] D(x,y,σ)=L(x,y,σ)-L(x,y,kσ)

[0154] Where (x, y) is the coordinate system of the image pixel; L(x, y, σ) is the Gaussian blurred image with σ as the parameter; D(x, y, σ) is the Gaussian difference image; the local extremum detection algorithm compares each pixel in D(x, y, σ) with its 26 spatial (position and scale) adjacent pixels. If it is larger or smaller than any adjacent pixel, it is used as a SIFT feature point, and all SIFT feature points are counted to obtain a feature point set;

[0155] Step 5.3: The size and direction of the gradient in the 16×16 region near the feature point are formed into a 128-dimensional vector according to the histogram statistics method, and it is normalized to obtain the feature point descriptor used for matching. The feature descriptor of the feature point is based on the following formula:

[0156]

[0157] θ(x,y)=arctan((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y)))

[0158] Calculate the distance matching feature points of the descriptors of all image pairs to obtain a set of matching feature point pairs. in

[0159] Step 5.4: Calculate the camera parameters based on the binocular stereo geometry method. The camera parameters are given by the matrix S λ K[R|t] is given by, where is the camera internal reference; is the camera external parameter, where is the rotation matrix, is the translation vector; S λ =diag(1 / λ,1 / λ,1) is the radial distortion matrix of the lens, where is a nonlinear polynomial function of the distance d from a pixel to the center of the image, is the radial distortion parameter; where S λ ,K can obtain the initial value in the Exchange Image File Format (EXIF) and perform global optimization later;

[0160] Step 5.5: Use multiple views to construct an over-constrained system and use the least squares method to solve X:

[0161] min X =||AX-b|| 2

[0162] Where A is a 3n×4 matrix consisting of the projection equations of all views; b is a 3n×1 vector consisting of the coordinates of the image points in all views; X is the coordinate of the three-dimensional point, that is, the spatial position information of the sparse point cloud;

[0163] Step 5.6: Use the bundle adjustment algorithm (BA) to minimize reprojection, optimize the position information of the sparse point cloud, and finally output the sparse point cloud.

[0164] Furthermore, the step 6 specifically includes the following steps:

[0165] Step 6.1: Fill and average pool the probability volume P obtained in step 3.5 corresponding to each depth map, calculate the depth regression index and output the photometric consistency confidence map to measure the similarity of color information at the same point from different perspectives. The specific formula is as follows:

[0166]

[0167] C(h,w)=P″(depth_index(h,w),h,w)

[0168] Where P″(d,h,w) represents average pooling; pad represents padding; depth_index(h,w) represents depth index; C(h,w) is the photometric consistency confidence map, ranging from 0 to 1. The higher the value, the higher the photometric consistency and the more reliable it is. The photometric consistency mask is set to pixels with a confidence greater than 0.9.

[0169] Step 6.2: Perform geometric consistency check on the obtained depth map on no less than 4 views. The specific method is to project any point on the reference view depth map into the world coordinate system, then project it from the world coordinate system to the source view, and then reversely project it back to the reference view, and calculate the degree of deviation of the pixel from the original pixel before and after the projection. The smaller the deviation, the better the geometric consistency; the specific method is as follows:

[0170] ||pp″||2≤τp and p″ represent the pixel positions before and after reprojection; τ is the threshold; τ=0.25 is selected as the geometric consistency mask.

[0171] Step 6.3: Filter the depth map using the masks obtained in step 6.1 and step 6.2 to obtain a filtered depth map;

[0172] Step 6.4: Using the filtered depth map obtained in step 6.3 and its corresponding camera parameters, project the points on the depth map into the world coordinate system, and assign the color information of the RGB image corresponding to each point to the point cloud to obtain the final dense building point cloud model; the specific formula is as follows:

[0173] P′=K[RT]P W

[0174] Where P′ is the homogeneous coordinate of the pixel coordinate system; K is the camera intrinsic parameter matrix; R is the camera rotation matrix; T is the camera translation matrix; P W The coordinates of the world coordinate system.

[0175] Specifically, this embodiment is implemented based on the above-mentioned specific implementation mode 1, and is implemented based on a computer program written in Python and C++ languages. The target implemented is a DTU dataset, and it can also be used for a picture set collected by a drone. The following specific examples illustrate the effects of the present invention.

[0176] The hardware environment of this example is shown in Table 1.

[0177] Table 1 Hardware environment parameters

[0178]

[0179] To expand the DTU dataset, first read the point cloud file corresponding to each scene in the dataset, and use a downsampling ratio of 0.1 to randomly downsample the point cloud so that the number of point clouds in each scene is only 10% of the original to restore the sparse point cloud. Then, using the camera parameters corresponding to each image, the sparse point cloud is projected to the shooting angle of each image based on the projection matrix to obtain a sparse depth map. The extended DTU sparse depth map dataset can be obtained by traversing the 49 camera parameters corresponding to all scenes.

[0180] All the image datasets of the DTU dataset are calculated based on the Canny operator, with a low threshold of 150 and a high threshold of 200, to obtain the edge map datasets corresponding to all the images.

[0181] 80% of the image dataset, sparse depth map dataset, and edge map dataset of the extended DTU dataset are used for training, and 20% are used for testing. During training, the corresponding training sets are sent to the ISENet network in batches for iterative training. The network flow chart is as follows: Figure 1 As shown in Figure 2, the training set has 27,097 samples and the test set has 7,546 samples (each image corresponds to 7 different lighting conditions). The training samples are as follows: Figure 2 The feature extraction module is shown in Figure 3 As shown, the adaptive depth partitioning is Figure 4 shown.

[0182] The relevant parameter settings for model training are shown in Table 2.

[0183] Table 2 Model related parameter settings

[0184]

[0185] In order to make the model converge better, this model uses learning rate decay and Adam optimizer. The learning rate is adjusted to 0.5 times the original value at 10, 12, and 14 rounds of training. The loss function curve during training is as follows: Figure 5 As shown in the figure, during the training process, the loss function of the training set quickly dropped to below 11 after 16 rounds of iterations. As the iterations progressed, the error tended to stabilize after 10 iterations, indicating that the model reached convergence.

[0186] The test set is input into the network using the trained weight parameters for testing. The output depth map error is as follows: Figure 6 As shown, compared with the best current similar method MVSFormer++, the 2mm error of the present invention is reduced from 15.6% to 8.7%, and the 8mm error is reduced from 6.7% to 3.5%, and the overall accuracy is improved by about 47%.

[0187] Figure 7 It shows the image of the input method and the output depth map. The output depth map has high smoothness and low noise. Figure 8 The left picture shows the point cloud model of the three-dimensional reconstruction of the building in PLY format output by the present invention, and the right picture shows the three-dimensional point cloud model of the real building obtained by structured light scanning. In comparison, the building point cloud model output by the present invention is more complete.

[0188] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

[0189] Implementation Method 2

[0190] An embodiment of the present invention provides a building 3D model reconstruction system based on improved MVSNet, the system uses the building 3D model reconstruction method based on improved MVSNet as described in embodiment 1, the system comprises:

[0191] The dataset expansion module uses the DTU dataset to expand the sparse depth map dataset and edge map dataset corresponding to the image set;

[0192] The optimal weight calculation module uses the expanded DTU dataset to train the proposed network ISENet to obtain the optimal weight;

[0193] A multi-view image acquisition module acquires the building appearance from different viewpoints to obtain a multi-view image set;

[0194] The sparse depth map and edge map calculation module calculates and optimizes the camera parameters and sparse depth map corresponding to each image in the multi-view image set obtained by the multi-view image acquisition module based on the known camera intrinsic parameters and the SfM (Structure from Motion) algorithm, and uses the Canny operator to calculate the edge map corresponding to each image;

[0195] A depth map calculation module uses the multi-view sample set obtained by the multi-view image acquisition module to obtain a corresponding depth map through the ISENet model using the trained weights, wherein the multi-view sample set includes an RGB image of the building, a sparse depth map, and an edge map;

[0196] The dense building 3D point cloud model module is based on the multi-view image set obtained by the multi-view image acquisition module, the depth map calculated by the depth map calculation module and the camera parameters, and obtains a dense building 3D point cloud model based on the depth map synthesis point cloud algorithm.

[0197] Implementation Method 3

[0198] An embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory is used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected via a bus. Specifically, the processor implements any step in the first embodiment above by running the computer program stored in the memory.

[0199] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0200] The memory may include a read-only memory, a flash memory, and a random access memory, and provides instructions and data to the processor. A part or all of the memory may also include a nonvolatile random access memory.

[0201] It can be seen from the above that the electronic device provided by the embodiment of the present invention can implement the method for reconstructing a three-dimensional building model based on the improved MVSNet as described in the first embodiment by running a computer program. It can make full use of the advantages of neural networks in high-dimensional data modeling, fully learn the characteristics of complex structure and depth discontinuity of general buildings, and realize accurate reconstruction from multi-view images to complete point cloud models, thereby reducing manpower while ensuring the authenticity and reliability of the model.

[0202] It should be understood that if the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned implementation method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method implementations when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0203] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest range consistent with the principles and novel features disclosed herein.

[0204] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the implementation method can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method implementation method, which will not be repeated here.

[0205] It should be noted that the methods and detailed examples provided in the above embodiments can be combined with the devices and equipment provided in the embodiments, and references can be made to each other, and no further details will be given.

[0206] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0207] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal equipment and method can be implemented in other ways. For example, the device / equipment implementation described above is only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0208] The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above-mentioned embodiments, or replace some of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for reconstructing a three-dimensional building model based on an improved MVSNet, characterized in that: The method comprises the following steps, Step 1: Use the DTU dataset to expand the sparse depth map dataset and edge map dataset corresponding to the image set; Step 2: Use the expanded DTU dataset in step 1 to train the proposed network ISENet to obtain the optimal weight; Step 3: Use a drone equipped with a camera to fly around the building, and control the camera to take pictures of the building's appearance from different perspectives to obtain a multi-perspective picture set; Step 4: Based on the known camera intrinsic parameters and SfM algorithm, calculate and optimize the camera parameters and sparse depth map corresponding to each image in the multi-view image set obtained in step 3, and use the Canny operator to calculate the edge map corresponding to each image; Step 5: Using the multi-view sample set obtained in step 3 and step 4, the corresponding depth map is obtained through the ISENet model using the trained weights, wherein the multi-view sample set includes an RGB image of the building, a sparse depth map, and an edge map; Step 6: Based on the multi-view image set in step 3, the depth map obtained in step 5 and the camera parameters, a dense 3D point cloud model of the building is obtained based on the depth map synthesis point cloud algorithm.

2. The method for reconstructing a three-dimensional building model according to claim 1, characterized in that: The step 1 specifically comprises: Step 1.1: According to the 49 camera parameter projections given in the dataset, the coordinates of each sparse point cloud are converted from the world coordinate system to the pixel coordinate system to obtain 49 sparse depth maps corresponding to each scene, and a total of 128 sparse depth map datasets corresponding to the scenes can be obtained; Step 1.2: Perform Gaussian filtering on all the images in the DTU dataset to remove noise and smooth the images; Step 1.3: Using the Sobel operator, calculate the gradient strength and direction of each pixel in the image: Step 1.4: Use non-maximum suppression to remove points with weak gradient strength and only retain local maximum points; Step 1.5: Set the low threshold to 150 and the high threshold to 200, so that points with gradient strength less than 150 are marked as non-edge points, and points with gradient strength greater than 200 are marked as edge points; when a pixel gradient is between 150 and 200, it is marked as an edge point only when it is adjacent to a pixel with gradient strength greater than 200; Step 1.6: For all pixels marked as edge points, if their neighboring pixels are also edge points, they are connected; thus, the edge map dataset of the DTU dataset is obtained.

3. The method for reconstructing a three-dimensional building model according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: Input the sparse depth map dataset, edge map dataset, and original image dataset obtained in step 1 into the feature pyramid network in a multi-view manner to obtain multi-view multi-stage features corresponding to the three datasets; Step 2.2: Use the sparse depth map to guide the depth division to form a projection volume for subsequent projection; Step 2.3: For the reference view of the smallest coarse feature obtained in step 2.1, copy its features on each assumed surface of the projection body obtained in step 2.2, and project the coarse features of the source view onto the projection body according to the homography transformation to form a feature body; Step 2.4: Aggregate the features of different perspectives using variance to obtain the cost volume; Step 2.5: Use 3D U-Net to regularize the cost volume obtained in step 2.4 and fuse the features of the three modalities; after multiple convolutional layers and pooling layers, set the last layer of the model to the Softmax layer to obtain the probability volume P; Step 2.6: The depth map estimated in the first stage is obtained by weighted summation of the depth probabilities of each pixel, and back-propagated with the true depth map using L1 loss; Step 2.7: Calculate the gradients of the depth map estimated in the first stage in the u and v directions respectively, then synthesize them into the total gradient and perform logarithmic normalization: Set the threshold to 0.15 so that G t A pixel value greater than 0.15 indicates that the position is a depth edge, otherwise it is not a depth edge, thereby obtaining a depth edge mask; At the same time, the edge atlas obtained in step 2.7 is intersected with the corresponding depth edge mask to obtain the final edge mask; a point where the edge mask is 0 indicates that the pixel position is not an edge, and a point where it is 1 indicates that the pixel position belongs to an edge; Step 2.8: Take the probability volume of step 2.5 and the edge mask of step 2.7 and find the two depth hypothesis values ​​d with the highest probability at the edge top2 ; Step 2.9: Upsample the depth map and edge mask obtained in step 2.7 for the second stage of depth segmentation, where the depth map adopts trilinear interpolation and the edge mask adopts nearest neighbor interpolation; take the upsampled depth map as the reference, take the depth of each point as the center, divide 32 depth assumption planes along the depth direction, and set the depth interval to 5; in addition, for the edge position corresponding to the edge mask, take the d obtained in step 2.8 as the reference, top2 16 depth assumption planes are re-divided as the center of the two depth values, and the divisions at the original edge positions are replaced to obtain the projection body of the second stage; Step 2.10: Project the second-stage features obtained in step 2.1 in the same manner as step 2.3 to obtain the feature body of the second stage, then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the second stage, and continue to repeat steps 2.7, 2.8, and 2.9 to obtain the projection body of the third stage. At this time, the number of depth assumption planes is 8 and the depth interval is 2.5; Step 2.11: Project the third-stage features obtained in step 2.1 in the same manner as step 2.3 to obtain the feature body of the third stage, and then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the third stage; if it is the fourth stage, continue to repeat steps 2.7, 2.8, and 2.9 to obtain the fourth-stage projection body. At this time, the number of depth assumption planes is 8 and the depth interval is 1.25; Step 2.12: If it is four stages, project the fourth stage features obtained in step 2.1 according to the method of step 2.3 to obtain the feature body of the fourth stage, and then repeat steps 2.4, 2.5, and 2.6 to obtain the depth map of the fourth stage; Step 2.13: The total loss considering the losses at each stage is given by the following formula: In the formula, Loss total is the total loss; s represents the number of stages; p valid is a valid set of true value pixels, which means that only the subject within the mask is calculated, and the background outside the mask is not calculated; λ s is the weight of the loss at each stage. If it is a three-stage model, the weights of the first, second, and third stage losses are 0.5, 1.0, and 2.0 respectively. If it is a four-stage model, the weights of the first, second, third, and fourth stage losses are 0.5, 1.0, 2.0, and 4.0 respectively. d s (p) represents the true value of each stage, Indicates the estimated value of each pixel at each stage; Step 2.14: Follow the above steps to train ISENet for 16 rounds to obtain the final training weights, and select the optimal weights as model parameters; the initial learning rate is 0.001, and the learning rate is halved in the 10th and 12th rounds of training, and the optimizer is Adam.

4. The method for reconstructing a three-dimensional building model according to claim 1, characterized in that: Specifically, step 3 includes selecting clear weather with high visibility, controlling the drone equipped with a high-definition camera to fly multiple circles around the target building at different altitudes, and controlling the camera to shoot from multiple perspectives; The distance between the drone and the building surface was kept at about 4 meters, and the photo content overlap rate was controlled at about 50%. Based on the built-in fixed-point cruise algorithm of the drone and manual camera posture adjustment, the original multi-view picture set S0{X i |i=1,...,N0}, a total of N0 pictures.

5. The method for reconstructing a three-dimensional building model according to claim 2, characterized in that: The step 4 specifically includes the following steps: Step 4.1: Preprocess the images obtained from the multi-view sample set S0 to filter out meaningless scene images that do not contain the main building, and obtain the processed sample set S{X i |i=1,...,N}, a total of N pictures. Step 4.2: Obtain the equivalent focal length f and image resolution from the camera manufacturer's manual as the known initial parameters of the camera intrinsic parameters. Step 4.3: Calculate camera parameters and sparse point cloud based on SfM algorithm. Step 4.4: Restore the sparse depth map corresponding to each image based on the correspondence between the sparse point cloud and the image, specifically according to step 2.

3. Step 4.5: Obtain the edge map of each image based on the Canny operator, specifically according to steps 2.4, 2.5, 2.6, and 2.

7.

6. The method for reconstructing a three-dimensional building model according to claim 2, characterized in that: The step 5 specifically includes the following steps: Step 5.1: Preprocess the image and use the bilinear interpolation method to downsample the width and height of the image by 1 / 2; Step 5.2: Select the scale-invariant feature transformation algorithm to extract feature points; Step 5.3: The size and direction of the gradient in the 16×16 region near the feature point are formed into a 128-dimensional vector according to the histogram statistics method, and it is normalized to obtain the feature point descriptor used for matching. Step 5.4: Calculate the camera parameters based on the binocular stereo geometry method. Step 5.5: Use multiple views to construct an over-constrained system and use the least squares method to solve X: Step 5.6: Use the bundle adjustment algorithm to minimize reprojection, optimize the position information of the sparse point cloud, and finally output the sparse point cloud.

7. The method for reconstructing a three-dimensional building model according to claim 2, characterized in that: The step 6 specifically includes the following steps: Step 6.1: Fill and average pool the probability volume P obtained in step 3.5 corresponding to each depth map, calculate the depth regression index and output the photometric consistency confidence map to measure the similarity of color information at the same point from different perspectives; Step 6.2: Perform geometric consistency check on the obtained depth map on no less than 4 views. Specifically, any point on the reference view depth map is projected into the world coordinate system, then projected from the world coordinate system to the source view, and then reversely projected back to the reference view, and the degree of deviation of the pixel from the original pixel before and after the projection is calculated. A small deviation indicates good geometric consistency; Step 6.3: Filter the depth map using the masks obtained in step 6.1 and step 6.2 to obtain a filtered depth map; Step 6.4: Using the filtered depth map obtained in step 6.3 and its corresponding camera parameters, project the points on the depth map into the world coordinate system, and assign the color information of the RGB image corresponding to each point to the point cloud to obtain the final dense building point cloud model.

8. A building 3D model reconstruction system based on improved MVSNet, characterized in that: The system uses the building three-dimensional model reconstruction method based on the improved MVSNet as described in any one of claims 1 to 7, and the system includes: The dataset expansion module uses the DTU dataset to expand the sparse depth map dataset and edge map dataset corresponding to the image set; The optimal weight calculation module uses the expanded DTU dataset to train the proposed network ISENet to obtain the optimal weight; A multi-view image acquisition module acquires the building appearance from different viewpoints to obtain a multi-view image set; The sparse depth map and edge map calculation module calculates and optimizes the camera parameters and sparse depth map corresponding to each image in the multi-view image set obtained by the multi-view image acquisition module based on the known camera intrinsic parameters and the SfM algorithm, and uses the Canny operator to calculate the edge map corresponding to each image; A depth map calculation module uses the multi-view sample set obtained by the multi-view image acquisition module to obtain a corresponding depth map through the ISENet model using the trained weights, wherein the multi-view sample set includes an RGB image of the building, a sparse depth map, and an edge map; The dense building 3D point cloud model module is based on the multi-view image set obtained by the multi-view image acquisition module, the depth map calculated by the depth map calculation module and the camera parameters, and obtains a dense building 3D point cloud model based on the depth map synthesis point cloud algorithm.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images