Method and device for multi-view 3D reconstruction of buildings based on sky artifact removal

Abstract

The present application relates to the technical field of three-dimensional reconstruction, and discloses a method and device for multi-view three-dimensional reconstruction of a building based on sky artifact removal. The method comprises: collecting a multi-view image set of a building, and performing resolution optimization processing on the multi-view image set to obtain input image data; performing sky region segmentation on the input image data through a U²‑Net network to generate a sky region mask map; performing depth map estimation analysis on the input image data through a block matching algorithm based on a multi-scale window to obtain a depth map set; based on the sky region mask map, filtering the depth estimation data corresponding to the sky region in the depth map set to obtain a depth map after the sky region is removed; performing frame-by-frame fusion on the depth map after the sky region is removed to generate a three-dimensional point cloud model. The present application improves the efficiency and accuracy of the multi-view three-dimensional reconstruction method of a building based on sky artifact removal.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction, and more particularly to a method for removing sky artifacts from a building multi-view image. Figure 3 Maintenance and reconstruction method and device. Background Art

[0002] Multi-view Figure 3 As an important technology in the field of computer vision, multi-dimensional reconstruction is the core of which is to restore three-dimensional spatial information from two-dimensional images taken from different angles, thereby generating a high-precision three-dimensional model. This technology has shown broad application prospects and great value in many fields such as building mapping, cultural heritage protection, virtual reality, game development, robot navigation, and medical image processing. However, despite the Figure 3 Although significant progress has been made in 3D reconstruction techniques, there are still some challenges in practical applications, especially when processing images containing sky areas.

[0003] When shooting complex scenes such as buildings, the sky area often inevitably appears in the image as a background. However, the sky area usually lacks sufficient texture and depth information, which makes it difficult to accurately calculate its depth value in the depth estimation stage. Traditional methods often generate a large amount of erroneous depth information when dealing with such weakly textured areas, thereby introducing obvious sky artifacts in the final 3D reconstruction results. These artifacts not only reduce the accuracy of the point cloud model, but also seriously affect the visual effect of the model, limiting the further application of 3D reconstruction technology in related fields. In addition to the sky artifact problem, traditional multi-view Figure 3 Multi-dimensional reconstruction methods also face other challenges in the depth estimation stage. For example, for complex structures such as buildings, their surfaces may contain multiple materials and colors, which increases the difficulty of feature extraction and matching. In addition, factors such as changes in lighting conditions, occlusion problems, and image noise may also affect the accuracy of depth estimation. Therefore, how to achieve accurate and robust depth estimation in complex scenes is an important issue for multi-viewing. Figure 3 It is one of the key issues that need to be solved in the reconstruction technology. Summary of the invention

[0004] This application provides a building multi-view based on sky artifact removal Figure 3 Dimensional reconstruction method and device for improving multi-viewing of buildings based on sky artifact removal Figure 3 The efficiency and accuracy of the reconstruction method.

[0005] In the first aspect, the present application provides a building multi-viewing method based on sky artifact removal. Figure 3 dimensional reconstruction method, the building multi-view based on sky artifact removal Figure 3The three-dimensional reconstruction method includes: collecting a multi-view image set of a building, and performing resolution optimization processing on the multi-view image set to obtain input image data; performing sky area segmentation on the input image data through a U²-Net network to generate a sky area mask map; performing depth map estimation analysis on the input image data through a block matching algorithm based on a multi-scale window to obtain a depth map set; based on the sky area mask map, filtering the depth estimation data corresponding to the sky area in the depth map set to obtain a depth map after the sky area is eliminated; and performing frame-by-frame fusion on the depth map after the sky area is eliminated to generate a three-dimensional point cloud model.

[0006] In the second aspect, the present application provides a building multi-viewing method based on sky artifact removal. Figure 3 dimensional reconstruction method system, the building multi-view based on sky artifact removal Figure 3 The maintenance and reconstruction method system includes:

[0007] The acquisition module is used to acquire a multi-view image set of a building, and perform resolution optimization processing on the multi-view image set to obtain input image data; the segmentation module is used to perform sky area segmentation on the input image data through a U²-Net network to generate a sky area mask map; the analysis module is used to perform depth map estimation analysis on the input image data through a block matching algorithm based on a multi-scale window to obtain a depth map set; the filtering module is used to filter the depth estimation data corresponding to the sky area in the depth map set based on the sky area mask map to obtain a depth map after the sky area is eliminated; the fusion module is used to perform frame-by-frame fusion on the depth map after the sky area is eliminated to generate a three-dimensional point cloud model.

[0008] In the technical solution provided by this application, a block matching algorithm based on a multi-scale window is used to adaptively adjust the hypothesized sampling area, and a multi-scale matching window is used for depth estimation, which significantly improves the depth estimation effect in weak texture areas, making the reconstructed three-dimensional model in these areas more accurate and complete. The U2-Net network is used to accurately segment the sky area in the image, generate a sky area mask, and filter out the depth estimation of the sky area in the point cloud reconstruction stage, effectively avoiding a large number of artifacts in the reconstructed point cloud and improving the quality of the reconstructed model. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0010] Figure 1The multi-view image of a building based on sky artifact removal in the embodiment of the present application Figure 3 A schematic diagram of an embodiment of a dimensional reconstruction method;

[0011] Figure 2 A schematic diagram of a checkerboard propagation method in an embodiment of the present application;

[0012] Figure 3 The multi-view image of a building based on sky artifact removal in the embodiment of the present application Figure 3 A schematic diagram of an embodiment of a dimensional reconstruction method system. DETAILED DESCRIPTION

[0013] The embodiment of the present application provides a building multi-view based on sky artifact removal. Figure 3 Dimensional reconstruction method and device. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0014] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, the multi-viewing method of buildings based on sky artifact removal is as follows: Figure 3 An embodiment of a dimensional reconstruction method includes:

[0015] Step S101, collecting a multi-view image set of a building, and performing resolution optimization processing on the multi-view image set to obtain input image data;

[0016] Step S102, performing sky area segmentation on the input image data through the U²-Net network to generate a sky area mask map;

[0017] Step S103, performing depth map estimation analysis on the input image data by using a multi-scale window-based block matching algorithm to obtain a depth atlas;

[0018] Step S104: filtering the depth estimation data corresponding to the sky area in the depth atlas based on the sky area mask image to obtain a depth map after the sky area is removed;

[0019] Step S105: fuse the depth map after the sky area is removed frame by frame to generate a three-dimensional point cloud model.

[0020] Specifically, it can be understood that the execution subject of the present application can be a building multi-view based on sky artifact removal. Figure 3 The maintenance and reconstruction method system may also be a terminal or a server, which is not limited here. The embodiment of the present application is described by taking a server as an execution subject as an example.

[0021] The building is photographed from different angles by a monocular camera to obtain a set of multi-view images. These images are denoised, color corrected, and sharpened to improve image quality. Subsequently, the processed images are adjusted to a uniform resolution, such as 384×384 pixels, to form standardized input image data. This process ensures consistency and efficiency in subsequent processing. Next, the U²-Net network is used to segment the sky region of the input image data. U²-Net is a deep learning model specifically designed for image segmentation tasks. The network extracts image features and generates pixel-level probability maps through multiple layers of convolution and pooling operations. The probability map represents the probability that each pixel belongs to the sky region. By setting a threshold (such as 0.5), the probability map is converted into a binary mask, where pixels above the threshold are marked as sky regions and pixels below the threshold are marked as non-sky regions. Finally, the mask is adjusted to the same resolution as the original input image to obtain an accurate sky region mask map. Subsequently, a multi-scale window-based block matching algorithm is used to analyze the depth map estimation of the input image data. The algorithm first randomly initializes the plane parameters for each pixel of each frame of the image, including the normal vector and the depth value. Then, adaptive checkerboard sampling is used to obtain the optimal plane parameters of eight regions in the neighborhood of each pixel. A structured information matrix is ​​constructed using these parameters, and the best matching image subset for each pixel is analyzed. Spatial propagation and random optimization are performed on these subsets to update and refine the plane parameters. A multi-scale strategy is adopted to gradually reduce the matching window size, iteratively optimize the plane parameters, and improve the depth estimation accuracy. Finally, the depth value is calculated based on the optimized plane parameters to generate the initial depth map, and the final depth atlas is obtained through median filtering.

[0022] Based on the sky area mask map, the depth estimation data corresponding to the sky area in the depth atlas is filtered. First, the mask map is aligned with each frame of the depth map at the pixel level to identify the sky area pixels in the depth map. The depth estimation data of these pixels are marked and removed to obtain a preliminary filtered depth map. Then edge smoothing is performed to ensure a natural transition between the sky area and the non-sky area, and a depth map after the sky area is removed is obtained. Finally, the depth map after the sky area is removed is fused frame by frame to generate a 3D point cloud model. This includes transforming the depth map from the camera coordinate system to the world coordinate system, using the iterative closest point (ICP) algorithm for multi-frame registration, and generating an initial 3D point cloud with color attributes based on the depth value and image color information. Then, noise removal and outlier filtering are performed, and the bundle adjustment algorithm is used for global optimization to minimize the reprojection error. Finally, surface reconstruction and texture mapping are performed to obtain a high-quality 3D point cloud model.

[0023] For example, in a quality control project at a construction site, this method can be used to effectively reconstruct a 3D model of the construction site. Engineers used a high-resolution camera to take 100 images of the building from different angles, with the original resolution of each image being 3000×2000 pixels. After preprocessing, the images were resized to 384×384 pixels, greatly reducing the amount of data while retaining key features. The U²-Net network successfully identified the sky area in the image, with an average of about 20% of the area in each image being marked as sky. In the depth estimation stage, the multi-scale block matching algorithm was used through three iterations, and the window size was gradually reduced from the initial 21×21 pixels to 5×5 pixels, significantly improving the accuracy of the depth map.

[0024] In the embodiment of the present application, a block matching algorithm based on a multi-scale window is used to adaptively adjust the assumed sampling area, and a multi-scale matching window is used for depth estimation, which significantly improves the depth estimation effect in weak texture areas, making the reconstructed three-dimensional model in these areas more accurate and complete. The U2-Net network is used to accurately segment the sky area in the image, generate a sky area mask, and filter out the depth estimation of the sky area in the point cloud reconstruction stage, which effectively avoids a large number of artifacts in the reconstructed point cloud and improves the quality of the reconstructed model.

[0025] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0026] (1) Use a monocular camera to photograph the building from different angles to obtain a multi-view image set;

[0027] (2) Denoising and color correction are performed on each image in the multi-view image set to obtain a corrected image set;

[0028] (3) performing image sharpening on the corrected image set to obtain a sharpened image set;

[0029] (4) Adjust the resolution size of the sharpened image set to obtain input image data.

[0030] Specifically, a monocular camera is used to shoot the building from different angles to obtain a multi-view image set. A monocular camera refers to a camera with only one lens, which obtains images of the building from different perspectives by changing the shooting position and angle. In actual operation, the photographer needs to move around the building and take photos at different positions to ensure that all sides and details of the building are covered. After obtaining the multi-view image set, the next step is to denoise and color correct each image to obtain a corrected image set. The denoising process aims to reduce random noise in the image and improve image quality. Common denoising methods include Gaussian filtering, median filtering, etc. Taking median filtering as an example, it reduces noise by replacing the value of each pixel with the median of the values ​​of its surrounding pixels. Color correction adjusts the color balance and saturation of the image to make the image more realistic and natural. It usually includes operations such as white balance adjustment and color space conversion.

[0031] Perform image sharpening on the corrected image set to obtain a sharpened image set. The purpose of image sharpening is to enhance the edges and details of the image and make the image clearer. Commonly used sharpening methods include Laplace operator, high-pass filtering, etc. For example, when using the Laplace operator for image sharpening, the Laplace transform of the image is first calculated, and then the result is added to the original image to enhance the edges and details. The last step is to adjust the resolution size of the sharpened image set to obtain the input image data. The purpose of this step is to unify the size of all images for subsequent processing. Commonly used image scaling methods include bilinear interpolation, bicubic interpolation, etc. Taking bilinear interpolation as an example, it calculates the value of the new pixel by weighted average of the surrounding four pixels, thereby achieving image enlargement or reduction.

[0032] For example, taking the facade of a five-story office building as an example, the photographer may need to take about 100 photos from the ground and different floors to cover all the external features of the building. These original images may have different degrees of noise, color distortion and blur. Through denoising, the number of noise points in each image may be reduced from the original 5000 or so to less than 500. The color correction process may adjust the color temperature of the image from a cool 7000K to a more natural 5500K. Image sharpening may increase the edge contrast of the image by 20%, making the outline and details of the building clearer. Finally, all images may be uniformly adjusted to a resolution of 3840x2160 to meet the needs of subsequent processing.

[0033] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0034] (1) Input the input image data into the U²-Net network for forward propagation processing to obtain a probability map representing the probability that each pixel belongs to the sky area;

[0035] (2) Convert the probability map into a binary mask to generate a sky area mask map, where pixels with probabilities higher than a preset threshold are marked as sky areas, and pixels with probabilities lower than the preset threshold are marked as non-sky areas;

[0036] (3) Adjust the sky area mask map to the same resolution as the input image data to obtain the sky area mask map.

[0037] Specifically, in the BIM-based intelligent supervision system for building construction quality information, the U²-Net network is used to segment the sky area of ​​the input image data. First, the input image data is input into the U²-Net network for forward propagation processing. U²-Net is a deep learning model specially designed for image segmentation tasks. During the forward propagation process, the input image undergoes multiple layers of convolution, pooling, and upsampling operations, and finally outputs a probability map of the same size as the input image. This probability map represents the probability that each pixel belongs to the sky area. U²-Net takes into account the 6 saliency probability maps output by the intermediate process. And the final significance probability map , the loss function is defined as:

[0038]

[0039] in, is the significance probability map The error, is the error term The weight of is the image size.

[0040] Each error term is calculated by binary cross entropy:

[0041]

[0042] in, are pixel coordinates, is the coordinate The true value of the pixel at is the significance probability map Central coordinates The pixel value at .

[0043] The Adam optimizer is used for model training, and the corresponding hyperparameters are set to , The resolution of input and output data is set to 384×384, the batch size is set to 10, and the learning rate is set to , model training is completed.

[0044] The view image set is preprocessed and adjusted to a resolution of 384 × 384. The preprocessed image is input into the trained U2-Net model to obtain the prediction result output by the model, which is a probability map of the same size as the input image, indicating the probability of each pixel belonging to the sky area.

[0045] After obtaining the probability map, the next step is to convert the probability map into a binary mask to generate a sky area mask map. This is done by setting a preset threshold. Specifically, for each pixel in the probability map, if its probability value is higher than the preset threshold (for example, 0.5), the pixel is marked as a sky area (value 1); conversely, if the probability value is lower than the threshold, it is marked as a non-sky area (value 0). In this way, a binary mask map is obtained, where 1 represents the sky area and 0 represents the non-sky area. Finally, the sky area mask map is adjusted to the same resolution as the input image data. This step is necessary because the image size may change during the processing of the neural network. By adjusting the mask map back to the resolution of the original input image, the exact correspondence between the sky area mask and the original image is ensured.

[0046] For example, in a quality control project at a construction site, U²-Net is used to segment the sky area of ​​a building image with a resolution of 3840x2160. First, the image is input into the U²-Net network, and a probability map of the same size is obtained after forward propagation. Assume that in this probability map, the pixel values ​​in the upper half are generally higher than 0.8, while those in the lower half are generally lower than 0.2. Set the threshold to 0.5 and binarize the probability map. After processing, about 40% of the pixels are marked as sky areas (value 1), mainly concentrated in the upper half of the image, which is consistent with the actual situation of the building image. Finally, confirm that the binarized mask image maintains a resolution of 3840x2160, which is completely consistent with the original input image.

[0047] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0048] (1) Randomly initialize each frame of the input image data and generate initial plane parameters for each pixel;

[0049] (2) Based on the initial plane parameters of each pixel, the optimal plane parameters of eight regions are obtained within the neighborhood of each pixel through an adaptive checkerboard sampling algorithm;

[0050] (3) constructing a structured information matrix using the optimal plane parameters of the eight regions of each pixel, and obtaining the best matching image subset for each pixel based on the structured information matrix analysis;

[0051] (4) Perform spatial propagation and random optimization on the best matching image subset of each pixel, update and refine the plane parameters of each pixel, and obtain the optimized plane parameters of each pixel;

[0052] (5) Based on the preset multi-scale strategy, the optimized plane parameters of each pixel are iteratively optimized by gradually reducing the matching window size to obtain the multi-scale optimized plane parameters of each pixel;

[0053] (6) Calculate the depth value based on the multi-scale optimized plane parameters of each pixel and generate an initial depth map for each pixel;

[0054] (7) Perform value filtering on the initial depth map of each pixel to obtain the target depth map of each pixel, and summarize the target depth maps of each pixel to obtain a depth atlas.

[0055] Specifically, for the current frame image , according to the mean distribution for each pixel in the image Generate a random plane hypothesis Under the condition of known plane assumption, the homography transformation matrix is ​​used You can get the current frame image Pixels In matching image The corresponding position , calculate the reference view With matching view The homography matrix :

[0056]

[0057] Among them, the matrix and vector are the rotation matrix and translation vector between the two views, is the camera internal parameter. The most widely used photometric matching cost for block matching is the normalized cross-correlation algorithm. In practical use, the introduction of bilateral filtering weighting can obtain a more reliable photometric matching cost. The bilateral weight function is defined as follows:

[0058]

[0059] in, is the neighborhood pixel in the window, is the gray value standard deviation 3.0, is the pixel distance standard deviation 5.0, is the two-norm of the vector.

[0060] Then calculate the reference view With matching view The photometric matching cost :

[0061]

[0062] in, In pixels The set of pixels within the matching window centered at .

[0063] Can count pixels and matching image collection All matching images in Photometric matching cost , and then aggregate all the photometric matching costs with equal weights to obtain the pixel The photometric aggregation cost :

[0064]

[0065] After random initialization, the block matching algorithm assumes that adjacent pixels are likely to be in the same plane. Therefore, for each pixel in the current frame image And its corresponding plane assumption , the block matching algorithm will be in the pixel Sample neighboring pixels in the neighborhood of The plane assumption , if the neighborhood pixels The plane assumption The plane assumption More pixel-perfect The support plane of the pixel The plane assumption Replace with neighboring pixels The plane assumption Block matching updates and refines the plane hypothesis of all pixels in the current frame image through this spatial propagation method. Figure 2 As shown in the figure, the pure black pixels are the sampled neighborhood pixels, and the triangles are the pixels to be estimated. In the odd iterations, the square pixels in the sampling area are used to update the plane hypothesis of the circular pixels, while in the even iterations, the sampled circular pixels are used to update the plane hypothesis of the square pixels. Adaptive checkerboard sampling divides the neighborhood pixels into 8 regions, and adopts the survival of the fittest strategy in each region to select the plane hypothesis with the minimum aggregation cost for propagation.

[0066] For each pixel of the current frame image After obtaining the photometric matching costs of the eight regions, the structured information matrix is ​​constructed through the matching costs to infer the matching image set. The weight of is used to embed the sampled matching cost into the structured cost matrix:

[0067]

[0068] in, Indicates area Intrinsic Plane Hypothesis and Matching Images The matching cost.

[0069] For inappropriate matching images, all the matching costs corresponding to them are always high. Appropriate matching views usually correspond to some smaller matching costs. Set a reasonable matching cost threshold and give weight to matching images through voting decisions. Reliable matching cost threshold Defined as: in, Indicates The iterative process, is the initial matching cost threshold of 0.8, is the constant 2.

[0070] For matching images , define its good matching cost set as , whose bad matching cost set is , the two sets are calculated as follows:

[0071]

[0072] in, is the bad match cost threshold of 45.

[0073] Calculate the photometric aggregation cost of the sampling hypothesis to realize the propagation and update of the plane hypothesis. Perform random optimization on the plane hypothesis and perform three iterative optimizations at each scale.

[0074] 1) Spatial propagation

[0075] For any pixel in the current frame image and the corresponding plane assumption , if the area of ​​the chessboard sampling The best hypothesis within satisfy , then the plane hypothesis Pixels The plane assumption More reliable, As a pixel The current plane assumption Block matching uses checkerboard sampling and spatial propagation to obtain as reliable a plane hypothesis as possible for each pixel in the reference image.

[0076] 2) Random Optimization

[0077] For any pixel in the current frame image , there may be no suitable plane assumption in its neighborhood. Therefore, after spatial propagation, the pixels need to be The current plane assumption Perform random optimization. One is to randomly generate a plane hypothesis or , and secondly, add disturbance to the current plane hypothesis or The randomly generated plane hypothesis set can be obtained as follows:

[0078] If the plane hypothesis generated by random optimization is different from the original plane hypothesis More pixel-perfect The support plane of , that is, the photometric aggregation cost is less than , then the best hypothesis in the random hypothesis set is selected as the pixel The plane assumption.

[0079] The matching window size of the block matching algorithm is usually fixed. It is difficult for a fixed-size matching window to take into account the depth estimation of both texture-rich areas and weak-texture areas. The calculation is as follows:

[0080]

[0081] in, Indicates the maximum scale level, Indicates the current scale level. Sets the maximum scale level The larger the window is, the richer the texture information it contains. Therefore, the depth of the weak texture area is estimated through the scale 2 window first, and then the window is gradually reduced to refine the depth estimation result of the previous level. Finally, a more accurate estimation result is obtained in the scale 0 window, thereby achieving reliable estimation of the weak texture area.

[0082] For example, in a high-rise building construction project, 100 building exterior images were taken with a high-resolution camera, each with a resolution of 4000×3000 pixels. After random initialization of each image, 12 million initial plane parameters were obtained. Through adaptive checkerboard sampling, each pixel obtains the optimal plane parameters in its 8 neighborhood areas, generating a total of 96 million candidate plane parameters. After constructing the structured information matrix, assuming that on average 5 best matching images are selected for each pixel, a total of 60 million pixel-image matching pairs are formed. In the spatial propagation and random optimization stages, the plane parameters of each pixel are updated about 10 times, and a total of 1.2 billion parameter updates are performed. In the multi-scale optimization process, the matching window size is gradually reduced from 41×41 pixels to 5×5 pixels, and a total of 3 iterations are performed. In each iteration, the plane parameters of all 12 million pixels are optimized, and a total of 360 million parameter optimization operations are performed. The resulting initial depth map contains 12 million depth values. After median filtering, a target depth map with less noise and smoother is obtained. The depth maps of 100 images are aggregated to form a depth atlas containing 1.2 billion depth values, providing an accurate depth information foundation for subsequent 3D reconstruction and quality control.

[0083] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0084] (1) Align the sky area mask image with each frame of the depth map set at the pixel level to obtain the aligned mask image;

[0085] (2) Based on the aligned mask image, identify the pixel position corresponding to the sky area in the depth image and obtain the sky area pixel set;

[0086] (3) Mark the depth estimation data in the sky area pixel set to obtain the data set to be filtered; (4) Remove the data set to be filtered from the depth atlas to obtain the depth atlas after preliminary filtering;

[0087] (5) Perform edge smoothing on the depth atlas after preliminary filtering to obtain a depth map after removing the sky area.

[0088] Specifically, the sky area mask map is aligned with each frame of the depth map in the depth map set at the pixel level. Ensure that the mask map and the depth map are completely spatially corresponding, and the information at each pixel position can be accurately matched. The aligned mask map provides an accurate reference for subsequent sky area recognition. Based on the aligned mask map, the pixel position corresponding to the sky area in the depth map is identified to obtain the sky area pixel set. This step is achieved by comparing the pixel values ​​in the mask map. Pixels with a value of 1 in the mask map represent the sky area, and pixels with a value of 0 represent the non-sky area. By traversing each pixel of the depth map and checking the value of the corresponding mask map position, it is possible to determine which pixels belong to the sky area. The coordinate information of these pixels is collected into the sky area pixel set.

[0089] Then, the depth estimation data in the sky area pixel set is marked to obtain the data set to be filtered. In fact, the depth values ​​corresponding to the pixels in the sky area pixel set in the depth map are marked. These marked depth values ​​are usually unreliable because the sky area lacks valid depth information. Next, the data set to be filtered is removed from the depth atlas to obtain the depth atlas after preliminary filtering. This step actually sets the depth values ​​of the pixels marked as the sky area to invalid values ​​or removes them. In this way, the depth atlas after preliminary filtering no longer contains erroneous depth information of the sky area. Finally, the depth atlas after preliminary filtering is edge smoothed to obtain a depth map after the sky area is removed. The purpose of edge smoothing is to eliminate the sudden changes that may occur between the sky area and the non-sky area, and to ensure the continuity and natural transition of the depth map. This is usually achieved by applying an edge-preserving filter, such as bilateral filtering or guided filtering.

[0090] For example, in a quality control project at a large construction site, a series of building exterior images were taken using a high-resolution camera, each with a resolution of 4000×3000 pixels. After preliminary processing, the corresponding sky area mask map and initial depth map set were obtained. In the pixel-level alignment process, the resolution and position of the mask map and each frame of the depth map were ensured to be exactly the same. By analyzing the aligned mask map, it was found that in the 4000×3000 image, about 3 million pixels were marked as sky areas. These pixels are mainly distributed in the upper half of the image, accounting for 25% of the total number of pixels. In the depth map, the depth values ​​of these 3 million pixel positions were marked as data to be filtered. After removing these data to be filtered, about 25% of the area in the depth map was set to invalid values. In order to smooth the transition between the sky area and the building outline, a bilateral filter was applied to the depth map. The spatial standard deviation of the filter was set to 5 pixels, and the range standard deviation was set to 0.1 (assuming that the depth value has been normalized to the range of 0-1). After this processing, the depth value at the junction of the top of the building and the sky presents a smooth transition, avoiding abrupt boundaries.

[0091] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0092] (1) Transform the depth map after removing the sky area from the camera coordinate system to the world coordinate system to obtain a depth map in a unified coordinate system;

[0093] (2) Use the iterative closest point algorithm to perform multi-frame registration of depth maps in a unified coordinate system to obtain an aligned depth atlas;

[0094] (3) Generate an initial 3D point cloud with color attributes based on the depth values ​​of the aligned depth atlas and the corresponding image color information;

[0095] (4) Remove noise and filter outliers from the initial 3D point cloud to obtain optimized point cloud data;

[0096] (5) Use the bundle adjustment algorithm to globally optimize the optimized point cloud data, minimize the reprojection error, and obtain a candidate 3D point cloud model;

[0097] (6) Perform surface reconstruction and texture mapping on the candidate 3D point cloud model to obtain a 3D point cloud model.

[0098] Specifically, in the BIM-based intelligent supervision system for building construction quality information, the depth map after the sky area is removed needs to undergo a series of processing to generate an accurate three-dimensional point cloud model. First, the depth map after the sky area is removed is transformed from the camera coordinate system to the world coordinate system to obtain a depth map in a unified coordinate system. This step uses the camera pose information of each frame image to transform the depth data from the camera local coordinate system to the global world coordinate system through rigid body transformation. The transformation process includes rotation and translation operations to ensure that all depth maps are represented in the same coordinate system. The iterative closest point (ICP) algorithm is used to perform multi-frame registration of the depth map in the unified coordinate system to obtain an aligned depth atlas. The ICP algorithm minimizes the distance between two sets of point clouds in an iterative manner to achieve accurate alignment. The algorithm first finds the nearest neighbor for each point, and then calculates and applies the optimal rigid body transformation to minimize the sum of the squared distances between the two sets of points. This is repeated until convergence or the preset number of iterations is reached. Based on the depth values ​​of the aligned depth atlas and the corresponding image color information, an initial three-dimensional point cloud with color attributes is generated. Each depth map pixel corresponds to a 3D space point, whose coordinates are determined by the depth value and pixel position, and the color is directly obtained from the original image. In this way, each point contains not only position information but also color information, laying the foundation for subsequent texture mapping.

[0099] The initial 3D point cloud is subjected to noise removal and outlier filtering to obtain optimized point cloud data. Noise removal usually uses the statistical outlier filtering method to calculate the average distance from each point to its neighboring points, and set a threshold to remove abnormal points. For outlier filtering, the radius outlier filtering method can be used to set a sphere radius. If the number of points in the sphere is less than a certain threshold, the center point is considered an outlier and deleted. Subsequently, the optimized point cloud data is globally optimized using the bundle adjustment algorithm to minimize the reprojection error and obtain a candidate 3D point cloud model. Bundle adjustment is a technique that simultaneously optimizes 3D point coordinates and camera parameters to improve reconstruction accuracy by minimizing the reprojection error of all observation points. The algorithm constructs and solves a large-scale nonlinear least squares problem, iterating the optimization until convergence.

[0100] Finally, the candidate 3D point cloud model is subjected to surface reconstruction and texture mapping to obtain the final 3D point cloud model. The Poisson reconstruction algorithm is commonly used for surface reconstruction, which converts the point cloud into a continuous surface by solving the Poisson equation. Texture mapping maps the color information in the original image to the reconstructed surface to improve the visual realism of the model.

[0101] For example, in a construction quality supervision project of a 30-story building, 200 high-resolution images (4000x3000 pixels each) were collected. After preliminary processing, 200 frames of depth maps with the sky area removed were obtained. First, the 200 frames of depth maps were transformed from the camera coordinate system to the world coordinate system using the camera posture information obtained by the COLMAP software. Assuming that each frame of the depth map contains 10 million valid depth points, a total of 2 billion 3D points of the initial point cloud are generated. The ICP algorithm is used for multi-frame registration, with an average of 50 iterations between every two frames, and a total of 9900 (199*50) registration operations. During the registration process, about 100,000 sampling points are selected for matching each iteration, and the average alignment error between adjacent frames is less than 1 mm. When generating the initial point cloud with color, the RGB value corresponding to each point is directly extracted from the original image. Assuming that 15% of the points in the point cloud lack color information due to occlusion or other reasons, about 1.7 billion colored 3D points are finally obtained. In the noise removal and outlier filtering stage, the threshold for statistical outlier filtering was set to 2 standard deviations, the radius of the sphere for radial outlier filtering was 0.1 meters, and the minimum number of neighbors was 10. After this step, the number of point clouds was reduced by about 5%, leaving about 1.615 billion points. The bundle adjustment optimization involves solving a nonlinear least squares problem containing about 1.615 billion 3D point coordinates and 600 camera parameters (200 frames × 3 rotation parameters + 3 translation parameters). After 100 iterations of optimization, the average reprojection error was reduced from the initial 2 pixels to less than 0.5 pixels.

[0102] Finally, the optimized point cloud is converted into a mesh model using the Poisson reconstruction algorithm. The number of mesh faces is about 1 / 10 of the number of point clouds, that is, 161.5 million triangular faces. During the texture mapping process, the image with the best viewing angle is selected for each face to project and generate a high-resolution texture map. The resulting three-dimensional point cloud model not only has high geometric accuracy, but also has realistic surface texture, providing a reliable digital foundation for the accurate evaluation of building construction quality.

[0103] In a specific embodiment, based on a preset multi-scale strategy, in the step of iteratively optimizing the optimized plane parameters of each pixel by gradually reducing the matching window size, the side length L of the matching window is calculated as follows:

[0104] ;

[0105] in Indicates the maximum scale level, Indicates the current scale level and sets the maximum scale level .

[0106] The above multi-view method of building based on sky artifact removal in the embodiment of the present application Figure 3 The multi-dimensional reconstruction method of the building based on sky artifact removal in the embodiment of the present application is described below. Figure 3 For a description of the system's reconstruction methods, see Figure 3 In the embodiment of the present application, the multi-viewing method of buildings based on sky artifact removal is as follows: Figure 3 An embodiment of the reconstruction method system includes:

[0107] The acquisition module 201 is used to acquire a multi-view image set of a building and perform resolution optimization processing on the multi-view image set to obtain input image data;

[0108] The segmentation module 202 is used to perform sky region segmentation on the input image data through a U²-Net network to generate a sky region mask map;

[0109] An analysis module 203 is used to perform depth map estimation analysis on the input image data by using a multi-scale window-based block matching algorithm to obtain a depth map set;

[0110] A filtering module 204 is used to filter the depth estimation data corresponding to the sky area in the depth atlas based on the sky area mask map to obtain a depth map after the sky area is removed;

[0111] The fusion module 205 is used to fuse the depth map after the sky area is removed frame by frame to generate a three-dimensional point cloud model.

[0112] Through the collaboration of the above components, the block matching algorithm based on multi-scale windows is used to adaptively adjust the assumed sampling area, and a multi-scale matching window is used for depth estimation, which significantly improves the depth estimation effect in weak texture areas, making the reconstructed 3D model more accurate and complete in these areas. The U2-Net network is used to accurately segment the sky area in the image, generate a sky area mask, and filter out the depth estimation of the sky area in the point cloud reconstruction stage, effectively avoiding a large number of artifacts in the reconstructed point cloud and improving the quality of the reconstructed model.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0114] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for multi-view 3D reconstruction of a building based on sky artifact removal, characterized in that: The following steps are involved: Collecting a multi-view image set of a building, and performing resolution optimization processing on the multi-view image set to obtain input image data; The input image data is segmented into a sky area through a U²-Net network to generate a sky area mask map, including: inputting the input image data into the U²-Net network for forward propagation processing to obtain a probability map representing the probability that each pixel belongs to the sky area; converting the probability map into a binary mask to generate a sky area mask map, wherein pixels with probabilities higher than a preset threshold are marked as sky areas, and pixels with probabilities lower than the preset threshold are marked as non-sky areas; adjusting the sky area mask map to the same resolution as the input image data to obtain the sky area mask map; The input image data is subjected to depth map estimation and analysis by a block matching algorithm based on a multi-scale window to obtain a depth atlas, including: randomly initializing each frame of the input image data to generate initial plane parameters for each pixel; based on the initial plane parameters of each pixel, obtaining the optimal plane parameters of eight regions in the neighborhood of each pixel by an adaptive checkerboard sampling algorithm; constructing a structured information matrix using the optimal plane parameters of the eight regions of each pixel, and obtaining the best matching image subset of each pixel according to the structured information matrix analysis; performing spatial propagation and random optimization on the best matching image subset of each pixel, updating and refining the plane parameters of each pixel, and obtaining the optimized plane parameters of each pixel; based on a preset multi-scale strategy, the optimized plane parameters of each pixel are iteratively optimized by gradually reducing the matching window size to obtain the multi-scale optimized plane parameters of each pixel; calculating the depth value based on the multi-scale optimized plane parameters of each pixel to generate the initial depth map of each pixel; performing value filtering on the initial depth map of each pixel to obtain the target depth map of each pixel, and summarizing the target depth map of each pixel to obtain the depth atlas; Based on the preset multi-scale strategy, by gradually reducing the size of the matching window, in the step of iteratively optimizing the optimized plane parameters of each pixel, the side length L of the matching window is calculated as follows: in Indicates the maximum scale level, Indicates the current scale level and sets the maximum scale level ; Based on the sky area mask map, the depth estimation data corresponding to the sky area in the depth map set is filtered to obtain a depth map after the sky area is removed, including: pixel-level alignment of the sky area mask map with each frame depth map in the depth map set to obtain an aligned mask map; based on the aligned mask map, the pixel position corresponding to the sky area in the depth map is identified to obtain a sky area pixel set; the depth estimation data in the sky area pixel set is marked to obtain a data set to be filtered; the data set to be filtered is removed from the depth map set to obtain a preliminarily filtered depth map set; edge smoothing is performed on the preliminarily filtered depth map set to obtain a depth map after the sky area is removed; The depth map after the sky area is removed is fused frame by frame to generate a 3D point cloud model.

2. The method according to claim 1, characterized in that The step of collecting a multi-view image set of a building and performing resolution processing on the multi-view image set to obtain input image data includes: Using a monocular camera to photograph the building from different angles to obtain a multi-view image set; Denoising and color correcting each image in the multi-view image set to obtain a corrected image set; Performing image sharpening on the corrected image set to obtain a sharpened image set; The resolution and size of the sharpened image set are adjusted to obtain the input image data.

3. The method according to claim 1, characterized in that The step of fusing the depth map after the sky area is removed frame by frame to generate a three-dimensional point cloud model includes: The depth map after the sky area is removed is transformed from the camera coordinate system to the world coordinate system to obtain a depth map in a unified coordinate system; Using an iterative closest point algorithm to perform multi-frame registration on the depth map in the unified coordinate system to obtain an aligned depth atlas; Based on the depth values ​​of the aligned depth atlas and the corresponding image color information, generating an initial three-dimensional point cloud with color attributes; Performing noise removal and outlier filtering on the initial three-dimensional point cloud to obtain optimized point cloud data; Using a bundle adjustment algorithm to globally optimize the optimized point cloud data, minimize the reprojection error, and obtain a candidate three-dimensional point cloud model; Performing surface reconstruction and texture mapping on the candidate three-dimensional point cloud model to obtain the three-dimensional point cloud model.

4. A device for multi-view 3D reconstruction of buildings based on sky artifact removal, characterized in that: The building multi-view 3D reconstruction device based on sky artifact removal comprises: An acquisition module, used for acquiring a multi-view image set of a building, and performing resolution optimization processing on the multi-view image set to obtain input image data; A segmentation module, for performing sky area segmentation on the input image data through a U²-Net network to generate a sky area mask map, comprising: inputting the input image data into the U²-Net network for forward propagation processing to obtain a probability map representing the probability that each pixel belongs to the sky area; converting the probability map into a binary mask to generate a sky area mask map, wherein pixels with probabilities higher than a preset threshold are marked as sky areas, and pixels with probabilities lower than the preset threshold are marked as non-sky areas; adjusting the sky area mask map to the same resolution as the input image data to obtain the sky area mask map; An analysis module is used to perform depth map estimation analysis on the input image data through a block matching algorithm based on a multi-scale window to obtain a depth atlas, including: randomly initializing each frame of the input image data to generate initial plane parameters for each pixel; based on the initial plane parameters of each pixel, obtaining the optimal plane parameters of eight regions in the neighborhood of each pixel through an adaptive checkerboard sampling algorithm; constructing a structured information matrix using the optimal plane parameters of the eight regions of each pixel, and obtaining the best matching image subset for each pixel according to the structured information matrix analysis; performing spatial propagation and random optimization on the best matching image subset of each pixel, updating and refining the plane parameters of each pixel, and obtaining the optimized plane parameters of each pixel; based on a preset multi-scale strategy, iteratively optimizing the optimized plane parameters of each pixel by gradually reducing the matching window size to obtain the multi-scale optimized plane parameters of each pixel; calculating the depth value based on the multi-scale optimized plane parameters of each pixel to generate an initial depth map for each pixel; performing value filtering on the initial depth map of each pixel to obtain a target depth map of each pixel, and summarizing the target depth maps of each pixel to obtain the depth atlas; Based on the preset multi-scale strategy, by gradually reducing the size of the matching window, in the step of iteratively optimizing the optimized plane parameters of each pixel, the side length L of the matching window is calculated as follows: in Indicates the maximum scale level, Indicates the current scale level and sets the maximum scale level ; A filtering module is used to filter the depth estimation data corresponding to the sky area in the depth map set based on the sky area mask map to obtain a depth map after the sky area is eliminated, including: performing pixel-level alignment on the sky area mask map with each frame of the depth map in the depth map set to obtain an aligned mask map; based on the aligned mask map, identifying the pixel position corresponding to the sky area in the depth map to obtain a sky area pixel set; marking the depth estimation data in the sky area pixel set to obtain a data set to be filtered; removing the data set to be filtered from the depth map set to obtain a depth map set after preliminary filtering; performing edge smoothing on the depth map set after preliminary filtering to obtain a depth map after the sky area is eliminated; The fusion module is used to fuse the depth map after the sky area is removed frame by frame to generate a three-dimensional point cloud model.

Images