A method for completing three-dimensional building models

Abstract

The present invention provides a method for completing a three-dimensional model of a building, which belongs to the technical field of three-dimensional modeling. The method obtains an image dataset through aerial photography, obtains a primary depth map based on 2D images in the image dataset, performs hole identification and contour generation based on the primary depth map, and searches for effective views that can fully reflect the different contour surface features of the building to be modeled. These effective views can be a small number of key views, which are used for completing the depth information. Only a small amount of calculation is required to complete a very accurate completion update. At the same time, since a small number of key views are processed, enhanced feature recognition can be introduced in the completion process, which helps to understand the true meaning of each object in the view, such as glass, and achieve more accurate completion operations.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional modeling, and in particular relates to a method for completing a three-dimensional model of a building. Background Art

[0002] 3D building reconstruction technology is a highly efficient digital simulation method that accurately captures real-world buildings and converts them into 3D models in a computer. This technology has been applied in a number of key areas, including but not limited to precise analysis of urban planning, digital preservation of architectural heritage, visualization of real estate development, creation of visual effects for the gaming and film industries, and the creation of immersive experiences in augmented reality and virtual reality.

[0003] During the three-dimensional reconstruction process, the quality of the model is affected by many factors such as sensor resolution, lighting conditions, building occlusion, and surface reflection and transparency. The initial model is often incomplete or inaccurate. In order to solve these problems, the completion step is particularly critical. In the past, the completion of three-dimensional models was mainly done manually. Although this method is meticulous, it is inefficient and time-consuming. With the development of technology, automated completion methods using machine learning have significantly improved efficiency and effectiveness. However, current machine learning algorithms still have limitations in model completion, especially in identifying the true meaning of each part of the model. This leads to the completion of data void areas often relying on similarity matching principles, and sometimes it is impossible to accurately restore the original characteristics of the structure. For example, Chinese patent ZL202310902348.9 discloses a method for reconstructing a three-dimensional model, which extends the adjacent structure to the void and determines the target structure of the void based on the similarity between the extended part and the adjacent structure, thereby restoring, extending or remodeling the void in the original spatial model. This method restores the material of the void by referring to the surrounding material of the void, and the material patched at the void is the same or similar to the material of the adjacent structure. Since modern buildings usually contain a large number of weak texture structures, such as glass, these weak texture structures are difficult to be recognized by machine vision. If the completion method in the above patent is used, it is easy to complete the glass in the way of walls or other objects, resulting in inconsistency with the actual building situation. For another example, Chinese patent ZL202311664268.0 discloses a method for completing missing data in ancient building point clouds. Although it utilizes semantic information, this information is manually constructed rather than generated by machine autonomous learning. This not only requires a huge workload, but may also be impractical when processing large-scale data sets.

[0004] In addition, in order to improve the quality of the reconstructed model, the current mainstream 3D reconstruction technologies, such as 3D reconstruction based on LiDAR and 3D reconstruction based on multi-view stereo vision (MVS), usually require a large amount of parameters or image data. These methods rely on rich inputs to ensure the accuracy and details of the model, but also bring high demands on storage and computing resources. In the completion operation, if all these data and parameters are taken into account, a lot of computing resources will undoubtedly be wasted. In addition, existing technologies still face challenges in identifying the true meaning of specific areas of the image, especially low-texture areas; these areas lack sufficient visual clues, making it difficult for automatic reconstruction algorithms to accurately infer their 3D structure; this leads to the current automatic completion methods still facing challenges.

[0005] In view of this, the existing technology needs to be further improved. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention provides a method for completing a three-dimensional building model, the purpose of which is to solve at least one of the above problems.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] A method for completing a three-dimensional building model comprises the following steps:

[0009] Step S1: using drone aerial photography to obtain an image dataset of the building to be modeled;

[0010] Step S2: generating a corresponding primary depth map according to the acquired image data set;

[0011] Step S3: completing the depth information of the generated primary depth map to obtain an updated depth map;

[0012] Step S4: Generate first 3D point cloud data using the updated depth map, and then generate an initial query proposal set using the first 3D point cloud data;

[0013] Step S5: performing primary feature extraction on the 2D image in the image dataset to obtain a plurality of first image features;

[0014] Step S6: generating an updated query proposal set based on the initial query proposal and the plurality of first image features;

[0015] Step S7: Generate and model second 3D point cloud data using the updated query proposal set;

[0016] In step S3, completing the depth information of the primary depth map includes at least the following steps in sequence:

[0017] S31, using the primary depth map to select from the image dataset a valid view that can fully reflect the visual features and geometric properties of the building to be modeled in different directions;

[0018] S32, performing secondary feature extraction on the valid view to obtain a second image feature with semantics;

[0019] S33: Update the primary depth map based on the second image feature with semantics to obtain an updated depth map.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects:

[0021] The present invention proposes a method for completing a three-dimensional model during model generation. The method obtains an image dataset through aerial photography, obtains a primary depth map based on the 2D images in the image dataset, and then uses a depth map update module to complete and update the primary depth map. The completed and updated depth map is used to generate first 3D point cloud data, and then further processing is used to reconstruct the three-dimensional model. Since depth maps have higher computational efficiency and simpler data processing than point clouds, this helps to achieve completion operations with less computation. During the depth map update process, valid views corresponding to each contour surface are obtained through hole identification and contour generation. The number of valid view images is greatly reduced compared to the number of images in the image dataset, so only a small number of valid views need to be processed to perform the completion operation. In addition, by introducing the GDnet network, two-level feature extraction is achieved. The first-level feature extraction is used to obtain the first image feature, and the first image feature is used to guide subsequent three-dimensional reconstruction. The second-level feature extraction is enhanced feature extraction, which can identify weak texture structures in the image, such as glass. Due to the small number of effective views, although this enhanced feature extraction requires more calculations, it does not lead to a significant increase in the amount of calculation overall. On this basis, the understanding of the true meaning of each object in the effective view is achieved, which helps to obtain a more accurate depth map. On this basis, depth information is completed, and the result will be more accurate.

[0022] In general, the present invention achieves a more accurate completion operation of the three-dimensional model while using fewer computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0024] Figure 1 This is the main flow chart of the present invention for completing the three-dimensional model of a building;

[0025] Figure 2This is a principle framework diagram of the present invention for completing a three-dimensional building model;

[0026] Figure 3 This is a schematic diagram of the principle of updating the depth map of the present invention;

[0027] Figure 4 It is a schematic diagram of the depth information that can be obtained at the glass position;

[0028] Figure 5 This is a schematic diagram of the glass position where depth information cannot be obtained;

[0029] Figure 6 yes Figure 2 Schematic framework diagram of the feature extraction network shown;

[0030] Figure 7 is a schematic diagram of the present invention for selecting effective views;

[0031] Among them, 1-wall, 2-glass, 3-drone flight trajectory, 4-camera, 5-actual depth distance, 6-calculated depth distance, 7-multi-level feature extractor, 8-LCFI module group, 9-low-level image features, 10-high-level image features, 11-attention features, 12-fusion features, 13-feature output operation, 14-hole position, 15-valid view, 100-depth map update module. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples. The illustrative embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0033] like Figures 1 to 7 As shown, the present invention provides a method for completing a three-dimensional building model, which includes the following steps:

[0034] Step S1: using a drone to obtain an image dataset of the building to be modeled; the image dataset is composed of 2D image data, and the image dataset covers all directions of the building to be modeled (for example, the top, left, right, front, and back);

[0035] Step S2: generating a corresponding primary depth map according to the acquired image data set;

[0036] Step S3: completing the depth information of the generated primary depth map to obtain an updated depth map;

[0037] Step S4: Generate first 3D point cloud data using the updated depth map, and then generate an initial query proposal set using the first 3D point cloud data;

[0038] Step S5: performing primary feature extraction on the 2D image in the image dataset to obtain a plurality of first image features;

[0039] Step S6: generating an updated query proposal set based on the initial query proposal set and the plurality of first image features;

[0040] Step S7: Generate and model second 3D point cloud data using the updated query proposal set;

[0041] In step S3, completing the depth information of the primary depth map includes at least the following steps in sequence:

[0042] S31, using the primary depth map to select from the image dataset a valid view 15 that can fully reflect the visual features and geometric properties of the building to be modeled in different directions;

[0043] S32, performing secondary feature extraction on the effective view 15 to obtain a second image feature with semantics;

[0044] S33: Update the primary depth map based on the second image feature with semantics to obtain an updated depth map.

[0045] It should be noted that in the existing technology, the completion of three-dimensional models is either done manually or through deep learning machine algorithms. Manual three-dimensional completion is inefficient and time-consuming, and is limited by the operator's technical level and subjective judgment, which can easily lead to inconsistencies and errors in the completion results. In contrast, machine learning algorithms can quickly process large amounts of data, automatically complete the completion task, and significantly improve the completion efficiency. However, most of the current machine learning algorithms are unable to identify the true meaning of each part of the model, and, in order to improve the quality of the reconstructed model, the current mainstream three-dimensional reconstruction methods require a large amount of parameters or image data. These methods rely on rich inputs to ensure the accuracy and details of the model. In the process of model completion, it is usually necessary to consider the above-mentioned large amount of parameters or image data, which undoubtedly increases the requirements for storage and computing resources. See Figure 2In the depth map update module 100, the present invention obtains the effective view 15 through the primary depth map. The effective view 15 can be only a part of the key view in the image data set. In this way, in the process of depth information completion, the computer only needs to process a small number of key pictures in the image data set to complete the completion work. Therefore, the amount of calculation can be reduced as much as possible. On this basis, the enhanced extraction algorithm of the secondary feature extraction is used to obtain the true meaning of each part, and then the depth information is completed based on the true meaning of each part. This method is more accurate and reliable than the previous completion method. It should be further explained that the enhanced extraction algorithm of the secondary feature extraction often requires a lot of calculation to ensure the recognition of the true meaning of the object in the image. If too many pictures are processed, the amount of calculation will be greatly increased. For this reason, the present invention selects the effective view 15 and only performs feature enhancement extraction on the effective view 15. In this way, it can ensure that the accurate completion operation is achieved with less calculation. It should be further explained that the query proposal set (Query Proposals) can be, for example, a set of binary data streams, which have a corresponding relationship with the corresponding 3D point cloud data. For example, the first 3D point cloud data corresponds to the initial query proposal set, and the second 3D point cloud data corresponds to the updated query proposal set. The query proposal network can be, for example, a neural network, such as a CNN. In a preferred example, the query proposal network is used to process three-dimensional 3D point cloud data and convert it into a one-dimensional binary data stream (for example, with the help of a Reshape function). This conversion of data dimensions can facilitate data processing.

[0046] Furthermore, in step S1, the drone is equipped with a high-definition digital camera for oblique photography. The drone flies around the building to be modeled according to a predetermined route and altitude; the collected image data set is a collection of 2D image data.

[0047] Furthermore, in step S2, when processing the image dataset to obtain a primary depth map, depth estimation is performed using methods such as stereo vision and deep learning. Before depth estimation, preprocessing operations such as denoising, illumination correction, and contrast adjustment are also performed on the image dataset. After the primary depth map is generated, distortion correction and filtering operations are also performed. The above is prior art and will not be further elaborated.

[0048] In a preferred embodiment, the primary feature extraction and the secondary feature extraction are implemented using the same feature extraction network. Figure 6The feature extraction network adopts the GDNet (Glass Detection Network) network architecture, which includes a multi-level feature extractor 7, an LCFI module (Large-field Contextual Feature Integration Module) group 8, and a feature fusion and output module. The multi-level feature extractor 7 is used to receive an input image and extract multi-scale image features therefrom. The LCFI module group 8 is used to process the multi-scale image features from the multi-level feature extractor 7 and simultaneously capture contextual information from local to global. The LCFI module group 8 includes multiple LCFI modules, each of which is used to extract image features of a predetermined scale and integrate corresponding contextual information. At least one of the multiple LCFI modules outputs low-level image features 9, which are rich in local details. The outputs of the remaining LCFI modules are sent to the feature fusion and output module. After a first fusion process, high-level image features 10 are generated. Subsequently, after a second fusion process, the outputs are second image features with semantic meaning. Through this design, GDNet can efficiently extract and integrate image features at different levels from a single network, maintain consistency in feature extraction throughout the entire network, help learn generalized feature representations, avoid redundant calculations between models, and help ensure the detection and recognition quality of weak texture structures such as glass.

[0049] In a further preferred embodiment, the low-level image feature 9 can be selectively formed into a first image feature through a primary feature output, or, be fused for a second time with the high-level image feature 10 in a feature fusion and output module and formed into a semantic second image feature through a secondary feature output.

[0050] In a preferred embodiment, the number of LCFI modules is the same as the number of network layers of the multi-level feature extractor 7. Each layer of the multi-level feature extractor 7 is responsible for extracting features at different levels of abstraction. The shallow layers focus on extracting basic features such as edges and textures, while the deep layers extract more complex shapes, object parts, and scene information. Each LCFI module corresponds one-to-one to a layer output of the multi-level feature extractor 7.

[0051] In a preferred embodiment, the remaining LCFI modules include at least 3 LCFI modules ( Figure 6 The at least three LCFI modules process the image features of different levels from the multi-level feature extractor 7 respectively, and their outputs are subjected to a first fusion process to obtain high-level image features 10. The first fusion process includes at least concatenation. Figure 6symbol “C” in the figure) and attention fusion ( Figure 6 The symbol "F" in the operation.

[0052] In a further preferred embodiment, the high-level image features 10 and the low-level image features 9 are weighted by the attention weights and fused to form the attention features 11, which can be fused by element-wise multiplication (element-wise multiplication, Figure 6 The symbol "×" in the image is used to realize the attention feature 11 formed, and the high-level image feature 10 is further spliced ​​and fused with attention to obtain the fusion feature 12. The fusion feature 12 is formed into a second image feature with semantics through the secondary feature output after the feature output operation 13. The feature output operation 13 includes using the sigmoid activation function to convert the network output into a probability map, which helps to convert the glass detection problem into a binary classification problem. Furthermore, the feature output operation 13 also includes introducing semantic suggestions. Through semantic suggestions, the image features output by the secondary feature extraction can be assigned semantic labels, which makes the semantic information of the output image features more accurate and is conducive to the subsequent update of the depth map.

[0053] In a preferred embodiment, the feature extraction network only performs secondary feature extraction on the effective view 15. The effective view 15 can be indexed (before entering the feature extraction network), for example, by using a special string of numbers or letters for indexing (the purpose of indexing is to distinguish it from other input images). As long as the feature extraction network can recognize it, when the feature extraction network is working, its multi-level feature extractor 7 can start the corresponding network layer according to the indexing information. For example, when the indexed effective view 15 enters the feature extraction network, each level of the multi-level feature extractor 7 works, and each LCFI module in the LCFI module group 8 also works after receiving the output of the multi-level feature extractor 7. The output of the LCFI module group 8 is detected by the feature extraction network. After processing by the fusion and output module, the second image feature with semantic meaning is output through the secondary feature. When the unindexed image enters the feature extraction network, only a portion of the network layers (e.g., only one network layer) of the multi-level feature extractor 7 is operational, and only a portion of the LCFI modules in the LCFI module group 8 is operational (e.g., only one LCFI module). At this time, the network layers in the multi-level feature extractor 7 and the LCFI modules in the LCFI module group 8 that are in operation are only used to generate low-level image features 9. Since the secondary feature output requires the use of high-level image features 10, at this time, only the primary feature output can be used to form the first image feature. By adopting the above technical settings, in the process of depth information completion, in order to generate the second image feature, only a small number of images (effective views) need to be processed. Although the enhanced extraction algorithm of the secondary feature extraction requires more computation than other algorithms for a single image, the total number of images is greatly reduced, so the overall computational complexity will be very small. This helps to reduce the computational resources of the completion operation, improve efficiency, and reduce hardware requirements.

[0054] In order to better achieve the purpose of the present invention, step S31 specifically includes:

[0055] S311, pre-processing:

[0056] Gaussian filtering is performed on the primary depth map and the 2D images in the image dataset to reduce noise;

[0057] S312, contour generation and contour surface determination:

[0058] The contour of the building to be modeled is extracted using the primary depth map preprocessed in step S311. For example, a deep learning-based image edge and contour extraction method such as DeepEdge and CEDN (fully convolutional encoder-decoder network) is used to obtain the basic contour of the building to be modeled. The number of contour surfaces of the building to be modeled and the orientation of the contour surfaces (e.g., the normal direction of the contour surface) are determined based on the obtained basic contour. It should be noted that the contour generated here is only a preliminary contour, and its accuracy does not need to be too high, as long as it can reflect the basic shape of the building to be modeled as a whole. Therefore, a large amount of calculation is not required.

[0059] S313. Establishment of correspondence between 2D image and contour:

[0060] The orientation of the drone when taking the image is compared with the orientation of each contour surface. Based on the results of the orientation comparison, the image dataset is divided into multiple subsets, each subset corresponding to a specific contour surface.

[0061] S314, Hollow Area Identification:

[0062] Use threshold method, deep learning method, clustering algorithm, connected domain analysis algorithm, etc. to identify the hole area of ​​the primary depth map. Taking the threshold method as an example, in the primary depth map, a threshold interval of the depth value can be set, and the depth value of each pixel in the primary depth map is compared with the threshold interval, and the pixel point area whose depth value exceeds the threshold interval (that is, is not within the threshold interval) is identified as a hole area; it should be noted that when performing hole area identification, multiple methods can be used for joint identification to ensure the accuracy of the identified hole area. There are many specific identification methods in the existing technology, which will not be repeated here;

[0063] S315: Determine the location of the cavity area:

[0064] The hole area identified in the primary depth map is projected onto the corresponding 2D image, thereby determining the position of the hole area on the corresponding 2D image. Since each subset of the image data set corresponds to a specific contour surface, the contour surface corresponding to the hole area can be further determined by determining the position of the hole area on the corresponding 2D image. On this basis, the position of the hole area on the corresponding contour surface can be further determined according to the position of the hole area on the corresponding 2D image. For example, see Figure 7 Hole position 14 in the middle;

[0065] S316, selection of valid view 15:

[0066] For a contour surface with a hole position 14, a first weighted scoring method is used to select valid views 15 from the corresponding subset that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface; for a contour surface without a hole position 14, a second weighted scoring method is used to select valid views 15 from the corresponding subset that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface; wherein the first weighted scoring method is different from the second weighted scoring method, the first weighted scoring method is at least associated with the hole information of the hole position 14, and the second weighted scoring method is associated with the overall coverage of the view, the richness of the visual texture, the consistency of the illumination, and the complementarity between views. In general, the first weighted scoring method places special emphasis on the visibility and geometric features of the hole area, ensuring that the selected views can capture the boundaries and internal structures of the holes. When scoring, it will give higher weights to hole-related features, such as the clarity of the hole boundaries, the texture details inside the holes, and the contrast between the holes and the surrounding environment. The second weighted scoring method pays more attention to the global consistency and visual appeal of the views. Its scoring criteria include the lighting conditions of the views, the diversity of colors and textures, and the visual balance and symmetry of the views.

[0067] It should be noted that, by selecting the effective view 15, this solution can select a small number of key images related to the building to be modeled in various directions from the image data set, which helps to complete its completion with less computation. It should also be noted that although voids are an element that the present invention needs to focus on when performing the completion operation, in actual buildings, especially modern buildings, they often contain various weak texture structures such as glass. Affected by factors such as shooting angle, lighting conditions, glass transparency, and glass color, at certain glass locations, its features cannot be captured, thus forming void areas. In addition, at some glass locations, due to high transparency or higher indoor brightness than outdoor, the visual image easily captures indoor information at the glass location. In this case, although depth information is also obtained and voids will not be formed, this depth information is inaccurate and also requires completion operations. Figure 4 and Figure 5 A simple example is given, where a drone carrying a camera 4 flies along a drone flight trajectory 3 to acquire an image dataset. At the position of the glass 2 on the wall 1, in one case (refer to Figure 4), it can capture the feature point P, and obtain the calculated depth distance 6 through depth value calculation. There is a large difference between the calculated depth distance 6 and the actual depth distance 5. If the model is reconstructed using the calculated depth distance 6, it is easy to form a deformation that is inconsistent with the actual building at the corresponding position. In other cases, it cannot capture the feature point P', and thus cannot obtain the calculated depth distance 6 through calculation, and naturally cannot obtain the actual depth distance 5. At this time, model reconstruction is likely to form holes in the reconstructed three-dimensional model. To this end, the present invention adopts two weighted scoring methods in the selection of effective views 15, the purpose of which is to obtain effective views 15 related to the building to be modeled in different directions. Afterwards, the effective views 15 are used to perform secondary feature extraction through the GDnet network, so that the true meaning of each part on the effective view 15, especially structures such as glass, can be identified. For example, when the hole area is glass, or when the non-hole area is glass, corresponding understanding can be made, which helps to ensure that the model is closer to the actual situation during reconstruction.

[0068] As a preferred embodiment, a simple example of obtaining "selecting valid views 15 from the corresponding subset that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface by using the first weighted scoring method" is provided here (the principle of the second weighted scoring method is similar, except that the parameters or content considered are different, which will not be repeated here). The example is obtained by the following steps:

[0069] A. Quantitative evaluation of coverage

[0070] Calculate the pixel ratio of the hole area in the corresponding 2D image and use it as the coverage;

[0071] B. Multi-dimensional evaluation of information quantity

[0072] Methods such as local variance, number of feature points, and information entropy are used to evaluate the information content of each 2D image corresponding to the primary depth map;

[0073] C. Calculation of the first weighted score

[0074] The original values ​​of coverage and information are normalized to the range of [0, 1] for weighted summation. Then, weights are assigned to coverage and information, and the sum of the weights should be 1. In a preferred example, a machine learning model is used to predict the performance of different feature and weight combinations, and the weights are adjusted accordingly. A weighted score is calculated based on the normalized coverage, normalized information, and the corresponding weights, where the first weighted score = (normalized coverage × coverage weight) + (normalized information × information weight). In one example, the normalized coverage and normalized information can be normalized and calculated using the following formulas:

[0075]

[0076] Where C is the current coverage, C min is the minimum coverage, C max is the maximum coverage; where H is the information entropy of the image, H min and H max are the minimum and maximum values ​​of information entropy in the data set respectively;

[0077] D. Sort all images in the corresponding subset based on the calculated first weighted score, and select the view with the highest score as the best view, which is then used as valid view 15. It should be noted that sometimes a single valid view 15 may not fully reflect the corresponding contour surface. Therefore, to ensure reliable completion, after obtaining the best view, images with higher scores but in different orientations from the best view may be selected as needed and also used as valid views 15. For example, contour surfaces at hole locations 14 often require multiple valid views 15. Therefore, a threshold value may be set when selecting valid views 15, so that only views with the first weighted score above the threshold are considered.

[0078] In order to better achieve the purpose of the present invention, step S33 adopts Figure 3 The framework diagram shown is used to update the depth map, wherein, after obtaining the semantic image features in step S32, they are fed into a specific neural network model, such as a CNN model or a Transformer model, to update the depth map. When the corresponding semantic image features are identified as glass, since the glass is usually flat, it can be first flattened. Since the trajectory of the drone is determined, the coordinate position of the drone when collecting the corresponding image is known. At this time, several depth distances (for example, three) from the glass to the drone position can be found through the edge of the glass (for example, the glass frame). At this time, the depth values ​​at other positions on the glass can be obtained by simple geometric conversion, thereby obtaining an updated depth map. It should be understood that the acquisition of depth values ​​requires continuous feedback adjustment. It should also be understood that glass can also be non-planar, which should also fall within the scope of protection of the present invention. However, since the calculation of non-planar glass is more complicated, it will not be discussed here.

[0079] It should also be noted that, generally, the data processing of 3D point clouds is very complex. Relatively speaking, the computational cost of processing and analyzing depth maps is much lower and the response speed is faster. Therefore, the present invention uses depth information for completion before generating the first 3D point cloud data, and then uses the 3D point cloud data for subsequent modeling operations. This helps to improve the quality of three-dimensional model completion and reconstruction while minimizing the amount of computation and cost.

[0080] In addition, steps 5 to 7 are some common steps and will not be described in detail here.

[0081] In a further preferred embodiment, after the 3D model is reconstructed, texture mapping is also performed using the valid view 15. Because the valid view 15 has been screened using a weighted score, it can fully reflect the visual characteristics and geometric properties of the building being modeled from different directions. Therefore, by mapping each contour surface, the model can be given a more realistic appearance and details. Preferably, the texture mapping operation is also implemented using a deep learning-based method, which will not be further described here.

[0082] Finally, it should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. Portions not described in detail in this specification belong to the prior art known to those skilled in the art.

[0083] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to the embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for completing a three-dimensional building model, comprising the following steps: Step S1: using drone aerial photography to obtain an image dataset of the building to be modeled; Step S2: generating a corresponding primary depth map according to the acquired image data set; Step S3: completing the depth information of the generated primary depth map to obtain an updated depth map; Step S4: Generate first 3D point cloud data using the updated depth map, and then generate an initial query proposal set using the first 3D point cloud data; Step S5: performing primary feature extraction on the 2D image in the image dataset to obtain a plurality of first image features; Step S6: generating an updated query proposal set based on the initial query proposal set and the plurality of first image features; Step S7: Generate and model second 3D point cloud data using the updated query proposal set; It is characterized in that, in step S3, completing the depth information of the primary depth map at least sequentially includes: S31, using the primary depth map, selecting valid views (15) from the image dataset that can fully reflect the visual features and geometric properties of the building to be modeled in different directions; the valid views (15) are a small number of key images selected from the image dataset that are related to the building to be modeled in various directions; S32, performing secondary feature extraction on the valid view (15), assigning semantic labels to image features output by the secondary feature extraction, thereby obtaining second image features with semantics; wherein the secondary feature extraction is enhanced feature extraction, which is used to identify weak texture structures in the image; S33: Update the primary depth map based on the second image feature with semantics to obtain an updated depth map.

2. A building three-dimensional model completion method according to claim 1, characterized in that: The primary feature extraction and the secondary feature extraction are implemented using the same feature extraction network.

3. A building three-dimensional model completion method according to claim 2, characterized in that: The feature extraction network includes a multi-level feature extractor (7), an LCFI module group (8) and a feature fusion and output module, wherein the multi-level feature extractor (7) is used to receive an input image and extract multi-scale image features therefrom, the LCFI module group (8) is used to process the multi-scale image features from the multi-level feature extractor (7) and simultaneously capture context information from local to global, the LCFI module group (8) includes a plurality of LCFI modules, each LCFI module is used to extract image features of a predetermined scale and integrate corresponding context information, at least one of the plurality of LCFI modules outputs a low-level image feature (9), and the outputs of the remaining LCFI modules in the plurality of LCFI modules are fed into the feature fusion and output module, and after a first fusion process, a high-level image feature (10) is generated, and then after a second fusion process, the output is a second image feature with semantics.

4. A method for completing a three-dimensional building model according to any one of claims 2 to 3, characterized in that: The feature extraction network only performs secondary feature extraction on the valid view (15).

5. A building three-dimensional model completion method according to claim 4, characterized in that: The effective view (15) is indexed, and when the feature extraction network works, its multi-level feature extractor (7) can start the corresponding network layer according to the indexing information. When the indexed effective view (15) enters the feature extraction network, each layer of the multi-level feature extractor (7) works, and each LCFI module in the LCFI module group (8) also works after receiving the output of the multi-level feature extractor (7). The output of the LCFI module group (8) is formed into a second image feature with semantics through the secondary feature output after being processed by the feature fusion and output module; when the unindexed image enters the feature extraction network, only part of the network layer of the multi-level feature extractor (7) works, and only part of the LCFI modules in the LCFI module group (8) works. At this time, the network layer in the multi-level feature extractor (7) that is in a working state and the LCFI modules in the LCFI module group (8) that are in a working state are only used to generate low-level image features (9).

6. A building three-dimensional model completion method according to claim 1, characterized in that: Step S31 specifically includes: S311, preprocessing: filtering the primary depth map and the 2D image in the image dataset to reduce noise; S312, contour generation and contour surface determination: extracting the contour of the building to be modeled using the preprocessed primary depth map to obtain the basic contour of the building to be modeled, and determining the number of contour surfaces of the building to be modeled and the orientation of the contour surfaces based on the obtained basic contour; S313. Establishing the correspondence between 2D images and contours: Compare the orientation of the drone when taking the image with the orientation of each contour surface. Based on the results of the orientation comparison, divide the image dataset into multiple subsets, each subset corresponding to a specific contour surface. S314, hole area identification: performing hole identification on the primary depth map to identify the hole area in the primary depth map; S315: Determine the position of the hole region: Project the hole region identified in the primary depth map onto the corresponding 2D image, thereby determining the position of the hole region on the corresponding 2D image, and further determining the contour surface corresponding to the hole region and the position of the hole region on the corresponding contour surface; S316: Select the valid view (15).

7. A method for completing a three-dimensional building model according to claim 6, characterized in that: Step S316 includes, for the contour surface with a hole position (14), using a first weighted scoring method to select from the corresponding subset a valid view (15) that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface; for the contour surface without a hole position (14), using a second weighted scoring method to select a valid view (15) that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface; wherein, the first weighted scoring method is different from the second weighted scoring method, and the first weighted scoring method is associated with the hole information of the hole position (14).

Images