Building three-dimensional model completion method

Abstract

The invention provides a building three-dimensional model completion method, which belongs to the technical field of three-dimensional modeling, and comprises the following steps: obtaining an image data set through aerial photography, obtaining a primary depth map based on a 2D image in the image data set, and carrying out cavity identification and contour generation according to the primary depth map. According to the method, effective views capable of comprehensively reflecting different contour surface features of a building to be modeled are searched, the effective views can be a small number of key views, when the effective views are used for depth information completion operation, very accurate completion updating can be completed only by using a small amount of calculation amount, and meanwhile, due to the fact that a small number of key views are processed, the accuracy of depth information completion updating is greatly improved. Therefore, enhanced feature recognition can be introduced in the whole complementing process, which is helpful for understanding the real meaning of each object such as glass in the view and realizing more accurate complementing operation.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional modeling, and in particular relates to a method for completing a three-dimensional model of a building. Background Art

[0002] 3D reconstruction of buildings is an efficient digital simulation method that accurately captures buildings in the real world and converts them into 3D models in computers. This technology has been applied in many key areas, including but not limited to accurate analysis of urban planning, digital preservation of architectural heritage, visualization of real estate development, creation of visual effects in the game and film industries, and creation of immersive experiences in augmented reality and virtual reality.

[0003] In the process of 3D reconstruction, the quality of the model is affected by many factors such as sensor resolution, lighting conditions, building occlusion, and surface reflection and transparency. The initial model is often incomplete or inaccurate. In order to solve these problems, the completion step is particularly critical. In the past, the completion of 3D models was mainly done manually. Although this method is meticulous, it is inefficient and time-consuming. With the development of technology, the automated completion method using machine learning has significantly improved efficiency and effectiveness. However, current machine learning algorithms still have limitations in model completion, especially in identifying the true meaning of each part of the model. This leads to the fact that the completion of data hole areas often relies on the principle of similarity matching, and sometimes it is impossible to accurately restore the original characteristics of the structure. For example, Chinese patent ZL202310902348.9 discloses a method for reconstructing a three-dimensional model, which extends the adjacent structure to the void and determines the target structure of the void based on the similarity between the extended part and the adjacent structure, thereby restoring, extending or remodeling the void in the original space model. The method restores the material of the void by referring to the surrounding material of the void, and the material patched at the void is the same or similar to the material of the adjacent structure. Since modern buildings usually contain a large number of weak texture structures, such as glass, these weak texture structures are difficult to be recognized by machine vision. If the completion method in the above patent is used, it is easy to complete the glass in the way of walls or other objects, resulting in inconsistency with the actual building situation. For another example, Chinese patent ZL202311664268.0 discloses a method for completing missing data of ancient building point clouds. Although it uses semantic information, this information is manually constructed rather than generated by machine autonomous learning, which is not only a huge workload, but also may be impractical when processing large-scale data sets.

[0004] In addition, in order to improve the quality of the reconstructed model, the current mainstream 3D reconstruction technologies, such as 3D reconstruction based on LiDAR and 3D reconstruction based on multi-view stereo vision (MVS), usually require a large amount of parameters or image data. These methods rely on rich inputs to ensure the accuracy and details of the model, but also bring high demands on storage and computing resources. In the completion operation, if all these data and parameters are taken into account, a lot of computing resources will undoubtedly be wasted. In addition, the existing technology still faces challenges in identifying the true meaning of specific areas of the image, especially low-texture areas; these areas lack sufficient visual clues, making it difficult for automatic reconstruction algorithms to accurately infer their 3D structures; this leads to the current automatic completion methods are still challenging.

[0005] In view of this, the prior art needs to be further improved. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a method for completing a three-dimensional building model, the purpose of which is to solve at least one of the above problems.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] A method for completing a three-dimensional model of a building comprises the following steps:

[0009] Step S1, using drone aerial photography to obtain an image dataset of the building to be modeled;

[0010] Step S2: generating a corresponding primary depth map according to the acquired image data set;

[0011] Step S3: completing the depth information of the generated primary depth map to obtain an updated depth map;

[0012] Step S4: generating first 3D point cloud data using the updated depth map, and then generating an initial query proposal set using the first 3D point cloud data;

[0013] Step S5, performing primary feature extraction on the 2D image in the image data set to obtain a plurality of first image features;

[0014] Step S6: generating an updated query proposal set based on the initial query proposal and the plurality of first image features;

[0015] Step S7: Generate and model second 3D point cloud data using the updated query proposal set;

[0016] Among them, in step S3, completing the depth information of the primary depth map at least sequentially includes:

[0017] S31, selecting, from the image data set, effective views that can fully reflect the visual features and geometric properties of the building to be modeled in different directions using the primary depth map;

[0018] S32, performing secondary feature extraction on the effective view to obtain a second image feature with semantics;

[0019] S33: Update the primary depth map based on the second image feature with semantics to obtain an updated depth map.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects:

[0021] The present invention proposes a method for completing a three-dimensional model in the middle of model generation, which obtains an image data set through aerial photography, obtains a primary depth map based on a 2D image in the image data set, and then uses a depth map update module to complete and update the primary depth map. The completed and updated depth map is used to generate the first 3D point cloud data, and then the three-dimensional model is reconstructed through further processing. Since the depth map has higher computational efficiency and simpler data processing than the point cloud, this helps to achieve the completion operation with less calculation. In the process of updating the depth map, through hole identification and contour generation, valid views corresponding to each contour surface are obtained. The number of pictures of the valid view is greatly reduced compared to the number of pictures in the image data set, so only a small number of valid views need to be processed in the subsequent completion operation. In addition, by introducing the GDnet network, two-level feature extraction is achieved, in which the first-level feature extraction is used to obtain the first image feature, and the first image feature is used to guide the subsequent three-dimensional reconstruction. The second-level feature extraction is enhanced feature extraction, which can identify weak texture structures in the picture such as glass. Due to the small number of effective views, although this enhanced feature extraction requires more calculations, it does not lead to a significant increase in the amount of calculation as a whole. On this basis, the understanding of the true meaning of each object in the effective view is achieved, which helps to obtain a more accurate depth map. On this basis, depth information completion is performed, and the result will be more accurate.

[0022] In general, the present invention achieves a more accurate completion operation of the three-dimensional model while using less computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0024] Figure 1 It is the main flow chart of the present invention for completing the three-dimensional model of a building;

[0025] Figure 2It is a principle framework diagram of the present invention for completing the three-dimensional model of a building;

[0026] Figure 3 It is a principle framework diagram of the present invention for updating the depth map;

[0027] Figure 4 It is a schematic diagram of the depth information that can be obtained at the glass position;

[0028] Figure 5 It is a schematic diagram where depth information cannot be obtained at the glass position;

[0029] Figure 6 yes Figure 2 Schematic framework diagram of the feature extraction network shown;

[0030] Figure 7 is a schematic diagram of selecting effective views according to the present invention;

[0031] Among them, 1-wall, 2-glass, 3-drone flight trajectory, 4-camera, 5-actual depth distance, 6-calculated depth distance, 7-multi-level feature extractor, 8-LCFI module group, 9-low-level image features, 10-high-level image features, 11-attention features, 12-fusion features, 13-feature output operation, 14-hole position, 15-valid view, 100-depth map update module. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. The illustrative embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0033] like Figures 1 to 7 As shown, the present invention provides a method for completing a three-dimensional model of a building, which comprises the following steps:

[0034] Step S1, using drone aerial photography to obtain an image dataset of the building to be modeled; the image dataset is composed of 2D image data, and the image dataset covers all directions (for example, the top and left, right, front and back) of the building to be modeled;

[0035] Step S2: generating a corresponding primary depth map according to the acquired image data set;

[0036] Step S3: completing the depth information of the generated primary depth map to obtain an updated depth map;

[0037] Step S4: generating first 3D point cloud data using the updated depth map, and then generating an initial query proposal set using the first 3D point cloud data;

[0038] Step S5, performing primary feature extraction on the 2D image in the image data set to obtain a plurality of first image features;

[0039] Step S6: generating an updated query proposal set based on the initial query proposal set and the plurality of first image features;

[0040] Step S7: Generate and model second 3D point cloud data using the updated query proposal set;

[0041] Among them, in step S3, completing the depth information of the primary depth map at least sequentially includes:

[0042] S31, selecting, from the image data set, a valid view 15 that can fully reflect the visual features and geometric properties of the building to be modeled in different directions using the primary depth map;

[0043] S32, performing secondary feature extraction on the effective view 15 to obtain a second image feature with semantics;

[0044] S33: Update the primary depth map based on the second image feature with semantics to obtain an updated depth map.

[0045] It should be noted that in the prior art, the completion of three-dimensional models is either done manually or through deep learning machine algorithms. Manual three-dimensional completion is inefficient and time-consuming, and is limited by the operator's technical level and subjective judgment, which can easily lead to inconsistencies and errors in the completion results. In contrast, machine learning algorithms can quickly process large amounts of data and automatically complete completion tasks, significantly improving completion efficiency. However, most of the current machine learning algorithms are unable to identify the true meaning of each part of the model, and, in order to improve the quality of the reconstructed model, the current mainstream three-dimensional reconstruction methods require a large amount of parameters or image data. These methods rely on rich inputs to ensure the accuracy and details of the model. In the process of model completion, it is usually necessary to consider the above-mentioned large amount of parameters or image data, which undoubtedly increases the requirements for storage and computing resources. See Figure 2In the depth map update module 100, the present invention obtains the effective view 15 through the primary depth map. The effective view 15 can be only a part of the key view in the image data set. In this way, in the process of completing the depth information, the computer only needs to process a small number of key pictures in the image data set to complete the completion work. Therefore, the amount of calculation can be reduced as much as possible. On this basis, the enhanced extraction algorithm of the secondary feature extraction can be used to obtain the true meaning of each part, and then the depth information is completed based on the true meaning of each part. This method is more accurate and reliable than the previous completion method. It should be further explained that the enhanced extraction algorithm of the secondary feature extraction often requires a lot of calculations to ensure the recognition of the true meaning of the object in the image. If too many pictures are processed, the amount of calculation will be greatly increased. For this reason, the present invention selects the effective view 15 and only performs enhanced feature extraction on the effective view 15. In this way, it can ensure that accurate completion operations are achieved with less calculation. It should be further explained that the query proposal set (Query Proposals) can be, for example, a set of binary data streams, which have a corresponding relationship with the corresponding 3D point cloud data. For example, the first 3D point cloud data corresponds to the initial query proposal set, and the second 3D point cloud data corresponds to the updated query proposal set. The query proposal network can be, for example, a neural network, such as CNN. In a preferred example, the query proposal network is used to process three-dimensional 3D point cloud data and convert it into a one-dimensional binary data stream (for example, with the help of a Reshape function, etc.). This conversion of data dimensions can facilitate data processing.

[0046] Furthermore, in step S1, the drone is equipped with a high-definition digital camera for oblique photography, and the drone flies around the building to be modeled according to a predetermined route and altitude; the collected image data set is a collection of 2D image data.

[0047] Furthermore, in step S2, when processing the image data set to obtain the primary depth map, stereoscopic vision, deep learning and other methods are used to estimate the depth. Before estimating the depth, the image data set is also subjected to preprocessing operations such as denoising, correcting illumination, and adjusting contrast. After the primary depth map is generated, distortion correction and filtering operations are also performed. The above are prior arts and will not be described in detail.

[0048] In a preferred embodiment, the primary feature extraction and the secondary feature extraction are implemented using the same feature extraction network. Figure 6The feature extraction network adopts the GDNet (Glass Detection Network) network architecture, which includes a multi-level feature extractor 7, a LCFI module (Large-field Contextual Feature Integration Module) group 8 and a feature fusion and output module, wherein the multi-level feature extractor 7 is used to receive an input image and extract multi-scale image features therefrom, the LCFI module group 8 is used to process the multi-scale image features from the multi-level feature extractor 7 and simultaneously capture context information from local to global, the LCFI module group 8 includes a plurality of LCFI modules, each LCFI module is used to extract image features of a predetermined scale and integrate corresponding context information, at least one of the plurality of LCFI modules outputs a low-level image feature 9, and these low-level image features 9 are rich in local details, and the outputs of the remaining LCFI modules in the plurality of LCFI modules are sent to the feature fusion and output module, and after a first fusion process, a high-level image feature 10 is generated, and then after a second fusion process, the output is a second image feature with semantics. Through this design, GDNet can efficiently extract and integrate image features at different levels from a single network, maintain consistency in feature extraction throughout the entire network, help learn generalized feature representations, avoid redundant calculations between models, and help ensure the detection and recognition quality of weak texture structures such as glass.

[0049] In a further preferred embodiment, the low-level image feature 9 can be selectively formed into a first image feature through a primary feature output, or, can be fused for a second time with the high-level image feature 10 in a feature fusion and output module and formed into a semantic second image feature through a secondary feature output.

[0050] In a preferred embodiment, the number of LCFI modules is the same as the number of network layers of the multi-level feature extractor 7. Each level of the multi-level feature extractor 7 is responsible for extracting features at different abstraction levels. The shallow layers focus on extracting basic features such as edges and textures, while the deep layers extract more complex shapes, object parts and scene information. Each LCFI module corresponds one-to-one to a level output of the multi-level feature extractor 7.

[0051] In a preferred embodiment, the remaining LCFI modules include at least 3 LCFI modules ( Figure 6 The at least three LCFI modules process the image features of different levels from the multi-level feature extractor 7 respectively, and their outputs are subjected to a first fusion process to obtain a high-level image feature 10, and the first fusion process at least includes concatenation (concatenation, Figure 6symbol “C” in the figure) and attention fusion ( Figure 6 symbol "F" in the operation.

[0052] In a further preferred embodiment, the high-level image feature 10 and the low-level image feature 9 are weighted by the attention weight and fused to form the attention feature 11, and the fusion can be performed by element-wise multiplication (element-wise multiplication, Figure 6 The symbol "×" in the image is realized, and the formed attention feature 11 is further spliced ​​with the high-level image feature 10 and the attention fusion is obtained to obtain the fusion feature 12. The fusion feature 12 is formed as a second image feature with semantics through the secondary feature output after the feature output operation 13. The feature output operation 13 includes using the sigmoid activation function to convert the network output into a probability map, which helps to convert the glass detection problem into a binary classification problem. Furthermore, the feature output operation 13 also includes introducing semantic suggestions. Through semantic suggestions, the image features output by the secondary feature extraction can be assigned semantic labels, which makes the semantic information of the output image features more accurate and facilitates the subsequent update of the depth map.

[0053] In a preferred embodiment, the feature extraction network only performs secondary feature extraction on the effective view 15. The effective view 15 can be indexed (before entering the feature extraction network), for example, by using a special string of numbers or letters for indexing (the purpose of indexing is to distinguish it from other input images). As long as the feature extraction network can recognize it, when the feature extraction network is working, its multi-level feature extractor 7 can start the corresponding network level according to the indexing information. For example, when the indexed effective view 15 enters the feature extraction network, each level of the multi-level feature extractor 7 works, and each LCFI module in the LCFI module group 8 also works after receiving the output of the multi-level feature extractor 7. The output of the LCFI module group 8 is detected by the feature extraction network. After being processed by the fusion and output module, the second image feature with semantics is formed through the secondary feature output; when the unindexed image enters the feature extraction network, only part of the network level (for example, only one network level) of the multi-level feature extractor 7 works, and only part of the LCFI modules in the LCFI module group 8 work (for example, only one LCFI module), and at this time, the network level in the multi-level feature extractor 7 and the LCFI modules in the LCFI module group 8 that are in working state are only used to generate low-level image features 9; since the secondary feature output needs to use the high-level image feature 10, at this time, it can only perform the primary feature output to form the first image feature. By adopting the above technical settings, in the process of completing the depth information, in order to generate the second image feature, only a small amount of images (effective views) need to be processed. Although for a single picture, the enhanced extraction algorithm of the secondary feature extraction requires more calculation than other algorithms, but because the total number of images is greatly reduced, the overall calculation amount will be very small; this helps to reduce the computing resources of the completion operation, improve efficiency, and reduce the requirements for hardware, etc.

[0054] In order to better achieve the purpose of the present invention, step S31 specifically includes:

[0055] S311, pre-processing:

[0056] Gaussian filtering is performed on the primary depth map and 2D images in the image dataset to reduce noise;

[0057] S312, contour generation and contour surface determination:

[0058] The contour of the building to be modeled is extracted using the primary depth map preprocessed in step S311, for example, by using image edge and contour extraction methods based on deep learning such as DeepEdge and CEDN (fully convolutional encoder-decoder network), the basic contour of the building to be modeled is obtained, and the number of contour surfaces of the building to be modeled and the orientation of its contour surfaces (for example, the normal direction of the contour surface) are determined based on the obtained basic contour; it should be noted that the contour generated here is only a preliminary contour, and its accuracy does not need to be too high, as long as it can reflect the basic shape of the building to be modeled as a whole, therefore, it is not necessary to perform too much calculation;

[0059] S313. Establishment of the correspondence between 2D image and contour:

[0060] The orientation of the drone when taking the image is compared with the orientation of each contour surface, and the image data set is divided into multiple subsets according to the result of the orientation comparison, each subset corresponds to a specific contour surface;

[0061] S314, Hollow area identification:

[0062] Use threshold methods, deep learning methods, clustering algorithms, connected domain analysis algorithms, etc. to identify hole areas in the primary depth map. Taking the threshold method as an example, in the primary depth map, a threshold interval of the depth value can be set, and the depth value of each pixel in the primary depth map is compared with the threshold interval, and the pixel point area with a depth value exceeding the threshold interval (that is, not within the threshold interval) is identified as a hole area; it should be noted that when identifying the hole area, multiple methods can be used for joint identification to ensure the accuracy of the identified hole area. There are many specific identification methods in the prior art, which will not be repeated here;

[0063] S315: Determine the location of the cavity area:

[0064] The hole region identified in the primary depth map is projected onto the corresponding 2D image, thereby determining the position of the hole region on the corresponding 2D image. Since each subset of the image data set corresponds to a specific contour surface, the contour surface corresponding to the hole region can be further determined by determining the position of the hole region on the corresponding 2D image. On this basis, the position of the hole region on the corresponding contour surface can be further determined according to the position of the hole region on the corresponding 2D image. For example, see Figure 7 The hole position in 14;

[0065] S316, selection of effective view 15:

[0066] For a contour surface with a hole position 14, a first weighted scoring method is used to select valid views 15 from a corresponding subset that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface; for a contour surface without a hole position 14, a second weighted scoring method is used to select valid views 15 from a corresponding subset that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface; wherein the first weighted scoring method is different from the second weighted scoring method, the first weighted scoring method is at least associated with the hole information of the hole position 14, and the second weighted scoring method is associated with the overall coverage of the view, the richness of the visual texture, the consistency of illumination, and the complementarity between views. In general, the first weighted scoring method places special emphasis on the visibility and geometric features of the hole area, ensuring that the selected view can capture the boundary and internal structure of the hole. When scoring, it will give higher weights to features related to the hole, such as the clarity of the hole boundary, the texture details inside the hole, and the contrast between the hole and the surrounding environment; the second weighted scoring method pays more attention to the global consistency and visual appeal of the view. Its scoring criteria include the lighting conditions of the view, the diversity of color and texture, and the visual balance and symmetry of the view.

[0067] It should be noted that, through the selection of the effective view 15, the present solution can select a small number of key images related to the building to be modeled in all directions from the image data set, which helps to complete its completion with less calculation. It should also be noted that although the void is an element that the present invention needs to focus on when performing the completion operation, in actual buildings, especially modern buildings, there are often various weak texture structures such as glass. Affected by factors such as shooting angle, lighting conditions, glass transparency, and glass color, at some glass positions, its features cannot be captured, thus forming a void area. In addition, there are some glass positions, because of the high transparency or the indoor brightness is higher than the outdoor, which makes it easy for the visual image to capture the indoor information at the glass position. In this case, although the depth information is also obtained and no void is formed, this depth information is inaccurate and also requires a completion operation. Figure 4 and Figure 5 A simple example is given, where a drone carrying a camera 4 flies along a drone flight trajectory 3 to acquire an image data set, at a glass 2 on a wall 1, in one case (refer to Figure 4), it can capture the feature point P, and obtain the calculated depth distance 6 through depth value calculation. There is a large difference between the calculated depth distance 6 and the actual depth distance 5. If the model is reconstructed through the calculated depth distance 6, it is easy to form a deformation that does not match the actual building at the corresponding position. In other cases, it cannot capture the feature point P', and thus cannot obtain the calculated depth distance 6 through calculation, and naturally cannot obtain the actual depth distance 5. At this time, model reconstruction is easy to form holes in the reconstructed three-dimensional model. To this end, the present invention adopts two weighted scoring methods in the selection of effective views 15, the purpose of which is to obtain effective views 15 related to the building to be modeled in different directions. Afterwards, the effective views 15 are used for secondary feature extraction through the GDnet network, so that the true meaning of each part on the effective view 15, especially the structure such as glass, can be identified. For example, when the hole area is glass, or when the non-hole area is glass, a corresponding understanding can be made, which helps to ensure that the model is closer to the actual situation during reconstruction.

[0068] As a preferred embodiment, a simple acquisition example of "selecting effective views 15 that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface from the corresponding subset by the first weighted scoring method" is given here (the principle of the second weighted scoring method is similar, but the parameters or contents considered are different, which will not be repeated here), which is obtained by the following steps:

[0069] A. Quantitative evaluation of coverage

[0070] Calculate the pixel ratio of the hole area in the 2D image corresponding to the hole area, and use it as the coverage rate;

[0071] B. Multi-dimensional evaluation of information volume

[0072] The local variance, number of feature points, information entropy and other methods are used to evaluate the information content of each 2D image corresponding to the primary depth map;

[0073] C. Calculation of the first weighted score

[0074] The original values ​​of coverage and information are normalized to the range of [0, 1] for weighted summation. Then, weights are assigned to coverage and information, and the sum of the weights should be 1. In a preferred example, a machine learning model is used to predict the performance of different feature and weight combinations, and the weights are adjusted accordingly. A weighted score is calculated based on the normalized coverage, normalized information, and the corresponding weights, where the first weighted score = (normalized coverage × coverage weight) + (normalized information × information weight). In one example, the normalized coverage and normalized information can be normalized and calculated using the following formulas:

[0075]

[0076] Where C is the current coverage, C min is the minimum coverage, C max is the maximum coverage; where H is the information entropy of the image, H min and H max are the minimum and maximum values ​​of information entropy in the data set, respectively;

[0077] D. Sort all images in the corresponding subset according to the calculated first weighted score, and select the view with the highest score as the best view, which is used as the effective view 15. It should be noted that sometimes a single effective view 15 cannot reflect all the conditions of the corresponding contour surface. Therefore, in order to ensure the reliability of the completion, after obtaining the best view, an image with a higher score but in a different orientation from the best view can be selected as needed, and it can also be used as the effective view 15; for example, the contour surface at the hole position 14 often needs to determine more than one effective view 15. Therefore, when selecting the effective view 15, a threshold value can be set, and only when the first weighted score is higher than the threshold value, it will be considered.

[0078] In order to better achieve the purpose of the present invention, step S33 adopts Figure 3 The framework diagram shown in the figure is used to update the depth map, wherein, after obtaining the semantic image feature in step S32, it is sent to a specific neural network model such as a CNN model or a Transformer model to update the depth map. When the corresponding semantic image feature is identified as glass, since the glass is usually a plane, it can be first planarized. Since the trajectory of the drone is determined, the coordinate position of the drone when collecting the corresponding image is known. At this time, several depth distances (for example, three) from the glass to the drone position can be found through the edge of the glass (for example, the glass frame). At this time, the depth values ​​at other positions on the glass can be obtained by simple geometric conversion, thereby obtaining an updated depth map. It should be understood that the acquisition of the depth value requires continuous feedback adjustment. It should also be understood that the glass can also be non-planar, which should also belong to the scope of protection of the present invention. However, since the calculation of non-planar glass is more complicated, it will not be discussed here.

[0079] It should also be noted that, generally, the data processing of 3D point clouds is very complex. Relatively speaking, the computational cost of processing and analyzing depth maps is much lower and the response speed is faster. Therefore, the present invention uses depth information for completion before generating the first 3D point cloud data, and then uses the 3D point cloud data for subsequent modeling operations. This helps to improve the quality of three-dimensional model completion and reconstruction while minimizing the amount of calculation and cost.

[0080] In addition, steps 5 to 7 are some common steps and will not be described in detail here.

[0081] In a further preferred embodiment, after the three-dimensional model is reconstructed, texture mapping is also performed using the effective view 15. Since the effective view 15 has been screened by weighted scoring, it can fully reflect the visual features and geometric properties of the building to be modeled in different directions. Therefore, by mapping each contour surface, the model can be given a more realistic appearance and details. Preferably, the texture mapping operation is also implemented by a method based on deep learning, which will not be described in detail here.

[0082] Finally, it should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. Parts not described in detail in this specification belong to the prior art known to professionals in the field.

[0083] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to the embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for completing a three-dimensional model of a building, comprising the following steps: Step S1, using drone aerial photography to obtain an image dataset of the building to be modeled; Step S2: generating a corresponding primary depth map according to the acquired image data set; Step S3: completing the depth information of the generated primary depth map to obtain an updated depth map; Step S4: generating first 3D point cloud data using the updated depth map, and then generating an initial query proposal set using the first 3D point cloud data; Step S5, performing primary feature extraction on the 2D image in the image data set to obtain a plurality of first image features; Step S6: generating an updated query proposal set based on the initial query proposal set and the plurality of first image features; Step S7: Generate and model second 3D point cloud data using the updated query proposal set; It is characterized in that, in step S3, completing the depth information of the primary depth map at least sequentially includes: S31, using the primary depth map to select from the image data set a valid view (15) that can fully reflect the visual features and geometric properties of the building to be modeled in different directions; S32, performing secondary feature extraction on the effective view (15) to obtain a second image feature with semantics; S33: Update the primary depth map based on the second image feature with semantics to obtain an updated depth map.

2. A method for completing a three-dimensional building model as claimed in claim 1, characterized in that: The primary feature extraction and the secondary feature extraction are implemented using the same feature extraction network.

3. A method for completing a three-dimensional building model as claimed in claim 1, characterized in that: The feature extraction network comprises a multi-level feature extractor (7), an LCFI module group (8) and a feature fusion and output module, wherein the multi-level feature extractor (7) is used to receive an input image and extract multi-scale image features therefrom, the LCFI module group (8) is used to process the multi-scale image features from the multi-level feature extractor (7) and simultaneously capture context information from local to global, the LCFI module group (8) comprises a plurality of LCFI modules, each LCFI module is used to extract image features of a predetermined scale and integrate corresponding context information, at least one of the plurality of LCFI modules outputs a low-level image feature (9), the outputs of the remaining LCFI modules in the plurality of LCFI modules are sent to the feature fusion and output module, and after a first fusion process, a high-level image feature (10) is generated, and then after a second fusion process, the output is a second image feature with semantics.

4. A method for completing a three-dimensional building model according to any one of claims 1 to 3, characterized in that: The feature extraction network only performs secondary feature extraction on the effective view (15).

5. A method for completing a three-dimensional building model as claimed in claim 4, characterized in that: The effective view (15) is indexed, and when the feature extraction network is working, its multi-level feature extractor (7) can start the corresponding network layer according to the indexing information. When the indexed effective view (15) enters the feature extraction network, all layers of the multi-level feature extractor (7) are working, and all LCFI modules in the LCFI module group (8) are also working after receiving the output of the multi-level feature extractor (7). The output of the LCFI module group (8) is processed by the feature fusion and output module and is formed into a second image feature with semantics through the secondary feature output. When the unindexed image enters the feature extraction network, only part of the network layers of the multi-level feature extractor (7) are working, and only part of the LCFI modules in the LCFI module group (8) are working. At this time, the network layers in the multi-level feature extractor (7) and the LCFI modules in the LCFI module group (8) that are in working state are only used to generate low-level image features (9).

6. A method for completing a three-dimensional building model as claimed in claim 1, characterized in that: Step S31 specifically includes: S311, pre-processing: Filtering the primary depth map and the 2D images in the image dataset to reduce noise; S312, contour generation and contour surface determination: Extracting the outline of the building to be modeled by using the preprocessed primary depth map, obtaining the basic outline of the building to be modeled, and determining the number of outline surfaces of the building to be modeled and the orientation of the outline surfaces based on the obtained basic outline; S313. Establishment of the correspondence between 2D image and contour: The orientation of the drone when taking the image is compared with the orientation of each contour surface, and the image data set is divided into multiple subsets according to the result of the orientation comparison, each subset corresponds to a specific contour surface; S314, hole area identification: performing hole identification on the primary depth map to identify the hole area in the primary depth map; S315: Determine the location of the cavity area: Projecting the hole region identified in the primary depth map onto the 2D image corresponding thereto, thereby determining the position of the hole region on the corresponding 2D image, and further determining the contour surface corresponding to the hole region and the position of the hole region on the corresponding contour surface; S316: Select a valid view (15).

7. A method for completing a three-dimensional building model as claimed in claim 6, characterized in that: Step S316 includes, for a contour surface with a hole position (14), using a first weighted scoring method to select from a corresponding subset a valid view (15) that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface; for a contour surface without a hole position (14), using a second weighted scoring method to select a valid view (15) that can fully reflect the visual features and geometric properties of the building to be modeled on the current contour surface; wherein the first weighted scoring method is different from the second weighted scoring method, and the first weighted scoring method is associated with the hole information of the hole position (14).

Images