Building semantic three-dimensional model construction method, system, terminal and storage medium

Abstract

The application discloses a semantic three-dimensional model construction method and system of a building, a terminal and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring an oblique photograph image corresponding to the building, performing structured three-dimensional reconstruction based on the oblique photograph image, and obtaining a structured three-dimensional model corresponding to the building; performing structured facade analysis on the structured three-dimensional model to determine each facade in the structured three-dimensional model; determining target images corresponding to each facade from the oblique photograph image according to facade normal vectors corresponding to each facade in the structured three-dimensional model and image normal vectors corresponding to each oblique photograph image; performing semantic segmentation on the target images to obtain semantic information corresponding to the target images; and projecting the semantic information back to the structured three-dimensional model to obtain a semantic three-dimensional model corresponding to the building. In this way, the construction efficiency of the semantic three-dimensional model of the building can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, system, terminal, and storage medium for constructing a semantic 3D model of a building. Background Technology

[0002] With the development of science and technology, especially the development of 3D reconstruction technology, the application of 3D building models is becoming more and more widespread, and users' requirements for 3D building models are also getting higher and higher.

[0003] Currently, 3D reconstruction technology can generate 3D models of buildings, but the reconstruction results lack corresponding semantic information. Existing technologies rely on manual modeling, such as manual modeling based on computer-aided design (CAD) drawings. The problem with existing technologies is that the process of constructing semantic 3D models of buildings requires significant manual intervention, which hinders the improvement of the efficiency of semantic 3D model construction.

[0004] Therefore, the relevant technologies still need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide a method, system, terminal and storage medium for constructing semantic 3D models of buildings, aiming to solve the technical problem in related technologies that the construction of semantic 3D models of buildings relies on manual modeling, requires a lot of manual intervention, and is not conducive to improving the construction efficiency of semantic 3D models of buildings.

[0006] To achieve the above objectives, the first aspect of this application provides a method for constructing a semantic 3D model of a building, wherein the method includes:

[0007] Oblique photogrammetry images of the building are acquired, and structured 3D reconstruction is performed based on the oblique photogrammetry images to obtain the structured 3D model of the building.

[0008] The structured 3D model above is analyzed for its facade to determine each facade.

[0009] Based on the facade normal vectors corresponding to each facade in the above structured 3D model and the image normal vectors corresponding to each of the above oblique photographic images, the target images corresponding to each of the above facades are determined from the above oblique photographic images.

[0010] Semantic segmentation is performed on the target image to obtain the semantic information corresponding to the target image.

[0011] The semantic information is back-projected onto the structured 3D model to obtain the semantic 3D model corresponding to the building.

[0012] Optionally, the above-mentioned structured 3D reconstruction based on the oblique photogrammetric images to obtain the structured 3D model corresponding to the building includes:

[0013] Based on the above oblique photographic images, the sparse point cloud corresponding to the above buildings is obtained by the structure-recovery-motion algorithm.

[0014] Based on the above sparse point cloud, the dense point cloud corresponding to the above building is obtained by multi-view stereo matching.

[0015] Based on the above dense point cloud, a structured 3D model of the above building is obtained by performing structured reconstruction.

[0016] Optionally, before determining the target image corresponding to each of the aforementioned facades from the aforementioned oblique photographic images based on the facade normal vectors corresponding to each facade in the aforementioned structured 3D model and the image normal vectors corresponding to each of the aforementioned oblique photographic images, the above method further includes:

[0017] Spatial pose restoration is performed on the above oblique photographic images;

[0018] Based on the rotation matrix obtained after spatial pose recovery, the image normal vector corresponding to the above oblique photogrammetric image is determined.

[0019] Optionally, before determining the target image corresponding to each of the aforementioned facades from the aforementioned oblique photographic images based on the facade normal vectors corresponding to each facade in the aforementioned structured 3D model and the image normal vectors corresponding to each of the aforementioned oblique photographic images, the above method further includes:

[0020] For each of the above facades, the facade normal vector is determined based on the face normal vector and vertex normal vector of the triangular mesh corresponding to the facade.

[0021] Optionally, the above-mentioned determination of the target image corresponding to each of the aforementioned facades from the aforementioned oblique photographic images, based on the facade normal vectors corresponding to each facade in the aforementioned structured 3D model and the image normal vectors corresponding to each of the aforementioned oblique photographic images, includes:

[0022] Based on the vertex coordinates of the above structured 3D model, determine the geometric center point of the building;

[0023] Obtain the center point of each of the above oblique photographic images;

[0024] For each facade in the above structured 3D model, the plane equation of the facade is constructed according to the geometric boundary corresponding to the facade. The intersection point of the target line of sight and the plane where the facade is located is determined according to the plane equation. The oblique photographic image whose intersection point falls within the geometric boundary corresponding to the facade is used as the candidate image corresponding to the facade. The target line of sight is the line of sight from the center point of the above oblique photographic image to the geometric center point of the building.

[0025] Based on the facade normal vector corresponding to the above facade and the image normal vector of the candidate image corresponding to the above facade, the target image corresponding to the above facade is determined from the candidate image corresponding to the above facade.

[0026] Optionally, determining the target image corresponding to the facade from the candidate images corresponding to the facade based on the facade normal vector and the image normal vector of the candidate images corresponding to the facade includes:

[0027] Calculate the angle between the above facade normal vector and the above image normal vector;

[0028] The candidate image with the smallest included angle among the candidate images corresponding to the above facade is taken as the target image corresponding to the above facade.

[0029] Optionally, the semantic information mentioned above includes component labels corresponding to various building components;

[0030] The above-mentioned semantic information is back-projected onto the above-mentioned structured 3D model to obtain the semantic 3D model corresponding to the above-mentioned building, including:

[0031] Based on the positional correspondence between the target image and the structured 3D model, the component labels are back-projected into the structured 3D model to obtain a semantic 3D model labeled with component labels.

[0032] A second aspect of this application provides a semantic 3D model construction system for buildings, wherein the semantic 3D model construction system for buildings includes:

[0033] The 3D reconstruction module is used to acquire oblique photogrammetric images of the building, perform structured 3D reconstruction based on the oblique photogrammetric images, and obtain the structured 3D model of the building.

[0034] The facade analysis module is used to perform structured facade analysis on the above structured 3D model to determine each facade in the above structured 3D model;

[0035] The target image determination module is used to determine the target image corresponding to each of the above-mentioned facades from the above-mentioned oblique photographic images based on the facade normal vectors corresponding to each facade in the above-mentioned structured 3D model and the image normal vectors corresponding to each of the above-mentioned oblique photographic images.

[0036] The semantic segmentation module is used to perform semantic segmentation on the target image to obtain the semantic information corresponding to the target image.

[0037] The semantic 3D model construction module is used to back-project the above semantic information onto the above structured 3D model to obtain the semantic 3D model corresponding to the above building.

[0038] A third aspect of this application provides a terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of any of the semantic three-dimensional model construction methods for the building.

[0039] A fourth aspect of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the semantic three-dimensional model construction methods for the aforementioned building.

[0040] As can be seen from the above, in this application, oblique photographic images of the building are acquired, and structured 3D reconstruction is performed based on the oblique photographic images to obtain a structured 3D model of the building; structured facade analysis is performed on the structured 3D model to determine each facade in the structured 3D model; based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image, target images corresponding to each facade are determined from the oblique photographic images; semantic segmentation is performed on the target images to obtain semantic information corresponding to the target images; and the semantic information is back-projected onto the structured 3D model to obtain a semantic 3D model of the building.

[0041] Thus, the construction of semantic 3D models of buildings can be achieved without relying on manual modeling or intervention. Instead, a structured 3D model is directly constructed based on oblique photogrammetry images, and the target image corresponding to the facade in the structured 3D model is automatically determined, achieving a match between the facade and the target image. Then, semantic segmentation is automatically performed on the target image to obtain semantic information, which is then projected into the structured 3D model through backprojection, ultimately yielding the semantic 3D model of the building. This allows for the automatic construction of semantic 3D models of buildings based on oblique photogrammetry images, eliminating the need for manual processing and significantly improving the efficiency of semantic 3D model construction. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating a method for constructing a semantic 3D model of a building, as provided in an embodiment of this application.

[0044] Figure 2 This is a schematic diagram of a structured 3D model construction process provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of a target image filtering process provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of a semantic mapping process provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the constituent modules of a semantic 3D model construction system for buildings provided in an embodiment of this application;

[0048] Figure 6 This is a block diagram illustrating the internal structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0049] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0050] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0051] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to classification." Similarly, the phrases "if determined" or "if classified to [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once classified to [the described condition or event]," or "in response to classification to [the described condition or event]."

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0056] Semantic 3D modeling of buildings is a core component of urban digital construction, and the integrated geometric and semantic 3D representation of buildings is becoming increasingly important. Although 3D reconstruction technology based on oblique photogrammetry is relatively mature and can quickly acquire geometric models of large-scale buildings, the reconstruction results lack component-level semantic information and face significant difficulties in reconstructing building facade details and semantic annotation. This severely limits the value of the models in refined applications.

[0057] In existing technologies, the construction of semantic 3D models of buildings relies on manual modeling, rule-driven, or semi-automated methods, such as manual modeling based on CAD drawings or manually adding semantic labels after generating a geometric model using photogrammetry. These methods require a large amount of manual intervention, are inefficient and costly, and lack sufficient automation in semantic information extraction.

[0058] To address at least one of the aforementioned technical problems, this application proposes a method for constructing a semantic 3D model of a building. The method involves: acquiring oblique photographic images corresponding to the building; performing structured 3D reconstruction based on the oblique photographic images to obtain a structured 3D model of the building; performing structured facade analysis on the structured 3D model to determine each facade; determining target images corresponding to each facade from the oblique photographic images based on the facade normal vectors and image normal vectors corresponding to each facade; performing semantic segmentation on the target images to obtain semantic information corresponding to the target images; and back-projecting the semantic information onto the structured 3D model to obtain the semantic 3D model of the building.

[0059] Thus, the construction of semantic 3D models of buildings can be achieved without relying on manual modeling or intervention. Instead, a structured 3D model is directly constructed based on oblique photogrammetry images, and the target image corresponding to the facade in the structured 3D model is automatically determined, achieving a match between the facade and the target image. Then, semantic segmentation is automatically performed on the target image to obtain semantic information, which is then projected into the structured 3D model through backprojection, ultimately yielding the semantic 3D model of the building. This allows for the automatic construction of semantic 3D models of buildings based on oblique photogrammetry images, eliminating the need for manual processing and significantly improving the efficiency of semantic 3D model construction.

[0060] like Figure 1 As shown in the embodiment of this application, a method for constructing a semantic 3D model of a building is provided. Specifically, the method includes the following steps:

[0061] Step S100: Obtain the oblique photogrammetry image corresponding to the building, and perform structured 3D reconstruction based on the oblique photogrammetry image to obtain the structured 3D model corresponding to the building.

[0062] Step S200: Perform structured facade analysis on the above structured 3D model to determine each facade in the above structured 3D model;

[0063] Step S300: Based on the facade normal vectors corresponding to each facade in the above structured 3D model and the image normal vectors corresponding to each of the above oblique photographic images, determine the target images corresponding to each of the above facades from the above oblique photographic images respectively.

[0064] Step S400: Perform semantic segmentation on the target image to obtain the semantic information corresponding to the target image;

[0065] Step S500: The semantic information is back-projected into the structured 3D model to obtain the semantic 3D model corresponding to the building.

[0066] The aforementioned buildings are those requiring semantic 3D model construction. Specific buildings can be designated or selected based on actual needs, and no specific limitations are imposed here. The aforementioned oblique photogrammetric images can be obtained through drone acquisition. For example, a drone equipped with image acquisition devices can be used to photograph the area where the buildings are located to obtain multi-view oblique photogrammetric images. Alternatively, other methods can be used, and no specific limitations are imposed here.

[0067] Specifically, the above-mentioned structured 3D reconstruction based on the oblique photogrammetry images to obtain the structured 3D model corresponding to the building includes:

[0068] Based on the above oblique photographic images, the sparse point cloud corresponding to the above buildings is obtained by the structure-recovery-motion algorithm.

[0069] Based on the above sparse point cloud, the dense point cloud corresponding to the above building is obtained by multi-view stereo matching.

[0070] Based on the above dense point cloud, a structured 3D model of the above building is obtained by performing structured reconstruction.

[0071] In order to provide a geometric reference framework for the subsequent semantic 3D model construction process, in this embodiment of the application, it is necessary to first establish a structured 3D model of the building. Figure 2 This is a schematic diagram of a structured 3D model construction process provided in an embodiment of this application, such as... Figure 2 As shown, in the 3D reconstruction stage, the oblique photogrammetric images are processed by the Structure from Motion (SfM) algorithm to obtain sparse point clouds of buildings, and then dense point clouds of buildings are obtained by multi-view stereo matching (MVS).

[0072] Furthermore, in the structured modeling stage, a pre-set structured reconstruction algorithm is used to extract regular geometric structures from the generated dense point cloud of buildings, resulting in a lightweight structured 3D model with high-order structural information. The pre-set structured reconstruction algorithm can be preset and adjusted according to actual needs; for example, the PolyFit algorithm can be used.

[0073] In this embodiment, semantic information extraction is based on the target image (i.e., the optimal facade image that matches the building facade) in the oblique photographic images of the building. Specifically, image filtering is performed based on the image normal vector and the facade normal vector. Therefore, in this embodiment, it is necessary to calculate the image normal vector and the facade normal vector first.

[0074] Specifically, before determining the target image corresponding to each of the aforementioned facades from the aforementioned oblique photographic images based on the facade normal vectors corresponding to each facade in the aforementioned structured 3D model and the image normal vectors corresponding to each of the aforementioned oblique photographic images, the aforementioned method further includes:

[0075] Spatial pose restoration is performed on the above oblique photographic images;

[0076] Based on the rotation matrix obtained after spatial pose recovery, the image normal vector corresponding to the above oblique photogrammetric image is determined.

[0077] In this embodiment, the spatial pose of the oblique photographic image is first restored, and then the image normal vector corresponding to the image is calculated using a rotation matrix. Furthermore, to ensure that the image normal vector always points to the center of the building and thus does not affect the subsequent selection of target images, this application introduces the coordinates of the building center to perform a direction consistency check, thereby ensuring that the photographic direction of the image always points to the geometric center of the building.

[0078] Specifically, calculate the vector from the center point of the photograph to the geometric center point of the building. :

[0079] ;

[0080] in, The coordinates of the geometric center point of the building are determined based on the structured three-dimensional model of the building. These are the coordinates of the center point of the oblique photographic image.

[0081] Define a front vector in the camera coordinate system Where T represents the vector transpose. Calculate the image normal vector. initial value ,in, . This is the third column of the rotation matrix. Calculate the dot product. If the calculated dot product d is greater than or equal to 0, it means that the angle between the two vectors is less than or equal to 90°, and their directions are basically the same. Otherwise, if the calculated dot product d is less than 0, it means the angle between the two vectors is greater than 90° and their directions are basically opposite. The final This serves as the final calculated image normal vector.

[0082] Furthermore, before determining the target image corresponding to each of the aforementioned facades from the aforementioned oblique photographic images based on the facade normal vectors corresponding to each facade in the aforementioned structured 3D model and the image normal vectors corresponding to each of the aforementioned oblique photographic images, the aforementioned method further includes:

[0083] For each of the above facades, the facade normal vector is determined based on the face normal vector and vertex normal vector of the triangular mesh corresponding to the facade.

[0084] For the facade normal vector corresponding to the structured 3D model, in this embodiment of the application, the normalized direction vector describing the geometric features of the building is extracted by weighted averaging of the face normal vector of the triangular mesh and the vertex normal vector, combined with deduplication processing, and used as the facade normal vector corresponding to the facade.

[0085] Specifically, based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each of the oblique photographic images, the target images corresponding to each facade are determined from the oblique photographic images, including:

[0086] Based on the vertex coordinates of the above structured 3D model, determine the geometric center point of the building;

[0087] Obtain the center point of each of the above oblique photographic images;

[0088] For each facade in the above structured 3D model, the plane equation of the facade is constructed according to the geometric boundary corresponding to the facade. The intersection point of the target line of sight and the plane where the facade is located is determined according to the plane equation. The oblique photographic image whose intersection point falls within the geometric boundary corresponding to the facade is used as the candidate image corresponding to the facade. The target line of sight is the line of sight from the center point of the above oblique photographic image to the geometric center point of the building.

[0089] Based on the facade normal vector corresponding to the above facade and the image normal vector of the candidate image corresponding to the above facade, the target image corresponding to the above facade is determined from the candidate image corresponding to the above facade.

[0090] The process of determining the target image corresponding to the facade from the candidate images corresponding to the facade, based on the facade normal vector and the image normal vector of the candidate images corresponding to the facade, includes:

[0091] Calculate the angle between the above facade normal vector and the above image normal vector;

[0092] The candidate image with the smallest included angle among the candidate images corresponding to the above facade is taken as the target image corresponding to the above facade.

[0093] In this embodiment of the application, for the selection of target images, in order to minimize perspective distortion, the angle between the aforementioned facade normal vector and the aforementioned image normal vector is... The smallest image is the target image corresponding to this facade. Specifically, let the facade normal vector be denoted as... The image normal vector (i.e., the photographic direction vector) is Its spatial angle The calculation formula is:

[0094] ;

[0095] in, Representative to and Perform vector dot product operation.

[0096] For standard cubic buildings, parallel facades may exist, which can lead to duplicate normal vectors, resulting in duplicate or missing target images. To further address this issue, this application employs a precise filtering method based on facade boundaries.

[0097] Specifically, in the embodiments of this application, the geometric center point of the building Calculated based on the following formula:

[0098] ;

[0099] in, This represents the coordinates of the i-th vertex in the structured 3D model, where n is the total number of vertices. The geometric center point of the building described above characterizes the overall spatial position of the building.

[0100] For each actual elevation j in the structured 3D model, the plane equation corresponding to elevation j is directly constructed based on its geometric boundaries. :

[0101] ;

[0102] Specifically, the above plane equations It can be used to characterize the plane to which elevation j belongs. Wherein, Let be a variable representing the coordinates of the point; This represents the coordinates of any vertex on elevation j; This represents the unit normal vector of facade j (i.e., the facade normal vector corresponding to facade j). It should be noted that, in this embodiment, the facade normal vector... In fact, it also represents the unit normal vector corresponding to the facade. and The meanings are similar, the only difference being whether it refers to a specific facade, i.e. This represents the facade normal vector corresponding to a general facade (i.e., without specifying a particular facade). The vector represents the facade normal vector corresponding to the specific facade j. The meanings, calculation processes, and usage of both can be used as a reference, and will not be elaborated here.

[0103] It should be further noted that for the i-th oblique photographic image, its photographic center point is denoted as... In the embodiments of this application, It also represents the center point of the oblique photographic image. and The meanings are similar, the only difference being whether it specifically refers to a particular oblique photographic image, i.e. This represents the center point of a general oblique photographic image (without specifying which oblique photographic image it refers to). The term represents the center point of the i-th oblique photographic image. The meanings, calculation processes, and usage of these terms can be used as a reference for each other, and will not be elaborated upon here.

[0104] For the center point of the i-th oblique photographic image Target line of sight Intersection with the plane to which elevation j belongs The position can be represented as:

[0105] ;

[0106] Where t represents the slope of the equation. The equation is valid if and only if t satisfies... Intersection When the image falls within the polygonal boundary corresponding to facade j, the center point of the photograph is... The corresponding oblique photographic image is included in the set of candidate images corresponding to facade j:

[0107] ;

[0108] in, This represents the set of candidate images corresponding to facade j. Represents the i-th candidate image; This represents the corresponding phase plane. The formula above indicates that when the line intersects the plane... When it appears in j, the corresponding image Added to the set as candidate images.

[0109] For each facade, the target image is searched only among its corresponding candidate images. Therefore, the target image corresponding to the j-th facade is... It can be represented as:

[0110] ;

[0111] in, represent and The included angle.

[0112] Figure 3 This is a schematic diagram of a target image filtering process provided in an embodiment of this application, such as... Figure 3 As shown in this embodiment, spatial pose recovery is performed on the oblique photographic image to obtain the corresponding image normal vector. Structured facade analysis is performed on the structured 3D model to obtain the facade normal vector. Then, based on the image normal vector and the facade normal vector, a screening based on the facade boundary is performed to finally determine the target image corresponding to the facade, thereby supplementing semantic information based on the target image.

[0113] In this embodiment of the application, the semantic information includes component labels corresponding to various building components; the above-projection of the semantic information back into the structured 3D model to obtain the semantic 3D model corresponding to the building includes:

[0114] Based on the positional correspondence between the target image and the structured 3D model, the component labels are back-projected into the structured 3D model to obtain a semantic 3D model labeled with component labels.

[0115] Figure 4 This is a schematic diagram of a semantic mapping process provided in an embodiment of this application, such as... Figure 4 As shown in this embodiment, for the target images corresponding to each selected facade, a deep learning method is used for semantic segmentation, and then the facade layout is regularized on the semantic segmentation results to obtain regularized semantic information. Finally, the regularized semantic information is back-projected onto a structured 3D model to construct a semantic 3D model of the building with complete semantic annotation.

[0116] This application employs deep learning methods to achieve accurate semantic segmentation of building facade elements. First, an improved deep learning segmentation network is designed and trained using publicly available building facade annotation datasets. For example, the deep learning segmentation network can adopt FastSCNN or DeepLabv3+ architectures, and its performance is optimized through data augmentation, transfer learning, and other techniques. The trained model can perform end-to-end semantic segmentation on input building facade images, outputting pixel-level label maps containing various building components (such as windows, doors, decorative components, etc.). A pixel-level label map refers to a map with semantic labels for each component; a single pixel-level label map includes semantic information for multiple components.

[0117] In this embodiment, facade layout rules are pre-set for regularization processing. Architectural geometric information automatically extracted from images often suffers from issues such as positional offsets, rough edges, and inconsistent dimensions. Therefore, regularization processing is necessary to ensure that the distribution of building components conforms to the structural rules of the building. In this embodiment, firstly, the predicted mask of the target component is extracted from semantic segmentation to generate the corresponding binary image. Then, morphological optimization is performed on the output binary image, eliminating noise through erosion followed by dilation. Next, connected component analysis is used to accurately locate each window, recording its position and size. A dynamic threshold grouping algorithm is employed, automatically calculating reasonable row height and column width thresholds by analyzing the relative distance between the center points of all building components, completing precise grouping, automatically identifying building components in the same row / column, and adjusting their center coordinates to a unified axis while maintaining the original spatial distribution pattern.

[0118] After regularization, pixel-level label images yield semantic images containing semantic information, such as component labels corresponding to various building components. These regularized semantic images are then back-projected onto the elevation of the corresponding structured 3D model of the building using the collinearity equation in photogrammetry, resulting in a model labeled with semantic information—the semantic 3D model of the building.

[0119] Thus, the construction of semantic 3D models of buildings can be achieved without relying on manual modeling or intervention. Instead, a structured 3D model is directly constructed based on oblique photogrammetry images, and the target image corresponding to the facade in the structured 3D model is automatically determined, achieving a match between the facade and the target image. Then, semantic segmentation is automatically performed on the target image to obtain semantic information, which is then projected into the structured 3D model through backprojection, ultimately yielding the semantic 3D model of the building. This allows for the automatic construction of semantic 3D models of buildings based on oblique photogrammetry images, eliminating the need for manual processing and significantly improving the efficiency of semantic 3D model construction.

[0120] Specifically, in this embodiment, firstly, multi-angle oblique photographic images of the building are acquired. A sparse point cloud is generated using a structure-of-motion (SfM) algorithm. Then, a dense point cloud is obtained using multi-view stereo matching technology. Finally, the PolyFit algorithm is used for structured geometric reconstruction to obtain a structured 3D model of the building. Based on this, the best facade image (i.e., the target image corresponding to each facade) is selected through optimal image matching. Automated semantic segmentation is performed using a deep learning algorithm, and standardized semantic annotation results are obtained by combining facade layout regularization processing. Finally, the regularized semantic information is back-projected onto the structured 3D model, achieving integrated fusion of geometric structure and semantic information, thereby constructing a semantic 3D model with complete semantic annotation, enabling efficient and convenient creation of a semantic 3D model of the building. Regarding the selection of the optimal facade image (i.e., the target image) from the multi-view images, this application defines the image with the smallest angle between the image normal vector and the facade normal vector of the structured facade as the optimal image for that facade. Firstly, an image normal vector calculation model based on camera extrinsic parameters recovered using SfM is established, and building center coordinate constraints are introduced to ensure directional consistency. Secondly, in the target image selection stage, to address the matching ambiguity caused by parallel facades, a precise matching algorithm based on the geometric features of facade boundaries is proposed, ensuring the uniqueness of the matched image for each facade. These measures fully guarantee the uniqueness and accuracy of target image selection.

[0121] This application proposes a method for constructing semantic 3D models of buildings by combining 3D structured models with regularized semantic information. This method maximizes the information value of UAV-collected data through a target image matching mechanism and significantly reduces manual intervention by introducing a deep learning network. Furthermore, this application significantly improves the engineering applicability of semantic annotation through facade layout regularization, accurately locates facade elements through morphological processing and connected component analysis, and achieves automatic row and column alignment using a dynamic threshold grouping algorithm. This method only requires the semantic segmentation results as input to automatically output a regularized facade layout conforming to the building module. Finally, through an innovative automated modeling process, the cost of semantic modeling is significantly reduced, achieving efficient, convenient, and scalable semantic 3D model construction.

[0122] like Figure 5 As shown in the figure, corresponding to the semantic 3D model construction method of the above-mentioned building, this application embodiment also provides a semantic 3D model construction system for buildings, the semantic 3D model construction system for buildings including:

[0123] The 3D reconstruction module 510 is used to acquire oblique photographic images corresponding to the building, and perform structured 3D reconstruction based on the oblique photographic images to obtain a structured 3D model corresponding to the building.

[0124] The facade analysis module 520 is used to perform structured facade analysis on the above-mentioned structured 3D model to determine each facade in the above-mentioned structured 3D model.

[0125] The target image determination module 530 is used to determine the target image corresponding to each of the above-mentioned facades from the above-mentioned oblique photographic images based on the facade normal vectors corresponding to each facade in the above-mentioned structured 3D model and the image normal vectors corresponding to each of the above-mentioned oblique photographic images.

[0126] The semantic segmentation module 540 is used to perform semantic segmentation on the target image to obtain the semantic information corresponding to the target image.

[0127] The semantic 3D model construction module 550 is used to back-project the above semantic information onto the above structured 3D model to obtain the semantic 3D model corresponding to the above building.

[0128] Thus, the construction of semantic 3D models of buildings does not rely on manual modeling or intervention. Instead, it directly constructs structured 3D models based on oblique photogrammetry images and automatically identifies the target images corresponding to the facades in the structured 3D model, achieving a match between the facades and the target images. Then, it automatically performs semantic segmentation on the target images to obtain semantic information, and projects this semantic information into the structured 3D model through backprojection, ultimately obtaining the semantic 3D model of the building. This allows for the automatic construction of semantic 3D models of buildings based on oblique photogrammetry images, eliminating the need for manual processing and significantly improving the efficiency of semantic 3D model construction.

[0129] It should be noted that the specific structure and implementation of the semantic 3D model construction system for the above-mentioned buildings and its various modules or units can be referred to the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0130] It should be noted that the division of the modules in the semantic 3D model construction system for the above-mentioned buildings is not unique and is not intended as a specific limitation.

[0131] Based on the above embodiments, this application also provides a terminal, the principle block diagram of which can be as follows: Figure 6As shown. The aforementioned terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of any of the aforementioned methods for constructing a semantic 3D model of a building. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0132] Those skilled in the art will understand that Figure 6 The block diagram shown is only a partial structural diagram related to the solution of this application and does not constitute a limitation on the terminal on which the solution of this application is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0133] In one embodiment, a terminal is provided, the terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of any of the semantic three-dimensional model construction methods for buildings provided in the embodiments of this application.

[0134] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the semantic three-dimensional model construction methods for buildings provided in this application.

[0135] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] In the embodiments provided in this application, it should be understood that the disclosed systems / terminal devices and methods can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0140] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0141] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of constructing a semantic three-dimensional model of a building, characterized by, The method includes: Oblique photogrammetry images of the building are acquired, and structured 3D reconstruction is performed based on the oblique photogrammetry images to obtain a structured 3D model of the building. The structured 3D model is subjected to structured elevation analysis to determine each elevation in the structured 3D model; Based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image, the target image corresponding to each facade is determined from the oblique photographic image. Perform semantic segmentation on the target image to obtain the semantic information corresponding to the target image; The semantic information is back-projected onto the structured 3D model to obtain the semantic 3D model corresponding to the building; The step of determining the target image corresponding to each facade from the oblique photographic images based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image includes: The geometric center point of the building is determined based on the vertex coordinates of the structured 3D model. Obtain the center point of each of the oblique photographic images; For each facade in the structured 3D model, the plane equation of the facade is constructed according to the geometric boundary corresponding to the facade. The intersection point of the target line of sight and the plane where the facade is located is determined according to the plane equation. The oblique photographic image whose intersection point falls within the geometric boundary corresponding to the facade is used as the candidate image corresponding to the facade. The target line of sight is the line of sight from the center point of the oblique photographic image to the geometric center point of the building. Based on the facade normal vector and the image normal vector of the candidate image corresponding to the facade, the target image corresponding to the facade is determined from the candidate image corresponding to the facade.

2. The method of claim 1, wherein, The step of performing structured 3D reconstruction based on the oblique photographic images to obtain a structured 3D model corresponding to the building includes: Based on the oblique photographic image, the sparse point cloud corresponding to the building is obtained by the structure-recovery-motion algorithm. Based on the sparse point cloud, the dense point cloud corresponding to the building is obtained by multi-view stereo matching; The structured 3D model of the building is obtained by performing structured reconstruction based on the dense point cloud.

3. The method of claim 1, wherein, Before determining the target image corresponding to each facade from the oblique photographic images based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image, the method further includes: Spatial pose recovery is performed on the oblique photographic images; Based on the rotation matrix obtained after spatial pose recovery, the image normal vector corresponding to the oblique photographic image is determined.

4. The method of claim 1, wherein, Before determining the target image corresponding to each facade from the oblique photographic images based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image, the method further includes: For each of the aforementioned facades, the facade normal vector is determined based on the face normal vector and vertex normal vector of the triangular mesh corresponding to the facade.

5. The method of claim 1, wherein, The step of determining the target image corresponding to the facade from the candidate images corresponding to the facade based on the facade normal vector and the image normal vector of the candidate images corresponding to the facade includes: Calculate the angle between the facade normal vector and the image normal vector; The candidate image with the smallest included angle among the candidate images corresponding to the facade is taken as the target image corresponding to the facade.

6. The method according to any one of claims 1 to 5, wherein The semantic information includes component tags corresponding to various building components; The step of back-projecting the semantic information onto the structured 3D model to obtain the semantic 3D model corresponding to the building includes: Based on the positional correspondence between the target image and the structured 3D model, the component labels are back-projected into the structured 3D model to obtain a semantic 3D model labeled with component labels.

7. A semantic three-dimensional model construction system of a building, characterized by, The system includes: The 3D reconstruction module is used to acquire oblique photogrammetric images corresponding to buildings, perform structured 3D reconstruction based on the oblique photogrammetric images, and obtain a structured 3D model corresponding to the building. The facade analysis module is used to perform structured facade analysis on the structured 3D model to determine each facade in the structured 3D model; The target image determination module is used to determine the target image corresponding to each facade from the oblique photogrammetry image based on the facade normal vector corresponding to each facade in the structured 3D model and the image normal vector corresponding to each oblique photogrammetry image. A semantic segmentation module is used to perform semantic segmentation on the target image to obtain semantic information corresponding to the target image; A semantic 3D model construction module is used to back-project the semantic information onto the structured 3D model to obtain the semantic 3D model corresponding to the building; The target image determination module is specifically used for: determining the geometric center point of the building based on the vertex coordinates of the structured 3D model; obtaining the photographic center point of each of the oblique photographic images; for each facade in the structured 3D model, constructing the plane equation of the facade based on the geometric boundary corresponding to the facade, determining the intersection point of the target line of sight and the plane where the facade is located based on the plane equation, and using the oblique photographic images whose intersection point falls within the geometric boundary corresponding to the facade as candidate images corresponding to the facade, wherein the target line of sight is the line of sight from the photographic center point of the oblique photographic image to the geometric center point of the building; and determining the target image corresponding to the facade from the candidate images corresponding to the facade based on the facade normal vector and the image normal vector of the candidate images corresponding to the facade.

8. A terminal, characterized by comprising: The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the semantic three-dimensional model construction method for a building as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the semantic three-dimensional model construction method for a building as described in any one of claims 1 to 6.

Images