Semantic three-dimensional model construction method and system of building, terminal and storage medium

Abstract

The invention discloses a semantic three-dimensional model construction method and system of a building, a terminal and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining an oblique photography image corresponding to the building, carrying out the structured three-dimensional reconstruction based on the oblique photography image, and obtaining a structured three-dimensional model corresponding to the building; performing structured facade analysis on the structured three-dimensional model, and determining each facade in the structured three-dimensional model; according to the facade normal vector corresponding to each facade in the structured three-dimensional model and the image normal vector corresponding to each oblique photography image, respectively determining a target image corresponding to each facade from the oblique photography images; performing semantic segmentation on the target image to obtain semantic information corresponding to the target image; and reversely projecting the semantic information into the structured three-dimensional model to obtain a semantic three-dimensional model corresponding to the building. Therefore, the construction efficiency of the semantic three-dimensional model of the building can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a building semantic three-dimensional model construction method and system, a terminal and a storage medium. BACKGROUND

[0002] With the development of science and technology, especially the development of three-dimensional reconstruction technology, the application of building three-dimensional models is more and more extensive, and the requirements of users for building three-dimensional models are also higher and higher.

[0003] At present, a building three-dimensional model can be generated based on three-dimensional reconstruction technology, but the reconstruction result lacks corresponding semantic information. In the prior art, the construction of a building semantic three-dimensional model depends on artificial modeling, such as manual modeling based on computer-aided design (CAD) drawings. The problem of the prior art is that the process of constructing a building semantic three-dimensional model needs a large amount of manual intervention, which is not conducive to improving the construction efficiency of the building semantic three-dimensional model.

[0004] Therefore, the related technology needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a building semantic three-dimensional model construction method and system, a terminal and a storage medium, which aims to solve the technical problem that the construction of a building semantic three-dimensional model in the related art depends on artificial modeling and needs a large amount of manual intervention, which is not conducive to improving the construction efficiency of the building semantic three-dimensional model.

[0006] In order to achieve the above purpose, the first aspect of the present application provides a building semantic three-dimensional model construction method, wherein the building semantic three-dimensional model construction method comprises: obtaining a structured three-dimensional model corresponding to a building by acquiring a tilt photography image corresponding to the building and performing structured three-dimensional reconstruction based on the tilt photography image; performing structured facade analysis on the structured three-dimensional model to determine each facade in the structured three-dimensional model; determining a target image corresponding to each facade from the tilt photography image according to a facade normal vector corresponding to each facade in the structured three-dimensional model and an image normal vector corresponding to each tilt photography image; performing semantic segmentation on the target image to obtain semantic information corresponding to the target image; projecting the semantic information into the structured three-dimensional model to obtain a semantic three-dimensional model corresponding to the building.

[0007] Optionally, the structured three-dimensional reconstruction based on the above oblique photography images obtains a structured three-dimensional model corresponding to the building, comprising: According to the above oblique photography images, a sparse point cloud corresponding to the building is obtained by a motion recovery structure algorithm; According to the above sparse point cloud, a dense point cloud corresponding to the building is obtained by multi-view stereo matching; According to the above dense point cloud, a structured three-dimensional model corresponding to the building is obtained by structured reconstruction.

[0008] Optionally, before determining the target image corresponding to each facade from the above oblique photography images according to the facade normal vector corresponding to each facade in the above structured three-dimensional model and the image normal vector corresponding to each oblique photography image, the method further comprises: Spatial pose recovery is performed on the above oblique photography images; According to the rotation matrix obtained after spatial pose recovery, the image normal vector corresponding to the above oblique photography image is determined.

[0009] Optionally, before determining the target image corresponding to each facade from the above oblique photography images according to the facade normal vector corresponding to each facade in the above structured three-dimensional model and the image normal vector corresponding to each oblique photography image, the method further comprises: For each facade, the facade normal vector corresponding to the facade is determined according to the face normal vector and the vertex normal vector of the triangular mesh corresponding to the facade.

[0010] Optionally, the determination of the target image corresponding to each facade from the above oblique photography images according to the facade normal vector corresponding to each facade in the above structured three-dimensional model and the image normal vector corresponding to each oblique photography image comprises: According to the vertex coordinates of the above structured three-dimensional model, a geometric center point of the building is determined; The photography center point of each of the above oblique photography images is obtained; For each facade in the above structured three-dimensional model, a plane equation of the facade is constructed according to the geometric boundary corresponding to the facade, the intersection point of the target line of sight and the plane on which the facade is located is determined according to the plane equation, and the oblique photography image in which the intersection point falls within the geometric boundary corresponding to the facade is taken as the candidate image corresponding to the facade, wherein the target line of sight is the line of sight from the photography center point of the oblique photography image to the geometric center point of the building; According to the facade normal vector corresponding to the facade and the image normal vector of the candidate image corresponding to the facade, the target image corresponding to the facade is determined from the candidate image corresponding to the facade.

[0011] Optionally, determining the target image corresponding to the facade from the candidate images corresponding to the facade based on the facade normal vector and the image normal vector of the candidate images corresponding to the facade includes: Calculate the angle between the above facade normal vector and the above image normal vector; The candidate image with the smallest included angle among the candidate images corresponding to the above facade is taken as the target image corresponding to the above facade.

[0012] Optionally, the semantic information mentioned above includes component labels corresponding to various building components; The above-mentioned semantic information is back-projected onto the above-mentioned structured 3D model to obtain the semantic 3D model corresponding to the above-mentioned building, including: Based on the positional correspondence between the target image and the structured 3D model, the component labels are back-projected into the structured 3D model to obtain a semantic 3D model labeled with component labels.

[0013] A second aspect of this application provides a semantic 3D model construction system for buildings, wherein the semantic 3D model construction system for buildings includes: The 3D reconstruction module is used to acquire oblique photogrammetric images of the building, perform structured 3D reconstruction based on the oblique photogrammetric images, and obtain the structured 3D model of the building. The facade analysis module is used to perform structured facade analysis on the above structured 3D model to determine each facade in the above structured 3D model; The target image determination module is used to determine the target image corresponding to each of the above-mentioned facades from the above-mentioned oblique photographic images based on the facade normal vectors corresponding to each facade in the above-mentioned structured 3D model and the image normal vectors corresponding to each of the above-mentioned oblique photographic images. The semantic segmentation module is used to perform semantic segmentation on the target image to obtain the semantic information corresponding to the target image. The semantic 3D model construction module is used to back-project the above semantic information onto the above structured 3D model to obtain the semantic 3D model corresponding to the above building.

[0014] A third aspect of this application provides a terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of any of the semantic three-dimensional model construction methods for the building.

[0015] A fourth aspect of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the semantic three-dimensional model construction methods for the aforementioned building.

[0016] As can be seen from the above, in the scheme, the corresponding oblique photography image of the building is acquired, the structured three-dimensional model corresponding to the building is obtained based on the structured three-dimensional reconstruction of the oblique photography image, the structured facade analysis of the structured three-dimensional model is performed, each facade in the structured three-dimensional model is determined, the target image corresponding to each facade is determined from the oblique photography image according to the facade normal vector corresponding to each facade in the structured three-dimensional model and the image normal vector corresponding to each oblique photography image, the semantic segmentation of the target image is performed to obtain the semantic information corresponding to the target image, and the semantic information is back-projected into the structured three-dimensional model to obtain the semantic three-dimensional model corresponding to the building.

[0017] In this way, in the process of constructing the semantic three-dimensional model of the building, artificial modeling is not needed, and manual intervention is not needed. Instead, the structured three-dimensional model is directly constructed based on the oblique photography image, the target image corresponding to each facade in the structured three-dimensional model is automatically determined, the matching between the facade and the target image is realized, the semantic segmentation of the target image is automatically performed to obtain the semantic information, and the semantic information is projected into the structured three-dimensional model through back-projection, so as to finally obtain the semantic three-dimensional model corresponding to the building. The construction of the semantic three-dimensional model of the building based on the oblique photography image can be automatically completed, manual processing is not needed, and the construction efficiency of the semantic three-dimensional model of the building is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a flowchart of a method for constructing a semantic three-dimensional model of a building provided by an embodiment of the present application; Figure 2 is a flowchart of a method for constructing a structured three-dimensional model provided by an embodiment of the present application; Figure 3 is a flowchart of a method for screening a target image provided by an embodiment of the present application; Figure 4 is a flowchart of a method for semantic mapping provided by an embodiment of the present application; Figure 5 is a schematic diagram of a component module of a system for constructing a semantic three-dimensional model of a building provided by an embodiment of the present application; Figure 6is a kind of internal structure principle block diagram of terminal provided in the embodiment of the application. DETAILED DESCRIPTION

[0020] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0021] It is to be understood that the terminology "including", when used in the present specification and in the accompanying claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] It is also to be understood that the terminology used in the present specification and the appended claims is for the purpose of describing the particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0023] It will be further understood that the terms "and / or", as used in the present specification and the appended claims, are to be interpreted as including any one or more of the listed potential combinations of elements and / or groups thereof.

[0024] As used in this specification and the appended claims, the term "if' can be construed to mean "when" or "once" or "in response to determining" or "in response to ascertaining," depending on the context. Similarly, the phrase "if it is determined" or "if it is ascertained" can be construed to mean "once it is determined" or "in response to determining" or "once it is ascertained" or "in response to ascertaining," depending on the context.

[0025] The technical solutions in the embodiments of the present application are clearly and completely described below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0026] Many specific details are set forth in the following description in order to provide a thorough understanding of the present application. However, the present application can be practiced according to other embodiments that can not be described in detail herein, and the present application is not limited to the embodiments described herein. It will be understood that the present application can be practiced with modification and alteration, and that the present application is not limited to the above-described embodiments.

[0027] Building semantic 3D modeling is a core step of urban digital construction, and it is increasingly important to express buildings in geometry and semantics. Although the current 3D reconstruction technology based on oblique photography is relatively mature, and can quickly obtain the geometric model of a large range of buildings, the reconstruction result lacks component-level semantic information, and there are great difficulties in building facade detail reconstruction and semantic annotation, which seriously restricts the value of the model in fine application.

[0028] In the prior art, the construction of a building semantic 3D model relies on manual modeling, rule-driven or semi-automatic methods, such as manual modeling based on CAD drawings or manually adding semantic labels after generating a geometric model using photogrammetry. These methods require a lot of manual intervention, are low in efficiency and high in cost, and the degree of automation of semantic information extraction is insufficient.

[0029] In order to solve at least one of the above technical problems, the present application proposes a building semantic 3D model construction method. In the present application, the oblique photography image corresponding to the building is obtained, the structured 3D reconstruction is carried out based on the oblique photography image, and the structured 3D model corresponding to the building is obtained. The structured facade analysis is carried out on the structured 3D model, and each facade in the structured 3D model is determined. According to the facade normal vector corresponding to each facade in the structured 3D model and the image normal vector corresponding to each oblique photography image, the target image corresponding to each facade is determined from the oblique photography image. The semantic segmentation is carried out on the target image, and the semantic information corresponding to the target image is obtained. The semantic information is back-projected into the structured 3D model, and the semantic 3D model corresponding to the building is obtained.

[0030] In this way, in the process of constructing the semantic 3D model of the building, it is not necessary to rely on manual modeling or manual intervention. Instead, the structured 3D model is directly constructed based on the oblique photography image, and the target image corresponding to the facade in the structured 3D model is automatically determined to realize the matching of the facade and the target image. Further, the semantic segmentation is automatically carried out on the target image to obtain the semantic information, and the semantic information is projected into the structured 3D model through back-projection, and finally the semantic 3D model corresponding to the building is obtained. The construction of the semantic 3D model of the building based on the oblique photography image can be automatically completed without manual processing, which is helpful to improve the construction efficiency of the semantic 3D model of the building.

[0031] As Figure 1 shown, the embodiment of the present application provides a semantic three-dimensional model construction method of a building, specifically, the above method comprises the following steps: Step S100, obtaining the corresponding oblique photography image of the building, performing structured three-dimensional reconstruction based on the above oblique photography image, and obtaining the corresponding structured three-dimensional model of the building; Step S200, performing structured facade analysis on the structured three-dimensional model to determine each facade in the structured three-dimensional model; Step S300, determining the target image corresponding to each facade from the above oblique photography image according to the facade normal vector corresponding to each facade in the structured three-dimensional model and the image normal vector corresponding to each of the above oblique photography images; Step S400, performing semantic segmentation on the target image to obtain the semantic information corresponding to the target image; Step S500, projecting the semantic information into the structured three-dimensional model to obtain the semantic three-dimensional model corresponding to the building.

[0032] Among them, the building is a building that needs to be constructed into a semantic three-dimensional model, which can be specified or selected according to actual needs, and is not limited here. The above oblique photography image can be obtained by a drone, for example, a drone equipped with an image acquisition device is used to take pictures of the area where the building is located, and multi-view drone oblique photography images are obtained, or other methods can also be used to obtain, which is not limited here.

[0033] Specifically, the above structured three-dimensional reconstruction based on the above oblique photography image to obtain the corresponding structured three-dimensional model of the building comprises: According to the above oblique photography image, the sparse point cloud corresponding to the building is obtained by a motion recovery structure algorithm; According to the above sparse point cloud, the dense point cloud corresponding to the building is obtained by multi-view stereo matching; According to the above dense point cloud, the structured three-dimensional model corresponding to the building is obtained by structured reconstruction.

[0034] In order to provide a geometric reference framework for the subsequent semantic three-dimensional model construction process, in the embodiment of the present application, it is necessary to first establish the structured three-dimensional model of the building. Figure 2 is a structured three-dimensional model construction process schematic diagram provided by the embodiment of the present application, as Figure 2As shown, in the 3D reconstruction stage, the oblique photogrammetric images are processed by the Structure from Motion (SfM) algorithm to obtain sparse point clouds of buildings, and then dense point clouds of buildings are obtained by multi-view stereo matching (MVS).

[0035] Furthermore, in the structured modeling stage, a pre-set structured reconstruction algorithm is used to extract regular geometric structures from the generated dense point cloud of buildings, resulting in a lightweight structured 3D model with high-order structural information. The pre-set structured reconstruction algorithm can be preset and adjusted according to actual needs; for example, the PolyFit algorithm can be used.

[0036] In this embodiment, semantic information extraction is based on the target image (i.e., the optimal facade image that matches the building facade) in the oblique photographic images of the building. Specifically, image filtering is performed based on the image normal vector and the facade normal vector. Therefore, in this embodiment, it is necessary to calculate the image normal vector and the facade normal vector first.

[0037] Specifically, before determining the target image corresponding to each of the aforementioned facades from the aforementioned oblique photographic images based on the facade normal vectors corresponding to each facade in the aforementioned structured 3D model and the image normal vectors corresponding to each of the aforementioned oblique photographic images, the aforementioned method further includes: Spatial pose restoration is performed on the above oblique photographic images; Based on the rotation matrix obtained after spatial pose recovery, the image normal vector corresponding to the above oblique photogrammetric image is determined.

[0038] In this embodiment, the spatial pose of the oblique photographic image is first restored, and then the image normal vector corresponding to the image is calculated using a rotation matrix. Furthermore, to ensure that the image normal vector always points to the center of the building and thus does not affect the subsequent selection of target images, this application introduces the coordinates of the building center to perform a direction consistency check, thereby ensuring that the photographic direction of the image always points to the geometric center of the building.

[0039] Specifically, calculate the vector from the center point of the photograph to the geometric center point of the building. : ; in, The coordinates of the geometric center point of the building are determined based on the structured three-dimensional model of the building. These are the coordinates of the center point of the oblique photographic image.

[0040] Define a front vector in the camera coordinate system Where T represents the transpose of a vector. Calculate the image normal vector. initial value ,in, . This is the third column of the rotation matrix. Calculate the dot product. If the calculated dot product d is greater than or equal to 0, it means that the angle between the two vectors is less than or equal to 90°, and their directions are basically the same. Otherwise, if the calculated dot product d is less than 0, it means the angle between the two vectors is greater than 90° and their directions are basically opposite. The final This serves as the final calculated image normal vector.

[0041] Furthermore, before determining the target image corresponding to each of the aforementioned facades from the aforementioned oblique photographic images based on the facade normal vectors corresponding to each facade in the aforementioned structured 3D model and the image normal vectors corresponding to each of the aforementioned oblique photographic images, the aforementioned method further includes: For each of the above facades, the facade normal vector is determined based on the face normal vector and vertex normal vector of the triangular mesh corresponding to the facade.

[0042] For the facade normal vector corresponding to the structured 3D model, in this embodiment of the application, the normalized direction vector describing the geometric features of the building is extracted by weighted averaging of the face normal vector of the triangular mesh and the vertex normal vector, combined with deduplication processing, and used as the facade normal vector corresponding to the facade.

[0043] Specifically, based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each of the oblique photographic images, the target images corresponding to each facade are determined from the oblique photographic images, including: Based on the vertex coordinates of the above structured 3D model, determine the geometric center point of the building; Obtain the center point of each of the above-mentioned oblique photographic images; For each facade in the above structured 3D model, the plane equation of the facade is constructed according to the geometric boundary corresponding to the facade. The intersection point of the target line of sight and the plane where the facade is located is determined according to the plane equation. The oblique photographic image whose intersection point falls within the geometric boundary corresponding to the facade is used as the candidate image corresponding to the facade. The target line of sight is the line of sight from the center point of the above oblique photographic image to the geometric center point of the building. Based on the facade normal vector corresponding to the above facade and the image normal vector of the candidate image corresponding to the above facade, the target image corresponding to the above facade is determined from the candidate image corresponding to the above facade.

[0044] The process of determining the target image corresponding to the facade from the candidate images corresponding to the facade, based on the facade normal vector and the image normal vector of the candidate images corresponding to the facade, includes: Calculate the angle between the above facade normal vector and the above image normal vector; The candidate image with the smallest included angle among the candidate images corresponding to the above facade is taken as the target image corresponding to the above facade.

[0045] In this embodiment of the application, for the selection of target images, in order to minimize perspective distortion, the angle between the aforementioned facade normal vector and the aforementioned image normal vector is... The smallest image is the target image corresponding to this facade. Specifically, let the facade normal vector be denoted as... The image normal vector (i.e., the photographic direction vector) is Its spatial angle The calculation formula is: ; in, Representative to and Perform vector dot product operation.

[0046] For standard cubic buildings, parallel facades may exist, which can lead to duplicate normal vectors, resulting in duplicate or missing target images. To further address this issue, this application employs a precise filtering method based on facade boundaries.

[0047] Specifically, in the embodiments of this application, the geometric center point of the building Calculated based on the following formula: ; in, This represents the coordinates of the i-th vertex in the structured 3D model, where n is the total number of vertices. The geometric center point of the building described above characterizes the overall spatial position of the building.

[0048] For each actual elevation j in the structured 3D model, the plane equation corresponding to elevation j is directly constructed based on its geometric boundaries. : ; Specifically, the above plane equations It can be used to characterize the plane to which elevation j belongs. Wherein, Let be a variable representing the coordinates of the point; This represents the coordinates of any vertex on elevation j; This represents the unit normal vector of facade j (i.e., the facade normal vector corresponding to facade j). It should be noted that, in this embodiment, the facade normal vector... In fact, it also represents the unit normal vector corresponding to the facade. and The meanings are similar, the only difference being whether it refers to a specific facade, i.e. This represents the facade normal vector corresponding to a general facade (i.e., without specifying a particular facade). The vector represents the facade normal vector corresponding to the specific facade j. The meanings, calculation processes, and usage of both can be used as a reference, and will not be elaborated here.

[0049] It should be further noted that for the i-th oblique photographic image, its photographic center point is denoted as... In the embodiments of this application, It also represents the center point of the oblique photographic image. and The meanings are similar, the only difference being whether it specifically refers to a particular oblique photographic image, i.e. This represents the center point of a general oblique photographic image (without specifying which oblique photographic image it refers to). The term represents the center point of the i-th oblique photographic image. The meanings, calculation processes, and usage of these terms can be used as a reference for each other, and will not be elaborated upon here.

[0050] For the center point of the i-th oblique photographic image Target line of sight Intersection with the plane to which elevation j belongs The position can be represented as: ; Where t represents the slope of the equation. The equation is valid if and only if t satisfies... Intersection When the image falls within the polygonal boundary corresponding to facade j, the center point of the photograph is... The corresponding oblique photographic image is included in the set of candidate images corresponding to facade j: ; in, This represents the set of candidate images corresponding to facade j. Represents the i-th candidate image; This represents the corresponding phase plane. The formula above indicates that when the line intersects the plane... When it appears in j, the corresponding image Added to the set as candidate images.

[0051] For each facade, the target image is searched only among its corresponding candidate images. Therefore, the target image corresponding to the j-th facade is... It can be represented as: ; in, represent and The included angle.

[0052] Figure 3 This is a schematic diagram of a target image filtering process provided in an embodiment of this application, such as... Figure 3 As shown in this embodiment, spatial pose recovery is performed on the oblique photographic image to obtain the corresponding image normal vector. Structured facade analysis is performed on the structured 3D model to obtain the facade normal vector. Then, based on the image normal vector and the facade normal vector, a screening based on the facade boundary is performed to finally determine the target image corresponding to the facade, thereby supplementing semantic information based on the target image.

[0053] In this embodiment of the application, the semantic information includes component labels corresponding to various building components; the above-projection of the semantic information back into the structured 3D model to obtain the semantic 3D model corresponding to the building includes: Based on the positional correspondence between the target image and the structured 3D model, the component labels are back-projected into the structured 3D model to obtain a semantic 3D model labeled with component labels.

[0054] Figure 4 This is a schematic diagram of a semantic mapping process provided in an embodiment of this application, such as... Figure 4 As shown in this embodiment, for the target images corresponding to each selected facade, a deep learning method is used for semantic segmentation, and then the facade layout is regularized on the semantic segmentation results to obtain regularized semantic information. Finally, the regularized semantic information is back-projected onto a structured 3D model to construct a semantic 3D model of the building with complete semantic annotation.

[0055] This application employs deep learning methods to achieve accurate semantic segmentation of building facade elements. First, an improved deep learning segmentation network is designed and trained using publicly available building facade annotation datasets. For example, the deep learning segmentation network can adopt FastSCNN or DeepLabv3+ architectures, and its performance is optimized through data augmentation, transfer learning, and other techniques. The trained model can perform end-to-end semantic segmentation on input building facade images, outputting pixel-level label maps containing various building components (such as windows, doors, decorative components, etc.). A pixel-level label map refers to a map with semantic labels for each component; a single pixel-level label map includes semantic information for multiple components.

[0056] In this embodiment, facade layout rules are pre-set for regularization processing. Architectural geometric information automatically extracted from images often suffers from issues such as positional offsets, rough edges, and inconsistent dimensions. Therefore, regularization processing is necessary to ensure that the distribution of building components conforms to the structural rules of the building. In this embodiment, firstly, the predicted mask of the target component is extracted from semantic segmentation to generate the corresponding binary image. Then, morphological optimization is performed on the output binary image, eliminating noise through erosion followed by dilation. Next, connected component analysis is used to accurately locate each window, recording its position and size. A dynamic threshold grouping algorithm is employed, automatically calculating reasonable row height and column width thresholds by analyzing the relative distance between the center points of all building components, completing precise grouping, automatically identifying building components in the same row / column, and adjusting their center coordinates to a unified axis while maintaining the original spatial distribution pattern.

[0057] After regularization, pixel-level label images yield semantic images containing semantic information, such as component labels corresponding to various building components. These regularized semantic images are then back-projected onto the elevation of the corresponding structured 3D model of the building using the collinearity equation in photogrammetry, resulting in a model labeled with semantic information—the semantic 3D model of the building.

[0058] Thus, the construction of semantic 3D models of buildings can be achieved without relying on manual modeling or intervention. Instead, a structured 3D model is directly constructed based on oblique photogrammetry images, and the target image corresponding to the facade in the structured 3D model is automatically determined, achieving a match between the facade and the target image. Then, semantic segmentation is automatically performed on the target image to obtain semantic information, which is then projected into the structured 3D model through backprojection, ultimately yielding the semantic 3D model of the building. This allows for the automatic construction of semantic 3D models of buildings based on oblique photogrammetry images, eliminating the need for manual processing and significantly improving the efficiency of semantic 3D model construction.

[0059] Specifically, in this embodiment, firstly, multi-angle oblique photographic images of the building are acquired. A sparse point cloud is generated using a structure-of-motion (SfM) algorithm. Then, a dense point cloud is obtained using multi-view stereo matching technology. Finally, the PolyFit algorithm is used for structured geometric reconstruction to obtain a structured 3D model of the building. Based on this, the best facade image (i.e., the target image corresponding to each facade) is selected through optimal image matching. Automated semantic segmentation is performed using a deep learning algorithm, and standardized semantic annotation results are obtained by combining facade layout regularization processing. Finally, the regularized semantic information is back-projected onto the structured 3D model, achieving integrated fusion of geometric structure and semantic information, thereby constructing a semantic 3D model with complete semantic annotation, enabling efficient and convenient creation of a semantic 3D model of the building. Regarding the selection of the optimal facade image (i.e., the target image) from the multi-view images, this application defines the image with the smallest angle between the image normal vector and the facade normal vector of the structured facade as the optimal image for that facade. Firstly, an image normal vector calculation model based on camera extrinsic parameters recovered using SfM is established, and building center coordinate constraints are introduced to ensure directional consistency. Secondly, in the target image selection stage, to address the matching ambiguity caused by parallel facades, a precise matching algorithm based on the geometric features of facade boundaries is proposed, ensuring the uniqueness of the matched image for each facade. These measures fully guarantee the uniqueness and accuracy of target image selection.

[0060] This application proposes a method for constructing semantic 3D models of buildings by combining 3D structured models with regularized semantic information. This method maximizes the information value of UAV-collected data through a target image matching mechanism and significantly reduces manual intervention by introducing a deep learning network. Furthermore, this application significantly improves the engineering applicability of semantic annotation through facade layout regularization, accurately locates facade elements through morphological processing and connected component analysis, and achieves automatic row and column alignment using a dynamic threshold grouping algorithm. This method only requires the semantic segmentation results as input to automatically output a regularized facade layout conforming to the building module. Finally, through an innovative automated modeling process, the cost of semantic modeling is significantly reduced, achieving efficient, convenient, and scalable semantic 3D model construction.

[0061] like Figure 5 As shown in the figure, corresponding to the semantic 3D model construction method of the above-mentioned building, this application embodiment also provides a semantic 3D model construction system for buildings, the semantic 3D model construction system for buildings including: The 3D reconstruction module 510 is used to acquire oblique photographic images corresponding to the building, and perform structured 3D reconstruction based on the oblique photographic images to obtain a structured 3D model corresponding to the building. The facade analysis module 520 is used to perform structured facade analysis on the above-mentioned structured 3D model to determine each facade in the above-mentioned structured 3D model. The target image determination module 530 is used to determine the target image corresponding to each of the above-mentioned facades from the above-mentioned oblique photographic images based on the facade normal vectors corresponding to each facade in the above-mentioned structured 3D model and the image normal vectors corresponding to each of the above-mentioned oblique photographic images. The semantic segmentation module 540 is used to perform semantic segmentation on the target image to obtain the semantic information corresponding to the target image. The semantic 3D model construction module 550 is used to back-project the above semantic information onto the above structured 3D model to obtain the semantic 3D model corresponding to the above building.

[0062] Thus, the construction of semantic 3D models of buildings does not rely on manual modeling or intervention. Instead, it directly constructs structured 3D models based on oblique photogrammetry images and automatically identifies the target images corresponding to the facades in the structured 3D model, achieving a match between the facades and the target images. Then, it automatically performs semantic segmentation on the target images to obtain semantic information, and projects this semantic information into the structured 3D model through backprojection, ultimately obtaining the semantic 3D model of the building. This allows for the automatic construction of semantic 3D models of buildings based on oblique photogrammetry images, eliminating the need for manual processing and significantly improving the efficiency of semantic 3D model construction.

[0063] It should be noted that the specific structure and implementation of the semantic 3D model construction system for the above-mentioned buildings and its various modules or units can be referred to the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0064] It should be noted that the division of the modules in the semantic 3D model construction system for the above-mentioned buildings is not unique and is not intended as a specific limitation.

[0065] Based on the above embodiments, this application also provides a terminal, the principle block diagram of which can be as follows: Figure 6 As shown. The aforementioned terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of any of the aforementioned methods for constructing a semantic 3D model of a building. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0066] Those skilled in the art will understand that Figure 6 The block diagram shown is only a partial structural diagram related to the solution of this application and does not constitute a limitation on the terminal on which the solution of this application is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0067] In one embodiment, a terminal is provided, the terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of any of the semantic three-dimensional model construction methods for buildings provided in the embodiments of this application.

[0068] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the semantic three-dimensional model construction methods for buildings provided in this application.

[0069] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0071] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0072] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0073] In the embodiments provided in this application, it should be understood that the disclosed systems / terminal devices and methods can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0074] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0075] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for constructing a semantic three-dimensional model of a building, characterized in that, The method includes: Oblique photogrammetry images of the building are acquired, and structured 3D reconstruction is performed based on the oblique photogrammetry images to obtain a structured 3D model of the building. The structured 3D model is subjected to structured elevation analysis to determine each elevation in the structured 3D model; Based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image, the target image corresponding to each facade is determined from the oblique photographic image. Semantic segmentation is performed on the target image to obtain the semantic information corresponding to the target image; The semantic information is back-projected into the structured 3D model to obtain the semantic 3D model corresponding to the building.

2. The method for constructing a semantic three-dimensional model of a building according to claim 1, characterized in that, The step of performing structured 3D reconstruction based on the oblique photographic images to obtain a structured 3D model corresponding to the building includes: Based on the oblique photographic image, the sparse point cloud corresponding to the building is obtained by the structure-recovery-motion algorithm. Based on the sparse point cloud, the dense point cloud corresponding to the building is obtained by multi-view stereo matching; The structured 3D model of the building is obtained by performing structured reconstruction based on the dense point cloud.

3. The method for constructing a semantic three-dimensional model of a building according to claim 1, characterized in that, Before determining the target image corresponding to each facade from the oblique photographic images based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image, the method further includes: Spatial pose recovery is performed on the oblique photographic images; Based on the rotation matrix obtained after spatial pose recovery, the image normal vector corresponding to the oblique photographic image is determined.

4. The method for constructing a semantic three-dimensional model of a building according to claim 1, characterized in that, Before determining the target image corresponding to each facade from the oblique photographic images based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image, the method further includes: For each of the aforementioned facades, the facade normal vector is determined based on the face normal vector and vertex normal vector of the triangular mesh corresponding to the facade.

5. The method for constructing a semantic three-dimensional model of a building according to claim 1, characterized in that, The step of determining the target image corresponding to each facade from the oblique photographic images based on the facade normal vectors corresponding to each facade in the structured 3D model and the image normal vectors corresponding to each oblique photographic image includes: The geometric center point of the building is determined based on the vertex coordinates of the structured 3D model. Obtain the center point of each of the oblique photographic images; For each facade in the structured 3D model, the plane equation of the facade is constructed according to the geometric boundary corresponding to the facade. The intersection point of the target line of sight and the plane where the facade is located is determined according to the plane equation. The oblique photographic image whose intersection point falls within the geometric boundary corresponding to the facade is used as the candidate image corresponding to the facade. The target line of sight is the line of sight from the center point of the oblique photographic image to the geometric center point of the building. Based on the facade normal vector and the image normal vector of the candidate image corresponding to the facade, the target image corresponding to the facade is determined from the candidate image corresponding to the facade.

6. The method for constructing a semantic three-dimensional model of a building according to claim 5, characterized in that, The step of determining the target image corresponding to the facade from the candidate images corresponding to the facade based on the facade normal vector and the image normal vector of the candidate images corresponding to the facade includes: Calculate the angle between the facade normal vector and the image normal vector; The candidate image with the smallest included angle among the candidate images corresponding to the facade is taken as the target image corresponding to the facade.

7. The method for constructing a semantic three-dimensional model of a building according to any one of claims 1 to 6, characterized in that, The semantic information includes component tags corresponding to various building components; The step of back-projecting the semantic information onto the structured 3D model to obtain the semantic 3D model corresponding to the building includes: Based on the positional correspondence between the target image and the structured 3D model, the component labels are back-projected into the structured 3D model to obtain a semantic 3D model labeled with component labels.

8. A semantic 3D model construction system for buildings, characterized in that, The system includes: The 3D reconstruction module is used to acquire oblique photogrammetric images corresponding to buildings, perform structured 3D reconstruction based on the oblique photogrammetric images, and obtain a structured 3D model corresponding to the building. The facade analysis module is used to perform structured facade analysis on the structured 3D model to determine each facade in the structured 3D model; The target image determination module is used to determine the target image corresponding to each facade from the oblique photogrammetry image based on the facade normal vector corresponding to each facade in the structured 3D model and the image normal vector corresponding to each oblique photogrammetry image. A semantic segmentation module is used to perform semantic segmentation on the target image to obtain semantic information corresponding to the target image; The semantic 3D model construction module is used to back-project the semantic information onto the structured 3D model to obtain the semantic 3D model corresponding to the building.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the semantic three-dimensional model construction method for a building as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the semantic three-dimensional model construction method for a building as described in any one of claims 1 to 7.

Images