Building facade modeling method, device and equipment based on adjunct identification and medium

Abstract

The invention provides a building facade modeling method, device and equipment based on adjunct identification and a medium, and relates to the technical field of image processing. The method comprises the steps of obtaining point cloud data of a building facade of a target building; performing semantic segmentation on the point cloud data through a double-flow decoupling network to obtain a host segmentation result and a parasitic segmentation result; wherein the double-flow decoupling network comprises a host mainstream branch, a parasitic flow branch and a hierarchical interaction module, and the hierarchical interaction module connects the host mainstream branch with the parasitic flow branch through a gating mechanism; generating an adjunct mask according to a parasitic segmentation result, and performing adjunct stripping on the point cloud data based on the adjunct mask to obtain a building facade point cloud containing holes; and performing geometric repair on the building facade point cloud, and constructing a building facade model of the target building based on the building facade point cloud after geometric repair. According to the method, high-precision appendage stripping and modeling can be completed, and the data preprocessing cost of old community digital reconstruction is reduced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, equipment and medium for building facade modeling based on appendage recognition. Background Technology

[0002] The facades of older residential buildings commonly suffer from high-density obstruction by attachments, including air conditioner units, security windows, and illegally installed cables. These attachments not only geometrically cover the building's structure, such as walls, window frames, and moldings, but also cause physical damage to the original walls during installation. Therefore, the digital reconstruction of old building facades faces a core challenge: how to accurately identify and remove various attachments from the walls using point cloud data while preserving the building's structural integrity.

[0003] Existing deep learning-based point cloud semantic segmentation methods, such as PointNet++ and RandLA-Net, primarily rely on point-by-point feature extraction or random sampling strategies. For targets with similar geometric shapes to wall surfaces, such as air conditioner outdoor units and security grille brackets, these methods struggle to distinguish boundaries due to a lack of sufficient modeling of local topological relationships. Even more problematic is that small attachments like cables and security grille rebar occupy only a tiny fraction of points in the point cloud, easily discarded during random sampling, resulting in extremely low recall. Using a general architecture with multi-classification networks treats all categories equally, failing to consider the significant imbalance in sample numbers between host classes (like walls) and parasitic classes (like cables). This leads to the network ignoring small targets, resulting in coarse segmentation edges that erode the building structure when stripping attachments, leaving data gaps in the point cloud caused by occlusion. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for building facade modeling based on appendage identification, to solve the problem of rough segmentation edges and erosion of the building body when removing appendages during the building facade modeling process.

[0005] In a first aspect, embodiments of the present invention provide a building facade modeling method based on appendage identification, comprising: Obtain point cloud data of the building facade of the target building; Semantic segmentation of point cloud data is performed using a two-stream decoupling network to obtain host segmentation results and parasitic segmentation results. The two-stream decoupling network includes a host stream branch, a parasitic stream branch, and a hierarchical interaction module. The hierarchical interaction module connects the host stream branch and the parasitic stream branch through a gating mechanism. An appendage mask is generated based on the parasitic segmentation results, and appendages are stripped from the point cloud data based on the appendage mask to obtain a building facade point cloud containing holes. Geometric repair is performed on the point cloud of the building facade, and a building facade model of the target building is constructed based on the geometrically repaired point cloud of the building facade.

[0006] In one possible implementation, both the host stream branch and the parasitic stream branch are sparse convolutional U-Net architectures, including encoders, decoders, and skip connections; the hierarchical interaction module includes a first gated convolutional unit and a second gated convolutional unit. The input of the first gated convolutional unit is connected to the output of the intermediate layer of the host stream branch decoder. The output generates a parasitic suppression mask. The parasitic suppression mask is multiplied element-wise with the feature map of the corresponding layer in the parasitic stream branch decoder and then fed into the next layer of the parasitic stream branch decoder. The input of the second gated convolutional unit is connected to the output of the intermediate layer of the parasitic stream branch decoder. The output generates an edge enhancement mask. The edge enhancement mask is added element-wise with the feature map of the corresponding layer in the host stream branch decoder and then fed into the next layer of the host stream branch decoder.

[0007] In one possible implementation, the host flow branch uses a planar consistency loss function, while the parasitic flow branch uses a class-balanced cross-entropy loss function and a focus loss function.

[0008] In one possible implementation, acquiring the point cloud data of the building facade of the target building includes: Acquire image sequences and point cloud data of the building facade of the target building; Based on depth distribution prediction, the two-dimensional image features of the image sequence are enhanced into three-dimensional view frustum features; By using a cross-attention mechanism, the features of the 3D view frustum are fused with the geometric features of the point cloud data to obtain multimodal fusion features; Accordingly, semantic segmentation of point cloud data is performed using a two-stream decoupling network, yielding host segmentation results and parasitic segmentation results, including: Based on multimodal fusion features, semantic segmentation of point cloud data is performed through a two-stream decoupling network to obtain host segmentation results and parasitic segmentation results.

[0009] In one possible implementation, geometric restoration of the building facade point cloud includes: Detecting void areas in point clouds on building facades; For symmetrical void regions, the global symmetry plane and local symmetry center of the building facade are detected. The features of the intact region are mapped to the void region through symmetry transformation. The void region is then repaired using a generative adversarial network combined with a symmetry consistency loss function. For void regions that lack symmetry, the contextual features of the void region's edge are extracted, similar components are retrieved from a pre-built building component library, and a deformation network is used to non-rigidly deform the retrieved components to fit the void boundary, resulting in point cloud data after void repair.

[0010] In one possible implementation, after obtaining the point cloud data after hole repair, the following is also included: The point cloud data after hole repair is transformed into an implicit representation of the symbolic distance field. A continuous symbolic distance function is learned using a neural network, and a watertight mesh model is generated by extracting the zero isosurface. Using the texture of the walls surrounding the repair area as a guide, a conditional diffusion model is used to generate texture maps of the hole areas, and Poisson fusion is used to eliminate lighting differences, resulting in point cloud data after texture completion.

[0011] In one possible implementation, a building facade model of the target building is constructed based on the geometrically repaired building facade point cloud, including: Instance segmentation is performed on the geometrically repaired point cloud data to separate independent building components; For each separated building component, the minimum volume bounding box algorithm based on the eigenvalue decomposition of the covariance matrix is ​​used to fit the geometric parameters to obtain the fitted parameterized object. The fitted parametric object is mapped to a standard entity of the industrial base class, and the building facade model of the target building is output.

[0012] Secondly, embodiments of the present invention provide a building facade modeling device based on appendage recognition, comprising: The acquisition module is used to acquire point cloud data of the building facade of the target building; The segmentation module is used to perform semantic segmentation on point cloud data through a two-stream decoupling network to obtain host segmentation results and parasitic segmentation results. The two-stream decoupling network includes a host stream branch, a parasitic stream branch, and a hierarchical interaction module. The hierarchical interaction module connects the host stream branch and the parasitic stream branch through a gating mechanism. The stripping module is used to generate an appendage mask based on the parasitic segmentation results, and strip the appendages from the point cloud data based on the appendage mask to obtain a building facade point cloud containing holes. The building module is used to perform geometric repair on the point cloud of the building facade and to build the building facade model of the target building based on the geometrically repaired point cloud of the building facade.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation thereof.

[0015] The building facade modeling method, apparatus, equipment, and medium based on appendage identification provided in this invention employ a dual-stream decoupled network to process the building host layer and the appendage parasitic layer separately. The host stream focuses on low-frequency, large-scale structures such as walls, while the parasitic stream focuses on high-frequency, small targets such as cables and supports. The two streams interact through a gating mechanism to mutually suppress false alarms. This architecture fundamentally solves the class imbalance problem in multi-class networks and improves the recognition rate of difficult-to-distinguish small targets such as cables. An appendage mask is generated based on the parasitic segmentation results and directly removed, avoiding the problem of rough segmentation edges eroding the building body in traditional methods. After stripping, a point cloud containing only the building body is obtained, and gaps are filled through geometric repair. The final output building facade model is clean, complete, and conforms to the building structure logic. This method can complete high-precision appendage stripping and modeling without manual intervention, significantly reducing the data preprocessing cost in the digital transformation of old residential areas and providing reliable technical support for the construction of building information models in large-scale urban renewal. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the implementation of the building facade modeling method based on appendage recognition provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the structure of the two-stream decoupling segmentation network provided in an embodiment of the present invention; Figure 3 This is an overall flowchart of building facade modeling provided in the embodiments of the present invention; Figure 4 This is a logic flowchart of cavity repair provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the frustum perception feature fusion network provided in an embodiment of the present invention; Figure 6 This is a comparison chart of the effects of minimum volume bounding box fitting and BIM object generation provided in the embodiments of the present invention; Figure 7 This is a schematic diagram of the building facade modeling device based on appendage recognition provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0018] See Figure 1 The document illustrates a flowchart of the building facade modeling method based on appendage recognition provided in an embodiment of the present invention, detailed below: Step 101: Obtain point cloud data of the building facade of the target building.

[0019] In this embodiment, a ground-based laser scanner, a mobile measurement system, or a lidar mounted on a drone can be used to scan the target building and directly acquire 3D point cloud data of its facade. The point cloud data records the spatial coordinates of each sampling point, and some devices can also simultaneously acquire reflection intensity or color information. To ensure the accuracy of subsequent segmentation and modeling, the point cloud density must be no less than 500 points per square meter during acquisition to fully preserve the geometric details of walls, window frames, and various accessories. The acquired raw point cloud data usually contains noise points and outliers. Preprocessing operations such as statistical filtering or radius filtering can be performed to remove isolated noise points and obtain clean point cloud data, which can then be used as input for subsequent semantic segmentation.

[0020] Step 102: Semantic segmentation of point cloud data is performed through a two-stream decoupling network to obtain host segmentation results and parasitic segmentation results; wherein, the two-stream decoupling network includes a host stream branch, a parasitic stream branch and a hierarchical interaction module, and the hierarchical interaction module connects the host stream branch and the parasitic stream branch through a gating mechanism.

[0021] In this embodiment, the dual-stream decoupling network includes a host flow branch, a parasitic flow branch, and a hierarchical interaction module. The hierarchical interaction module connects the host flow branch and the parasitic flow branch through a gating mechanism. In the building facade scenario, the building itself, such as walls, windows, and doors, constitutes the host layer, while unstructured attachments such as air conditioner outdoor units, security grilles, cables, and brackets constitute the parasitic layer. These two types of objects differ significantly in geometric scale and morphological characteristics: the host layer typically exhibits a large-scale, low-frequency, smooth planar or near-planar structure, while the parasitic layer is mostly a small-scale, high-frequency, discrete, or grid-like structure. Traditional multi-class segmentation networks treat all categories equally, easily ignoring the few and weakly characterized parasitic layer objects. The dual-stream decoupling network proposed in this invention addresses this characteristic by designing two parallel and interactive processing paths. The host flow branch focuses on extracting low-frequency features of the building itself, and its network structure tends to output smooth, continuous segmented regions; the parasitic flow branch focuses on extracting high-frequency features of attachments, and is more sensitive to edges, corners, and small structures. The hierarchical interaction module uses a gating mechanism to allow the two branches to exchange information at multiple levels of feature extraction: the prediction results of the host stream can help suppress false detections that may occur in the parasitic stream, such as misidentifying wall shadows as cables; the prediction results of the parasitic stream can also enhance the host stream's fine localization of building edges, such as distinguishing the boundary between window frames and security grilles. After such dual-stream interactive segmentation, the semantic labels of each point cloud point are finally output, where the host segmentation result includes point sets of building body categories such as walls, windows, and doors, and the parasitic segmentation result includes point sets of accessory categories such as air conditioners, security grilles, and cables. Figure 2The structure of the two-stream decoupled segmentation network is illustrated, clearly showing the host stream branch, the parasitic stream branch, and the hierarchical interaction modules between them.

[0022] Step 103: Generate an appendage mask based on the parasitic segmentation results, and strip appendages from the point cloud data based on the appendage mask to obtain a building facade point cloud containing holes.

[0023] In this embodiment, the appendage mask is a binary marker array of the same size as the point cloud, with points marked as true corresponding to appendage points in the parasitic segmentation result. The peeling operation removes all points marked as appendages by the mask from the original point cloud, leaving only the building body point cloud. Because the appendages obscure the walls or windows behind them during long-term attachment, the point cloud data that was originally covered in these areas no longer exists after peeling, thus leaving geometric holes in the building body point cloud. The shape and location of these holes correspond one-to-one with the removed appendages, such as the wall holes left after removing an air conditioner outdoor unit, or the window areas exposed but lacking point cloud data after removing security grilles. Figure 3 This demonstrates the position of the stripping operation in the overall process from a macro perspective. Although the point cloud data obtained after stripping is clean, it is not yet complete and needs further repair.

[0024] Step 104: Perform geometric repair on the building facade point cloud, and construct the building facade model of the target building based on the geometrically repaired building facade point cloud.

[0025] In this embodiment, the goal of geometric restoration is to logically and correctly fill in the missing geometric shapes based on the intact building structural information surrounding the void, ensuring that the restored walls, window frames, and other areas blend naturally with the surrounding environment. Restoration methods can utilize structural priors such as the symmetry and repetition of the building facade, or draw inspiration from typical forms of similar building components. After geometric restoration, surface smoothing and texturing are required to eliminate seam marks, ultimately generating a complete, continuous, and realistic 3D building facade model. This model can be directly used for engineering quantity calculations, landscape assessments, digital archiving, or subsequent building information modeling. Figure 4 A schematic diagram of the logic flow for cavity repair is given, in which different repair strategies are selected based on characteristics such as whether the cavity has symmetry, and finally the repaired complete model is output.

[0026] This invention employs a dual-stream decoupled network to process the building host layer and the parasitic appendage layer separately. The host stream focuses on low-frequency, large-scale structures such as walls, while the parasitic stream focuses on high-frequency, small targets such as cables and supports. The two streams interact through a gating mechanism to mutually suppress false alarms. This architecture fundamentally solves the class imbalance problem in multi-classification networks, improving the recognition rate of difficult-to-distinguish small targets such as cables. An appendage mask is generated based on the parasitic segmentation results and directly removed, avoiding the problem of rough segmentation edges eroding the building body in traditional methods. After stripping, a point cloud containing only the building body is obtained, and gaps are filled through geometric repair. The final output building facade model is clean, complete, and conforms to the building's structural logic. This method can achieve high-precision appendage stripping and modeling without manual intervention, significantly reducing the data preprocessing cost in the digital transformation of old residential areas and providing reliable technical support for building information model construction in large-scale urban renewal.

[0027] In one possible implementation, both the host stream branch and the parasitic stream branch are sparse convolutional U-Net architectures, including encoders, decoders, and skip connections; the hierarchical interaction module includes a first gated convolutional unit and a second gated convolutional unit. The input of the first gated convolutional unit is connected to the output of the intermediate layer of the host stream branch decoder. The output generates a parasitic suppression mask. The parasitic suppression mask is multiplied element-wise with the feature map of the corresponding layer in the parasitic stream branch decoder and then fed into the next layer of the parasitic stream branch decoder. The input of the second gated convolutional unit is connected to the output of the intermediate layer of the parasitic stream branch decoder. The output generates an edge enhancement mask. The edge enhancement mask is added element-wise with the feature map of the corresponding layer in the host stream branch decoder and then fed into the next layer of the host stream branch decoder.

[0028] In this embodiment, as Figure 2As shown, both the host flow branch and the parasitic flow branch adopt a sparse convolutional U-Net structure. Since point cloud data is sparse in 3D space, traditional dense convolution introduces a large amount of invalid computation, while sparse convolution only performs convolution operations on non-empty voxels, significantly improving computational efficiency. Each branch's U-Net contains an encoder, a decoder, and skip connections connecting corresponding layers. The encoder progressively downsamples to extract high-level semantic features, the decoder progressively upsamples to restore spatial resolution, and the skip connections pass the fine-grained geometric information retained by the encoder to the decoder, helping to accurately locate segmentation boundaries. A hierarchical interaction module is embedded between the decoders of the two branches, containing a first gated convolutional unit and a second gated convolutional unit. The input of the first gated convolutional unit is connected to the intermediate layer output of the host flow branch decoder, which contains strong response regions of building bodies such as walls and windows. The first gated convolutional unit performs convolution and sigmoid activation on this feature map to generate a parasitic suppression mask with a value between 0 and 1, representing the confidence that each spatial location belongs to a building body. By element-wise multiplying this mask with the feature map of the corresponding layer in the parasitic flow branch decoder, spurious responses that may occur inside the wall surface in the parasitic flow can be suppressed, such as misidentifying wall stains as cables. The multiplied feature map is then fed into the next layer of the parasitic flow branch decoder for further processing. Similarly, the input of the second gated convolutional unit is connected to the intermediate layer output of the parasitic flow branch decoder, which is sensitive to the edges and small structures of accessories such as cables and brackets. The second gated convolutional unit generates an edge enhancement mask, which is element-wise added to the feature map of the corresponding layer in the host flow branch decoder. This enhances the host flow's ability to locate building edges, such as helping to distinguish the boundary between window frames and window glass. The added feature map is then fed into the next layer of the host flow branch decoder. Through this bidirectional gated feature interaction, the two branches correct each other during the learning process, effectively improving segmentation accuracy, especially improving the recall rate of small objects in the parasitic flow and the edge sharpness in the host flow. Figure 2 The hierarchical interaction module in the document details the connection method and data flow of the first gated convolutional unit and the second gated convolutional unit.

[0029] In one possible implementation, the host flow branch uses a planar consistency loss function, while the parasitic flow branch uses a class-balanced cross-entropy loss function and a focus loss function.

[0030] In this embodiment, when training the two-stream decoupling network, differentiated loss functions need to be designed for the two branches to enhance their respective strengths in feature learning. For the host stream branch, the goal is to segment architectural entities with distinct planar geometric characteristics, such as walls and window glass. Therefore, this invention introduces a planar consistency loss function. Specifically, for the local neighborhood point set predicted by the network as a wall... Calculate the sum of the residuals for fitting an optimal plane to this point set: The smaller the residual, the closer the point set is to the ideal plane. This loss function encourages the host flow branch to output geometrically smooth segmented regions, suppressing local noise or fluctuations, resulting in more regular wall point clouds that can effectively fill in the small noise in LiDAR sampling.

[0031] For the parasitic flow branch, the goal is to segment sparse and unevenly distributed accessories such as air conditioners, cables, and brackets. Conventional cross-entropy loss functions tend to ignore minority classes when faced with extreme imbalances in the point cloud, where wall points may account for 99% while cable points account for less than 1%. Therefore, this invention employs a combination of class-balanced cross-entropy loss and focus loss functions. Class-balanced cross-entropy... Assign a weight to each category This weight is inversely proportional to the square root of the frequency of the class in the training set. Classes with lower frequencies, such as cables, receive higher loss weights, thus forcing the network to focus on these minority class samples during training. (Focus Loss) Building upon this foundation, the focus shifts to difficult-to-classify samples. The core idea is to reduce the loss contribution of easily classified samples and amplify the loss contribution of difficult-to-classify samples. When a sample is easily classified correctly by the network, its prediction probability is close to 1, and the modulation factor in the focus loss approaches 0, thus suppressing the loss for that sample. Conversely, when a sample is difficult to classify, such as a thin cable close to a wall, its prediction probability is low, the modulation factor is large, and the loss is amplified, allowing the network to focus more on learning these difficult samples. By weighted summing of class-balanced cross-entropy and focus loss, the parasitic flow branch effectively improves the ability to identify small appendages under extremely imbalanced data distributions. During training, the loss functions of the host flow branch and the parasitic flow branch are jointly optimized, sharing some of the underlying feature extraction network, ensuring the overall network maintains smooth building segmentation while also sensitively capturing weak signals from the parasitic layer.

[0032] During the stripping operation, an accessory mask is generated based on the segmentation results. .

[0033]

[0034]

[0035] in, The raw point cloud data, For the point cloud data corresponding to the appendages, The point cloud data that needs to be retained. After stripping, in The collection of voids formed by occlusion is detected above. And perform geometric repairs.

[0036] In one possible implementation, acquiring the point cloud data of the building facade of the target building includes: Acquire image sequences and point cloud data of the building facade of the target building; Based on depth distribution prediction, the two-dimensional image features of the image sequence are enhanced into three-dimensional view frustum features; By using a cross-attention mechanism, the features of the 3D view frustum are fused with the geometric features of the point cloud data to obtain multimodal fusion features; Accordingly, semantic segmentation of point cloud data is performed using a two-stream decoupling network, yielding host segmentation results and parasitic segmentation results, including: Based on multimodal fusion features, semantic segmentation of point cloud data is performed through a two-stream decoupling network to obtain host segmentation results and parasitic segmentation results.

[0037] In this embodiment, acquiring point cloud data also includes a multimodal fusion step, the specific implementation of which is given below. In this embodiment, acquiring the point cloud data of the building facade of the target building does not rely solely on single LiDAR data, but rather fuses multimodal information from the image sequence to improve the accuracy of semantic segmentation. Specifically, firstly, image sequences and raw point cloud data of the building facade of the target building are acquired. The image sequence is acquired by a high-resolution panoramic camera with an overlap rate of no less than 80%; the raw point cloud data is acquired by LiDAR with a density of no less than 500 points per square meter. The two are spatiotemporally registered using continuous time trajectory optimization to ensure that each point cloud point can find its precise corresponding pixel in the image. The specific registration process is as follows: Let a point in the LiDAR coordinate system be... The point in the camera coordinate system is Image pixel coordinates are .

[0038] The rigid transformation relationship is:

[0039] in For rotation matrix, It is a translation vector.

[0040] The projection relationship is as follows:

[0041] This invention employs an automatic calibration algorithm based on edge features, utilizing the correspondence between building outlines in point clouds and images to optimize... The parameters are used to minimize the reprojection error.

[0042] Because the data acquisition vehicle is in motion, each frame of the LiDAR point cloud is not an instantaneous exposure, but rather acquired gradually over approximately 100ms as the laser rotates. Directly matching this image with an instantaneous exposure (Exposure Time < 10ms) would result in motion distortion.

[0043] This invention introduces continuous time trajectory Each laser point Based on its precise launch time Perform exercise compensation:

[0044] This step ensures pixel-level alignment between point cloud edges and image texture edges under high-speed motion, laying the foundation for subsequent feature fusion.

[0045] Then, based on depth distribution prediction, the two-dimensional image features of the image sequence are enhanced into three-dimensional view frustum features. Input image The pre-trained Swin-Transformer-Base is used as the backbone network. Swin-Transformer utilizes a window-based self-attention mechanism, which can capture long-range dependencies.

[0046] Output feature map ,in This represents the number of feature channels.

[0047] To assign 2D features to 3D spatial locations, the depth of each pixel needs to be estimated. However, instead of regressing a single depth value, the network predicts a depth distribution. The network outputs a depth distribution tensor. ,in It is the number of discrete depth intervals (e.g., 40 intervals, ranging from 4m to 50m).

[0048]

[0049] Constructing the feature voxels of the view cone :

[0050] This is equivalent to applying 2D image features along the viewing direction. This step extends the semantic information of the image from the pixel plane to the entire view frustum space, making the image texture no longer limited by foreground occlusion. However, based on the predicted depth probability, the depth location feature value of the object is large, while the atmospheric location feature value is small.

[0051] Next, a cross-attention mechanism is used to fuse the 3D view frustum features with the geometric features of the point cloud data to obtain multimodal fusion features. For each point in the LiDAR point cloud... First, voxelization is used to obtain geometric features. Using the projected coordinates of this point and depth value ,exist The local frustum feature region corresponding to the index. Define the cross-attention operation: Query (Q): — Geometric fingerprints of 3D points.

[0052] Key (K): — Local frustum features.

[0053] Value (V): — Local frustum features.

[0054] Perform multi-head cross-attention computation so that each point cloud point can adaptively aggregate texture semantic information in the image based on its own geometric position. Figure 5 The structure of the frustum perception feature fusion network is shown on the left, illustrating the process of converting 2D features into frustum features, while the right side shows the cross-attention fusion module. After the multimodal fusion, each point cloud point obtains a fused feature vector containing both accurate 3D geometric information and rich texture semantics. Subsequently, based on this multimodal fused feature, the point cloud data is semantically segmented using the two-stream decoupling network described in step 102. Because the fused feature already embeds high-resolution texture information from the image, the network can better distinguish objects with similar geometric features but different semantics, such as differentiating the side of an air conditioner that is close to the wall from the actual wall surface. Simultaneously, the continuous texture features of the image help the network identify sparse cable regions in the point cloud, significantly improving the segmentation accuracy of small appendages. This multimodal fusion scheme fully utilizes the geometric accuracy of the LiDAR and the texture density of the optical image, providing the two-stream decoupling network with more comprehensive input features.

[0055] In one possible implementation, geometric restoration of the building facade point cloud includes: Detecting void areas in point clouds on building facades; For symmetrical void regions, the global symmetry plane and local symmetry center of the building facade are detected. The features of the intact region are mapped to the void region through symmetry transformation. The void region is then repaired using a generative adversarial network combined with a symmetry consistency loss function. For void regions that lack symmetry, the contextual features of the void region's edge are extracted, similar components are retrieved from a pre-built building component library, and a deformation network is used to non-rigidly deform the retrieved components to fit the void boundary, resulting in point cloud data after void repair.

[0056] In this embodiment, the repair process includes a generative model based on a generative adversarial network, which introduces a symmetric consistency loss function. The specific implementation is given below. First, global symmetry axes are extracted, and the point cloud is projected onto the XY plane to obtain the projected point cloud. The problem of symmetry axis detection is transformed into an optimization problem:

[0057] Find the optimal symmetry plane using gradient-based optimization or grid search.

[0058] Then, local symmetry detection is performed: for each void, its autocorrelation function is calculated to detect whether there is a repeating cycle (such as a window every 3 meters).

[0059] When symmetry is detected in void regions, such as rows of windows or centrally symmetrical doorways, the global symmetry plane or local symmetry center of the building facade is first accurately calculated using principal component analysis and random sample consensus algorithms. The point cloud of the intact region is then mapped to the void region through a symmetry transformation, resulting in a noisy symmetric prior point cloud. The generator G of the generative adversarial network employs an encoder-decoder structure, with the encoder using PointNet++ to extract a portion of the point cloud. Symmetric Prior Point Cloud The decoder uses a folded network architecture to fold a 2D mesh into a 3D shape, outputting a repaired point cloud. The discriminator D determines whether the input point cloud is a real, intact building point cloud or a generated point cloud. In addition to adversarial loss and reconstruction loss, the generator's loss function also introduces a symmetry consistency loss, calculated as follows: The loss mechanism forces the generated point cloud to maintain geometrical consistency with the symmetrical prior point cloud, thereby ensuring that the repaired area matches the topology of the building's symmetrical areas, such as accurately restoring obscured window frame moldings.

[0060] The entire generative adversarial network is trained on a dataset containing a large number of symmetrical building facades. During training, the generator and discriminator are alternately optimized until the discriminator can no longer distinguish between the repaired area and the truly intact area. This generative repair method based on symmetry priors can logically and correctly restore the fine structure that conforms to architectural principles, compared to traditional hole-filling algorithms. Figure 4 This illustrates the decision-making path for symmetry-aware repair.

[0061] For damaged areas with unique shapes, the specific repair steps are as follows: 1. Feature Library Construction: Collect 1000+ standard architectural components (Classical Elements) and extract their global feature vectors using a PointNet pre-trained network. Build the FAISS index library.

[0062] 2. Query and Retrieval: Extracting the edges of holes Features Search the database for the top 3 similar components.

[0063]

[0064] 3. Non-rigid Registration: The retrieved standard window may not match the actual size. The deformation field is calculated using the Coherent Point Drift (CPD) algorithm or a deep deformation network. This involves filling standard components into the holes while maintaining their topological structure (such as the connection relationship of window frames).

[0065] 4. SDF-based surface fusion: To address the seam issue between the repair patch and the original wall surface, this invention does not directly output point clouds, but instead outputs a symbolic distance field (SDF). An MLP network is then constructed. Training is performed by sampling near the stitching region. The loss function includes: At a known point cloud location, .

[0066] This ensures the mathematical properties of SDF.

[0067] Minimize the second derivative to ensure smooth curvature.

[0068] After training, Marching Cubes is used to extract the zero isosurface to obtain the completed repair mesh.

[0069] In one possible implementation, after obtaining the point cloud data after hole repair, the following is also included: The point cloud data after hole repair is transformed into an implicit representation of the symbolic distance field. A continuous symbolic distance function is learned using a neural network, and a watertight mesh model is generated by extracting the zero isosurface. Using the texture of the walls surrounding the repair area as a guide, a conditional diffusion model is used to generate texture maps of the hole areas, and Poisson fusion is used to eliminate lighting differences, resulting in point cloud data after texture completion.

[0070] In this embodiment, after completing the geometric restoration and extracting the zero isosurface to obtain a watertight mesh model, it is necessary to assign a texture to the restored area that is consistent with the surrounding walls. This embodiment uses a conditional diffusion model to achieve this goal. Specifically, the texture of the surrounding walls is used as a conditional cue, which can be the color distribution, gradient features, or high-level semantic description of several surrounding pixels. The conditional diffusion model gradually adds noise to the real texture image through a forward diffusion process, and then recovers the image from the random noise through a reverse denoising process, where each step of the reverse process is guided by the conditional cue. During the training phase, the diffusion model is trained on a large amount of intact building wall texture data, enabling it to learn the statistical distribution of wall textures. During inference, the noisy image of the hole area is input into the model, and the texture features of the surrounding walls are injected in each denoising step, ultimately generating a texture map that is highly consistent with the tone and material of the surrounding walls. After generation, the Poisson fusion algorithm is used to eliminate the lighting differences and splicing traces between the generated texture and the original texture. Poisson fusion achieves seamless stitching by solving a Poisson equation with Dirichlet boundary conditions, ensuring that the gradient within the fusion region matches the gradient of the source image as closely as possible, while the pixel values ​​at the boundary remain consistent with the target image boundary. This texture completion method requires no manual intervention and can automatically generate textures with repair traces that are visually indistinguishable, significantly enhancing the realism of the model.

[0071] In one possible implementation, a building facade model of the target building is constructed based on the geometrically repaired building facade point cloud, including: Instance segmentation is performed on the geometrically repaired point cloud data to separate independent building components; For each separated building component, the minimum volume bounding box algorithm based on the eigenvalue decomposition of the covariance matrix is ​​used to fit the geometric parameters to obtain the fitted parameterized object. The fitted parametric object is mapped to a standard entity of the industrial base class, and the building facade model of the target building is output.

[0072] In this embodiment, after completing geometric repair and surface smoothing, in order to transform the processing results into a format directly usable for engineering applications, post-processing is performed on the point cloud data after the attachments have been removed and repaired. First, the repaired complete point cloud model is segmented into instances. A clustering algorithm based on Euclidean distance or a graph cut algorithm is used to separate point clouds belonging to different building components, such as separating individual windows, doors, and wall blocks. Addressing the potential tilting and settlement phenomena in old buildings, a minimum volume bounding box algorithm based on covariance matrix eigenvalue decomposition is used to fit the geometric parameters of each separated component. Specifically, for a component's point set, its covariance matrix is ​​calculated and eigenvalue decomposition is performed, obtaining three eigenvectors as the three principal axes of the bounding box. Then, the point set is projected onto each principal axis direction to obtain the minimum and maximum values ​​of the projection interval, thereby determining the length, width, height, and center point coordinates of the bounding box. This algorithm can accurately fit the minimum directed bounding box of the component, accurately reflecting its spatial position and deflection angle. Finally, based on the semantic label of each component, the fitted parametric objects are mapped to corresponding entities in the industry-standard class. For example, a component with the semantic label "window" is mapped to an IfcWindow entity, and its geometric parameters are filled into the corresponding attribute fields; a component with the semantic label "wall" is mapped to an IfcWallStandardCase entity. The final output is an editable, lightweight Building Information Modeling (BIM) file. Figure 6 The left side shows the repair effect of the axis-aligned bounding box method, while the right side shows the point cloud model repaired by the minimum volume bounding box method and the fitted parametric and editable building information model object, intuitively demonstrating the transformation effect from unstructured point cloud to structured building information model. Figure 6 The symbols in the text are: length: L; width: W; height: H; deflection angle: θ.

[0073] In one specific embodiment, to verify the effectiveness of the present invention, a comparative experiment was conducted on a measured dataset containing 50 old buildings, and the results are shown below.

[0074] Table 1: Semantic segmentation performance comparison (mIoU, %)

[0075] Table 2: Comparison of Chamfer Distance Repair Quality )

[0076] As can be seen from the data in Tables 1 and 2, the present invention has achieved a multiple-level improvement in the segmentation accuracy of difficult categories such as cables, and has improved the repair geometric accuracy by about 4 times compared with the general algorithm, fully demonstrating the superiority of the "decoupling + structural prior" strategy.

[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0079] Figure 7 A schematic diagram of the building facade modeling device based on appendage recognition provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: like Figure 7 As shown, the building facade modeling device 7 based on appendage recognition includes: The acquisition module 71 is used to acquire point cloud data of the building facade of the target building; The segmentation module 72 is used to perform semantic segmentation on point cloud data through a two-stream decoupling network to obtain host segmentation results and parasitic segmentation results. The two-stream decoupling network includes a host stream branch, a parasitic stream branch, and a hierarchical interaction module. The hierarchical interaction module connects the host stream branch and the parasitic stream branch through a gating mechanism. The stripping module 73 is used to generate an appendage mask based on the parasitic segmentation result, and strip the appendages from the point cloud data based on the appendage mask to obtain a building facade point cloud containing holes. Module 74 is used to perform geometric repair on the point cloud of the building facade and to construct the building facade model of the target building based on the geometrically repaired point cloud of the building facade.

[0080] In one possible implementation, both the host stream branch and the parasitic stream branch are sparse convolutional U-Net architectures, including encoders, decoders, and skip connections; the hierarchical interaction module includes a first gated convolutional unit and a second gated convolutional unit. The input of the first gated convolutional unit is connected to the output of the intermediate layer of the host stream branch decoder. The output generates a parasitic suppression mask. The parasitic suppression mask is multiplied element-wise with the feature map of the corresponding layer in the parasitic stream branch decoder and then fed into the next layer of the parasitic stream branch decoder. The input of the second gated convolutional unit is connected to the output of the intermediate layer of the parasitic stream branch decoder. The output generates an edge enhancement mask. The edge enhancement mask is added element-wise with the feature map of the corresponding layer in the host stream branch decoder and then fed into the next layer of the host stream branch decoder.

[0081] In one possible implementation, the host flow branch uses a planar consistency loss function, while the parasitic flow branch uses a class-balanced cross-entropy loss function and a focus loss function.

[0082] In one possible implementation, module 71 is specifically used for: Acquire image sequences and point cloud data of the building facade of the target building; Based on depth distribution prediction, the two-dimensional image features of the image sequence are enhanced into three-dimensional view frustum features; By using a cross-attention mechanism, the features of the 3D view frustum are fused with the geometric features of the point cloud data to obtain multimodal fusion features; Accordingly, the segmentation module 72 is specifically used for: Based on multimodal fusion features, semantic segmentation of point cloud data is performed through a two-stream decoupling network to obtain host segmentation results and parasitic segmentation results.

[0083] In one possible implementation, building module 74 is specifically used for: Detecting void areas in point clouds on building facades; For symmetrical void regions, the global symmetry plane and local symmetry center of the building facade are detected. The features of the intact region are mapped to the void region through symmetry transformation. The void region is then repaired using a generative adversarial network combined with a symmetry consistency loss function. For void regions that lack symmetry, the contextual features of the void region's edge are extracted, similar components are retrieved from a pre-built building component library, and a deformation network is used to non-rigidly deform the retrieved components to fit the void boundary, resulting in point cloud data after void repair.

[0084] In one possible implementation, building module 74 is also used for: After obtaining the point cloud data after void repair, the void repair point cloud data is transformed into an implicit representation of the symbolic distance field. A continuous symbolic distance function is learned using a neural network, and a zero isosurface is extracted to generate a watertight mesh model. Using the texture of the walls surrounding the repair area as a guide, a conditional diffusion model is used to generate texture maps of the hole areas, and Poisson fusion is used to eliminate lighting differences, resulting in point cloud data after texture completion.

[0085] In one possible implementation, building module 74 is specifically used for: Instance segmentation is performed on the geometrically repaired point cloud data to separate independent building components; For each separated building component, the minimum volume bounding box algorithm based on the eigenvalue decomposition of the covariance matrix is ​​used to fit the geometric parameters to obtain the fitted parameterized object. The fitted parametric object is mapped to a standard entity of the industrial base class, and the building facade model of the target building is output.

[0086] This invention employs a dual-stream decoupled network to process the building host layer and the parasitic appendage layer separately. The host stream focuses on low-frequency, large-scale structures such as walls, while the parasitic stream focuses on high-frequency, small targets such as cables and supports. The two streams interact through a gating mechanism to mutually suppress false alarms. This architecture fundamentally solves the class imbalance problem in multi-classification networks, improving the recognition rate of difficult-to-distinguish small targets such as cables. An appendage mask is generated based on the parasitic segmentation results and directly removed, avoiding the problem of rough segmentation edges eroding the building body in traditional methods. After stripping, a point cloud containing only the building body is obtained, and gaps are filled through geometric repair. The final output building facade model is clean, complete, and conforms to the building's structural logic. This method can achieve high-precision appendage stripping and modeling without manual intervention, significantly reducing the data preprocessing cost in the digital transformation of old residential areas and providing reliable technical support for building information model construction in large-scale urban renewal.

[0087] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 8 As shown, the electronic device 8 of this embodiment includes a processor 80 and a memory 81. The memory 81 stores a computer program 82. When the processor 80 executes the computer program 82, it implements the steps in the various method embodiments described above. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the various device embodiments described above.

[0088] For example, computer program 82 may be divided into one or more modules / units, which are stored in memory 81 and executed by processor 80 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 82 in electronic device 8.

[0089] Electronic device 8 may include, but is not limited to, processor 80 and memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 8 may also include input / output devices, network access devices, buses, etc.

[0090] The processor 80 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0091] The memory 81 can be an internal storage unit of the electronic device 8, such as a hard disk or RAM. The memory 81 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 81 can include both internal and external storage units of the electronic device 8. The memory 81 is used to store the computer program 82 and other programs and data required by the electronic device 8. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0092] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0093] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0094] This invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0095] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0096] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0097] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A facade modeling method based on accessory recognition, characterized in that, The method comprises: obtaining point cloud data of a building facade of a target building; performing semantic segmentation on the point cloud data by using a dual-stream decoupling network to obtain a host segmentation result and a parasite segmentation result; wherein the dual-stream decoupling network comprises a host stream branch, a parasite stream branch and a hierarchical interaction module, and the hierarchical interaction module connects the host stream branch and the parasite stream branch through a gating mechanism; generating an accessory mask according to the parasite segmentation result, and performing accessory stripping on the point cloud data based on the accessory mask to obtain a building facade point cloud containing a cavity; performing geometric repair on the building facade point cloud, and constructing a building facade model of the target building based on the geometrically repaired building facade point cloud.

2. The fagade modeling method based on appendage recognition according to claim 1, characterized in that, The host stream branch and the parasite stream branch are both sparse convolution U-Net architectures, comprising an encoder, a decoder and a skip layer connection; the hierarchical interaction module comprises a first gating convolution unit and a second gating convolution unit; the input end of the first gating convolution unit is connected to the middle layer output of the host stream branch decoder, and the output end generates a parasite suppression mask; the parasite suppression mask is multiplied element by element with the feature map of the corresponding level in the parasite stream branch decoder, and then fed into the next layer of the parasite stream branch decoder; the input end of the second gating convolution unit is connected to the middle layer output of the parasite stream branch decoder, and the output end generates an edge enhancement mask; the edge enhancement mask is added element by element with the feature map of the corresponding level in the host stream branch decoder, and then fed into the next layer of the host stream branch decoder.

3. The fagade modeling method based on appendage recognition according to claim 1, characterized in that, The host stream branch adopts a plane consistency loss function, and the parasite stream branch adopts a class balanced cross entropy loss function and a focal loss function.

4. The fagade modeling method based on appendage recognition according to claim 1, characterized in that, The method comprises: obtaining an image sequence and point cloud data of a building facade of a target building; lifting two-dimensional image features of the image sequence to three-dimensional frustum features based on depth distribution prediction; fusing the three-dimensional frustum features and geometric features of the point cloud data through a cross-attention mechanism to obtain multi-modal fusion features; Correspondingly, the method of performing semantic segmentation on the point cloud data by using a dual-stream decoupling network to obtain a host segmentation result and a parasite segmentation result comprises: performing semantic segmentation on the point cloud data by using a dual-stream decoupling network based on the multi-modal fusion features to obtain a host segmentation result and a parasite segmentation result.

5. The facade modeling method based on identification of appendages according to any of claims 1 to 4, characterized in that, The method of performing geometric repair on the building facade point cloud comprises: detecting a cavity region of the building facade point cloud; for a cavity region with symmetry, detecting a global symmetry plane and a local symmetry center of the building facade, mapping the features of an intact region to the cavity region through a symmetry transformation, and repairing the cavity region by using a generative adversarial network combined with a symmetry consistency loss function; for a cavity region without symmetry, extracting context features of the edge of the cavity region, searching for similar components in a pre-constructed building component library, and deforming the retrieved components by using a deformation network to adapt to the cavity boundary to obtain point cloud data after cavity repair.

6. The fagade modeling method based on appendage recognition according to claim 5, characterized in that, After obtaining the point cloud data after hole repair, the following is also included: The point cloud data after hole repair is transformed into an implicit representation of the symbolic distance field. A continuous symbolic distance function is learned using a neural network, and a watertight mesh model is generated by extracting the zero isosurface. Using the texture of the walls surrounding the repair area as a guide, a conditional diffusion model is used to generate texture maps of the hole areas, and Poisson fusion is used to eliminate lighting differences, resulting in point cloud data after texture completion.

7. The fagade modeling method based on appendage recognition according to claim 1, characterized in that, The construction of the building facade model of the target building based on the geometrically repaired building facade point cloud includes: Instance segmentation is performed on the geometrically repaired point cloud data to separate independent building components; For each separated building component, the minimum volume bounding box algorithm based on the eigenvalue decomposition of the covariance matrix is ​​used to fit the geometric parameters to obtain the fitted parameterized object. The fitted parametric object is mapped to a standard entity of the industrial base class, and the building facade model of the target building is output.

8. A facade modeling device based on accessory recognition, characterized by, include: The acquisition module is used to acquire point cloud data of the building facade of the target building; The segmentation module is used to perform semantic segmentation on the point cloud data through a two-stream decoupling network to obtain host segmentation results and parasitic segmentation results; wherein, the two-stream decoupling network includes a host stream branch, a parasitic stream branch, and a hierarchical interaction module, and the hierarchical interaction module connects the host stream branch and the parasitic stream branch through a gating mechanism; The stripping module is used to generate an appendage mask based on the parasitic segmentation result, and to strip the appendages from the point cloud data based on the appendage mask to obtain a building facade point cloud containing holes. The construction module is used to perform geometric repair on the point cloud of the building facade and construct the building facade model of the target building based on the geometrically repaired point cloud of the building facade.

9. An electronic device, comprising: It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images