A method and apparatus for segmenting building roof instances

Abstract

A method and apparatus for segmenting building roof instances are disclosed. The method includes: determining a target connected component; acquiring a first RGB top view and a first depth map corresponding to the target connected component in a target building; inputting the first RGB top view and the first depth map into a target model; determining instance segmentation data of the target connected component based on the target model; the instance segmentation data of the target connected component is used to indicate the target mask sequence and target height information sequence of N roof instances contained in the target connected component; the target model is a model learned based on training samples, used to determine the instance segmentation data of the connected component based on the multimodal information provided by the input RGB top view and depth map of the connected component to be analyzed. Thus, by using a deep learning model to fuse RGB and depth information, accurate separation of complex roofs is achieved, improving the problem of contiguous noise in the segmentation of adjacent roof instances.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for segmenting building rooftop instances. Background Technology

[0002] In smart city 3D modeling, roof segmentation technology is the core foundation for achieving accurate digital representation and intelligent applications of the city.

[0003] Multi-sloped roofs (such as gable roofs, hip roofs, and hipped roofs) have numerous slopes and complex shapes. Dividing these diverse roofs into individual roof instances provides the smallest operational unit for building lifecycle management. For example, analyzing the effective sunlight-receiving area and orientation of different segmented slopes allows for accurate estimation of photovoltaic (PV) installed capacity, promoting urban green energy planning. Furthermore, analyzing the impact of different roof types on rainwater runoff paths enables models to more realistically simulate urban flooding risks.

[0004] However, traditional image segmentation logic is generally based on the aggregation of roof units by edge proximity. It cannot solve the problem of adjacent noise being connected in multi-slope roofs, which leads to multiple independent roof instances (especially slopes with indistinct height differences) being forcibly merged into a single contiguous block.

[0005] Therefore, how to provide a roof instance segmentation method to improve the problem of noise concatenation between adjacent roof instance segments is a key research topic for those skilled in the art. Summary of the Invention

[0006] In a first aspect, this application provides a method for segmenting building roof instances. The method includes: determining a target connected component, wherein the target connected component belongs to a preset connected component from which roof instance segmentation data cannot be accurately obtained through connected component analysis methods, and the target connected component corresponds to a part or all of a target building; obtaining a first RGB top view and a first depth map corresponding to the target connected component in the target building; inputting the first RGB top view and the first depth map into a target model; determining instance segmentation data of the target connected component based on the target model, so as to perform roof instance segmentation based on the instance segmentation data of the target connected component, wherein the instance segmentation data of the target connected component is used to indicate the N roof instances contained in the target connected component. The system comprises: a target mask sequence of N roof instances and a target height information sequence of the N roof instances, wherein N is greater than or equal to 2; wherein, the first mask in the target mask sequence is used to mark the outline and position of the first roof instance in the target connected component, and the first mask is any mask element in the target mask sequence; the target model is a model learned based on training samples, used to determine the instance segmentation data of the connected component based on the multimodal information provided by the RGB top view and depth map of the input connected component to be analyzed, wherein each training sample includes an RGB top view and depth map associated with labeled data, and the labeled data includes the segmentation mask of the real roof instance corresponding to the training sample and the height information of each roof instance.

[0007] The roof instance segmentation method provided in this application addresses several key issues. First, it directly tackles scenarios where traditional connected component analysis fails (e.g., complex roof structures with severe adhesion). Traditional methods cannot accurately separate independent roof instances or precisely depict their outlines in these scenarios. This solution introduces a neural network deep learning model, leveraging its powerful feature extraction and pattern recognition capabilities to overcome the limitations of traditional pixel / region similarity-based segmentation methods. It achieves accurate acquisition of instance segmentation data (mask sequence + height information sequence) for adhered and overlapping roofs, improving the problem of noise concatenation between adjacent roof instances. Second, the target model simultaneously acquires key height information for each roof instance. Therefore, the roof instance segmentation method provided in this application can offer a high-precision, integrated solution for subsequent 3D roof modeling.

[0008] On the other hand, mainstream methods rely on traditional image segmentation logic, aggregating roof units solely based on planar texture, color similarity, or edge proximity, completely ignoring three-dimensional spatial hierarchical relationships. When dealing with vertically adjacent roof structures (such as equipment rooms on the roofs of twin towers or stepped industrial plant roofs), the algorithm fails to perceive the spatial separation at different heights, causing multiple independent roof instances to be forcibly merged into a single, contiguous block. This spatial obfuscation severely distorts the true structure of the building, depriving subsequent critical applications such as load-bearing analysis, solar simulation, and photovoltaic deployment of their data foundation, significantly reducing the reliability of engineering analysis.

[0009] However, the target model provided in this application can simultaneously learn the semantic information of RGB and the geometric spatial features of depth, which is equivalent to indirectly encoding three-dimensional information. This ensures that the roof instance segmentation is carried out in three-dimensional space, which can significantly improve the accuracy of segmentation and the adaptability to complex scenes. For example, the target model can better distinguish adjacent roofs with similar color and texture (by utilizing height difference).

[0010] On the other hand, traditional solutions employ a fragmented pipeline approach: first, two-dimensional planar segmentation is performed to generate the roof boundary, and then the height value is estimated independently through point cloud fitting or shadow analysis. These two tasks lack a collaborative optimization mechanism, resulting in spatial misalignment between the segmentation results and the height prediction. Furthermore, manual intervention is required to preset key parameters: for example, the threshold for the normal vector angle needs to be adjusted empirically based on the roof slope, and the area range threshold needs to be manually configured according to the building scale. This highly manual reliance leads to three major drawbacks: first, the missegmentation rate fluctuates drastically when applied across different building types (such as low-rise residential buildings and super high-rise office buildings); second, height value offsets cause significant discrepancies between the BIM model and the actual scene; and third, each building requires repeated parameter adjustments and verification, making it impossible to achieve both efficiency and accuracy in large-scale applications.

[0011] However, by using the target model provided in this application, the two major tasks of accurately segmenting the roof instance (outline and location) and obtaining the instance's height information are optimized collaboratively, eliminating the need for subsequent re-alignment or estimation steps and improving both efficiency and accuracy. Furthermore, during the model application phase, it does not rely on manual intervention, reducing the workload of manually correcting the segmentation results later, further improving efficiency and accuracy.

[0012] In some possible implementations, determining the target connected component includes: acquiring an initial 3D model of the target building; performing voxelization on the initial 3D model of the target building to obtain a first voxel mesh; determining a set of candidate roof voxels based on the voxels with upward-facing normal vectors in the first voxel mesh; performing connected component analysis using a region growing clustering algorithm based on the candidate roof voxels to obtain at least one connected component; and determining, from the at least one connected component, other connected components besides those meeting a first preset condition and a second preset condition as the target connected component; wherein, a connected component meeting the first preset condition indicates that the connected component belongs to a noisy connected component, and a connected component meeting the second preset condition indicates that the connected component is a single unconnected roof.

[0013] This approach effectively removes non-roof structures by filtering the noise connected components based on the first preset condition, thus avoiding invalid misjudgments of the noise connected components and improving the purity of subsequent processing, preventing noise from interfering with the judgment of the deep learning model.

[0014] On the other hand, based on the second preset condition, individual disconnected roofs are filtered to identify and separate those roofs that are simple in structure, have clear boundaries, and exist independently. These simple roofs can usually obtain sufficiently accurate segmentation results and height information through traditional, low-computational-cost methods (such as region growing clustering), without the need to use complex deep learning models.

[0015] This avoids feeding simple, easily processed regions (noise and individual disconnected roofs) into the computationally intensive target model, which significantly reduces the number of connected domains requiring in-depth processing, greatly improves the overall system's operating efficiency, and saves computational resources.

[0016] In some possible implementations, the first preset condition includes: the area of ​​the connected component is less than a first area threshold, the standard deviation of the normal vector is greater than a first standard deviation threshold, the aspect ratio is greater than a first ratio, and the plane fitting error is greater than a first error value; the second preset condition includes: the area of ​​the connected component is greater than or equal to a second area threshold and less than or equal to a third area threshold, the standard deviation of the normal vector is less than a second standard deviation threshold, the aspect ratio is less than or equal to a second ratio, and the plane fitting error is less than a second error value; the third area threshold is greater than the first area threshold, the second standard deviation threshold is less than or equal to the first standard deviation threshold, the second ratio is less than the first ratio, and the second error value is less than or equal to the first error value.

[0017] In some possible implementations, after determining the N roof instances contained in the target connected component, the target mask sequence of the N roof instances, and the target height information sequence of the N roof instances based on the target model, the method further includes: obtaining a data-enhanced first voxel grid; obtaining the data-enhanced first voxel grid includes: based on the target mask sequence and the target height information sequence, marking voxels in the first voxel grid located within the region of the first mask and whose height information matches the first height information as belonging to the first roof instance, thereby obtaining the data-enhanced first voxel grid, wherein the first height information is the height information in the target height information sequence corresponding to the first mask; performing connected component analysis on the data-enhanced first voxel grid to remove noisy connected components, thereby obtaining a second voxel grid; converting the second voxel grid back to the initial 3D model to obtain a reconstructed 3D model, wherein different types of roofs are accurately segmented into different roof instances in the reconstructed 3D model.

[0018] This method involves converting the target building mesh model into a 3D voxel mesh, performing voxel mesh connectivity analysis (region growth and filtering based on voxel normal vectors and 3D geometric attributes), and indirectly encoding 3D information by combining the target model's RGB top view and depth map. This allows for the extraction of instance segmentation data in 3D space. The cutting and reconstruction steps are based on height sequence (3D attribute) voxels and converted back to the mesh model. Through 3D data processing at each step, high-fidelity reconstruction is achieved, eliminating the distortion of traditional dimensionality reduction.

[0019] In some possible implementations, after obtaining at least one connected component, the method further includes: identifying noisy connected components in the at least one connected component that meet the first preset condition, and removing the noisy connected components; identifying a first connected component in the at least one connected component that meets the second preset condition; searching for a reference adjacent connected component that meets a preset merging condition within a preset voxel radius of the first connected component, merging the reference adjacent connected component into the first connected component, and recalculating and updating the normal vector of the first connected component to obtain the updated first connected component; obtaining the data-enhanced first voxel mesh further includes: marking the voxels corresponding to the first connected component in the first voxel mesh as belonging to the second roof instance to obtain the data-enhanced first voxel mesh.

[0020] This method further processes connected domains (single non-connected roofs) that meet the second preset condition, merging truly adjacent and reasonable neighboring small roofs into a more reasonable instance, while avoiding the erroneous merging of truly different roofs. This further improves the geometric integrity and structural consistency of the roof instance segmentation of the target building, and can effectively handle different situations such as single pitched roofs and multiple pitched roofs (with obvious and indistinct height differences), thereby improving the accuracy and efficiency of roof segmentation.

[0021] In some possible implementations, the preset merging condition includes: the angle between the normal vectors corresponding to the reference adjacent connected components is less than a first angle threshold; the reference adjacent connected components and the first connected component satisfy the formula: >0.4, where perimeter 1 and perimeter 2 are the perimeters corresponding to the outer contours of the reference adjacent connected domain and the first connected domain, respectively, and the roof slope change after the first connected domain and the reference adjacent connected domain are merged is less than the second angle threshold.

[0022] In some possible implementations, the target model includes a feature fusion network, a candidate region network, a region feature alignment module, a three-task optimization network, and an instance decoupling network. The feature fusion network fuses the multimodal information contained in the RGB top view and depth map of the connected components to be analyzed, outputting a feature map. The candidate region network generates candidate region proposals based on the feature map output by the feature fusion network. The region feature alignment module constructs a fixed-size feature vector corresponding to each candidate region proposal based on the feature map output by the feature fusion network and the candidate region proposals output by the candidate region network. The three-task optimization network includes a classification head, a masking head, and a first height regression head. The classification head is used for... The feature vector of the candidate area proposal determines the binary classification result of whether the candidate area proposal belongs to the roof or the background. The masking head is used to determine the mask map corresponding to the candidate area proposal based on the feature vector of the candidate area proposal. The first height regression head is used to determine the initial height information of the candidate area proposal based on the feature vector of the candidate area proposal. The instance decoupling network is used to generate a spatial attention map and a center point heatmap based on the feature vector and center point coordinates of each candidate area proposal. The spatial attention map and the center point heatmap are combined to determine at least two roof instances contained in the connected domain to be analyzed and the mask sequence of the at least two roof instances. The second height regression head and the initial height information of each candidate area are used to determine the height information sequence of the at least two roof instances.

[0023] In this approach, the classification head uses a lightweight MLP network, and the masking head uses a lightweight UNet decoder to reduce the number of parameters and avoid introducing excessive computational burden.

[0024] In some possible implementations, the feature fusion network is a ResNet residual network, and the loss of the feature fusion network is implicit in the training process, including standard classification loss and regression loss; the candidate region network is a Region Proposal Network (RPN), and the loss of the candidate region network includes classification loss for the foreground and background of the candidate boxes, as well as regression loss; the region feature alignment module is a Region of Interest (ROIAlign) module; the classification head in the three-task optimization network is a lightweight multilayer perceptron (MLP) model, and the loss of the classification head includes cross-entropy loss; the masking head in the three-task optimization network is a lightweight U-shaped network (UNet) decoder; the first height regression head in the three-task optimization network is a noise-resistant regression network; the instance decoupling network is a UNet decoder, and the loss of the masking head and the loss of the mask sequence output by the UNet decoder include binary cross-entropy loss (BCE), the loss of the first height regression head and the loss of the second height regression head in the UNet decoder include smoothing L1 loss, and the total loss of the target model satisfies the formula: The and stated To balance the hyperparameters.

[0025] Using this approach, the target model is based on ResNet, which integrates the multimodal features of the RGB top view and depth map of the connected domain to be analyzed. It also introduces three parallel tasks—classification, segmentation, and height regression—on top of ResNet-RPN, and utilizes the UNet decoder and spatial attention mechanism to achieve accurate instance segmentation, count, and height regression of the bonded roof.

[0026] Secondly, this application also provides a building roof instance segmentation apparatus, including a unit for performing any of the building roof instance segmentation methods in the first aspect.

[0027] Thirdly, this application also provides a computer storage medium that can store multiple instructions adapted for loading and execution by a processor of any of the building roof instance segmentation methods in the first aspect.

[0028] Fourthly, embodiments of this application also provide a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to execute any of the building roof instance segmentation methods in the first aspect.

[0029] Fifthly, embodiments of this application also provide a chip module, including a transceiver component and a chip, wherein the chip is used to execute any of the building roof instance segmentation methods in the first aspect.

[0030] It is understood that the building roof instance segmentation device, computer storage medium, computer program, computer program product, and chip system provided above are all used to execute the method shown in any implementation of the first aspect of the embodiments of this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the method flow for a building roof instance segmentation method provided in an embodiment of this application;

[0032] Figure 2 This is a schematic flowchart of a method for determining a target connected component provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of a method for roof instance segmentation of a Mesh model of a target building based on instance segmentation data, provided in an embodiment of this application.

[0034] Figure 4 This is a schematic diagram of the architecture of a target model provided in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of a training method for a target model provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of a building roof instance segmentation device provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of another building roof example segmentation device provided in the embodiments of this application;

[0038] Figure 8 This is a schematic diagram of another building roof instance segmentation device provided in the embodiments of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below in conjunction with the accompanying drawings.

[0040] Example 1:

[0041] Please see Figure 1 , Figure 1 A flowchart illustrating a method for segmenting a building rooftop instance, as provided in an embodiment of this application. Figure 1 As shown, the method for segmenting building roof instances includes the following steps:

[0042] S101, the electronic device determines the target connected component.

[0043] In this embodiment of the application, the target connected component corresponds to a part of the target building, or the target connected component corresponds to the entire target building, and the target connected component belongs to a preset connected component where the roof instance cannot be accurately obtained through connected component analysis methods.

[0044] In some possible implementations, the target connected component is at least one connected component that satisfies the target preset condition obtained by performing connected component analysis on the candidate voxel set of the roof. The candidate voxel set of the roof includes voxels with upward-facing normal vectors in a first voxel mesh, which is obtained by voxelizing the initial 3D model of the target building.

[0045] As an example, a connected component that satisfies the target preset condition can be any connected component that does not satisfy either the first or the second preset condition. Specifically, a connected component satisfying the first preset condition indicates that it belongs to a noisy connected component, and a connected component satisfying the second preset condition indicates that it contains a single unconnected roof.

[0046] For the sake of simplicity, the connected domains that meet the second preset condition are referred to as valid single unconnected roofs, and the other connected domains besides the noisy connected domains and valid single unconnected roofs are referred to as large connected domains.

[0047] As an example, the first preset condition mentioned above includes:

[0048] 1) The area of ​​the connected component is less than the first area threshold, for example, the first area threshold is A_min, where A_min = max(10, 0.1% × building base area);

[0049] 2) The standard deviation of the normal vector is greater than the first standard deviation threshold (indicating that the normal vector is disordered), for example, the first standard deviation threshold is 0.3;

[0050] 3) Aspect ratio > first ratio value (indicating an abnormal aspect ratio), for example, the first ratio value is 5;

[0051] 4) Plane fitting error > first error value (indicating non-planar structure, uneven), for example, the first error value is the product of 0.3 and voxel size.

[0052] As an example, the second preset condition mentioned above includes:

[0053] 1) The area of ​​the connected component is greater than or equal to the second area threshold and less than or equal to the third area threshold, where the third area threshold is greater than the first area threshold mentioned above. For example, the second area threshold is 0.8A_min and the third area threshold is 1.2A_min.

[0054] 2) The standard deviation of the normal vector is less than the second standard deviation threshold, which is less than or equal to the first standard deviation threshold mentioned above. For example, the second standard deviation threshold is 0.2.

[0055] 3) The aspect ratio is greater than or equal to the second ratio, which is less than the first ratio mentioned above. For example, the second ratio is 3.

[0056] 4) Plane fitting error < second error value (implicit plane fitting error < 0.3voxel_size, indicating that the roof is flat or gently sloping), the second error value is less than or equal to the first error value mentioned above, for example, the second error value is 0.1.

[0057] In other possible implementations, the first and second preset conditions can be designed based on specific requirements. For example, the first and second preset conditions may only include area threshold conditions. For instance, connected components with an area less than 0.8A_min belong to noisy connected components that meet the first preset condition, connected components with an area greater than or equal to 0.8A_min and less than or equal to 1.2A_min belong to connected components that meet the second preset condition, and connected components with an area greater than 1.2A_min belong to target connected components. This paper does not limit this.

[0058] It should be noted that in some other possible implementations, the connected components that satisfy the target preset conditions can also be any connected components other than noisy connected components. Because a valid single unconnected roof is generally a single-slope roof or a multi-slope roof with significant height differences, it is generally considered that accurate roof instance segmentation data can be obtained through connected component analysis. The target model provided in this application mainly targets connected components for which accurate roof instance segmentation data cannot be obtained through connected component analysis. However, this does not mean that connected components that meet the above-mentioned second preset conditions cannot obtain roof instance segmentation data through the target model provided in this application. Connected components that meet the above-mentioned second preset conditions can also obtain roof instance segmentation data through the target model provided in this application; this is not a limitation in this paper.

[0059] As an example, let's take any connected component whose target is neither the first nor the second preset condition as an example, combined with... Figure 2 This describes how electronic devices specifically determine the target connected component. For example... Figure 2 As shown, determining the target connected components includes:

[0060] S1011, The electronic device acquires the initial mesh model of the target building.

[0061] In this embodiment of the application, the initial three-dimensional model of the target building is a polygon mesh model (hereinafter referred to as the initial mesh model for ease of description).

[0062] S1012, the electronic device performs voxelization on the initial mesh model of the target building to obtain the first voxel mesh.

[0063] For example, before voxelization, the electronic device reads relevant parameters for voxelization, including voxel size and voxel output storage path. For example, the voxel size can be 0.5m or other suitable size.

[0064] For example, the electronic device performs voxelization on the initial Mesh model based on the subdivide algorithm to obtain a first voxel mesh with a voxel side length of 0.5m.

[0065] S1013, the electronic device determines the candidate voxel set for the roof based on the voxels with upward normal vectors in the first voxel grid.

[0066] For example, the electronic device calculates the normal vectors of each voxel in the voxel mesh, filters reference voxels whose normal vectors make an angle less than a third angle threshold (e.g., the third angle threshold is 18°) with the vertical direction ([0,0,1]) as candidate roof regions, and uses dot product to calculate the verticality (∣). · |≥cosθthreshold), noise is eliminated by verifying the consistency of the normal vectors of adjacent voxels, and candidate voxels of the roof are merged to form a complete candidate set of roofs for subsequent connected component analysis.

[0067] S1014, the electronic device uses a region growing clustering algorithm to perform connected component analysis based on the candidate voxel set of the roof to obtain at least one connected component.

[0068] The region growing clustering algorithm starts from random seed voxels and recursively merges neighboring voxels with similar attributes based on the principle of spatial proximity.

[0069] 1) Regarding random seed selection: Prioritize voxels with high confidence (normal vector perpendicularity > 0.98) as initial seeds.

[0070] 2) Regarding the regional growth criterion (dual constraint mechanism):

[0071] Spatial constraint: Only merge neighboring voxels whose Euclidean distance is less than or equal to a first distance threshold, for example, the first distance threshold is... ×Voxenon size;

[0072] Normal vector constraint: The angle between the normal vectors is less than the fourth angle threshold, for example, the fourth angle threshold is 15° (which can also be understood as cosθ>0.96), to ensure topological continuity;

[0073] 3) Dynamic threshold: The mean normal vector is automatically updated every time a preset number of voxels are expanded (e.g., the preset number is 10 voxels) to adapt to changes in the surface;

[0074] 4) Parallel acceleration: Using an octree spatial index reduces the complexity of neighborhood search (O(n) → O(log n));

[0075] 5) Termination condition: No new voxels are added or all candidate voxels are covered in 100 consecutive iterations.

[0076] S1015, the electronic device determines a noisy connected component that meets the first preset condition in at least one of the connected components, and removes the noisy connected component.

[0077] S1016, the electronic device determines the first connected component that meets the second preset condition among the at least one connected component.

[0078] S1017, the electronic device determines any connected component other than the first connected component and the noisy connected component as the target connected component.

[0079] In the embodiments of this application, each target connected component is a separate connected component. For the target connected component, the building roof instance segmentation method of this application determines its roof instance segmentation information based on the target model through steps S102 to S103.

[0080] In some possible implementations, the electronic device directly uses the first connected domain that meets the second preset condition described above as a roof instance.

[0081] In some other possible implementations, the electronic device further merges the first connected domain that meets the above-mentioned second preset conditions through steps S1018 to S1019.

[0082] S1018, the electronic device searches for a reference adjacent connected region that meets the preset merging conditions within the preset voxel radius of the first connected region.

[0083] For example, the preset voxel radius is 10 voxels, or it could be any other suitable voxel radius.

[0084] For example, the above-mentioned preset merging conditions include:

[0085] 1) The angle between the normal vectors of the reference adjacent connected components is less than the first angle threshold. For example, the first angle threshold is 15°, that is, cosθ>0.96, where θ is the angle between the reference adjacent connected component and the normal vector.

[0086] 2) Structural compatibility assessment: The following formula 1 is considered to apply to the relationship between adjacent connected components and the first connected component:

[0087]

[0088] The common boundary length is the total length of the adjacent boundaries of the reference adjacent connected domain and the first connected domain. Perimeter 1 and perimeter 2 are the perimeters corresponding to the outer contours of the reference adjacent connected domain and the first connected domain, respectively.

[0089] 3) Geometric constraint merging: The roof slope change after merging the first connected domain with the reference adjacent connected domain is less than the second angle threshold, for example, the second angle threshold is 10°.

[0090] S1019, the electronic device merges the reference adjacent connected components into the first connected component, and recalculates and updates the normal vector of the first connected component to obtain the updated first connected component.

[0091] For example, after merging, the normal vector of the first connected component is recalculated by weighted averaging (the weight of any connected component within the merged region is a percentage of the area).

[0092] Using this method, for a connected domain (a single unconnected roof) that meets the second preset condition, considering the possibility of local breaks, adjacent unconnected roofs, minor normal perturbations, or slightly unreasonable structures, other roofs or voxel regions with consistent normals, structural compatibility, and reasonable slope changes are searched within its vicinity. These are then reasonably merged into a more complete, coherent, and geometrically consistent roof instance. Simultaneously, the weighted average normal vector is recalculated after merging to ensure overall orientation uniformity.

[0093] S102, the electronic device acquires the first RGB top view and the first depth map corresponding to the target connected component in the target building.

[0094] Specifically, the electronic device generates the aforementioned first RGB top view and first depth map from the local 3D mesh model corresponding to the target connected region in the 3D mesh model of the target building through rendering or projection. In this embodiment, the first RGB top view is used to provide visual information such as the color, texture, and shadow of the roof, helping to distinguish different roof materials and structures. The first RGB top view and the first depth map provide geometric information, especially the relative height, undulation, and slope of various parts of the roof, which is crucial for distinguishing roofs with similar heights but different structures. Although both the RGB top view and the depth map are two-dimensional images, their combination already contains three-dimensional information: color, texture, shape, and height.

[0095] In some possible implementations, the first RGB top view and depth map can be models of the upper 70% of the local mesh model corresponding to the target connected component in the mesh model of the target building. Understandably, the upper 70% of the building focuses on the roof structure; this approach reduces background noise and improves training efficiency and accuracy.

[0096] S103, the electronic device inputs the first RGB top view and the first depth map into the target model, and determines the instance segmentation data of the target connected domain based on the target model, so as to perform roof instance segmentation based on the instance segmentation data of the target connected domain.

[0097] In this embodiment of the application, the instance segmentation data of the target connected region is used to indicate the N roof instances contained in the target connected region, the target mask sequence of the N roof instances, and the target height information sequence of the N roof instances.

[0098] In this embodiment of the application, N is greater than or equal to 2, and the goal of the target model is to accurately separate each individual roof instance in a dense, cohesive region.

[0099] In this embodiment, the first mask in the target mask sequence is used to mark the outline and position of the first roof instance in the target connected region. The first mask is any mask element in the target mask sequence, and the first height information corresponding to the first mask in the target height information sequence is used to mark the height information of the first roof instance. The height information of the roof instance can be the height value of the lowest point of the roof instance.

[0100] In this embodiment, the target model is learned based on training samples. It is used to determine instance segmentation data (roof instance mask sequence and height information sequence) of the connected components based on the multimodal information provided by the RGB top view and depth map of the connected components to be analyzed. Each training sample includes an RGB top view and a depth map associated with labeled data. This labeled data includes the segmentation mask of the real roof instances corresponding to the training sample and the height information of each roof instance. For example, for each RGB top view and its corresponding depth map, there are matching manually labeled mask data and height information data.

[0101] Understandably, through training with a large amount of labeled data (including complex roofs, adhesions, and other difficult-to-handle samples), the model can learn various complex roof shapes, structural combinations, and their performance patterns under different lighting and viewing angles. This allows the trained model to not only accurately extract instance segmentation data for the target connected domain of roofs with known complex structures, but also accurately obtain instance segmentation data for unknown buildings and roof types with new features, demonstrating good generalization ability.

[0102] The building roof instance segmentation method provided in this application utilizes a deep learning model to fuse RGB and depth information, breaking through the technical bottleneck of traditional connected component analysis in complex roof instance segmentation. It achieves accurate separation (instance segmentation) of adhered and overlapping roofs, improves the problem of noise concatenation in the segmentation of adjacent roof instances, and simultaneously obtains the key height information of each roof instance, establishing an end-to-end segmentation-height joint optimization model to achieve parameter adaptation and cross-scene generalization.

[0103] In some possible implementations, the target connected component is a connected component that does not meet either the first preset condition or the second preset condition. For noisy connected components that meet the first preset condition, the electronic device removes the noisy connected component; for first connected components that meet the second preset condition, the electronic device... Figure 2 The merging strategy shown is further processed to obtain the updated first connected component. After obtaining this updated first connected component and the instance segmentation data (target mask sequence and target height information sequence) of the target connected component, the electronic device can perform roof instance segmentation on the mesh model of the target building. For example... Figure 3 As shown, as an example, the specific steps include:

[0104] S301, based on the target mask sequence and target height information sequence corresponding to the target connected components, mark the voxels in the first voxel grid that are located within the first mask region and whose height information matches the first height information as belonging to the first roof instance.

[0105] The first voxel mesh is the voxelized mesh obtained by voxelizing the initial mesh model of the target building in step S1012 above. The first mask (two-dimensional, top view projection plane) is any mask element in the target mask sequence, and the first height information is the height information corresponding to the first mask in the target height information sequence.

[0106] S302, based on the first connected component that meets the second preset condition, mark the voxel corresponding to the first connected component in the first voxel mesh as belonging to the second roof instance.

[0107] The first connected component can be one that meets the second preset condition and has not been traversed. Figure 2 The connected components that are merged in steps S1018 and S1019 can also be based on meeting the second preset condition and having undergone... Figure 2 The first connected component after merging in steps S1018 and S1019 is not limited in this paper.

[0108] It should be noted that since the first connected region itself is a set of voxels, the electronic device can directly mark the voxels corresponding to the first connected region as the second roof instance.

[0109] S303, the first voxel mesh after data augmentation is obtained based on S301 and S302.

[0110] Understandably, steps S301 and S302 can be executed simultaneously or sequentially, and the order is not limited.

[0111] S304. Perform connected component analysis on the data-enhanced first voxel mesh, remove noisy connected components, and obtain the second voxel mesh.

[0112] Specifically, removing noisy connected components from the first voxel mesh after data augmentation can be done by removing noisy connected components that meet a first preset condition. The first preset condition can be found in the relevant explanation above and will not be elaborated upon here. This method, through secondary denoising, improves the accuracy and efficiency of subsequent processing (such as segmentation and 3D reconstruction) by filtering out invalid regions.

[0113] It should be noted that electronic devices may also perform secondary connectivity analysis only on the voxels corresponding to the target connected components in the first voxel mesh after data augmentation, and remove noisy connected components. This paper does not limit this.

[0114] S305 converts the second voxel mesh back to the initial Mesh model, resulting in the reconstructed Mesh model.

[0115] In this embodiment of the application, different types of roofs in the reconstructed 3D model are accurately segmented into different roof instances.

[0116] Traditional methods, in order to simplify the processing, generally compress and project the 3D building roof structure onto a single plane (such as the ground plane or a horizontal section), and simulate 3D modeling using 2D image algorithms (contour extraction, region segmentation). This forced dimensionality reduction operation strips away height information, causing the complete loss of key geometric features such as the curvature changes of curved roofs and the stepped drops of multi-level sloping roofs. During reconstruction, the sloping structure, lacking continuous height support, exhibits topological distortions such as large-area slope fractures and eave misalignment. Especially for complex buildings such as stadiums and irregularly shaped commercial buildings, the reconstructed model often presents discontinuous and broken planes with severely insufficient geometric fidelity, making it difficult to meet the rigid requirements of millimeter-level precision models for urban renewal and ancient building restoration.

[0117] However, the building roof instance segmentation method provided in this application involves converting the target building mesh model into a three-dimensional voxel mesh, performing voxel mesh connectivity analysis (region growth and filtering based on voxel normal vectors and three-dimensional geometric attributes), and indirectly encoding three-dimensional information by combining the target model's RGB top view and depth map, enabling the extraction of instance segmentation data in three-dimensional space. The cutting and reconstruction steps are based on height sequence (three-dimensional attributes) to mark voxels and convert them back to the mesh model. High-fidelity reconstruction is achieved through three-dimensional data processing in each step, eliminating the distortion of traditional dimensionality reduction.

[0118] In some possible implementations, the target model includes a feature fusion network, a candidate region network, a region feature alignment module, a three-task optimization network, and an instance decoupling network. The feature fusion network fuses the multimodal information contained in the RGB top-view and depth maps of the connected components to be analyzed, outputting a feature map. The candidate region network generates candidate region proposals based on the feature map output by the feature fusion network; each candidate region proposal is a rectangle corresponding to the RGB top-view. The region feature alignment module constructs a fixed-size feature vector corresponding to the candidate region of each candidate region proposal based on the feature map output by the feature fusion network and the candidate region proposals output by the candidate region network. The three-task optimization network includes a classification head, a masking head, and a first height regression head. The classification head determines whether a candidate region belongs to the roof or the background based on the feature vector of the candidate region proposal (binary classification result). The masking head determines the mask map of the candidate region based on the feature vector of the candidate region proposal. The first height regression head determines the initial height information of the candidate region based on the feature vector of the candidate region proposal. The instance decoupling network is used to generate a spatial attention map and a center point heatmap based on the feature vector and center coordinates of each candidate region proposal. It combines the spatial attention map and the center point heatmap to determine at least two roof instances contained in the connected region to be analyzed and the mask sequence of the at least two roof instances. It also determines the height information sequence of the at least two roof instances based on the second height regression head and the initial height information of each candidate region.

[0119] In some possible implementations, the feature fusion network is a ResNet residual network, and the loss of the feature fusion network is implicit in the training process, including standard classification loss and regression loss (the parameters of the ResNet backbone network are optimized through backpropagation of the losses of downstream tasks (such as RPN, classification head, regression head, etc.)). The candidate region network is a region proposal network (RPN), and the loss of the candidate region network includes classification loss for the foreground and background of the candidate boxes, as well as regression loss. The region feature alignment module is the Region of Interest (ROIAlign) module; the classification head in the three-task optimization network is a lightweight multilayer perceptron (MLP) model, and the loss of the classification head includes cross-entropy loss; the masking head in the three-task optimization network is a lightweight UNet decoder; the first height regression head in the three-task optimization network is a noise-resistant regression network; the instance decoupling network is a UNet decoder, and the loss of the masking head and the loss of the mask sequence output by the UNet decoder include binary cross-entropy loss (BCE), the loss of the first height regression head and the loss of the second height regression head in the UNet decoder include smoothing L1 loss, and the total loss of the target model satisfies Formula 2:

[0120] ,

[0121] and To balance the hyperparameters.

[0122] The following combination Figure 4 This application provides a schematic diagram illustrating the architecture of a target model.

[0123] like Figure 4 As shown, the feature fusion network in this target model uses the ResNet residual network (also known as the ResNet backbone network).

[0124] The candidate area network adopts the Area Proposal Network (RPN).

[0125] The region feature alignment module uses the ROIAlign technique, which aligns features to regions of interest.

[0126] The classification head (Box Head) in the three-task optimization network uses a lightweight MLP model. The fully connected layers of this lightweight MLP model are: FC-256 → ReLU → Dropout → FC-2. Using a lightweight MLP network results in a very small number of parameters, used solely for the classification task, thus avoiding excessive computational burden.

[0127] The Mask Head algorithm employs a lightweight UNet decoder. The network structure of this lightweight UNet decoder includes: input feature broadcasting + center coordinate heatmap → upsampling convolution → output attention map. It is implemented in a lightweight manner and improves segmentation accuracy and localization accuracy by explicitly introducing center point location information for guidance.

[0128] The Height Head employs a noise-resistant regression network, whose structure consists of: FC-128 → LayerNorm → LeakyReLU → FC-1. Because roof height data may contain noise, inconsistent labeling, or uneven distribution, this module is called a "noise-resistant regression network." LayerNorm balances feature stability, and the smaller FC layer helps avoid overfitting.

[0129] The instance decoupled network uses the UNet decoder, which includes an attention weight map generation module, a center point heatmap generation module, and a height regression head.

[0130] It should be noted that the network models and internal hierarchical designs used in the various modules of the target model are merely examples. Other suitable network models or internal hierarchical designs can also be used, and this article does not limit them.

[0131] exist Figure 4 Based on the schematic diagram of the target model shown, combined with Figure 5 This describes the input, output, and training process of each module in the target model.

[0132] Phase 1: Construction of multimodal training set.

[0133] In this embodiment of the application, the input samples of the target model (which can also be understood as the input of ResNet) are RGB top view and depth map (corresponding to the connected component to be analyzed). The input samples correspond to the reference connected component, which belongs to the preset connected component that cannot accurately obtain roof instance segmentation data through connected component analysis methods.

[0134] For example, the reference connected component is a connected component that meets the above-mentioned target preset conditions after voxelization and region growth clustering of the reference building.

[0135] For example, select the upper 70% of the local 3D mesh model corresponding to the reference connected component in the 3D mesh model of the reference building. Render or project this upper 70% of the mesh model to generate an RGB top view and a depth map corresponding to the reference connected component. Input sample: RGB top view of the upper 70% of the building + depth map of the upper 70% of the building matching the RGB top view.

[0136] In this embodiment of the application, the input samples are associated with manually labeled data for supervised learning. This manually labeled data includes:

[0137] 1) Instance segmentation mask (precisely distinguishes adhered roofs).

[0138] 2) The lowest point height (GT_Height_i) of each roof instance.

[0139] Phase 2: Dual-stream feature extraction and candidate generation.

[0140] Dual-stream feature extraction and candidate generation refers to: fusing RGB and deep features through the ResNet backbone network (layers C2-C5), generating candidate bounding boxes through the Region Proposal Network (RPN), and constructing a 256-dimensional ROI feature vector for each proposal based on the ROIAlign module. Specifically, this stage 2 includes S502-S504:

[0141] S501 inputs the RGB top view and depth map of a single training sample into the ResNet backbone network. The RGB top view and depth map are associated with manually labeled data.

[0142] S502 uses the ResNet backbone network to fuse features from the RGB top view and depth map, extracting shared feature maps (C2-C5 layers) containing multimodal information.

[0143] In this approach, features from layers C2 to C5 of the ResNet backbone are preserved and used for subsequent candidate region proposal generation. These feature maps are rich in multimodal information (RGB texture + depth geometry), and features from different layers support subsequent multi-scale roof candidate detection.

[0144] S503, the Region Proposal Network (RPN), generates candidate region proposals based on the feature maps output from the ResNet backbone network.

[0145] For example, RPN extracts candidate region proposals (also known as candidate bounding boxes) based on the receptive fields corresponding to the feature maps output by layers C2 to C5 in the ResNet backbone network. These candidate region proposals are rectangular boxes corresponding to the RGB top view.

[0146] S504, the ROIAlign module maps each candidate region proposal to a feature map, constructs a 256-dimensional region feature vector (ROI feature) of a fixed-size feature block corresponding to each proposal, and represents the corresponding proposal based on this ROI feature vector.

[0147] Understandably, since the feature map is the result of downsampling the input image (e.g., downsampling by 16 times), it is necessary to map the candidate region proposals to the corresponding positions in the feature map based on the RIOAlign module, and construct a 256-dimensional ROI feature vector for each proposal with a fixed-size feature block.

[0148] Phase 3: Multi-task head collaborative training. This phase 3 specifically includes:

[0149] S505 inputs the ROI feature vector of each candidate region proposal into the classification head, masking head, and height regression head of the three-task optimization network for simultaneous optimization.

[0150] Classification head: Performs binary classification on each candidate area proposal: roof / background binary classification, outputs the probability that each candidate area proposal belongs to the roof, and the supervision signal is manually labeled data.

[0151] Roof Masking: Generates pixel-level instance segmentation masks for each candidate roof (focusing on learning the adhesion boundaries).

[0152] Height Regression Head: Regresses the height value of the roof instance corresponding to the candidate area proposal. The input source can be from the depth map or obtained from the ROI features. The supervision signal is the labeled true height.

[0153] In the embodiments of this application, during the training process of the target model, the three-task optimization network calculates the error between the prediction result and the real labeled data through loss functions (such as cross-entropy loss, L1 loss, and binary cross-entropy loss), and updates the parameters of RPN and ResNet through backpropagation to gradually optimize the model performance.

[0154] Phase 4: Spatial Awareness Instance Decoupling. Core Technology: UNet Decoder + Height Regression Head Parallel Mechanism.

[0155] Phase 4 specifically includes:

[0156] S506 inputs the ROI feature vector of the foreground proposal, the initial height information generated in stage 3, and the center coordinates into the UNet decoder.

[0157] The inputs to the UNet decoder include:

[0158] ROI features of the prospect proposal: 256-dimensional region feature vectors from ResNet-RPN.

[0159] Initial height information: The initial height prediction value generated by the height regression head in the previous stage.

[0160] Center coordinates: (center_x, center_y) Spatial positioning information, determined based on candidate region proposals (candidate bounding boxes).

[0161] In the embodiments of this application, the input of the UNet decoder only contains necessary, characterized or context-enhanced information (ROI features, center coordinates, initial height), which is reasonable and concise.

[0162] A UNet decoder is used to generate a spatial attention map, which is then combined with a center point detection head and a height regression head to achieve parallel instance localization.

[0163] Attention weight map: Highlights the boundaries of adhesion regions (supervised: truth boundary map).

[0164] Center point heat map: Marks the centroid location of each roof instance.

[0165] Height Regression Header: Outputs the precise height value for each instance (supervised by: GT_Height_i).

[0166] Training objectives:

[0167] The autoregressive generation of the mask sequence {Roof_Mask_t} (spatial attention map → binarized instance mask) and the height information sequence, in some other possible implementations, may also include the number of roof instances in the output.

[0168] In this embodiment of the application, {Roof_Mask_t} is the set of segmentation results of all independent roofs in the current building area (each mask represents a roof instance).

[0169] For example, UNet uses an autoregressive loop to generate {Roof_Mask_t} based on the attention weight map and the center point heatmap:

[0170] Step 1: Output the mask of the highest roof;

[0171] Step 2: Erase the predicted area;

[0172] Step 3: Continue outputting the second highest roof mask in a loop until there are no remaining roofs.

[0173] This method generates masks sequentially from high to low height, forming an ordered mask output and preventing duplication.

[0174] In this embodiment of the application, the loss function design of the target model includes:

[0175] Attention map (mask): Binary cross-entropy loss (BCE).

[0176] High regression: smoothed L1 loss.

[0177] (λ is the equilibrium hyperparameter).

[0178] The target model provided in this application integrates the multimodal features of the RGB top view and depth map of the connected domain to be analyzed. Based on ResNet-RPN, it introduces three parallel tasks: classification, segmentation, and height regression. It also utilizes the UNet decoder and spatial attention mechanism to achieve accurate instance segmentation, number statistics, and height regression of the bonded roof.

[0179] The roof instance segmentation method provided in this application achieves a breakthrough improvement in the segmentation effect of pitched roofs by fusing voxelization processing and multimodal neural network prediction.

[0180] ① A leap in reconstruction accuracy:

[0181] Compared with the traditional BBox segmentation method, the segmentation error rate was reduced from 25% to below 5% (based on statistics from 1083 buildings in the test set), which is 4 times better than the traditional method;

[0182] ②Full-process automation and efficiency innovation:

[0183] The dynamic cutting plane generation module transforms the ridge line into geometric constraints, enabling precise one-time cutting of the pitched roof block (1083 buildings took a total of 1.5 hours, with an average of 6 seconds per building, which is 120 times more efficient than manual modeling, which takes 2 hours per building).

[0184] The output model is compatible with the CityGML LOD2.1 standard, meeting the high-precision modeling requirements of smart cities.

[0185] It is understood that the embodiments of this application use an electronic device as an example to illustrate the execution subject of the building roof instance segmentation method provided in this application. This electronic device can also be understood as the building roof instance segmentation device shown in the embodiments of this application. In the embodiments of this application, the electronic device can be a microprocessor or computer for executing program code, etc. Any electronic device that can be used to execute the method provided in the embodiments of this application falls within the protection scope of the embodiments of this application, and this application does not impose any limitations. For example, the electronic device can be a desktop computer, a laptop, a mobile terminal, a 32-bit microprocessor, or a 64-bit microprocessor, etc., and the embodiments of this application do not limit this.

[0186] This application embodiment also provides a building roof instance segmentation device, including a device for performing... Figures 1 to 3 A unit for any building roof instance segmentation method.

[0187] Please refer to Figure 6 This is a structural schematic diagram of a building roof instance segmentation device provided in an embodiment of this application. Figure 6 As shown, the device may include:

[0188] The first determining unit 601 is used to determine the target connected component;

[0189] The first acquisition unit 602 is used to acquire the first RGB top view and the first depth map corresponding to the target connected component in the target building;

[0190] The second determining unit 603 is used to input the first RGB top view and the first depth map into the target model, and determine the instance segmentation data of the target connected component based on the target model.

[0191] In some possible implementations, such as Figure 7 As shown, the first determining unit 601 specifically includes:

[0192] Acquire subunit 6011, used to acquire the initial three-dimensional model of the target building;

[0193] The voxelization processing subunit 6012 is used to perform voxelization processing on the initial three-dimensional model of the target building to obtain the first voxel mesh.

[0194] The first determining subunit 6013 is used to determine the roof candidate voxel set based on the voxels with upward normal vectors in the first voxel grid;

[0195] The connected component analysis subunit 6014 is used to perform connected component analysis based on the roof candidate voxel set using a region growing clustering algorithm to obtain at least one connected component.

[0196] The second determining subunit 6015 is used to determine, from the at least one connected component, other than the connected components that meet the first preset condition and the second preset condition, as the target connected component.

[0197] In some possible implementations, the device further includes:

[0198] The second acquisition unit 604 is used to acquire the data-enhanced first voxel mesh; the second acquisition unit 604 specifically includes:

[0199] The first marking subunit 6041 is used to mark voxels in the first voxel grid that are located within the first mask region and whose height information matches the first height information as belonging to the first roof instance, based on the target mask sequence and the target height information sequence.

[0200] The elimination unit 605 is used to perform connected component analysis on the data-enhanced first voxel mesh, eliminate noisy connected components, and obtain the second voxel mesh.

[0201] Reconstruction unit 606 is used to convert the second voxel mesh back to the initial three-dimensional model to obtain the reconstructed three-dimensional model, in which different types of roofs are accurately segmented into different roof instances.

[0202] In some possible implementations, the second determining subunit 6015 is further configured to determine a noisy connected component in the at least one connected component that meets the first preset condition, and to remove the noisy connected component; and is further configured to determine a first connected component in the at least one connected component that meets the second preset condition;

[0203] In some possible implementations, the device further includes: a merging unit 607, configured to find a reference adjacent connected region that satisfies a preset merging condition within a preset voxel radius of the first connected region, merge the reference adjacent connected region into the first connected region, and recalculate and update the normal vector of the first connected region to obtain the updated first connected region.

[0204] In some possible implementations, the second acquisition unit 604 further includes a second marking subunit 6042, used to mark the voxels corresponding to the first connected domain in the first voxel grid as belonging to the second roof instance, so as to obtain the data-enhanced first voxel grid.

[0205] For explanations of names such as target connected components and target model, please refer to the relevant descriptions in the above method embodiments, which will not be elaborated here.

[0206] Please refer to Figure 8 This is a structural schematic diagram of another building roof segmentation device provided in an embodiment of this application, as shown below. Figure 8As shown, the building roof instance segmentation device 800 may include: at least one processor 801, such as a CPU, at least one communication interface 803, a memory 804, and at least one communication bus 802. The communication bus 802 is used to enable communication between these components. The communication interface 803 may optionally include a standard wired interface, a wireless interface (such as a Wi-Fi interface or a Bluetooth interface), etc. The memory 804 may be a high-speed RAM memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory 804 may also be at least one storage device located remotely from the aforementioned processor 801. Figure 8 As shown, the memory 804, which serves as a computer storage medium, may include an operating system, a network communication module, and program instructions.

[0207] exist Figure 8 In the building roof instance segmentation device 800 shown, the processor 801 can be used to load program instructions stored in the memory 804 and specifically perform the following operations:

[0208] Determine the target connected components;

[0209] Obtain the first RGB top view and the first depth map corresponding to the target connected component in the target building;

[0210] Input the first RGB top view and the first depth map into the target model, and determine the instance segmentation data of the target connected component based on the target model.

[0211] It should be noted that the specific execution process can be found in the detailed description of the above method embodiments, and will not be elaborated here.

[0212] For specific execution steps, please refer to the description of the foregoing method embodiments, which will not be detailed here.

[0213] This application also provides a computer storage medium that can store multiple instructions. These instructions are adapted to be loaded and executed by a processor using the building roof instance segmentation method provided in this application. For details of the execution process, please refer to the specific description of the method embodiments shown above, which will not be elaborated here.

[0214] This application also provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to execute the method steps of the method embodiments shown above.

[0215] This application also provides a chip module, including a transceiver component and a chip, wherein the chip is used to execute the method steps of the above-described method embodiments.

[0216] It is understood that the building roof instance segmentation system, building roof instance segmentation device, computer storage medium, computer program, computer program product, and chip provided above are all used to execute the method shown in any implementation of the corresponding aspect of the embodiments of this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding method, and will not be detailed here.

[0217] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes the processes of the embodiments of the above methods.

[0218] The term "at least one" in this application refers to one or more items. "More than one item" means two or more items. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, it should be understood that although the terms "first," "second," etc., may be used to describe objects in this application, these objects should not be limited to these terms. These terms are only used to distinguish the objects from each other.

[0219] The terms “including” and “having” mentioned above, and any variations thereof, are intended to cover non-exclusive inclusion.

[0220] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for building roof instance segmentation, characterized in that, The method comprises: determining a target connected domain, the target connected domain belonging to a preset connected domain that cannot accurately obtain roof instance segmentation data by a connected domain analysis method, the target connected domain corresponding to a part of an area or a whole area of a target building; obtaining a first RGB overhead view and a first depth map corresponding to the target connected domain in the target building; inputting the first RGB overhead view and the first depth map into a target model, and determining instance segmentation data of the target connected domain based on the target model, the instance segmentation data of the target connected domain being used to indicate N roof instances contained in the target connected domain, a target mask sequence of the N roof instances, and a target height information sequence of the N roof instances, the N being greater than or equal to 2; wherein a first mask in the target mask sequence is used to mark an outline and a position of a first roof instance in the target connected domain, and the first mask is any mask element in the target mask sequence; the target model is obtained based on learning of training samples, and is a model for providing multi-modal information based on input of an RGB overhead view and a depth map of a connected domain to be analyzed, and determining instance segmentation data of the connected domain, each training sample comprising an RGB overhead view and a depth map associated with labeled data, the labeled data comprising a segmentation mask of a real roof instance corresponding to the training sample and height information of each roof instance.

2. The method of claim 1, wherein, The determination of the target connected domain comprises: obtaining an initial three-dimensional model of the target building; performing voxelization processing on the initial three-dimensional model of the target building to obtain a first voxel grid; determining a roof candidate voxel set based on voxels with upward normal vectors in the first voxel grid; performing connected domain analysis on the roof candidate voxel set using a region growing clustering algorithm to obtain at least one connected domain; from the at least one connected domain, determining other connected domains except for connected domains meeting a first preset condition and connected domains meeting a second preset condition as the target connected domain; wherein a connected domain meeting the first preset condition indicates that the connected domain belongs to a noise connected domain, and a connected domain meeting the second preset condition indicates that the connected domain is a single unconnected roof.

3. The method of claim 2, wherein: the first preset condition comprises that an area of the connected domain is less than a first area threshold, a normal vector standard deviation is greater than a first standard deviation threshold, an aspect ratio is greater than a first ratio, and a plane fitting error is greater than a first error value; the second preset condition comprises that the area of the connected domain is greater than or equal to a second area threshold and less than or equal to a third area threshold, the normal vector standard deviation is less than a second standard deviation threshold, the aspect ratio is less than or equal to a second ratio, and the plane fitting error is less than a second error value; the third area threshold is greater than the first area threshold, the second standard deviation threshold is less than or equal to the first standard deviation threshold, the second ratio is less than the first ratio, and the second error value is less than or equal to the first error value.

4. The method of claim 2 or 3, wherein, After the target model is determined based on the target model, the target connected domain contains N roof instances, the target mask sequence of the N roof instances, and the target height information sequence of the N roof instances, the method further comprises: obtaining a first voxel grid after data enhancement; The first voxel grid after data enhancement comprises: based on the target mask sequence and the target height information sequence, the voxels in the first voxel grid located in the region of the first mask and the height information matching the first height information are marked as belonging to the first roof instance, to obtain the first voxel grid after data enhancement, and the first height information is the height information corresponding to the first mask in the target height information sequence; The data enhanced first voxel grid is analyzed by connected domain analysis, and noise connected domains are removed to obtain a second voxel grid; The second voxel grid is converted back to the initial three-dimensional model to obtain a reconstructed three-dimensional model, and different types of roofs in the reconstructed three-dimensional model are accurately segmented into different roof instances.

5. The method of claim 4, wherein, After the at least one connected domain is obtained, the method further comprises: determining the noise connected domain in the at least one connected domain that meets the first preset condition, and removing the noise connected domain; determining the first connected domain in the at least one connected domain that meets the second preset condition; In the first connected domain, a reference adjacent connected domain that meets a preset merging condition is searched for within a preset voxel radius, the reference adjacent connected domain is merged into the first connected domain, and the normal vector of the first connected domain is recalculated and updated to obtain an updated first connected domain. The first voxel grid after data enhancement further comprises: The voxels in the first voxel grid corresponding to the first connected domain are marked as belonging to the second roof instance to obtain the first voxel grid after data enhancement.

6. The method of claim 5, wherein, The preset merging condition comprises: The included angle of the normal vector corresponding to the reference adjacent connected domain is less than a first angle threshold; The reference adjacent connected domain and the first connected domain satisfy the formula: > 0.4, wherein the perimeter 1 and the perimeter 2 are the perimeters corresponding to the outer contours of the reference adjacent connected domain and the first connected domain respectively, The roof slope change after the first connected domain and the reference adjacent connected domain are merged is less than a second angle threshold.

7. The method of any one of claims 1, 2, 3, 5, or 6, wherein, The target model comprises a feature fusion network, a candidate region network, a region feature alignment module, a three-task optimization network, and an instance decoupling network, The feature fusion network is used to fuse the multi-modal information contained in the RGB overhead view and the depth map of the connected domain to be analyzed, and outputs a feature map; The candidate region network is used to generate a candidate region proposal based on the feature map output by the feature fusion network; The region feature alignment module is used to construct a fixed-size feature vector corresponding to each candidate region proposal based on the feature map output by the feature fusion network and the candidate region proposal output by the candidate region network; The three-task optimization network comprises a classification head, a mask head, and a first height regression head, the classification head is used to determine a binary classification result of whether a candidate region proposal belongs to a roof or a background based on a feature vector of the candidate region proposal, the mask head is used to determine a mask corresponding to the candidate region proposal based on the feature vector of the candidate region proposal, and the first height regression head is used to determine initial height information of the candidate region proposal based on the feature vector of the candidate region proposal; The instance decoupling network is used to generate a spatial attention map and a center point heat map based on the feature vector and the center coordinates of each candidate region proposal, and determine at least two roof instances contained in the to-be-analyzed connected domain and a mask sequence of the at least two roof instances in combination with the spatial attention map and the center point heat map, and determine a height information sequence of the at least two roof instances based on the second height regression head and the initial height information of each candidate region.

8. The method of claim 7, wherein, The feature fusion network is a residual network ResNet, and a loss of the feature fusion network is implicit in a training process and comprises a standard classification loss and a regression loss; The candidate region network is a region proposal network RPN, and a loss of the candidate region network comprises a classification loss of foreground and background of a candidate frame and a regression loss; The region feature alignment module is a region of interest feature alignment ROIAlign module; The classification head in the three-task optimization network is a lightweight multilayer perceptron MLP model, and a loss of the classification head comprises a cross-entropy loss; The mask head in the three-task optimization network is a lightweight U-shaped network UNet decoder; The first height regression head in the three-task optimization network is a noise-resistant regression network; The instance decoupling network is a UNet decoder, a loss of the mask head and a loss of a mask sequence output by the UNet decoder comprise a binary cross-entropy loss BCE, a loss of the first height regression head and a loss of a second height regression head in the UNet decoder comprise a smooth L1 loss, and a total loss of the target model satisfies a formula: , said and said are balancing hyperparameters.

9. A building roof instance segmentation device, characterized by, The computer readable storage medium is used to store a computer program, when the computer program is executed, the method of any one of claims 1 to 8 is executed.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is used to store a computer program, when the computer program is executed, the method of any one of claims 1 to 8 is executed.

Images