Structured image parsing method and system facing fine-grained structure boundary and small target segmentation

Abstract

The application belongs to the technical field of image semantic segmentation, and particularly relates to a structured image analysis method and system for fine-grained structure boundary and small target segmentation, which comprises the following steps: constructing a semantic segmentation network HDA UNet based on an encoder-decoder architecture, embedding a hierarchical deformable attention HDA module in a skip connection, and embedding a deformable attention DA module in a bottleneck layer to enhance the modeling capability of fine and long objects and fuzzy boundary features, and realize adaptive alignment and fusion of cross-scale features; training the network based on a pre-constructed multi-class edge guided loss function, weighting the cross-entropy loss according to the weight map generated according to the real label edge distance, and driving the network to focus on the learning of the boundary area; inputting the image to be analyzed into the trained network, and outputting a pixel-level semantic segmentation result. The application effectively improves the segmentation accuracy of load-bearing wall boundaries, door and window contours, sliding doors, railings and other small targets in building plans.

Description

Technical Field

[0001] This invention belongs to the field of image semantic segmentation technology, specifically involving a structured image parsing method and system for segmenting fine-grained structural boundaries and small targets. Background Technology

[0002] Early plan view recognition methods primarily relied on geometric heuristics and low-level image processing techniques, such as line detection, edge extraction, and morphological operations. These methods, based on the structural line features and spatial topological relationships of images, achieved room area division and building component localization, achieving certain analytical results in drawings with standardized layers and regular structures. Subsequently, techniques combining connected component analysis, primitive linewidth recognition, and optical character recognition emerged, further enabling the detection of door and window symbols and semantic annotation of room functions. In recent years, with the rapid development of deep learning technology, semantic segmentation models based on encoder-decoder architectures have been widely applied to plan view parsing tasks. Building upon this, a series of improved methods have been proposed, including introducing deeper network structures, integrating skeletonized post-processing, using orientation-aware convolution to enhance geometric structure modeling, and combining boundary-guided attention mechanisms, multi-task learning frameworks, and generative adversarial networks, thus achieving significant progress in the recognition and reconstruction of major architectural elements such as walls and rooms.

[0003] However, most existing methods focus on the main structure, such as walls and rooms, and lack the ability to model fine-grained structural boundaries (such as the boundary between load-bearing and non-load-bearing walls, and the detailed outlines of doors and windows) and small objects (such as sliding doors and railings). Because these small sample categories are scarce in the training data and their boundaries are often blurred, the model struggles to converge during training and is easily ignored by the backbone network, leading to missed detections and misclassifications. Furthermore, although some methods introduce attention or boundary-aware mechanisms, their feature alignment and fusion capabilities are limited, making it difficult to adaptively focus on key details in complex structural contexts. This results in weak feature representation capabilities for slender objects and blurred boundaries, and the segmentation results are often inaccurate at the boundaries. Therefore, existing methods have poor generalization ability when faced with diverse and complex residential floor plans in real-world scenarios, making it difficult to meet the practical application requirements for high-precision, fine-grained structured analysis. Summary of the Invention

[0004] The purpose of this invention is to provide a structured image parsing method and system for segmenting fine-grained structural boundaries and small targets, so as to solve the technical problem that existing image semantic segmentation methods have insufficient segmentation accuracy for fine-grained structural boundaries (such as load-bearing wall boundaries, door and window outlines) and small targets (such as sliding doors and railings).

[0005] The present invention achieves the above objectives through the following technical solutions:

[0006] Firstly, this invention proposes a structured image parsing method for segmenting fine-grained structural boundaries and small targets, the method comprising:

[0007] Construct a semantic segmentation network based on an encoder-decoder architecture, embedding a hierarchical deformable attention (HDA) module in the skip connections and a deformable attention (DA) module in the bottleneck layer;

[0008] The semantic segmentation network is trained based on a pre-constructed loss function, which is constructed by cross-entropy loss and a weight graph dynamically generated based on structural boundary information in the real labels, so as to drive the network to focus on learning fine-grained structural boundaries.

[0009] The structured image to be parsed is input into the trained semantic segmentation network, which outputs pixel-level semantic segmentation results.

[0010] The encoder extracts multi-scale features from the input image. The feature map output by the encoder is processed by the DA module to output enhanced bottleneck features. The decoder upsamples the enhanced bottleneck features step by step and combines the upsampled feature map with the corresponding encoder feature map after alignment and fusion by the HDA module at each level to improve the segmentation accuracy of fine-grained boundaries and small targets.

[0011] Furthermore, the semantic segmentation network is an HDA-UNet (Hierarchical Deformable Attention UNet) network, wherein:

[0012] The encoder consists of four sequentially connected coding layers. Each coding layer contains the following sequentially executed operations: two convolution operations with a 3×3 kernel, batch normalization and ReLU activation operations, and one max pooling operation with a 2×2 kernel.

[0013] The bottleneck layer connects the encoder and the decoder, wherein the embedded DA module is used to dynamically model and enhance the features of slender objects and blurred boundaries in the image;

[0014] The decoder mirrors the encoder structure and restores the feature map resolution by upsampling step by step; each decoding layer concatenates the upsampled feature map of its previous decoding layer with the corresponding encoder feature processed by the HDA module, and outputs it after convolution operation;

[0015] The skip connection processes the output features of each coding layer through the HDA module and then passes them to the corresponding decoding layer for fusion. Finally, the decoder outputs a semantic segmentation result that matches the resolution of the input image.

[0016] Furthermore, the steps of the DA module processing the input feature map X include:

[0017] The feature map X is input into two parallel branches. The first branch flattens it into a query vector. The second branch generates a value vector through a linear transformation. C represents the number of channels, and V is divided into M groups on an average basis along the channel dimension;

[0018] For each query position q, the corresponding M×K sampling offsets are predicted from Q through two linear transformations. With attention weight Where K is the number of sampling points for each attention head, and:

[0019] ;

[0020] In the formula, Indicates the first indivual , No. The query location, the first The offset vector of each sampling point; After normalization, it satisfies ;

[0021] Reference point coordinates constructed based on each query position q With normalized offset Calculate the final sampling position As shown in the following formula:

[0022] ;

[0023] At each sampling location Features are sampled from V and weighted according to attention. Perform a weighted summation to obtain the output of each attention head. As shown in the following formula: ,

[0024] in, for Attention output corresponding to reference point q;

[0025] After concatenating the outputs of all attention points, performing a linear projection transformation, and then combining it with the original query... The intermediate feature Z1 is obtained by summing the residuals and performing layer normalization. q As shown in the following formula:

[0026] ;

[0027] In the formula, This is the output of the first attention head for the q-th query position. This is the output of the m-th attention head at the q-th query position;

[0028] Finally, the output is reshaped into a two-dimensional feature map to obtain the enhanced features.

[0029] Furthermore, reference point coordinates are constructed based on each query location q. ,include:

[0030] For each query location q, construct a normalized coordinate within a unit grid, as follows:

[0031] ;

[0032] In the formula, H is the width of the feature map, and H is the height of the feature map.

[0033] Furthermore, the steps of the HDA module in processing the upsampled feature map include:

[0034] Flatten the upsampled feature map of the current stage of the decoder and use it as the query vector. For L feature maps of different scales from the encoder First, a 1×1 convolution is performed on each scale to achieve channel alignment, and then the data are flattened and concatenated to form a value vector V. ;

[0035] Based on the query location, a deformable attention mechanism is used to predict the sampling offset for features at different scales in the value vector V. With attention weight Based on the coordinates of the reference point Calculate the cross-scale sampling position with offset As shown in the following formula:

[0036] ;

[0037] At each sampling location Features are sampled from V through interpolation and weighted according to attention. After performing weighted summation and aggregating the outputs of all attention heads and scales, intermediate features are obtained through linear transformation. As shown in the following formula:

[0038] ;

[0039] The intermediate features The data is reshaped into a two-dimensional spatial form and then concatenated with the upsampled feature map Y from the current stage of the decoder to form a fused feature. As shown in the following formula:

[0040] .

[0041] Furthermore, the construction of the pre-built loss function includes the following steps:

[0042] For the input segmentation label map Each category Generate its binary image The Laplacian kernel k is used for convolution, and then the edge map of this category is extracted using the ReLU activation function. As shown in the following formula:

[0043] ;

[0044] Merge edge graphs of all categories Obtain multi-class edge masks ;

[0045] Calculate the distance from each non-edge pixel to the nearest edge pixel to generate a normalized distance map. Using the edge decay function for each pixel Assign an edge-aware weight As shown in the following formula:

[0046] ;

[0047] Where, 𝛼 is the edge weighting coefficient, For the first The distance from a pixel to its nearest edge pixel. Indicates the first Pixel loss amplification factor;

[0048] Based on the edge-aware weights For pixel-wise cross-entropy loss We then perform weighted analysis to obtain the final multi-class edge-guided loss. As shown in the following formula:

[0049] ;

[0050] In the formula, Indicates the location The real label of the place Indicates the location Based on network prediction results, .

[0051] Furthermore, the structured image is a building floor plan; the small targets include at least one of sliding doors and railings; the fine-grained structural boundaries include at least one of load-bearing wall boundaries, non-load-bearing wall boundaries, and door and window outlines.

[0052] Secondly, this invention proposes a structured image parsing system for fine-grained structural boundary and small target segmentation, used to implement the steps of the structured image parsing method described above. The system includes:

[0053] The network building module is used to build a semantic segmentation network based on an encoder-decoder architecture. It embeds a hierarchical deformable attention (HDA) module in the skip connections and a deformable attention (DA) module in the bottleneck layer.

[0054] The network training module is used to train a semantic segmentation network based on a pre-built loss function, which is constructed by cross-entropy loss and a weight graph dynamically generated based on structural boundary information in the real labels, so as to drive the network to focus on learning fine-grained structural boundaries.

[0055] The image parsing module is used to input the structured image to be parsed into the trained semantic segmentation network and output pixel-level semantic segmentation results;

[0056] The encoder is used to extract multi-scale features from the input image. The feature map output by the encoder end is processed by the DA module to output enhanced bottleneck features. The decoder is used to upsample the enhanced bottleneck features step by step, and at each level, the upsampled feature map is combined with the corresponding encoder feature map after alignment and fusion by the HDA module to improve the segmentation accuracy of fine-grained boundaries and small targets.

[0057] Furthermore, the system also includes a loss function construction module for constructing the joint loss function, including:

[0058] The edge extraction unit is used to generate binary images of each category from the input segmentation label map, and extract the edge maps of each category through Laplacian convolution and ReLU activation;

[0059] The edge fusion unit is used to merge edge maps of all categories to obtain a multi-category edge mask.

[0060] The weight generation unit is used to calculate the distance from each non-edge pixel to the nearest edge pixel, generate a normalized distance map, and assign edge-aware weights to each pixel using the edge decay function.

[0061] The loss calculation unit is used to weight the pixel-wise cross-entropy loss based on the edge-aware weights to obtain the final multi-class edge-guided loss.

[0062] The beneficial effects of this invention are as follows:

[0063] This invention effectively improves the accuracy of structured image parsing by constructing the HDA-UNet network, which integrates a hierarchical deformable attention mechanism. The network introduces a deformable attention module in the bottleneck layer to enhance the feature modeling ability for slender structures and blurred boundaries. In the skip connections, a hierarchical deformable attention module is used to achieve accurate alignment and fusion of cross-scale features. Furthermore, a multi-class edge-guided loss function is combined to strengthen the model's attention to boundary regions, thereby significantly improving the segmentation performance of fine-grained structural boundaries and small targets. Attached Figure Description

[0064] Figure 1 A flowchart illustrating a structured image parsing method for segmenting fine-grained structural boundaries and small targets, provided in one embodiment of this application;

[0065] Figure 2 This is another flowchart of a structured image parsing method for segmenting fine-grained structural boundaries and small targets provided in one embodiment of this application;

[0066] Figure 3 This is a diagram illustrating the overall architecture of the HDA-UNet model proposed in one embodiment of this application.

[0067] Figure 4 This is a flowchart of the processing of a deformable attention DA module proposed in one embodiment of this application;

[0068] Figure 5 This is a flowchart illustrating the multi-scale feature fusion method of the hierarchical deformable attention (HDA) module proposed in one embodiment of this application.

[0069] Figure 6 This is an example of some images from the FloorStruct1025 dataset in this application. The upper half of the image is the original image of the residential floor plan, and the lower half is the labeled image of the residential floor plan.

[0070] Figure 7 This is a comparative illustration of the visualization segmentation results of HDA-UNet on five residential floor plans in the FloorStruct1025 dataset in this application. Detailed Implementation

[0071] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0072] Example 1

[0073] Please see Figures 1-5This invention proposes a structured image parsing method for segmenting fine-grained structural boundaries and small objects. The method first constructs a semantic segmentation network based on an encoder-decoder architecture. The encoder extracts multi-scale features from the input image through four levels of downsampling, while the decoder progressively restores spatial resolution through symmetrical upsampling operations. In the network architecture, a deformable attention (DA) module is embedded in the bottleneck layer to enhance the deepest features output by the encoder, focusing on modeling slender walls and blurred boundary features in the image. Simultaneously, a hierarchical deformable attention (HDA) module is embedded in all skip connections to perform cross-scale spatial alignment of the feature maps from each stage of the encoder with the corresponding upsampled features from the decoder before feature fusion.

[0074] The semantic segmentation network is trained using a pre-built multi-class edge-guided loss function. This loss function generates an edge-aware weight map by calculating the normalized distance from each pixel in the ground truth label to the nearest edge, and then combines it with the standard cross-entropy loss function to form a dynamically weighted loss calculation method. This design allows the network to focus more on learning the pixels in the boundary region during training, especially the feature representation of fine-grained structural boundaries.

[0075] During the inference phase, structured images such as building floor plans to be parsed are input into the trained semantic segmentation network. The network, through the forward propagation process of encoder-DA module-decoder, combined with the multi-scale feature alignment capability of HDA module, finally outputs pixel-level semantic segmentation results with the same resolution as the input image, achieving accurate segmentation of building components such as load-bearing walls, non-load-bearing walls, doors and windows, sliding doors, and railings.

[0076] Further preferably, the semantic segmentation network is the HDA-UNet network, wherein:

[0077] The encoder consists of four sequentially connected coding layers. Each coding layer contains the following sequentially executed operations: two convolution operations with a 3×3 kernel, batch normalization and ReLU activation operations, and one max pooling operation with a 2×2 kernel.

[0078] The bottleneck layer connects the encoder and decoder, where the embedded DA module is used to dynamically model and enhance features of elongated objects and blurred boundaries in the image;

[0079] The decoder mirrors the encoder structure, recovering the feature map resolution through step-by-step upsampling; each decoding layer concatenates the upsampled feature map of its previous decoding layer with the corresponding encoder feature processed by the HDA module, and outputs it after convolution operation;

[0080] Skip Connections process the output features of each coding layer through the HDA module and then pass them to the corresponding decoding layer for fusion. Finally, the decoder outputs a semantic segmentation result that matches the resolution of the input image.

[0081] Please see Figure 3 The HDA-UNet model consists of four parts: Encoder, Bottleneck, Decoder, and Skip Connection, each corresponding to a color region. For a residential floor plan X, its dimensions are W×H×3, representing the image's width, height, and number of channels. For semantic segmentation, X is first input to the encoder, which extracts multi-scale features from the input image through four encoding layers. Each encoding layer contains two convolutional operations (kernel size: 3×3) + batch normalization + ReLU activation (Conv2d+BN+ReLU) and one max pooling operation (kernel size: 2×2). After passing through the entire encoder, the number of channels increases from 3... The image size is increased to 512, and the image size is reduced from W×H to W / 16×H / 16, capturing shallow texture, mid-level shape, and high-level semantic features respectively. The Bottleneck connects the encoder and decoder. To solve the problem of fixed receptive field of convolutional kernels in the encoding stage of existing models, a DA module is introduced to dynamically model and enhance the features of complex regions such as slender objects and blurred boundaries in residential floor plans. The decoder mirrors the encoder structure. By upsampling features from the encoder, the feature map is gradually restored to the original image size. Each decoding layer receives the feature map from the HDA module and the feature map from the previous decoding layer. After concatenation and convolution operations, a semantic segmentation result matching the input resolution is finally generated. Skip connections fuse feature information from the encoding stage into the decoding stage. Unlike most U-Net series methods that directly connect the encoding layer and the corresponding decoding layer features, this application embeds a specially designed HDA module in each skip connection layer. Through dynamic feature alignment and fusion across scales and levels, the key features of feature maps from different decoding layers are continuously enhanced.

[0082] Please see Figure 4 First, the feature map X is fed into two parallel branches. The first branch flattens X along the channel dimension to form the query vector Q∈R. C Where C is the number of channels; the other branch also uses a linear transformation of X according to the channel dimension to generate a value vector V∈R. C Furthermore, they are evenly divided into multiple groups along the channel dimension, called headi, i=1, 2, ..., m, which can effectively capture important features from different angles and suppress noise through adaptive and variable convolution range and shape.

[0083] The steps of the DA module in processing the input feature map X include:

[0084] The feature map X is input into two parallel branches. The first branch flattens it into a query vector. The second branch generates a value vector through a linear transformation. C represents the number of channels, and V is divided into M groups on an average basis along the channel dimension;

[0085] Construct reference point coordinates based on each query location q. ,include:

[0086] For each query location q, construct a normalized coordinate within a unit grid, as follows:

[0087] ;

[0088] In the formula, H is the width of the feature map, and H is the height of the feature map.

[0089] For each query position q, each query is predicted using two linear transformations. Sampling offset coordinates , and corresponding attention weight Predict the corresponding M×K sampling offsets from Q. With attention weight Where K is the number of sampling points for each attention head, and:

[0090] ;

[0091] In the formula, Indicates the first indivual , No. The query location, the first The offset vector of each sampling point; After normalization, it satisfies ;

[0092] Based on each query position Constructed reference point coordinates With normalized offset Calculate the final sampling position As shown in the following formula:

[0093] ;

[0094] At each sampling location Features are sampled from V and weighted according to attention. Perform a weighted summation to obtain the output of each attention head. As shown in the following formula: ,

[0095] in, for Corresponding reference point Attention output;

[0096] After concatenating the outputs of all attention points, performing a linear projection transformation, and then combining it with the original query... The intermediate feature Z1 is obtained by summing the residuals and performing layer normalization. q As shown in the following formula:

[0097] ;

[0098] In the formula, This is the output of the first attention head for the q-th query position. For the first The attention head is for the first Output results for each query position;

[0099] Finally, the output is reshaped into a two-dimensional feature map to obtain the enhanced features.

[0100] Understandably, the core of the DA module lies in leveraging learnable convolutional kernels to sample locations, thereby enhancing the ability to model spatial structures. In building floor plans, structural boundaries such as doors, windows, and railings often appear as slender, slanted, or irregular lines. Traditional convolutional methods often suffer from insufficient expressive power when modeling these forms, while the DA module effectively improves sensitivity to such structures through dynamic sampling and weighted aggregation.

[0101] In this embodiment, the HDA module is used to dynamically sample and fuse features from different scales of the coding layer. This module is embedded in the skip connection path. Its design concept is to combine multi-scale receptive fields with deformable attention mechanisms to improve the consistency of cross-scale features in spatial location and semantic expression.

[0102] In the network's decoder, the HDA module accepts the output features of two- or four-layer encoders (such as...). Figure 3 Hierarchical multi-scale feature fusion can fully utilize the spatial details in shallow features and the semantic information in deep features, while avoiding spatial structure distortion caused by direct upsampling.

[0103] Please see Figure 5 Furthermore, the steps for the HDA module to process the upsampled feature map include:

[0104] Flatten the upsampled feature map of the current stage of the decoder and use it as the query vector. For L feature maps of different scales from the encoder First, a 1×1 convolution is performed on each scale to achieve channel alignment, and then the data are flattened and concatenated to form a value vector V. :

[0105] ;

[0106] in, Indicated in scale The feature map is aligned with the channel above. and These represent splicing and flattening operations, respectively.

[0107] Based on the query location, a deformable attention mechanism is used to predict the sampling offset for features at different scales in the value vector V. With attention weight Based on the coordinates of the reference point Calculate the cross-scale sampling position with offset As shown in the following formula:

[0108] ;

[0109] At each sampling location Features are sampled from V through interpolation and weighted according to attention. After performing weighted summation and aggregating the outputs of all attention heads and scales, intermediate features are obtained through linear transformation. As shown in the following formula:

[0110] ;

[0111] intermediate features The data is reshaped into a two-dimensional spatial form and then concatenated with the upsampled feature map Y from the current stage of the decoder to form a fused feature. As shown in the following formula:

[0112] ;

[0113] Finally, the fusion result, along with the upsampled output Y, is sent to the subsequent decoder module to complete the information integration.

[0114] Understandably, the aforementioned multi-scale feature aggregation method can perceive structural boundaries and details at different spatial granularities, helping the model capture morphological features in sparse categories (such as sliding doors and balustrades). Furthermore, the introduction of this module significantly improves the synergy and structural alignment capabilities between cross-layer features, enhances the model's perception and representation of structures at different spatial scales, and also significantly improves the recognition performance for categories with few samples and blurred boundaries.

[0115] In this embodiment, to further improve the accuracy of identifying building structure boundaries, a multi-class edge-guided loss function (MCEGL) is designed. This loss function introduces an edge-aware weighting mechanism on top of the traditional cross-entropy, thereby guiding the HDA-UNet model to pay more attention to fine-grained edge regions in the residential floor plan, such as wall edges and door and window outlines. This loss function mainly includes three key steps: edge extraction, distance attenuation modeling, and weighted loss calculation.

[0116] A further preferred embodiment of the pre-constructed loss function includes the following steps:

[0117] Edge extraction: This involves processing the input segmentation label map. Each category Generate its binary image The Laplacian kernel k is used for convolution, and then the edge map of this category is extracted using the ReLU activation function. As shown in the following formula:

[0118] ;

[0119] Merge edge graphs of all categories Obtain multi-class edge masks .

[0120] Distance weight modeling: To extend the edge perception range, the distance from each non-edge pixel to the nearest edge pixel is further calculated, generating a normalized distance map. An edge-aware weight is assigned to each pixel (i, j) using an edge decay function. As shown in the following formula:

[0121] ;

[0122] in, These are the edge weighting coefficients. For the first The distance from a pixel to its nearest edge pixel. Indicates the first The loss amplification factor of a pixel.

[0123] Edge-weighted cross-entropy loss: based on edge-aware weights For pixel-wise cross-entropy loss We then perform weighted analysis to obtain the final multi-class edge-guided loss. As shown in the following formula:

[0124] ;

[0125] In the formula, Indicates the location The real label of the place Indicates the location Based on network prediction results, .

[0126] Understandably, this loss function allows the model to focus more effectively on edge structures in architectural images during training, thereby improving segmentation accuracy and boundary clarity, especially in structures such as doors, windows, and small walls.

[0127] Further preferably, the structured image is an architectural floor plan, specifically including floor plans of apartments, villas, and high-rise residential buildings. The small target refers to architectural components that occupy a small pixel area and have a slender shape in the image, including but not limited to at least one of sliding doors, railings, pipe shafts, and decorative lines. The fine-grained structural boundary refers to the boundary areas between different architectural components and the contour details of the components themselves, specifically including at least one of the material boundaries and geometric intersections between load-bearing and non-load-bearing walls, the precise outer contours of doors and windows, and the topological structure of wall corners and connections.

[0128] Example 2

[0129] Based on the same inventive concept, another specific embodiment of the present invention proposes a structured image parsing system for fine-grained structural boundary and small target segmentation, used to implement the steps of the structured image parsing method as described in Embodiment 1. The system includes:

[0130] The network building module is used to build a semantic segmentation network based on an encoder-decoder architecture. It embeds a hierarchical deformable attention (HDA) module in the skip connections and a deformable attention (DA) module in the bottleneck layer.

[0131] The network training module is used to train a semantic segmentation network based on a pre-built loss function. The loss function is constructed by cross-entropy loss and a weight graph dynamically generated based on structural boundary information in the real labels, so as to drive the network to focus on learning fine-grained structural boundaries.

[0132] The image parsing module is used to input the structured image to be parsed into the trained semantic segmentation network and output pixel-level semantic segmentation results;

[0133] The encoder is used to extract multi-scale features from the input image. The feature map output by the encoder is processed by the DA module to output the enhanced bottleneck features. The decoder is used to upsample the enhanced bottleneck features step by step, and at each level, the upsampled feature map is combined with the corresponding encoder feature map after alignment and fusion by the HDA module to improve the segmentation accuracy of fine-grained boundaries and small targets.

[0134] The system also includes a loss function construction module for constructing the joint loss function, including:

[0135] The edge extraction unit is used to generate binary images of each category from the input segmentation label map, and extract the edge maps of each category through Laplacian convolution and ReLU activation;

[0136] The edge fusion unit is used to merge edge maps of all categories to obtain a multi-category edge mask.

[0137] The weight generation unit is used to calculate the distance from each non-edge pixel to the nearest edge pixel, generate a normalized distance map, and assign edge-aware weights to each pixel using the edge decay function.

[0138] The loss calculation unit is used to weight the pixel-wise cross-entropy loss based on the edge-aware weights to obtain the final multi-class edge-guided loss.

[0139] For specific limitations regarding the structured image parsing system for fine-grained structural boundary and small target segmentation, please refer to the limitations of the structured image parsing method for fine-grained structural boundary and small target segmentation above, which will not be repeated here. It should be noted that each module in the above parsing system corresponds to multiple steps in implementing the above parsing method. The instances and application scenarios implemented by multiple modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above.

[0140] To more clearly illustrate the present invention and its advantages, the method provided by the present invention will be further explained below with reference to specific experimental examples and related partial figures.

[0141] This application uses the following experimental steps to verify the effectiveness of the proposed method (HDA-UNet) in residential floor plan structure segmentation. Specifically, this includes the construction and preprocessing of the FloorStruct1025 dataset, experimental setup and evaluation metrics, comparative experiments with existing mainstream methods, module ablation experiments, and loss function comparison experiments.

[0142] 1. Dataset

[0143] Currently available residential floor plan datasets, such as CVC-FP, CubiCasa, and ROBIN, suffer from limitations such as limited quantity, lack of detailed annotations, or relatively monotonous color styles. To address these issues, this application constructs the dataset FloorStruct1025 to support the refined identification and modeling of multi-category elements in residential floor plans.

[0144] Furthermore, to more intuitively compare the characteristics of existing publicly available residential floor plan datasets with the FloorStruct1025 dataset constructed in this application, Table 1 summarizes the key attributes of each dataset, including the number of images, annotation type, annotation detail, drawing style, and its limitations. As can be seen from the table, existing datasets are insufficient in terms of scale, annotation granularity, or style diversity, while FloorStruct1025 has a significant advantage in terms of category richness and relevance to practical applications.

[0145] Table 1. Commonly Used Datasets for Plane Map Analysis

[0146] ;

[0147] like Figure 6 As shown, the original floor plans in the FloorStruct1025 dataset are mainly derived from online residential floor plans. The drawings are diverse in style and cover a variety of building types, such as apartments, villas, and high-rise residences. They are characterized by complex structures, diverse line styles, and inconsistent color representations, fully preserving the noise and chaos present in actual drawings, thus creating a more challenging testing environment.

[0148] All images were manually annotated pixel-by-pixel using the Labelme tool and then proofread and corrected by professionals to ensure semantic consistency and annotation quality. The dataset contains six common building structural elements: load-bearing walls, non-load-bearing walls, windows, side-opening doors, sliding doors, and railings. Wall elements are more abundant, while sliding doors and railings are relatively scarce, indicating a significant class imbalance. This places higher demands on the segmentation model in terms of sparse object recognition, class differentiation, and boundary modeling. The final dataset was divided into training and test sets at a ratio of 85% and 15%, respectively.

[0149] FloorStruct1025 contains various types of architectural structural elements, with significant stylistic differences between images, such as wall textures, door and window representations, line thickness, and color distribution. Furthermore, similar objects (such as sliding doors and railings) exhibit varying forms and blurred boundaries across different drawings, further increasing the difficulty of semantic segmentation. These complexities fully demonstrate the representativeness and challenge of this dataset in real-world scenarios.

[0150] 2. Experimental Setup and Evaluation Indicators

[0151] All experiments were conducted on a computer equipped with an NVIDIA GeForce RTX 3090 (24 GB VRAM) GPU. During training, images were uniformly scaled to 512×512 resolution before being input into the network. Network parameters were randomly initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. The Adam optimizer was used during training, with an initial learning rate of 1e-4, which was gradually decayed to a minimum of 1e-6 using a Cosine Annealing strategy. No freeze policy was used during the entire training process; the network was trained end-to-end from start to finish. To ensure the reproducibility of the training process, a fixed random seed of 11 was used.

[0152] For evaluation metrics, the Intersection over Union (IoU), Mean IoU (mIoU), and Accuracy, commonly used in image segmentation tasks, are employed as performance metrics for the model. They are defined as follows:

[0153] Category IoU, for a given category c, is defined as the ratio of the intersection to the union of the predicted and actual regions for that category. The formula is as follows:

[0154] ;

[0155] in, : will the first The correct prediction for class is the first. The number of pixels in the class (True Positive); : Predict other types of errors as the first The number of pixels in the class; : will the first The number of pixels that were incorrectly predicted as belonging to another class.

[0156] mIoU is the average IoU across all classes, used to evaluate the overall performance of the segmentation model.

[0157] ;

[0158] Where N is the number of categories.

[0159] Furthermore, pixel accuracy reflects the model's overall prediction accuracy across all pixels, and is calculated using the following formula:

[0160] ;

[0161] 3. Experimental Results

[0162] 3.1 Ablation Experiment and Comparative Experiment

[0163] To evaluate the impact of each new module in HDA-UNet, this application designed three sets of ablation and comparison experiments for verification.

[0164] First, the effectiveness of the proposed HDA in building structure segmentation tasks is verified. This application is compared with several classic methods combining attention mechanisms with UNet published in recent years, including ATT-UNet, MTUNet, SA-UNet, META-UNet, and CSwinUNet. All models are trained on FloorStruct1025 for 100 epochs and achieve stable performance to ensure the fairness and consistency of the experiment.

[0165] Table 2. Comparison of semantic segmentation performance under different attention modules

[0166] ;

[0167] As can be seen from the table, although attention mechanisms such as MTUNet and SA-UNet perform well in other segmentation tasks, they have limited performance in building structure segmentation tasks aimed at fine-grained boundary recognition, making it difficult to meet the requirements for accurate recognition of complex structures. In contrast, the HDA module proposed in this application, while introducing multi-scale information, enhances the perception of local shapes and structural boundaries through a deformable mechanism. Ultimately, HDA-UNet outperforms other methods in both mIoU and Accuracy metrics, demonstrating its significant advantage in extracting features from complex building structures.

[0168] To further verify the contribution of each component of HDA-UNet to the overall performance, we sequentially selected one component from each of the three components and added it to the base UNet: DA, HDA, and MCEGL, forming three comparison methods: "+HDA", "+DA", and "+MCEGL". As shown in Table 3, DA and HDA are newly added components, while MCEGL is used to replace the baseline loss function (cross-entropy loss) in UNet. Each experiment was trained for 100 epochs on FloorStruct1025 to ensure the comparability of the results.

[0169] Table 3 Ablation Experiment Results

[0170] ;

[0171] As shown in Table 3, introducing either the DA module or HDA alone improves the model's structural recognition ability to some extent, but there is still a significant gap compared to the complete model. In particular, when the three modules are combined to form the complete HDA-UNet, the model's mIoU is significantly improved to 81.09%, which is more than 29 percentage points higher than the basic model UNet. This fully demonstrates that the modules have a good synergistic effect in structural detail modeling, spatial perception, and boundary enhancement.

[0172] Finally, the effectiveness of the proposed Multi-Class Edge Enhancement Loss Function (MCEGL) in building structure segmentation was verified. Using UNet as the backbone network, the performance of various classic loss functions and their combinations on building structure segmentation tasks was compared under the same training conditions.

[0173] Table 4. Comparison of semantic segmentation performance under different loss functions

[0174] ;

[0175] As shown in the table, the classic cross-entropy loss (CE) performs well in overall accuracy, but it has shortcomings in identifying structural boundaries and small sample classes. Adding Focal loss (Lrecognition) or Dice Loss slightly improves mIoU, indicating some improvement in handling class imbalance. Introducing MCEGL significantly improves mIoU, raising it to 69.22% while maintaining high overall accuracy, effectively enhancing the model's ability to distinguish multi-class structural edges.

[0176] 3.2 Comparative Experiment with Existing Methods

[0177] To further validate the effectiveness of the proposed HDA-Unet model in the task of structural segmentation of building floor plans, this application selects methods used in this field, including UNet, VGG-UNet, and DeepLabv3+, as the basic comparison models. Meanwhile, due to the scarcity of high-quality publicly available segmentation methods for residential floor plan research, this application introduces representative methods in the field of semantic segmentation in recent years as supplementary comparisons, covering convolutional neural networks (such as PSPNet, HRNet, and UNet++) and hybrid models combining the advantages of CNN and Transformer (such as SegFormer, TransUNet, and Swin-UNet). Considering the similarity in structural organization and boundary features between remote sensing image segmentation and residential floor plan segmentation, this application also introduces networks that have performed well in the field of remote sensing image segmentation in recent years (such as PyramidMamba, RS3Mamba, and CMTFNet). These selected comparisons will provide different comparative perspectives to validate the methods provided in this application. All models were trained for 300 epochs on the FloorStruct1025 dataset and achieved stable performance. Evaluation metrics include IoU for each category and its average mIoU, used to comprehensively measure the performance of different methods in structure segmentation tasks. Experimental results are shown in Table 5.

[0178] Table 5. Comparison of semantic segmentation performance of different models (FloorStruct1025 dataset)

[0179] ;

[0180] The results show that the proposed HDA-UNet exhibits leading segmentation performance across all categories, particularly in the "sliding door" and "balustrade" categories, where structural boundaries are blurred and shapes vary significantly. Compared to the traditional UNet, HDA-UNet improves mIoU by 17.13 percentage points; even compared to the current state-of-the-art remote sensing segmentation model CMTFNet, it still achieves a 5 percentage point improvement. These results demonstrate that the hierarchical deformable attention mechanism introduced by HDA-UNet effectively enhances the decoder's ability to model structural differences, significantly improves segmentation performance for complex structural regions, and validates its strong adaptability and generalization capabilities in planar image semantic parsing.

[0181] Furthermore, this application also compares the number of parameters and floating-point operations (FLOPs) of different methods from the perspective of model complexity. The results are shown in Table 6, and all measurements were performed with an input size of 224 × 224.

[0182] Table 6. Comparison of parameter size and FLOP of different models, measured at a resolution of 224 × 224.

[0183] ;

[0184] As shown in the table, HDA-UNet has a parameter size of 31.14M, which is at a moderate level. It is significantly lower than models such as SegFormer (84.60M) and TransUNet (77.37M), but also better than several remote sensing segmentation methods such as PyramidMamba (109.99M). Although HDA-UNet's FLOPs (60.28G) are relatively high, mainly due to the hierarchical deformable attention module introduced in the decoder stage, it is precisely these mechanisms that significantly improve its ability to model complex structural boundaries and small-scale targets.

[0185] Finally, the visualization results of HDA-UNet on the dataset are as follows: Figure 7 As shown. Five residential floor plans were randomly selected from these for display. Figure 7 From left to right, the images show the original image, the labeled image, and the network segmentation images of Vgg-Une, CMTFNe, and HDA-UNet, respectively. The results show that the HDA-UNet network can restore the structural outline well, with most components having clear edges and accurate category recognition, further verifying its application potential in practical tasks.

[0186] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0187] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0188] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A structured image parsing method for fine-grained structural boundaries and small target segmentation, characterized in that, The method includes: Construct a semantic segmentation network based on an encoder-decoder architecture, embedding a hierarchical deformable attention (HDA) module in the skip connections and a deformable attention (DA) module in the bottleneck layer; The semantic segmentation network is trained based on a pre-constructed loss function, which is constructed by cross-entropy loss and a weight graph dynamically generated based on structural boundary information in the real labels, so as to drive the network to focus on learning fine-grained structural boundaries. The structured image to be parsed is input into the trained semantic segmentation network, which outputs pixel-level semantic segmentation results. The encoder extracts multi-scale features from the input image. The feature map output by the encoder is processed by the DA module to output enhanced bottleneck features. The decoder upsamples the enhanced bottleneck features step by step and combines the upsampled feature map with the corresponding encoder feature map after alignment and fusion by the HDA module at each level to improve the segmentation accuracy of fine-grained boundaries and small targets. The semantic segmentation network is the HDA-UNet network, wherein: The encoder consists of four sequentially connected coding layers. Each coding layer contains the following sequentially executed operations: two convolution operations with a 3×3 kernel, batch normalization and ReLU activation operations, and one max pooling operation with a 2×2 kernel. The bottleneck layer connects the encoder and the decoder, wherein the embedded DA module is used to dynamically model and enhance the features of slender objects and blurred boundaries in the image; The decoder mirrors the encoder structure and restores the feature map resolution by upsampling step by step; each decoding layer concatenates the upsampled feature map of its previous decoding layer with the corresponding encoder feature processed by the HDA module, and outputs it after convolution operation; The skip connection processes the output features of each coding layer through the HDA module and then passes them to the corresponding decoding layer for fusion. Finally, the decoder outputs a semantic segmentation result that matches the resolution of the input image.

2. The structured image parsing method for fine-grained structural boundary and small target segmentation according to claim 1, characterized in that, The steps of the DA module in processing the input feature map X include: The feature map X is input into two parallel branches. The first branch flattens it into a query vector. The second branch generates a value vector through a linear transformation. C represents the number of channels, and V is divided into M groups on an average basis along the channel dimension; For each query position q, the corresponding M×K sampling offsets are predicted from Q through two linear transformations. With attention weight Where K is the number of sampling points for each attention head, and: ； In the formula, Indicates the first The offset vector of the head, the qth query position, and the kth sampling point; After normalization, it satisfies ; Reference point coordinates constructed based on each query position q With normalized offset Calculate the final sampling position As shown in the following formula: ； At each sampling location Features are sampled from V and weighted according to attention. Perform a weighted summation to obtain the output of each attention head. As shown in the following formula: ； in, for head m Attention output corresponding to reference point q; After concatenating the outputs of all attention points, performing a linear projection transformation, and then combining it with the original query... The intermediate feature Z1 is obtained by summing the residuals and performing layer normalization. q As shown in the following formula: ； In the formula, This is the output of the first attention head for the q-th query position. This is the output of the m-th attention head at the q-th query position; Finally, the output is reshaped into a two-dimensional feature map to obtain the enhanced features.

3. The structured image parsing method for fine-grained structural boundary and small target segmentation according to claim 2, characterized in that, Construct reference point coordinates based on each query location q. ,include: For each query location q, construct a normalized coordinate within a unit grid, as follows: ； In the formula, H is the width of the feature map, and H is the height of the feature map.

4. The structured image parsing method for fine-grained structural boundary and small target segmentation according to claim 3, characterized in that, The steps of the HDA module in processing the upsampled feature map include: Flatten the upsampled feature map of the current stage of the decoder and use it as the query vector. For L feature maps of different scales from the encoder First, a 1×1 convolution is performed on each scale to achieve channel alignment, and then the data are flattened and concatenated to form a value vector V. ; Based on the query location, a deformable attention mechanism is used to predict the sampling offset for features at different scales in the value vector V. With attention weight Based on the coordinates of the reference point Calculate the cross-scale sampling position with offset As shown in the following formula: ； At each sampling location Features are sampled from V through interpolation and weighted according to attention. After performing weighted summation and aggregating the outputs of all attention heads and scales, intermediate features are obtained through linear transformation. As shown in the following formula: ； The intermediate features The data is reshaped into a two-dimensional spatial form and then concatenated with the upsampled feature map Y from the current stage of the decoder to form a fused feature. As shown in the following formula: 。 5. The structured image parsing method for fine-grained structural boundary and small target segmentation according to claim 4, characterized in that, The construction of the pre-built loss function includes the following steps: For the input segmentation label map Each category Generate its binary image The Laplacian kernel k is used for convolution, and then the edge map of this category is extracted using the ReLU activation function. As shown in the following formula: ； Merge edge graphs of all categories Obtain multi-class edge masks ; Calculate the distance from each non-edge pixel to the nearest edge pixel to generate a normalized distance map. An edge-aware weight is assigned to each pixel (i, j) using an edge decay function. As shown in the following formula: ； Where, 𝛼 is the edge weighting coefficient, Let be the distance from the i-th pixel to the nearest edge pixel. Indicates the first Pixel loss amplification factor; Based on the edge-aware weights The pixel-wise cross-entropy loss CE(·) is weighted to obtain the final multi-class edge-guided loss. As shown in the following formula: ； In the formula, This represents the actual label at position (i, j). This represents the network prediction result at position (i, j). .

6. The structured image parsing method for fine-grained structural boundary and small target segmentation according to claim 1, characterized in that, The structured image is a building floor plan; the small targets include at least one of sliding doors and railings; the fine-grained structural boundaries include at least one of load-bearing wall boundaries, non-load-bearing wall boundaries, and door and window outlines.

7. A structured image parsing system for fine-grained structural boundary and small target segmentation, used to implement the steps of the structured image parsing method as described in any one of claims 1-6, characterized in that, The system includes: The network building module is used to build a semantic segmentation network based on an encoder-decoder architecture. It embeds a hierarchical deformable attention (HDA) module in the skip connections and a deformable attention (DA) module in the bottleneck layer. The network training module is used to train a semantic segmentation network based on a pre-built loss function, which is constructed by cross-entropy loss and a weight graph dynamically generated based on structural boundary information in the real labels, so as to drive the network to focus on learning fine-grained structural boundaries. The image parsing module is used to input the structured image to be parsed into the trained semantic segmentation network and output pixel-level semantic segmentation results; The encoder is used to extract multi-scale features from the input image. The feature map output by the encoder end is processed by the DA module to output enhanced bottleneck features. The decoder is used to upsample the enhanced bottleneck features step by step, and at each level, the upsampled feature map is combined with the corresponding encoder feature map after alignment and fusion by the HDA module to improve the segmentation accuracy of fine-grained boundaries and small targets.

8. The structured image parsing system for fine-grained structural boundary and small target segmentation according to claim 7, characterized in that, The system also includes a loss function construction module for constructing a loss function, including: The edge extraction unit is used to generate binary images of each category from the input segmentation label map, and extract the edge maps of each category through Laplacian convolution and ReLU activation; The edge fusion unit is used to merge edge maps of all categories to obtain a multi-category edge mask. The weight generation unit is used to calculate the distance from each non-edge pixel to the nearest edge pixel, generate a normalized distance map, and assign edge-aware weights to each pixel using the edge decay function. The loss calculation unit is used to weight the pixel-wise cross-entropy loss based on the edge-aware weights to obtain the final multi-class edge-guided loss.

Images