End-to-end wireframe analysis method and system based on dynamic deformable attention mechanism

Abstract

The invention discloses an end-to-end wireframe analysis method and system based on a dynamic deformable attention mechanism, and aims to realize scene understanding and reconstruction by detecting line segments and connection points. The specific implementation process is as follows: inputting an image of a building wireframe to be extracted into a trained wireframe analysis model, and outputting a wireframe analysis result of the image. The wireframe analysis model integrates a feature extraction and enhancement module based on Swin Transform and a line segment reasoning module based on a dynamic deformable attention mechanism, the feature extraction and enhancement module can efficiently capture a global structure and local details, and the dynamic deformable attention mechanism is focused on a sparse key area related to a line segment, so that the overall structure and local details are effectively captured. And the feature extraction precision and efficiency are further improved. In addition, an iterative optimization strategy is introduced, and the accuracy and stability of detection are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision technology, and in particular relates to an end-to-end wireframe parsing method and system based on a dynamic deformable attention mechanism. Background Art

[0002] Wireframes (including line segments and their connecting points) are the basis for low-level feature extraction and 3D scene reconstruction, playing a vital role in computer vision tasks such as camera pose estimation, image matching, and visual SLAM. Wireframe parsing aims to analyze visual features and contextual information in images, detect meaningful and significant wireframes, and reveal and restore the geometric structure of scenes or objects through the coupling relationship between line segments and their intersections.

[0003] The wireframe parsing task typically consists of two main stages: feature extraction and line segment inference. During wireframe parsing, the line segments within the wireframe often exhibit significant scale variations: long segments spanning the entire image, while short segments are confined to small regions. This multi-scale nature requires the parsing algorithm to simultaneously address both global and local features, capturing the overall relationships of long segments while preserving the local details of short segments. However, existing wireframe parsing methods have not yet effectively addressed this issue.

[0004] Existing wireframe parsing methods can be divided into two categories. The first category, CNN-based methods, often rely on complex post-processing or multi-step processes to associate points and lines, which limits their performance and application effectiveness. Furthermore, these methods struggle to balance modeling the relationships between full-length line segments while preserving the details of short, localized line segments, and perform poorly when dealing with varying line scales. Some methods primarily target the feature extraction stage, introducing global features by improving the feature extraction backbone network to improve performance. However, these methods rely on prior knowledge such as frequency-domain geometry and the Hough transform, increasing computational complexity and information redundancy, and still fail to effectively address the problem of multi-scale feature extraction. The second category, Transformer-based methods, focuses on the line segment inference stage and utilizes global dense modeling to process image contextual information, but these approaches still have limitations. Since important regions in wireframe parsing are often sparse, achieving multi-scale extraction incurs extremely high computational and memory overhead, which limits multi-scale feature extraction and leads to poor performance in small object detection.

[0005] Therefore, designing a wireframe parsing method that does not rely on prior information and can efficiently extract multi-scale features to improve wireframe parsing performance is a technical problem that needs to be solved urgently. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to achieve efficient multi-scale feature extraction without increasing computational and memory overhead, without relying on prior information, thereby solving the problem of significant scale changes in line segments in wireframe parsing tasks, and providing an end-to-end wireframe parsing method and system based on a dynamic deformable attention mechanism.

[0007] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0008] In the first aspect, the present invention provides an end-to-end wireframe parsing method based on a dynamic deformable attention mechanism, the specific process of which is as follows:

[0009] An image of a building wireframe to be extracted is input into a trained wireframe parsing model. First, the feature extraction and enhancement module performs multi-scale feature extraction and feature enhancement on the input image to generate a set of enhanced multi-scale feature maps. The enhanced multi-scale feature maps are then input into the line segment inference module. After being processed by a deformable encoder and a deformable decoder in sequence, optimized line segment entities are obtained. The optimized line segment entities are then passed through a feedforward network to predict the coordinates and confidence scores of each line segment entity and output the wireframe parsing result of the image. The deformable encoder is obtained by replacing the self-attention mechanism in the Transformer encoder with a dynamic deformable attention mechanism. The deformable decoder is obtained by replacing the cross-attention mechanism in the Transformer decoder with a dynamic multi-scale deformable attention mechanism. During training, the deformable decoder uses an iterative optimization mechanism to gradually optimize the line segment representation.

[0010] In the feature extraction and enhancement module, the input image is first divided into non-overlapping patches of fixed size, and then the obtained patches are mapped to a high-dimensional feature space through a linear embedding layer. Subsequently, the patch-based feature representation is downsampled to different resolutions, and a Swin Transformer block is used at each resolution to extract feature maps of the corresponding scale and form a set of initial multi-scale feature maps. The initial multi-scale feature maps are then input into the feature enhancement module for feature enhancement, and finally the enhanced multi-scale feature maps are generated.

[0011] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0012] As a preferred embodiment of the above-mentioned first aspect, in the feature enhancement module, the initial multi-scale feature map is composed of a first feature map, a second feature map, a third feature map and a fourth feature map with resolution from high to low. The first feature map is first processed by the first channel and the spatial convolution block, and the second feature map and the processed first feature map are spliced ​​to obtain an enhanced second feature map and input it into the second channel and the spatial convolution block to obtain the processed second feature map. The third feature map and the processed second feature map are spliced ​​to obtain an enhanced third feature map and input it into the third channel and the spatial convolution block to obtain the processed third feature map. The fourth feature map and the processed third feature map are spliced ​​as the input of the depth-separable partial self-attention block, and the depth-separable partial self-attention block outputs the enhanced fourth feature map. The first feature map is used as the enhanced first feature map, and finally the enhanced multi-scale feature map is composed of the enhanced first feature map, the enhanced second feature map, the enhanced third feature map and the enhanced fourth feature map.

[0013] As a preferred embodiment of the first aspect, each channel and spatial convolution block is composed of a depthwise separable convolution layer and a spatial convolution layer cascaded in sequence.

[0014] As a preferred embodiment of the above-mentioned first aspect, the depth-separable partial self-attention block is composed of a depth-separable convolutional layer, a PSA module and a depth-separable convolutional layer cascaded in sequence, and the PSA module is composed of N PSA layers cascaded in sequence; in each PSA layer, the input feature map is processed by the multi-head self-attention module and the feedforward network in sequence to obtain an intermediate feature map, and the intermediate feature map is spliced ​​with the input feature map and the spliced ​​feature map is used as the output feature map.

[0015] As a preferred embodiment of the first aspect, the wireframe parsing model is pre-trained using labeled training data.

[0016] As a preferred embodiment of the first aspect, when training the wireframe parsing model, a weighted sum of classification loss and distance loss is used as the total loss function.

[0017] Furthermore, the classification loss The calculation formula is as follows:

[0018]

[0019] in, represents the indicator function; σ(i) represents the optimal matching target index of the prediction index i; M represents the number of positive prediction indices; α1 and α2 are the weighting coefficients of positive and negative samples, respectively, which are used to balance the contribution of positive and negative samples in the loss function; p (i)represents the prediction confidence of index i; γ represents the adjustment factor, which is used to control the sensitivity of the loss function to the prediction confidence, and is usually used to avoid excessive punishment of difficult-to-classify samples during training.

[0020] Furthermore, the distance loss The calculation formula is as follows:

[0021]

[0022] in, Represents the sum of the L1 distances between the predicted coordinates and the target coordinates.

[0023] Furthermore, the total loss function of the wireframe parsing model is The calculation formula is as follows:

[0024]

[0025] Among them, λ cls and λ dist Represent the weights of classification loss and distance loss respectively.

[0026] In a second aspect, the present invention provides an end-to-end wireframe parsing system based on a dynamic deformable attention mechanism, comprising:

[0027] A data acquisition module, used to acquire an image of a wireframe of a building to be extracted;

[0028] The result acquisition module is used to input the image of the building wireframe to be extracted into the trained wireframe parsing model. First, the feature extraction and enhancement module performs multi-scale feature extraction and feature enhancement on the input image to generate a set of enhanced multi-scale feature maps. The enhanced multi-scale feature maps are then input into the line segment inference module. After being processed by the deformable encoder and the deformable decoder in sequence, optimized line segment entities are obtained. The optimized line segment entities are then used to predict the coordinates and confidence scores of each line segment entity through a feedforward network, and the wireframe parsing results of the image are output. The deformable encoder is obtained by replacing the self-attention mechanism in the Transformer encoder with a dynamic deformable attention mechanism; the deformable decoder is obtained by replacing the cross-attention mechanism in the Transformer decoder with a dynamic multi-scale deformable attention mechanism. The deformable decoder adopts an iterative optimization mechanism to gradually optimize the line segment representation during training.

[0029] In the feature extraction and enhancement module, the input image is first divided into non-overlapping patches of fixed size, and then the obtained patches are mapped to a high-dimensional feature space through a linear embedding layer. Subsequently, the patch-based feature representation is downsampled to different resolutions, and a Swin Transformer block is used at each resolution to extract feature maps of the corresponding scale and form a set of initial multi-scale feature maps. The initial multi-scale feature maps are then input into the feature enhancement module for feature enhancement, and finally the enhanced multi-scale feature maps are generated.

[0030] In a third aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, can implement the end-to-end wireframe parsing method based on a dynamic deformable attention mechanism as described in any one of the solutions in the first aspect above.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the end-to-end wireframe parsing method based on a dynamic deformable attention mechanism as described in any of the schemes in the first aspect above is implemented.

[0032] In a fifth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0033] The memory is used to store computer programs;

[0034] The processor is configured to implement the end-to-end wireframe parsing method based on a dynamic deformable attention mechanism as described in any one of the solutions of the first aspect above when executing the computer program.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention introduces Swin Transformer as the network backbone in the wireframe parsing task, and combines it with the Feature Enhancement Module (FEM) and the Dynamic Deformable Attention Mechanism (DDA), which significantly improves the performance of the two stages of feature extraction and enhancement and line segment reasoning, realizes multi-scale feature extraction and reasoning of line segments, and effectively solves the performance bottleneck of traditional methods when processing line segments with significant scale differences. Compared with traditional wireframe parsing methods, the present invention addresses the problem of line segment scale differences and achieves efficient multi-scale feature extraction. Relying on a layered architecture and sliding window mechanism, Swin Transformer gradually expands the receptive field through cross-window connections and hierarchical feature extraction, thereby obtaining a more refined multi-scale feature representation. The feature enhancement module combines depthwise separable convolution and spatial convolution to effectively remove redundant information and enhance the expression of local details. The dynamic deformable attention mechanism avoids the computational overhead and resource waste caused by global intensive modeling by dynamically selecting key areas, thereby improving the efficiency and accuracy of wireframe parsing. By combining global and local information, the proposed method simultaneously focuses on the global structure of long segments and the local details of short segments when parsing wireframes, improving the ability to parse segments of varying sizes. This approach is particularly suitable for computer vision tasks such as visual SLAM and image matching. Furthermore, the proposed method is suitable for line segment reasoning in complex scenes, capable of handling complex scenes containing long and short segments and multi-scale features, improving the accuracy and robustness of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of the steps of the present invention;

[0038] Figure 2 It is a schematic diagram of the wireframe parsing model structure of the present invention;

[0039] Figure 3 This is a structural diagram of a feature enhancement module of the present invention;

[0040] Figure 4 Schematic diagram of the dynamic deformable attention mechanism structure of the present invention;

[0041] Figure 5 A training and testing flow chart of a wireframe parsing model according to an embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram of the wireframe analysis results of the present invention;

[0043] Figure 7 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0045] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0046] In the wireframe parsing task, although existing methods have made improvements in feature extraction and line segment reasoning, they have not effectively solved the problem of significant line segment scale changes. Existing convolutional neural network (CNN)-based methods are limited by fixed convolution kernels and complex post-processing, making it difficult to strike a balance between global structure modeling and local detail preservation; while Transformer-based methods are limited by the high computational cost of global dense modeling and find it difficult to efficiently handle multi-scale feature extraction. Therefore, developing a method that can efficiently handle multi-scale line segment changes remains an important challenge. To this end, the present invention proposes an end-to-end wireframe parsing method based on a dynamic deformable attention mechanism. This method innovatively combines a Swin Transformer-based feature extraction and enhancement module with a dynamic deformable attention mechanism (DDA) to improve the wireframe parsing model's ability to handle line segment scale differences, and thereby constructs a wireframe parsing model.

[0047] like Figure 1 As shown, in a preferred implementation of the present invention, the specific steps of the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism are as follows:

[0048] An image of a building wireframe to be extracted is input into a trained wireframe parsing model (Wireframe Parsing using Deformable Transformers, WETR). The Feature Extraction and Enhancement module first performs multi-scale feature extraction and enhancement on the input image to generate a set of enhanced multi-scale feature maps. The enhanced multi-scale feature maps are then input into the line segment inference module, which is sequentially processed by a deformable encoder and a deformable decoder to obtain optimized line segment entities. The optimized line segment entities are then passed through a feed-forward network (FFN) to predict the coordinates and confidence scores of each line segment entity and output the wireframe parsing result of the image. The deformable encoder is obtained by replacing the self-attention mechanism in the Transformer encoder with a dynamic deformable attention mechanism (DDA). The deformable decoder is obtained by replacing the cross-attention mechanism in the Transformer decoder with a dynamic deformable attention mechanism. During training, the deformable decoder uses an iterative optimization mechanism to gradually optimize the line segment representation.

[0049] In the feature extraction and enhancement module, the input image is first divided into non-overlapping patches of fixed size, and then the obtained patches are mapped to a high-dimensional feature space through a linear embedding layer. Subsequently, the patch-based feature representation is downsampled to different resolutions, and a Swin Transformer block is used at each resolution to extract feature maps of the corresponding scale and form a set of initial multi-scale feature maps. The initial multi-scale feature maps are then input into the Feature Enhancement Module for feature enhancement, and finally the enhanced multi-scale feature maps are generated.

[0050] It should be noted that in the present invention, the wireframe parsing model employs a Transformer encoder-decoder architecture, whose internal processing consists of two stages: feature extraction and enhancement, and line segment inference. In the first stage, the input image is first segmented into small blocks. These blocks are then processed through a linear embedding layer and the Swine Transformer feature extraction backbone. A layered architecture and sliding window mechanism are used to efficiently extract multi-scale feature maps. These feature maps, which contain both local and global hierarchical information, are further refined by a feature enhancement module (FEM) to produce enhanced feature maps. In the second stage, the enhanced feature maps are processed by a deformable encoder, which employs a dynamic deformable attention mechanism to improve line segment detection by focusing on sparse key regions. The encoded dynamic feature maps are decoded by a deformable decoder, which integrates an iterative improvement strategy and a DDA mechanism to gradually refine the line segment representation while interacting with binary learnable line segment entities. A feedforward network (FFN) predicts the coordinates and confidence score of each line segment entity, providing the corresponding inference results.

[0051] To facilitate those skilled in the art to better understand the present invention, each module in the wireframe parsing model is described in detail below.

[0052] It should be noted that, in the present invention, the feature extraction and enhancement module uses Swin Transformer as the backbone network for high-quality multi-scale feature extraction in wireframe parsing. The implementation of Swin Transformer belongs to the prior art. The model realizes efficient multi-scale feature extraction through a layered architecture and a sliding window attention mechanism. It has a strong global perception capability and is suitable for processing complex scenes where long and short line segments coexist. It effectively solves the scale differences of line segments and improves the computational efficiency in visual tasks. The sliding window attention mechanism extracts features at multiple scales through a sliding window method, effectively capturing global relationships and local details. Especially when processing long and short line segments, it can avoid the performance degradation caused by scale differences. Specifically, if Figure 2 As shown, the feature extraction and enhancement module of the present invention first obtains a wireframe image I to be extracted of size (B×3×H×W), where B is the input Batch size, which depends on the sample size of each batch during the training phase and can be set to 1 during the prediction phase. H and W are the height and width of the input image, respectively. The input image is then divided into non-overlapping patches of fixed size, which are then mapped to a high-dimensional feature space through a linear embedding layer. These patch-based feature representations are then downsampled through a merging layer with a downsampling rate of {4×, 8×, 16×, 32×}, and a Swin Transformer block (STB) is used at each resolution i∈[1,4] to extract the corresponding feature map f i. Among them, each Swin Transformer block first uses the window-based multi-head self-attention (W-MSA) mechanism to calculate self-attention within a regular window configuration to capture local features. Subsequently, the shifted window multi-head self-attention (SW-MSA) mechanism is used to calculate self-attention across windows to achieve a wider representation. Through this process, the Swin Transformer blocks with successive resolutions gradually expand the receptive field through cross-window connections and multi-scale representations, and finally generate a set of initial multi-scale feature maps f = {f1, f2, f3, f4}, whose channel dimensions C are expanded from 3 to {192, 384, 768, 256}, respectively, significantly improving the accuracy and robustness of wireframe parsing.

[0053] It should be noted that in the feature enhancement module of the present invention, the initial multi-scale feature map is composed of the first feature map f1, the second feature map f2, the third feature map f3 and the fourth feature map f4 with the resolution from high to low. The first feature map is first processed by the first channel and the spatial convolution block, and the second feature map and the processed first feature map are spliced ​​to obtain the enhanced second feature map. And input it to the second channel and spatial convolution block to obtain the processed second feature map, and concatenate the third feature map and the processed second feature map to obtain the enhanced third feature map And input it to the third channel and spatial convolution block to obtain the processed third feature map, and concatenate the fourth feature map and the processed third feature map as the input of the depth-wise separable partial self-attention (DSPSA) block, which outputs the enhanced fourth feature map. The first feature map is used as the enhanced first feature map Finally, the enhanced multi-scale feature map is composed of the enhanced first feature map, the enhanced second feature map, the enhanced third feature map and the enhanced fourth feature map. This feature map contains both local and global hierarchical information.

[0054] It should be noted that each channel and spatial convolution (CSConv) block is composed of a depthwise separable convolution layer (DSC) and a spatial convolution layer (Spatial Convolution, SConv) cascaded in sequence.

[0055] It should be noted that the depth-wise separable partial self-attention block is composed of a depth-wise separable convolutional layer, a PSA module and a depth-wise separable convolutional layer cascaded in sequence, and the PSA module is composed of N PSA layers cascaded in sequence; in each PSA layer, the input feature map is processed by the multi-head self-attention module (MHSA) and the feed-forward network (FFN) in sequence to obtain the intermediate feature map, and the intermediate feature map is spliced ​​with the input feature map and the spliced ​​feature map is used as the output feature map.

[0056] In this embodiment, if Figure 3 As shown in the figure, the Feature Enhancement Module (FEM) uses a discriminative strategy to enhance feature maps at different scales while enriching local information to address the significant differences in line segment scales in wireframe parsing tasks. To enhance low-level and mid-level feature maps {f1, f2, f3} and capture rich local details of edges and corners while maintaining high resolution, the Feature Enhancement Module employs channel and spatial convolution blocks. The CSConv block contains depthwise separable convolution and spatial convolution. By integrating these two blocks, it effectively filters out interfering information and enhances key features. Depthwise separable convolution is used to reduce channel interference and computational cost, improve feature representation, and suppress background noise. The DSC consists of depthwise convolution and pointwise convolution. The depthwise convolution processes each channel of the input feature map independently and performs convolution with the corresponding filter. The pointwise convolution fuses information across channels, allowing features at each location to combine information from different channels. In this way, depthwise separable convolution effectively improves feature extraction efficiency and suppresses interference between channels. Spatial convolution is used to further refine the intermediate feature maps, focusing on local areas, optimizing spatial details, and highlighting important features, significantly improving wireframe parsing performance. This step enhances the local information of the feature maps, ensuring that important spatial structures and details are better preserved.

[0057] Furthermore, for the high-level feature map f4, which contains global context and semantic information, the present invention designs a depthwise separable partial self-attention block to enhance its low-resolution details. This block uses depthwise separable convolutional layers to improve local feature detection and reduce computational complexity, while employing a partial self-attention (PSA) mechanism to effectively capture key regions of line segments and junctions. Specifically, the DSPSA block first performs initial processing on the input feature map, followed by sequential optimization of the feature representation through multiple PSA layers. Each PSA layer divides the input feature map into two parts along the channel. One part calculates attention and extracts key information through a multi-head self-attention mechanism and a feed-forward network (FFN), while the other part is directly passed on. The two parts are then reconstructed and fused as input to the next PSA layer. Finally, the multi-layer optimized feature map is further processed by depthwise separable convolutional layers to generate the final enhanced feature map. The depthwise separable convolution further improves the ability to capture local features, while the PSA layer adaptively focuses on key regions, strengthening attention to line segments and intersections. Of particular note, the present invention introduces a depthwise separable partial self-attention block that processes only the lowest-resolution high-level feature map f4, avoiding the excessive overhead associated with the quadratic computational complexity of the self-attention mechanism. Furthermore, by eliminating redundant features and highlighting key features, it significantly optimizes low-, mid-, and high-level features, enhancing their expressive power while minimizing potential interference.

[0058] It should be noted that in the processing of the deformable encoder of the present invention, the correlation information between different scales in the input image is first captured by learning the multi-scale feature map, and multiple dynamic feature maps with the same resolution are generated. Each dynamic feature map contains detailed information about line segments at different scales, and these features are optimized using a dynamic deformable attention mechanism. This mechanism enables the deformable encoder to not only effectively extract features at different scales, but also dynamically select regions of interest for specific line segment features, dynamically adjust sampling positions, focus on sparse key regions, and extract contextual information related to the line segment, thereby providing the decoder with accurate multi-scale feature representations.

[0059] It should be noted that in the processing of the deformable decoder of the present invention, each decoder layer takes the dynamic feature map output from the deformable encoder and the line segment entity (initialized or from the previous decoder layer) as input. The deformable decoder combines the iterative optimization strategy and the dynamic deformable attention mechanism to gradually refine the line segment representation and combine it with N learnable line segment entities. Interact. Among them, the dynamic deformable attention mechanism dynamically guides each line segment entity to pay attention to the relevant areas in the dynamic feature map. First, the line segment entity is processed by the self-attention mechanism, so that it adaptively adjusts the attention area according to the line segment features, and then further optimizes these areas through the dynamic deformable attention mechanism to ensure accurate capture of relevant multi-scale features to better understand contextual information. Finally, for each line segment entity y∈y, the coordinates and confidence scores of each line segment entity are predicted through the feedforward network (FFN), thereby providing the corresponding wireframe parsing results. Through the dynamic deformable attention mechanism, the present invention can flexibly adapt to the changes in line segment scale, focus on sparse key areas, significantly reduce computational complexity, and improve the accuracy and efficiency of wireframe parsing.

[0060] It should be noted that, in the present invention, the implementation of the above-mentioned dynamic deformable attention mechanism belongs to the existing technology. The mechanism can dynamically adjust the feature focus area according to the characteristics of the input data, focus on the line segments in the sparse key area, and avoid the problem of excessive calculation when the traditional method processes multiple scales. In order to facilitate those skilled in the art to better understand the present invention, the processing process of the dynamic deformable attention mechanism is briefly described below. Figure 4 As shown in the figure, the implementation process of the dynamic deformable attention mechanism introduced in this invention is as follows: First, the input is the enhanced multi-scale feature map Each enhanced feature map has different resolutions and number of channels; for each query element, the sampling offset and attention weight are calculated by linear projection, where the sampling offset is used to dynamically adjust the sampling position, and the attention weight is normalized to ensure its rationality. q Based on the sampling offset, the sampling position is determined, and the feature value is obtained by bilinear interpolation. The output of each attention head is calculated by combining the attention weight and the learnable weight. The normalized coordinates are remapped to the coordinate system of each feature map through a coordinate mapping function, and multi-scale sampling offsets and attention weights are combined to achieve multi-scale feature fusion and enhancement. Finally, the outputs of all attention heads are concatenated and linearly transformed to generate a new multi-scale feature representation.

[0061] It should be noted that in the present invention, the above-mentioned iterative optimization mechanism (iterative optimization strategy, IRS) is a strategy that optimizes the performance of the model in multi-scale line segment processing by continuously adjusting model parameters. This mechanism enables each decoder layer in the wireframe parsing model to gradually optimize the line segment representation based on the prediction of the previous layer, adapt to the changes in line segments in complex scenes, and thus improve the prediction accuracy. Specifically, the iterative optimization mechanism acts on the standardized line segment representation. Each decoder layer adjusts the position information and size of the line segment according to the prediction of the previous layer, gradually approaching the true annotation. In this process, each prediction head learns independently, avoiding the limitations brought by shared parameters. In addition, considering the high sensitivity of the line segment parsing task to position information, directly propagating gradients across layers may lead to unstable training. To this end, the gradient is only back-propagated through the offset of the line segment, thereby maintaining the stability of the training. The present invention continuously optimizes the parameters of the model through an iterative optimization strategy, so that the stability and accuracy of the model when processing line segments of different scales are significantly improved.

[0062] It should be noted that the wireframe parsing model described above is pre-trained using annotated training data before being used for actual wireframe parsing. The loss function used in training the wireframe parsing model consists of a combination of classification loss and distance loss. The calculation process of these two losses is described below.

[0063] 1) Classification loss

[0064] The classification loss in this invention is an improvement on the binary cross-entropy loss. After learning rate decay, some difficult-to-classify samples are not sufficiently optimized during model learning. Focal loss, on the other hand, is a loss function designed to handle class imbalance and difficult and easy samples, allowing the model to focus more on difficult-to-classify samples. Similar to the concept of focal loss, the classification loss calculation in this invention improves the classification loss term by introducing an adaptive coefficient to better handle difficult-to-classify samples.

[0065] For N line segment predictions and M true target values ​​{L (j) ; j = 1, ..., M}, the present invention arranges the function To determine the best match between the line segment prediction and the true target value. It maps the prediction index {1,…,N} to the potential target index {1,…,N}, where the potential target index includes the true target value {1,…,M} and the unmatched prediction {M+1,…,N}. The specific implementation process is:

[0066]

[0067]

[0068] in, represents the matching loss that comprehensively considers distance and confidence; λ1 and λ2 are balance coefficients; represents the L1 distance between coordinates; Indicates the line segment prediction with index i; L (j) represents the true target value of index j; Represents a mapping from the set of positive integers to the set of positive integers.

[0069] Optimal permutation σ * It is calculated using the Hungarian algorithm, mapping M positive prediction indices to target indices {1,…,M}. In the inference phase, since the true value is not provided, it can be calculated by using the confidence level p (i) A fixed threshold is set on to filter N line segment predictions.

[0070] Classification loss The calculation formula is as follows:

[0071]

[0072] in, represents the indicator function; σ(i) represents the optimal matching target index of the prediction index i; M represents the number of positive prediction indices; α1 and α2 are the weighting coefficients of positive and negative samples, respectively, which are used to balance the contribution of positive and negative samples in the loss function; p (i) represents the prediction confidence of index i; γ represents the adjustment factor, which is used to control the sensitivity of the loss function to the prediction confidence, and is usually used to avoid excessive punishment of difficult-to-classify samples during training.

[0073] 2) Distance loss

[0074] Distance loss is only used for positive prediction. This method directly applies L1 distance loss as distance loss The calculation formula is as follows:

[0075]

[0076] in, Represents the sum of the L1 distances between the predicted coordinates and the target coordinates.

[0077] Thus, the total loss of the wireframe parsing model is It can be expressed as:

[0078]

[0079] Among them, λ cls and λ dist Represent the weights of classification loss and distance loss respectively.

[0080] The present invention will now use a specific example to demonstrate the application effect of the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism described in the above embodiment on a specific dataset, so as to facilitate understanding of the essence of the present invention.

[0081] Example

[0082] The specific implementation process of the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism adopted in this embodiment is as described above and will not be repeated here.

[0083] The overall process of this embodiment can be divided into three stages: data preprocessing, model training, and image prediction. Figure 5 shown.

[0084] 1. Data preprocessing

[0085] Step 1: For the original image obtained, image preprocessing is performed, first performing image cropping, flipping and other operations, and then performing data enhancement and processing into images of the same size (256×256).

[0086] Step 2: Label the image and use tools such as Labelme to perform vector wireframe annotation to generate GroundTruth for subsequent model training.

[0087] 2. Model training

[0088] Step 1: Build a training dataset and divide the training dataset into batches according to a fixed batch size, with a total of N.

[0089] Step 2: Sequentially select a batch of training samples with index i from the training dataset, where i∈{0,1,…,N}. Use each batch of training samples to train the wireframe parsing model WETR. The specific structure of WETR is as described above and will not be repeated here. During the training process, the classification loss of each training sample is calculated. and distance loss And based on the total loss of all training samples in the batch The network parameters of the entire model are adjusted until all batches of the training dataset participate in the model training. After reaching the specified number of iterations, the model converges and the training is completed.

[0090] 3. Image Prediction

[0091] The image of the test set is directly used as input to the trained wireframe parsing model WETR, and the coordinates of the two endpoints of the line segment are finally predicted, that is, This can describe the corresponding line segment This enables wireframe parsing.

[0092] In this embodiment, the test results are as follows: Figure 6 As shown. It can be seen that the wireframe parsing model can extract wireframe structures in complex indoor and outdoor scenes, and the connection relationship between line segments can be correctly reconstructed, with accurate and complete visualization results. By introducing Swin Transformer as the network backbone in the wireframe parsing task, and combining the feature enhancement module (FEM) and the dynamic deformable attention mechanism (DDA), the performance of the two stages of feature extraction and enhancement and line segment reasoning is significantly improved. Compared with the traditional wireframe parsing method, the present invention aims at the problem of line segment scale difference and realizes efficient multi-scale feature extraction, which provides a promising possibility for solving the problem of line segment scale difference of the wireframe parsing method.

[0093] It should also be noted that the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an end-to-end wireframe parsing system based on the dynamic deformable attention mechanism corresponding to the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism provided in the above embodiment, such as Figure 7 As shown, it includes:

[0094] A data acquisition module, used to acquire an image of a wireframe of a building to be extracted;

[0095] The result acquisition module is used to input the image of the building wireframe to be extracted into the trained wireframe parsing model. First, the feature extraction and enhancement module performs multi-scale feature extraction and feature enhancement on the input image to generate a set of enhanced multi-scale feature maps. The enhanced multi-scale feature maps are then input into the line segment inference module. After being processed by the deformable encoder and the deformable decoder in sequence, optimized line segment entities are obtained. The optimized line segment entities are then used to predict the coordinates and confidence scores of each line segment entity through a feedforward network, and the wireframe parsing results of the image are output. The deformable encoder is obtained by replacing the self-attention mechanism in the Transformer encoder with a dynamic deformable attention mechanism; the deformable decoder is obtained by replacing the cross-attention mechanism in the Transformer decoder with a dynamic multi-scale deformable attention mechanism. The deformable decoder adopts an iterative optimization mechanism to gradually optimize the line segment representation during training.

[0096] In the feature extraction and enhancement module, the input image is first divided into non-overlapping patches of fixed size, and then the obtained patches are mapped to a high-dimensional feature space through a linear embedding layer. Subsequently, the patch-based feature representation is downsampled to different resolutions, and a Swin Transformer block is used at each resolution to extract feature maps of the corresponding scale and form a set of initial multi-scale feature maps. The initial multi-scale feature maps are then input into the feature enhancement module for feature enhancement, and finally the enhanced multi-scale feature maps are generated.

[0097] It is understood that the above-mentioned end-to-end wireframe parsing method based on a dynamic deformable attention mechanism can be implemented via a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the end-to-end wireframe parsing method based on a dynamic deformable attention mechanism provided in the above-mentioned embodiment. The computer program product includes a computer program / instructions. When executed by a processor, the computer program / instructions implement the end-to-end wireframe parsing method based on a dynamic deformable attention mechanism as described in the above-mentioned embodiment.

[0098] Similarly, based on the same inventive concept, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism provided in the above embodiment, which includes a memory and a processor;

[0099] The memory is used to store computer programs;

[0100] The processor is configured to implement the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism in the above embodiment when executing the computer program.

[0101] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0102] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism provided in the above embodiment, and the storage medium stores a computer program. When the computer program is executed by the processor, it can implement the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism in the above embodiment.

[0103] It is understood that the storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0104] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0105] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0106] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. An end-to-end wireframe parsing method based on a dynamic deformable attention mechanism, characterized by: The specific process is as follows: An image of a building wireframe to be extracted is input into a trained wireframe parsing model. First, the feature extraction and enhancement module performs multi-scale feature extraction and feature enhancement on the input image to generate a set of enhanced multi-scale feature maps. The enhanced multi-scale feature maps are then input into the line segment inference module. After being processed by a deformable encoder and a deformable decoder in sequence, optimized line segment entities are obtained. The optimized line segment entities are then passed through a feedforward network to predict the coordinates and confidence scores of each line segment entity and output the wireframe parsing result of the image. The deformable encoder is obtained by replacing the self-attention mechanism in the Transformer encoder with a dynamic deformable attention mechanism. The deformable decoder is obtained by replacing the cross-attention mechanism in the Transformer decoder with a dynamic multi-scale deformable attention mechanism. During training, the deformable decoder uses an iterative optimization mechanism to gradually optimize the line segment representation. In the feature extraction and enhancement module, the input image is first divided into non-overlapping patches of fixed size, and then the obtained patches are mapped to a high-dimensional feature space through a linear embedding layer. Subsequently, the patch-based feature representation is downsampled to different resolutions, and a Swin Transformer block is used at each resolution to extract feature maps of the corresponding scale and form a set of initial multi-scale feature maps. The initial multi-scale feature maps are then input into the feature enhancement module for feature enhancement, and finally the enhanced multi-scale feature maps are generated.

2. The end-to-end wireframe parsing method based on a dynamic deformable attention mechanism according to claim 1, characterized in that: In the feature enhancement module, the initial multi-scale feature map is composed of the first feature map, the second feature map, the third feature map and the fourth feature map with resolution from high to low. The first feature map is first processed by the first channel and the spatial convolution block. The second feature map and the processed first feature map are spliced ​​to obtain the enhanced second feature map and input it to the second channel and the spatial convolution block to obtain the processed second feature map. The third feature map and the processed second feature map are spliced ​​to obtain the enhanced third feature map and input it to the third channel and the spatial convolution block to obtain the processed third feature map. The fourth feature map and the processed third feature map are spliced ​​as the input of the depth-separable partial self-attention block. The depth-separable partial self-attention block outputs the enhanced fourth feature map, and the first feature map is used as the enhanced first feature map. Finally, the enhanced first feature map, the enhanced second feature map, the enhanced third feature map and the enhanced fourth feature map constitute the enhanced multi-scale feature map.

3. The end-to-end wireframe parsing method based on a dynamic deformable attention mechanism according to claim 2, characterized in that: Each channel and spatial convolution block is composed of a depthwise separable convolution layer and a spatial convolution layer cascaded in sequence.

4. The end-to-end wireframe parsing method based on a dynamic deformable attention mechanism according to claim 2, characterized in that: The depth-wise separable partial self-attention block is composed of a depth-wise separable convolutional layer, a PSA module and a depth-wise separable convolutional layer cascaded in sequence, and the PSA module is composed of N PSA layers cascaded in sequence; in each PSA layer, the input feature map is processed by the multi-head self-attention module and the feedforward network in sequence to obtain an intermediate feature map, and the intermediate feature map is spliced ​​with the input feature map and the spliced ​​feature map is used as the output feature map.

5. The end-to-end wireframe parsing method based on a dynamic deformable attention mechanism according to claim 1, characterized in that: The wireframe parsing model is pre-trained using labeled training data.

6. The end-to-end wireframe parsing method based on a dynamic deformable attention mechanism according to claim 1, characterized in that: When training the wireframe parsing model, the weighted sum of classification loss and distance loss is used as the total loss function.

7. An end-to-end wireframe parsing system based on a dynamic deformable attention mechanism, characterized by: include: A data acquisition module, used to acquire an image of a wireframe of a building to be extracted; The result acquisition module is used to input the image of the building wireframe to be extracted into the trained wireframe parsing model. First, the feature extraction and enhancement module performs multi-scale feature extraction and feature enhancement on the input image to generate a set of enhanced multi-scale feature maps. The enhanced multi-scale feature maps are then input into the line segment inference module. After being processed by the deformable encoder and the deformable decoder in sequence, optimized line segment entities are obtained. The optimized line segment entities are then used to predict the coordinates and confidence scores of each line segment entity through a feedforward network, and the wireframe parsing results of the image are output. The deformable encoder is obtained by replacing the self-attention mechanism in the Transformer encoder with a dynamic deformable attention mechanism; the deformable decoder is obtained by replacing the cross-attention mechanism in the Transformer decoder with a dynamic multi-scale deformable attention mechanism. The deformable decoder adopts an iterative optimization mechanism to gradually optimize the line segment representation during training. In the feature extraction and enhancement module, the input image is first divided into non-overlapping patches of fixed size, and then the obtained patches are mapped to a high-dimensional feature space through a linear embedding layer. Subsequently, the patch-based feature representation is downsampled to different resolutions, and a Swin Transformer block is used at each resolution to extract feature maps of the corresponding scale and form a set of initial multi-scale feature maps. The initial multi-scale feature maps are then input into the feature enhancement module for feature enhancement, and finally the enhanced multi-scale feature maps are generated.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, it can implement the end-to-end wireframe parsing method based on a dynamic deformable attention mechanism as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the end-to-end wireframe parsing method based on the dynamic deformable attention mechanism according to any one of claims 1 to 6 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the end-to-end wireframe parsing method based on a dynamic deformable attention mechanism as described in any one of claims 1 to 6 when executing the computer program.

Images