A deployment-oriented line frame parsing method and system

Abstract

The application discloses a kind of line frame analysis method and system for deployment, belong to line frame analysis technical field, method includes: collection original image, and pre-process it, obtain the image to be parsed;Line frame analysis framework is constructed and trained, and line frame analysis model is obtained;The image to be parsed is input into line frame analysis model, through mobile terminal line analysis Vit through local convolution and token interaction mixed processing, multi-scale features with long-range context dependence in the image to be parsed are extracted, through multi-scale feature line decoder, multi-scale features are aligned and structure perception enhancement is carried out, and line frame in original image is parsed and obtained.The method of the application enhances the structure continuity and intersection positioning stability in line frame analysis process, obtains clearer performance improvement in line segment decoding process compared with simple splicing mode, while maintaining real-time processing speed;Support plug and play, without additional supervision mechanism, and maintain compatibility with standard training and inference process.

Description

Technical Field

[0001] This invention belongs to the field of wireframe parsing technology, specifically relating to a deployment-oriented wireframe parsing method and system. Background Technology

[0002] Wireframe (line segment) representations provide a compact, topology-aware mid-level cue for geometry-centric vision tasks, significantly benefiting fields such as pose estimation, mapping, augmented reality (AR), and mobile robotics. However, in practical applications, challenges such as strong lighting fluctuations, weak or repetitive textured scenes, and rapid viewpoint changes, coupled with stringent latency and power consumption constraints on edge hardware, make reliable deployment difficult. These constraints require detectors to maintain long-range structural continuity and achieve stable intersection localization without relying on heavyweight models or weakly supervised schemes.

[0003] In existing technologies, the classic L-CNN predicts connection point heatmaps and line / affinity evidence through a stacked CNN backbone network, followed by grouping and validation, establishing a detection-grouping paradigm. While structurally simple and reasonably fast, its pure convolutional features provide limited global contextual information, and the multi-stage post-processing is susceptible to cluttered regions and scale variations. HAWP, through global attracting line representation and lightweight validation techniques, jointly utilizes endpoint and midline evidence, significantly improving structural fidelity and reducing false positive rates. Building upon this, unified point-line frameworks such as PLNet, through a shared point structure encoder, achieve improved computational efficiency in the real-time visual optimization / spatial localization and mapping front-end.

[0004] Despite the advantages of the aforementioned existing technologies, such architectures still have two major limitations: (i) encoders that use only convolutions have insufficient long-range dependencies and are prone to structural instability under strong light and low-texture conditions; (ii) multi-scale coupling is usually achieved by simple splicing, which will mix different semantic and spatial scale deviations and propagate noise near edges and intersections. These problems are more prominent on resource-constrained devices. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, the deployment-oriented wireframe parsing method and system provided by this invention solves the problems of insufficient stability of existing model structures for wireframe parsing under strong light environments and low texture conditions, as well as the noise generated during use due to device resource limitations, which in turn affects the wireframe parsing effect.

[0006] To achieve the aforementioned objectives, the present invention employs the following technical solution: a deployment-oriented wireframe parsing method, comprising:

[0007] Acquire the original image and preprocess it to obtain the image to be analyzed;

[0008] A wireframe parsing framework is constructed and trained to obtain a wireframe parsing model; the wireframe parsing framework includes a mobile wire parsing ViT and a multi-scale feature line decoder.

[0009] The image to be parsed is input into the wireframe parsing model. The mobile wireframe parsing ViT uses local convolution and token interaction to extract multi-scale features with long-range context dependence from the image to be parsed. The multi-scale features are aligned and structure-aware enhancement is performed by the multi-scale feature line decoder to obtain the wireframe in the original image.

[0010] Furthermore, in the wireframe analytical model:

[0011] The mobile-end line parsing ViT includes a first convolutional layer, a first inverted residual block, a second inverted residual block, a third inverted residual block, a fourth inverted residual block, a first MobileViT module, a fifth inverted residual block, a second MobileViT module, a sixth inverted residual block, and a third MobileViT module connected in sequence.

[0012] The third MobileViT module, the second MobileViT module, and the third inverse residual block each output corresponding scale feature maps, which are combined with the image to be parsed to form multi-scale features, and then input into the multi-scale feature line decoder.

[0013] The multi-scale feature line decoder includes a feature refinement unit and a wireframe generation unit;

[0014] The feature refinement unit includes a first branch, a second branch, a third branch, a fourth branch, a splicing layer, and a 3×3 convolutional layer, all arranged in parallel and having the same structure.

[0015] The first branch, the second branch, the third branch, and the fourth branch each include a 1×1 convolutional layer, a mapping operation layer, a spatial attention mechanism, and a channel attention mechanism connected in sequence.

[0016] The input of the 1×1 convolutional layer in the first branch is connected to the output of the third MobileViT module; the input of the 1×1 convolutional layer in the second branch is connected to the output of the second MobileViT module; the input of the 1×1 convolutional layer in the third branch is connected to the output of the third inverted residual block; and the input of the 1×1 convolutional layer in the fourth branch is connected to the input of the first convolutional layer.

[0017] The feature maps output from the first branch, the second branch, the third branch, and the fourth branch are spliced ​​along the channel dimension by a splicing layer and refined by a 3×3 convolutional layer to obtain refined features;

[0018] The wireframe generation unit includes a first optimizer, a second optimizer, a line segment decoder, a connection point decoder, and an LOI verifier connected in sequence.

[0019] The first optimizer and the second optimizer process the refined features step by step and output the dense prediction tensor. The linear decoder and the connector decoder perform line segment decoding and connector decoding on the dense prediction tensor, respectively, and output the corresponding line segment candidate set and connector candidate set. The LOI validator sorts the line segments in the line segment candidate set and scales the sorted line segments and connectors in the connector candidate set to the corresponding coordinates in the original image, and parses the wireframe in the original image.

[0020] Furthermore, the first, second, third, fourth, fifth, and sixth inverted residual blocks have the same structure, and they are identical for the input image. The processing procedure is represented as follows:

[0021]

[0022]

[0023]

[0024] In the formula, This represents the SiLU activation function. express Pointwise convolution, express Depthwise convolution, Indicates that the input image X passes through Pointwise convolution increases the channel dimension, resulting in an expanded feature map. This indicates that the extended feature map is being processed. Spatial convolution captures feature maps of spatial features. This represents the feature map of the final output.

[0025] Furthermore, the mapping operation layer adjusts the resolution of the feature map output by the 1×1 convolutional layer to... To achieve scale feature alignment;

[0026] Among them, the feature map after scale feature alignment Represented as:

[0027]

[0028] In the formula, Indicates a mapping operation. This indicates that multi-scale features are projected onto a uniform number of channels. A 1×1 convolutional layer, Representing multi-scale features, Indicates batch size. Indicates the first The number of channels in the layer feature map. Indicates the height of the original image. Indicates the width of the original image. Representing multi-scale features The corresponding downsampling factor;

[0029] Among them, the current sampling factor At that time, mapping operation For bilinear upsampling; when At that time, mapping operation To maintain identity; when At that time, mapping operation For step anti-aliasing pooling.

[0030] Further, the wireframe generation unit parses and obtains the wireframes in the original image, including:

[0031] The first optimizer processes the refined features to generate preliminary unit prediction tensors. ;in, Indicates batch size. Indicates the height of the original image. Indicates the width of the original image;

[0032] The second optimizer, based on the first optimizer, predicts the tensor of the initial cell. Perform fine-tuning and output the corresponding cell prediction tensor. ;

[0033] Predict tensors for each unit Encode the original image at grid locations into a dense prediction tensor with nine physically meaningful channels. ;

[0034] On a quarter grid, the dense prediction tensor is processed by the line segment decoder and the connection point decoder, respectively. Decode the corresponding line segments and connection points to form a candidate set of line segments and a candidate set of connection points;

[0035] On a quarter-resolution feature map, line segments in the candidate set are sorted using the LOI validator;

[0036] Redundancy is suppressed on the sorted line segments, and the final retained line segments and the connection points in the candidate connection point set are scaled to the corresponding coordinates of the original image by a global step of 4, and the wireframe in the original image is obtained by parsing.

[0037] Furthermore, the dense prediction tensor Represented as:

[0038]

[0039] In the formula, Indicates the main direction. Indicates the initial side tangent. Indicates the termination of the side tangent. Represents the distance parameter. Represents the residual. This represents the coordinates of the connection points on the bidirectional softmax heatmap. This represents the subpixel offset.

[0040] Furthermore, the dense prediction tensor Decode the corresponding line segments and connection points to form a set of line segments and a candidate set of connection points, including:

[0041] On a quarter grid, based on dense prediction tensors The formula for calculating the coordinates of the endpoints is as follows:

[0042]

[0043]

[0044] In the formula, Indicates the starting endpoint. Indicates the termination endpoint. Indicates the center point of the cell. Indicates the distance scaling factor. Represents the horizontal component of the main direction of the line segment. Represents the perpendicular component of the principal direction of a line segment. Indicates the initial side slope. Indicates the slope of the terminating side. Indicates the normalization extension range. , Indicates a range-limiting function;

[0045] Candidate line segments are formed based on the calculated endpoint coordinates. ;

[0046] Geometric anchoring snaps the endpoints of candidate line segments to the nearest connection point, thereby anchoring the candidate line segments. Replace with It then determines whether there exists a pair of endpoints where the Euclidean square distance between any endpoint of a candidate line segment and any point in the candidate connection point set is less than a set distance threshold; if so, the candidate line segment is retained; otherwise, it is deleted. Represents points in the candidate set of connection points;

[0047] For the retained candidate line segments, by surrounding Add signed residuals, enumerate the extension range of candidate line segments after disturbance, and form a candidate line segment set;

[0048] Connection points are obtained based on bidirectional softmax heatmaps, and connection points above the confidence threshold are suppressed and filtered by eight-neighbor nonmaximum suppression.

[0049] Based on the selected connection points, sub-pixel offsets are used. The local probability maxima on the bidirectional softmax heatmap are refined to form a candidate set of points.

[0050] Furthermore, the line segments are sorted, including:

[0051] On the original image at quarter resolution, bilinear feature sampling is performed on each line segment in the candidate line segment set at its two endpoints and at equally spaced internal sampling points.

[0052] The sampled features are concatenated into a descriptor and input into the classifier to obtain the corresponding confidence score;

[0053] Sort the line segments according to their confidence level.

[0054] A deployment-oriented wireframe parsing system, comprising:

[0055] The image acquisition unit is used to acquire raw images and preprocess them into images to be analyzed.

[0056] Wireframe parsing models, including mobile wire parsing ViT and multi-scale feature line decoders;

[0057] The mobile terminal line parsing ViT is used to extract multi-scale features with long-range context dependence in the image to be parsed by a hybrid processing of local convolution and token interaction.

[0058] The multi-scale feature line decoder is used to align multi-scale features and perform structure-aware enhancement to parse and obtain wireframes in the original image.

[0059] Furthermore, the multi-scale feature line decoder includes a feature refinement unit and a wireframe generation unit;

[0060] The feature refining unit is used to align multi-scale features through a four-branch structure and output refined features;

[0061] The wireframe generation unit is used to perform stepwise optimization of the refined features, output dense prediction tensors, and perform line segment decoding and connection point decoding on the dense prediction tensors through a linear decoder and a connection point decoder, respectively. The decoded line segments are sorted through an LOI validator, and the sorted line segments and decoded connection points are scaled to the corresponding coordinates of the original image to obtain the wireframe in the original image.

[0062] The beneficial effects of this invention are as follows:

[0063] (1) The mobile wire-parsing ViT proposed in this invention is a mobile-friendly ViT backbone network customized for wireframe perception. It provides stronger long-range context dependencies at a relatively low computational cost, and enhances the structural continuity and intersection localization stability during wireframe parsing.

[0064] (2) The multi-scale feature line decoder provided by the present invention is a structure-aware multi-scale decoder that uniformly implements feature alignment, edge-aware upsampling and lightweight attention mechanism before the line segment prediction head. Compared with the simple splicing method, it achieves a clearer performance improvement while maintaining real-time processing speed.

[0065] (3) This invention is a plug-and-play solution that does not require additional supervision mechanisms and retains the standard training / inference interface for rapid portability. Furthermore, experiments on standard wireframe benchmarks and illumination stress datasets show that the method continuously improves the SAP index while maintaining a near-constant frame rate. Attached Figure Description

[0066] Figure 1 The flowchart illustrates the deployment-oriented wireframe parsing method provided by this invention.

[0067] Figure 2 The diagram shows the structural block of the wireframe analytical model provided by this invention. Detailed Implementation

[0068] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0069] Example 1:

[0070] This invention provides a deployment-oriented wireframe parsing method that supports plug-and-play functionality, requires no additional supervision mechanisms, and maintains compatibility with standard training and inference processes.

[0071] See Figure 1 ,include:

[0072] Acquire the original image and preprocess it to obtain the image to be analyzed;

[0073] A wireframe parsing framework is constructed and trained to obtain a wireframe parsing model; the wireframe parsing framework includes mobile wire parsing ViT and a multi-scale feature line decoder.

[0074] The image to be parsed is input into the wireframe parsing model. The mobile wireframe parsing ViT uses local convolution and token interaction to extract multi-scale features with long-range context dependence from the image to be parsed. The multi-scale features are aligned and structure-aware enhancement is performed by the multi-scale feature line decoder to obtain the wireframe in the original image.

[0075] In this embodiment, the acquired original image is converted into a corresponding grayscale image and adjusted to a uniform resolution, such as 512×512.

[0076] In embodiments of the present invention, such as Figure 2 As shown in the wireframe parsing model:

[0077] The mobile-end line parsing ViT includes a first convolutional layer, a first inverted residual block, a second inverted residual block, a third inverted residual block, a fourth inverted residual block, a first MobileViT module, a fifth inverted residual block, a second MobileViT module, a sixth inverted residual block, and a third MobileViT module connected in sequence.

[0078] The third MobileViT module, the second MobileViT module, and the third inverse residual block output corresponding scale feature maps, which are combined with the image to be parsed to form multi-scale features, and then input into the multi-scale feature line decoder.

[0079] The multi-scale feature line decoder includes a feature refinement unit and a wireframe generation unit;

[0080] The feature refinement unit includes a first branch, a second branch, a third branch, a fourth branch, a splicing layer, and a 3×3 convolutional layer, all arranged in parallel and with the same structure.

[0081] The first branch, the second branch, the third branch, and the fourth branch all consist of a 1×1 convolutional layer, a mapping operation layer, a spatial attention mechanism, and a channel attention mechanism connected in sequence.

[0082] In the first branch, the input of the 1×1 convolutional layer is connected to the output of the third MobileViT module; in the second branch, the input of the 1×1 convolutional layer is connected to the output of the second MobileViT module; in the third branch, the input of the 1×1 convolutional layer is connected to the output of the third inverted residual block; and in the fourth branch, the input of the 1×1 convolutional layer is connected to the input of the first convolutional layer.

[0083] The feature maps output from the first, second, third, and fourth branches are concatenated along the channel dimension by a concatenation layer and refined by a 3×3 convolutional layer to obtain refined features.

[0084] The wireframe generation unit includes a first optimizer, a second optimizer, a line segment decoder, a connection point decoder, and an LOI verifier connected in sequence.

[0085] The first and second optimizers process the refined features step by step, outputting a dense prediction tensor; the linear decoder and the connector decoder perform line segment decoding and connector decoding on the dense prediction tensor, respectively, outputting the corresponding line segment candidate set and connector candidate set; the LOI validator sorts the line segments in the line segment candidate set and scales the sorted line segments and connectors in the connector candidate set to the corresponding coordinates in the original image, parsing to obtain the wireframe in the original image.

[0086] In this embodiment of the invention, the mobile-end line parsing ViT injects global context at mobile cost by fusing local convolution and lightweight token interaction, generating a compact multi-scale hierarchical structure for downstream decoding.

[0087] Based on this, the first, second, third, fourth, fifth, and sixth inverted residual blocks in this embodiment have the same structure, and they are identical for the input image. The processing procedure is represented as follows:

[0088]

[0089]

[0090]

[0091] In the formula, This represents the SiLU activation function. express Pointwise convolution, express Depthwise convolution, Indicates that the input image X passes through Pointwise convolution increases the channel dimension, resulting in an expanded feature map. This indicates that the extended feature map is being processed. Spatial convolution captures feature maps of spatial features. This represents the feature map of the final output.

[0092] Furthermore, to enhance long-range dependencies without altering spatial resolution, MobileViT modules (i.e., the first to third MobileViT modules) are introduced in the mid-to-high-level stages of the mobile-end linear ViT parsing process. Within the MobileViT modules, for a given input... Divide it into sizes of × Non-overlapping image patches, and reshaped as:

[0093]

[0094]

[0095]

[0096] In the formula, Represents a block sequence tensor. Represents a block sequence tensor. This represents the total number of pixels within a single block. Indicates the length of the block sequence. This represents the number of feature channels for each pixel. Indicates the number of pixels in the block height direction. Indicates the number of pixels in the block width direction. Indicates the height of the input image. This indicates the width of the input image.

[0097] For each patch index, the transformer layer in the MobileViT module performs self-attention computation among N tokens arranged on a spatial grid. It is then processed by a position-wise feedforward network. The tokens after attention processing are then restored to the spatial layout of the original image. After being projected through a 1×1 convolution, they are concatenated with local convolution branches and feature fusion is achieved through a 3×3 convolution.

[0098] In this embodiment of the invention, the MobileViT module, through the above-described processing, injects global structural information while maintaining the resolution unchanged, at a lower computational cost.

[0099] In this embodiment of the invention, robust wireframe parsing requires combining fine-grained connection point clues with long-distance line segment evidence under strict latency and memory constraints. Traditional heavy decoders often sacrifice speed for improved accuracy, while simple multi-scale fusion leads to blurred edges and weakened corners. In this embodiment, a compact multi-scale line decoder is used to unify multi-resolution features on a shared quarter-resolution canvas, enhance edge-like and corner-like responses through a lightweight attention mechanism, and employ deep supervision for cascaded prediction.

[0100] In this embodiment of the invention, based on Figure 2 The wireframe model structure shown, in the feature refinement unit of the multi-scale feature line decoder, the mapping operation layer adjusts the resolution of the 1×1 convolutional layer output feature map to [resolution missing]. To achieve scale feature alignment;

[0101] Among them, the feature map after scale feature alignment Represented as:

[0102]

[0103] In the formula, Indicates a mapping operation. This indicates that multi-scale features are projected onto a uniform number of channels. A 1×1 convolutional layer, Representing multi-scale features, Indicates batch size. Indicates the first The number of channels in the layer feature map. Indicates the height of the original image. Indicates the width of the original image. Representing multi-scale features The corresponding downsampling factor;

[0104] Among them, the current sampling factor At that time, mapping operation For bilinear upsampling; when At that time, mapping operation To maintain identity; when At that time, mapping operation For step anti-aliasing pooling.

[0105] In this embodiment of the invention, by using the above mapping operation, all scale features are placed on the same spatial grid, achieving feature alignment, and the computational overhead is negligible.

[0106] Furthermore, to enhance information response and suppress redundancy, each aligned feature map is reweighted through two lightweight gating mechanisms: a channel attention mechanism for global statistics and a spatial attention mechanism for densely populated edges and connection points.

[0107] In this embodiment of the invention, in the multi-scale feature line decoder, the first optimizer and the second optimizer in the wireframe generation unit are two cascaded shallow encoder-decoder optimizers; based on this, the wireframe generation unit parses and obtains the wireframes in the original image, including:

[0108] The first optimizer processes the refined features to generate preliminary unit prediction tensors. ;in, Indicates batch size. Indicates the height of the original image. Indicates the width of the original image;

[0109] The second optimizer, based on the first optimizer, predicts the tensor of the initial cell. Perform fine-tuning and output the corresponding cell prediction tensor. ;

[0110] Predict tensors for each unit Encode the original image at grid locations into a dense prediction tensor with nine physically meaningful channels. ;

[0111] On a quarter grid, the dense prediction tensor is processed by the line segment decoder and the connection point decoder, respectively. Decode the corresponding line segments and connection points to form a candidate set of line segments and a candidate set of connection points;

[0112] On a quarter-resolution feature map, line segments in the candidate set are sorted using the LOI validator;

[0113] Redundancy is suppressed on the sorted line segments, and the final retained line segments and the connection points in the candidate connection point set are scaled to the corresponding coordinates of the original image by a global step of 4, and the wireframe in the original image is obtained by parsing.

[0114] In this embodiment of the invention, the intermediate features generated by the first optimizer Improved through a 1×1 bottleneck layer And output the preliminary unit prediction tensor The second optimizer mirrors this process, starting from D1, and ultimately generates the corresponding cell prediction tensor. It should be noted that during the model training phase... and Simultaneously apply deep supervision, while only using [other methods] during the inference phase. .

[0115] In this embodiment of the invention, the dense prediction tensor obtained by encoding the unit prediction tensor is... Represented as:

[0116]

[0117] In the formula, Indicates the main direction. Indicates the initial side tangent. Indicates the termination of the side tangent. Represents the distance parameter. Represents the residual. This represents the coordinates of the connection points on the bidirectional softmax heatmap. This represents the subpixel offset.

[0118] Among them, the main direction Parameterize the direction of the local line segment as lateral tangent Controlling the initial / terminal slope using tangent value Provides stable angular parameterization around the vertical line. Distance parameter Define the normalization endpoint extension range and use the residuals. Perform smooth modulation. Based on The bidirectional softmax generates a heatmap of connection points on a quarter-mesh grid, while the sub-pixel offset... Refine the coordinates of the connection points within each cell.

[0119] In this embodiment of the invention, the dense prediction tensor Decode the corresponding line segments and connection points to form a set of line segments and a candidate set of connection points, including:

[0120] On a quarter grid, based on dense prediction tensors The formula for calculating the coordinates of the endpoints is as follows:

[0121]

[0122]

[0123] In the formula, Indicates the starting endpoint. Indicates the termination endpoint. Indicates the center point of the cell. Indicates the distance scaling factor. Represents the horizontal component of the main direction of the line segment. Represents the perpendicular component of the principal direction of a line segment. Indicates the initial side slope. Indicates the slope of the terminating side. Indicates the normalization extension range. , Indicates a range-limiting function;

[0124] Candidate line segments are formed based on the calculated endpoint coordinates. ;

[0125] Geometric anchoring snaps the endpoints of candidate line segments to the nearest connection point, thereby anchoring the candidate line segments. Replace with It then determines whether there exists a pair of endpoints where the Euclidean square distance between any endpoint of a candidate line segment and any point in the candidate connection point set is less than a set distance threshold; if so, the candidate line segment is retained; otherwise, it is deleted. Represents points in the candidate set of connection points;

[0126] For the retained candidate line segments, by surrounding Add signed residuals, enumerate the extension range of candidate line segments after disturbance, and form a candidate line segment set;

[0127] Connection points are obtained based on bidirectional softmax heatmaps, and connection points above the confidence threshold are suppressed and filtered by eight-neighbor nonmaximum suppression.

[0128] Based on the selected connection points, sub-pixel offsets are used. The local probabilistic maxima on the bidirectional softmax heatmap are refined to form a candidate set of points; the local probabilistic maxima on the bidirectional softmax heatmap represent the potential locations of the connection points.

[0129] In this embodiment of the invention, the process of sorting line segments includes:

[0130] On the original image at quarter resolution, bilinear feature sampling is performed on each line segment in the candidate line segment set at its two endpoints and at equally spaced internal sampling points.

[0131] The sampled features are concatenated into a descriptor and input into the classifier to obtain the corresponding confidence score;

[0132] Sort the line segments according to their confidence level.

[0133] Example 2:

[0134] This embodiment is a further limitation based on Embodiment 1. Its purpose is to specifically provide a wireframe parsing system based on the wireframe parsing method. Other parts not mentioned refer to Embodiment 1 or the prior art.

[0135] The wireframe parsing system in this embodiment of the invention includes:

[0136] The image acquisition unit is used to acquire raw images and preprocess them into images to be analyzed.

[0137] Wireframe parsing models, including mobile wire parsing ViT and multi-scale feature line decoders;

[0138] ViT, a mobile-end line parsing tool, is used to extract multi-scale features with long-range context dependencies from the image to be parsed by mixing local convolution with token interaction.

[0139] A multi-scale feature line decoder is used to align multi-scale features and perform structure-aware enhancement to parse the wireframes in the original image.

[0140] In embodiments of the present invention, such as Figure 2 As shown, the multi-scale feature line decoder includes a feature refinement unit and a wireframe generation unit;

[0141] The feature refinement unit is used to align multi-scale features through a four-branch structure and output refined features;

[0142] The wireframe generation unit is used to perform progressive optimization of refined features, output dense prediction tensors, and then perform line segment decoding and connection point decoding on the dense prediction tensors through linear decoders and connection point decoders, respectively. The decoded line segments are sorted by the LOI validator, and the sorted line segments and decoded connection points are scaled to the corresponding coordinates in the original image to obtain the wireframe in the original image.

[0143] In this embodiment of the invention, the mobile terminal line parsing ViT includes a first convolutional layer, a first inverted residual block, a second inverted residual block, a third inverted residual block, a fourth inverted residual block, a first MobileViT module, a fifth inverted residual block, a second MobileViT module, a sixth inverted residual block, and a third MobileViT module connected in sequence.

[0144] The third MobileViT module, the second MobileViT module, and the third inverse residual block output corresponding scale feature maps, which are combined with the image to be parsed to form multi-scale features, and then input into the multi-scale feature line decoder.

[0145] In this embodiment of the invention, the feature refinement unit includes a first branch, a second branch, a third branch, a fourth branch, a splicing layer, and a 3×3 convolutional layer that are parallel and have the same structure.

[0146] The first branch, the second branch, the third branch, and the fourth branch all consist of a 1×1 convolutional layer, a mapping operation layer, a spatial attention mechanism, and a channel attention mechanism connected in sequence.

[0147] In the first branch, the input of the 1×1 convolutional layer is connected to the output of the third MobileViT module; in the second branch, the input of the 1×1 convolutional layer is connected to the output of the second MobileViT module; in the third branch, the input of the 1×1 convolutional layer is connected to the output of the third inverted residual block; and in the fourth branch, the input of the 1×1 convolutional layer is connected to the input of the first convolutional layer.

[0148] The feature maps output from the first, second, third, and fourth branches are concatenated along the channel dimension by a splicing layer and refined by a 3×3 convolutional layer to obtain refined features.

[0149] In this embodiment of the invention, the wireframe generation unit includes a first optimizer, a second optimizer, a line segment decoder, a connection point decoder, and an LOI verifier connected in sequence.

[0150] The first and second optimizers process the refined features step by step, outputting a dense prediction tensor; the linear decoder and the connector decoder perform line segment decoding and connector decoding on the dense prediction tensor, respectively, outputting the corresponding line segment candidate set and connector candidate set; the LOI validator sorts the line segments in the line segment candidate set and scales the sorted line segments and connectors in the connector candidate set to the corresponding coordinates in the original image, parsing to obtain the wireframe in the original image.

[0151] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0152] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A deployment-oriented wireframe parsing method, characterized in that, include: Acquire the original image and preprocess it to obtain the image to be analyzed; Build and train the wireframe parsing framework to obtain the wireframe parsing model; The wireframe parsing framework includes mobile wire parsing ViT and a multi-scale feature line decoder; The image to be parsed is input into the wireframe parsing model. The mobile wireframe parsing ViT uses local convolution and token interaction to extract multi-scale features with long-range context dependence in the image to be parsed. The multi-scale features are aligned and structure-aware enhancement is performed by the multi-scale feature line decoder to obtain the wireframe in the original image. In the wireframe analytical model: The mobile-end line parsing ViT includes a first convolutional layer, a first inverted residual block, a second inverted residual block, a third inverted residual block, a fourth inverted residual block, a first MobileViT module, a fifth inverted residual block, a second MobileViT module, a sixth inverted residual block, and a third MobileViT module connected in sequence. The third MobileViT module, the second MobileViT module, and the third inverse residual block each output corresponding scale feature maps, which are combined with the image to be parsed to form multi-scale features, and then input into the multi-scale feature line decoder. The multi-scale feature line decoder includes a feature refinement unit and a wireframe generation unit; The feature refinement unit includes a first branch, a second branch, a third branch, a fourth branch, a splicing layer, and a 3×3 convolutional layer, all arranged in parallel and having the same structure. The first branch, the second branch, the third branch, and the fourth branch each include a 1×1 convolutional layer, a mapping operation layer, a spatial attention mechanism, and a channel attention mechanism connected in sequence. The input of the 1×1 convolutional layer in the first branch is connected to the output of the third MobileViT module; the input of the 1×1 convolutional layer in the second branch is connected to the output of the second MobileViT module; the input of the 1×1 convolutional layer in the third branch is connected to the output of the third inverted residual block; and the input of the 1×1 convolutional layer in the fourth branch is connected to the input of the first convolutional layer. The feature maps output from the first branch, the second branch, the third branch, and the fourth branch are spliced ​​along the channel dimension by a splicing layer and refined by a 3×3 convolutional layer to obtain refined features; The wireframe generation unit includes a first optimizer, a second optimizer, a line segment decoder, a connection point decoder, and an LOI verifier connected in sequence. The first optimizer and the second optimizer process the refined features step by step and output the dense prediction tensor. The linear decoder and the connector decoder perform line segment decoding and connector decoding on the dense prediction tensor, respectively, and output the corresponding line segment candidate set and connector candidate set. The LOI validator sorts the line segments in the line segment candidate set and scales the sorted line segments and connectors in the connector candidate set to the corresponding coordinates in the original image, and parses the wireframe in the original image.

2. The deployment-oriented wireframe parsing method according to claim 1, characterized in that, The first, second, third, fourth, fifth, and sixth inverted residual blocks have the same structure and are identical to the input image. The processing procedure is represented as follows: In the formula, This represents the SiLU activation function. express Pointwise convolution, express Depthwise convolution, Indicates that the input image X passes through Pointwise convolution increases the channel dimension, resulting in an expanded feature map. This indicates that the extended feature map is being processed. Spatial convolution captures feature maps of spatial features. This represents the feature map of the final output.

3. The deployment-oriented wireframe parsing method according to claim 1, characterized in that, The mapping operation layer adjusts the resolution of the feature map output by the 1×1 convolutional layer to... To achieve scale feature alignment; Among them, the feature map after scale feature alignment Represented as: In the formula, Indicates a mapping operation. This indicates that multi-scale features are projected onto a uniform number of channels. A 1×1 convolutional layer, Representing multi-scale features, Indicates batch size. Indicates the first The number of channels in the layer feature map. Indicates the height of the original image. Indicates the width of the original image. Representing multi-scale features The corresponding downsampling factor; Among them, the current sampling factor At that time, mapping operation For bilinear upsampling; when At that time, mapping operation To maintain identity; when At that time, mapping operation For step anti-aliasing pooling.

4. The deployment-oriented wireframe parsing method according to claim 1, characterized in that, The wireframe generation unit parses and obtains the wireframes from the original image, including: The first optimizer processes the refined features to generate preliminary unit prediction tensors. ;in, Indicates batch size. Indicates the height of the original image. Indicates the width of the original image; The second optimizer, based on the first optimizer, predicts the tensor of the initial cell. Perform fine-tuning and output the corresponding cell prediction tensor. ; Predict tensors for each unit Encode the original image at grid locations into a dense prediction tensor with nine physically meaningful channels. ; On a quarter grid, the dense prediction tensor is processed by the line segment decoder and the connection point decoder, respectively. Decode the corresponding line segments and connection points to form a candidate set of line segments and a candidate set of connection points; On a quarter-resolution feature map, line segments in the candidate set are sorted using the LOI validator; Redundancy is suppressed on the sorted line segments, and the final retained line segments and the connection points in the candidate connection point set are scaled to the corresponding coordinates of the original image by a global step of 4, and the wireframe in the original image is obtained by parsing.

5. The deployment-oriented wireframe parsing method according to claim 4, characterized in that, The dense prediction tensor Represented as: In the formula, Indicates the main direction. Indicates the initial side tangent. Indicates the termination of the side tangent. Represents the distance parameter. Represents the residual. This represents the coordinates of the connection points on the bidirectional softmax heatmap. This represents the subpixel offset.

6. The deployment-oriented wireframe parsing method according to claim 5, characterized in that, Dense prediction tensors Decode the corresponding line segments and connection points to form a set of line segments and a candidate set of connection points, including: On a quarter grid, based on dense prediction tensors The formula for calculating the coordinates of the endpoints is as follows: In the formula, Indicates the starting endpoint. Indicates the termination endpoint. Indicates the center point of the cell. Indicates the distance scaling factor. Represents the horizontal component of the main direction of the line segment. Represents the perpendicular component of the principal direction of a line segment. Indicates the initial side slope. Indicates the slope of the terminating side. Indicates the normalization extension range. , Indicates a range-limiting function; Candidate line segments are formed based on the calculated endpoint coordinates. ; Geometric anchoring snaps the endpoints of candidate line segments to the nearest connection point, thereby anchoring the candidate line segments. Replace with It then determines whether there exists a pair of endpoints where the Euclidean square distance between any endpoint of a candidate line segment and any point in the candidate connection point set is less than a set distance threshold; if so, the candidate line segment is retained; otherwise, it is deleted. Represents points in the candidate set of connection points; For the retained candidate line segments, by surrounding Add signed residuals, enumerate the extension range of candidate line segments after disturbance, and form a candidate line segment set; Connection points are obtained based on bidirectional softmax heatmaps, and connection points above the confidence threshold are suppressed and filtered by eight-neighbor nonmaximum suppression. Based on the selected connection points, sub-pixel offsets are used. The local probability maxima on the bidirectional softmax heatmap are refined to form a candidate set of points.

7. The deployment-oriented wireframe parsing method according to claim 4, characterized in that, Sort the line segments, including: On the original image at quarter resolution, bilinear feature sampling is performed on each line segment in the candidate line segment set at its two endpoints and at equally spaced internal sampling points. The sampled features are concatenated into a descriptor and input into the classifier to obtain the corresponding confidence score; Sort the line segments according to their confidence level.

8. A deployment-oriented wireframe parsing system, implemented based on the wireframe parsing method according to any one of claims 1 to 7, characterized in that, include: The image acquisition unit is used to acquire raw images and preprocess them into images to be analyzed. Wireframe parsing models, including mobile wire parsing ViT and multi-scale feature line decoders; The mobile terminal line parsing ViT is used to extract multi-scale features with long-range context dependence in the image to be parsed by a hybrid processing of local convolution and token interaction. The multi-scale feature line decoder is used to align multi-scale features and perform structure-aware enhancement to parse and obtain wireframes in the original image.

9. The wireframe parsing system according to claim 8, characterized in that, The multi-scale feature line decoder includes a feature refinement unit and a wireframe generation unit; The feature refining unit is used to align multi-scale features through a four-branch structure and output refined features; The wireframe generation unit is used to perform stepwise optimization of the refined features, output dense prediction tensors, and perform line segment decoding and connection point decoding on the dense prediction tensors through a linear decoder and a connection point decoder, respectively. The decoded line segments are sorted through an LOI validator, and the sorted line segments and decoded connection points are scaled to the corresponding coordinates of the original image to obtain the wireframe in the original image.

Images