Dual-branch complementary semantic line detection method based on transformer

Abstract

The semantic line is of great significance to advanced visual tasks such as scene perception and scene understanding. Due to the ability to use simple line structures to give intuitive visual interpretation to image scenes, semantic line detection tasks have received more attention in recent years. However, the existing semantic line detection methods have the problems of difficult end-to-end optimization of the model, imbalance of the number of positive and negative class samples, and low line semantic correlation caused by the lack of global modeling. To solve the above problems, the invention designs and implements a double-branch complementary semantic line detection model and a complementary detection method based on the Transformer model as the main structure. The overall detection process of the method can be divided into three stages: (1) extraction and use of multi-scale image features and spatial transformation of the depth Hough transform (2) double-branch decoding from coarse granularity to fine granularity (3) semantic line complementary detection by implementing heat map prediction method and parameter regression method.

Description

TECHNICAL FIELD

[0001] The application provides a dual-branch complementary semantic line detection method based on a Transformer, and belongs to the field of pattern recognition and devices. BACKGROUND

[0002] As a key visual clue for conveying semantic structure layout information in natural images, semantic lines are of great significance to advanced visual tasks such as scene perception and scene understanding. In recent years, with the vigorous development of computer vision technology, and more and more scholars shifting their research focus from scene perception to scene understanding, the semantic line detection task has therefore received more attention. However, the existing semantic line detection methods have the problems of difficulty in end-to-end optimization of the model, imbalance in the number of positive and negative class samples, and lack of global modeling leading to low line semantic correlation.

[0003] Therefore, it is necessary to propose a semantic line detection model that can be optimized end-to-end and consider global modeling to improve line semantic correlation.

[0004] Current semantic line detection related research mainly has two directions, namely the semantic line detection method based on candidate lines and the multi-stage detection process, and the semantic line detection method based on deep Hough transform and heat map prediction: 1) The semantic line detection method based on candidate lines follows the idea of target detection algorithm, uses a deep neural network to extract the local features of randomly sampled candidate lines, and then performs classification and regression tasks to obtain the detection results. However, the method based on candidate lines and the multi-stage detection process cannot perform end-to-end training, and non-end-to-end optimization can achieve optimal solutions on sub-problems, but due to the inconsistent training objectives of each sub-model, performance is often compromised when the system is composed. 2) The semantic line detection method based on deep Hough transform and heat map prediction inherits the idea of traditional line segment detection algorithms, maps image features to Hough space, represents a line in image space with a point in Hough space, and then performs a binary classification task on each pixel point to obtain a parameterized representation of semantic lines, and finally inversely maps the detection results back to the original space. However, this method is easily affected by the imbalance between sample classes, as the distribution of semantic lines in the image is sparse and each semantic line is parameterized as a coordinate point, resulting in a generated heat map supervision signal with only a few positive pixel labels, further leading the model to prefer to predict samples as negative class and lose recall rate. SUMMARY

[0005] In view of the above-mentioned problems, the application takes the Transformer model in the paper "Attention is all you need" (Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [J]. Advances in neural information processing systems, 2017, 30.) as the main structure, and designs and implements a double-branch complementary semantic line detection model and a complementary detection method based on the Transformer. The overall detection process of the method can be divided into three stages: (1) extraction of multi-scale image features and spatial transformation using a deep Hough transform (2) double-branch decoding from coarse granularity to fine granularity (3) semantic line complementary detection by implementing a heat map prediction method and a parameter regression method. The overall semantic line detection method is as shown in Figure 1 The feature extraction and spatial transformation of the image can extract multi-scale image features and convert the depth features to the Hough parameter space. The double-branch decoding from coarse granularity to fine granularity obtains semantic content decoding features and line query decoding features through the interaction of the semantic content branch and the semantic query branch. The semantic line complementary detection is based on the output of the model double-branch decoder, adopts a heat map prediction method and a parameter regression method for prediction, and can realize the complementary fusion of the two prediction sets. In the last actual detection process, the semantic line parameters are inversely mapped back to the original image space to obtain the four-tuple coordinate representation of each semantic line, and the whole detection process is completed.

[0006] The overall structure and specific detection steps of the double-branch complementary semantic line detection method based on the Transformer proposed by the application are as follows:

[0007] Step 1, use the model encoder to extract multi-scale image features, and use the deep Hough transform to convert the depth feature representation from the image space to the Hough parameter space. Since the coupling degree of the two steps is low, the backbone network of the encoder can adopt any multi-scale feature extraction network. This step specifically includes:

[0008] Step 1.1 The backbone network inputs the original image I and obtains a set of multi-scale feature maps after four-stage feature encoding

[0009] Step 1.2 After completing the image feature extraction step, the deep Hough transform operation obtained by extending the traditional Hough transform method in the field of image processing in the paper “Deep Hough transform for semantic line detection” (Zhao K, Han, Zhang C B, et al. Deep Hough transform for semantic line detection [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021) is used. Since it is hoped that the transformed feature map can still be represented by a three-dimensional tensor, it is necessary to set the quantization interval to quantize the continuous line parameter values to a discrete space.

[0010] Step 1.3 The deep Hough transform operation is used to spatially transform the image features so that the line structure semantic information can be more compactly represented in the subsequent decoding process.

[0011] Step 1.4 The number of quantization levels is calculated simultaneously in the quantization process. The smaller the quantization interval, the greater the number of quantization levels, which means the smaller the quantization error, but also results in a larger size of the transformed feature map. In order to balance the performance and efficiency of the model and match the original feature map size, the deep Hough transform adopts a fixed number of angle quantization levels for each layer of feature maps and a distance quantization level that increases layer by layer.

[0012] Step 1.5 For any straight line l, its corresponding image feature is aggregated to a point in the parameter space feature map H after deep Hough transform, where i is the position index, is the quantized line parameter corresponding to the straight line l. Finally, the output of the deep Hough transform is obtained by transformation to obtain a multi-scale feature map in the Hough parameter space where DHT(·) represents the deep Hough transform, is a multi-scale feature map.

[0013] Step 2, dual-branch decoding from coarse to fine granularity. After obtaining the line structure feature representation in the Hough parameter space, the model decoder is used to start decoding from the highest level of coarse-grained feature map and iteratively receive more fine-grained semantic information layer by layer. Overall, the decoder has a dual-branch structure, which is called the semantic content branch and the semantic query branch, respectively; from a local point of view, these two branches are respectively built by stacking semantic content modules and semantic query modules at each level of the decoder. This step specifically includes:

[0014] Step 2.1. The semantic content module takes the Transformer block as the main body to mine valuable semantic content information contained in the features. On the basis of the Hoop parameter space feature map output by the model encoder providing a good representation ability for line structure features, the semantic content module continues to strengthen the feature expression of the region position with large semantic difference in the parameter space. Since the information that the parameter space feature map can represent is the basic unit of line structure, that is, each pixel in the parameter space feature map aggregates the feature information contained in a group of adjacent straight lines in the original image space feature map, therefore, applying local region operation to the parameter space feature map is actually equivalent to applying linear region operation to the image space feature map. This module specifically includes:

[0015] Step 2.1.1. A bilateral region attention mechanism is designed in the semantic content module to replace the multi-head self-attention mechanism originally used in the Transformer block to improve its performance and efficiency in processing dense feature representation. As shown in Figure 2 is the structure diagram of the bilateral region attention module.

[0016] Step 2.1.2. The bilateral region attention module directly receives the parameter space feature map as input and performs bilateral region feature generation according to the flow shown in (a). In the figure, θ and r are the line parameters corresponding to the straight line l, and the purpose is to gather bilateral region feature information for each implicit candidate line to form a group of bilateral region feature maps that can respectively describe the left and right region features of the line structure. First, generate a Gaussian weighted feature map. Figure 2

[0017] Step 2.1.3. Each obtained Gaussian weighted feature map is split into left and right two-part region features according to the reference axis to obtain the split region feature map. Then, each region feature is aggregated into a pixel point representation. The region aggregation features calculated with different columns as the reference axis are spliced in the column dimension to form a bilateral region feature map with the same size as the parameter space feature map.

[0018] Step 2.1.4. Cosine similarity is used to measure the semantic similarity of the bilateral region, where i represents the index of the bilateral region feature map, and ‖·‖ represents the modulus of the vector, and is a group of bilateral region feature maps finally obtained by the bilateral region feature generation process. The meaning of each pixel point feature is the bilateral region semantic information of the corresponding line structure. This similarity is used to measure the semantic similarity of the bilateral region, thereby comparing the heterogeneity degree of the bilateral region semantic information. This process calculates the semantic similarity of the bilateral region of the corresponding line structure for each pixel position in the bilateral region feature map, and the higher the similarity, the smaller the semantic difference between the left and right regions.

[0019] Step 2.1.5. By​ Computing bilateral region attention map A c , wherein and is a set of bilateral region feature maps finally obtained by the bilateral region feature generation process. After obtaining the bilateral region attention map, a simple and efficient attention weighting method H a = (1 + A c ) o H is used to calculate the attention weighted fusion result H a , wherein o represents element multiplication operation.

[0020] Step 2.1.6, except that the bilateral region attention mechanism is used instead of the multi-head self-attention mechanism, the semantic content module also adds a deep convolution to the feedforward neural network constructed by the full connection layer in the standard Transformer block.

[0021] Step 2.1.7 The last part of the semantic content module is consistent with the Transformer block, which consists of a skip connection and layer normalization.

[0022] The semantic query module in step 2.2 implements line structure feature extraction on the parameter space feature map in another decoding manner. It mainly learns a set of line query embeddings as candidate line agents, queries and aggregates the region features of interest in the feature map through cross-attention mechanism, and finally directly predicts the semantic line parameters through parameter regression. The present application adopts an improved scheme named Conditional DETR in the paper “Conditional detr for fast training convergence” (Meng D, Chen X, Fan Z, et al. Conditional detr for fast training convergence [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 3651-3660.) as the basic structure of the semantic query module, and designs and proposes a Gaussian prior cross-attention mechanism to replace the multi-head cross-attention mechanism in the original model, which reduces the attention space range of the line query by constraint to reduce its dependence on the semantic content features, so that it can focus more on the features near the predicted line structure position, forming a more efficient detection mode.

[0023] Step 2.2.1 The semantic content module first receives the Hough parameter space feature map Hi from the model encoder output of the corresponding level, and obtains the semantic content intermediate feature representation after model decoding, and combines it with the line query decoding feature and line query embedding Q l together as the input of the semantic query module, where N denotes the total number of line queries.

[0024] Step 2.2.2. Intermediate feature of semantic content input line query decoding feature output by the previous layer decoder and line query embedding Q l After decoding by the semantic query module, the updated line query decoding feature is obtained and the attention map generated by the Gaussian prior cross-attention

[0025] Step 2.2.3. Then, in order to promote the exchange of information between the two branches of the network and make full use of the supervision signal to improve the generalization ability of the model, the model decoder uses an attention fusion method to introduce complementary information to the semantic content intermediate feature representation during the decoding process. The attention fusion method uses a concatenation operation in the channel dimension to concatenate the semantic content intermediate feature representation with the attention map , which means that the attention area information of the line query on the feature map is supplemented to the semantic content intermediate feature representation.

[0026] Step 2.2.4. The semantic content decoding features output by different decoding layers are fused together using a feature pyramid network-based method to form multi-scale decoding feature representations with rich high-level semantic and spatial detail information. The fusion process of adjacent decoding layers is where Upsample(·) represents an upsampling operation that uses a bilinear interpolation method to upsample the feature map of the previous layer to align with the feature map scale of the current layer, denotes the semantic content decoding features output by different decoding layers, denotes the semantic content decoding features. The output of the final model decoder contains the decoding results of the two branches, i.e., the semantic content decoding features obtained after four decoding layers and the line query decoding feature

[0027] Step 3. Semantic line complement detection as the last stage of the overall model detection process, mainly undertakes the post-processing steps such as result prediction, result complement, and line parameter inverse mapping. Based on the semantic content decoding features S c and the line query decoding feature S q with different semantic properties and meanings, the heat map prediction method and the parameter regression method are used for prediction in the prediction stage, and the heat map prediction head F h and the parameter regression head F p are designed correspondingly. This step specifically includes:

[0028] Step 3.1 In the result prediction process, the model is equipped with a heat map prediction head and a parameter regression head for the heat map prediction task and the parameter regression task, respectively. For the heat map prediction task, it can be regarded as performing a binary classification task on each pixel in the feature map, and the prediction head adopts a two-layer perceptron composed of convolutional layers.

[0029] Step 3.2 By selecting a heat map prediction threshold τ h as the division standard for predicting positive and negative samples, and connecting all adjacent sample points predicted as positive to form several connected regions, the centroid coordinates and corresponding probabilities of each connected region are calculated as the parameters and confidence results of the predicted semantic lines. By forming the parameter set of the predicted semantic lines and the confidence result set C c from all the prediction results in the heat map, the final detection results of the semantic content branch output can be obtained. At the same time, in the heat map prediction task, given the real heat map label G h , the classical binary cross-entropy loss can be adopted, in addition to which, considering the large gap in the proportion of positive and negative samples in the real heat map label and the certain aggregation of positive samples, and the cross-entropy loss is calculated independently for each pixel prediction result, which is a region-independent loss, therefore, the region-related Dice loss is also added as a supplement, where G h is the given real heat map label, P h is the predicted parameter space heat map, and |·| represents the sum of all element values.

[0030] Step 3.3 For the parameter regression task, two three-layer perceptrons are used to predict the line parameters and confidence of all line queries, and in particular, the parameter coordinates directly predicted by the line query decoding feature are relative position coordinates based on the reference point, so the absolute position coordinates need to be obtained by adding the reference point coordinates. In the parameter regression task, the bipartite graph matching algorithm is used to find a best match for the bipartite graph composed of the line query prediction results and the real semantic line label, which contains the M line queries with the highest overall relevance to the real label. After the bipartite graph is constructed, the maximum weight matching of the graph is calculated using , where is the indicator function, σ(·) is the permutation function defined on the set of natural numbers, and w(i, j) is the weight of the undirected edge between each line query vertex and all real label vertices. Based on the optimal permutation function, the L1 distance loss and the Focal loss can be calculated for the semantic line parameters and confidence of the line query prediction, respectively.

[0031] Step 3.4. The two branches of the network independently complete the semantic line detection task by using the features obtained by decoding, and thus there are certain differences in the detection results. After the two branches of the network complete the semantic line detection task, the detection results are first combined into the same set, C u =C c ∪C q , wherein and C u represent the union of the detection results of the two branches of the semantic line, C c , C q respectively represent the semantic line parameters and the confidence obtained by the heat map prediction method and the parameter regression method.

[0032] Step 3.5. With reference to the idea of removing redundant bounding boxes in the target detection task by using the non-maximum suppression algorithm, the algorithm can also be used to suppress the coordinates of the semantic lines close to each other in the neighborhood by modifying the suppression judgment condition, so as to remove the repeated semantic line detection results.

[0033] Step 4. By applying the inverse mapping of the parameterized mapping function, the semantic line parameters are converted back to the original image space, and the final semantic line detection result L is obtained, and thus the entire detection task is completed.

[0034] The beneficial effects of the present application are as follows:

[0035] 1. The model of the present application supports efficient end-to-end training optimization and prediction. The proposed method couples the outputs of the two branches by using the attention fusion method to form complementary feature representations, combines the two prediction heads to undertake the heat map prediction and parameter regression tasks, and uses the complementary detection method to weaken the influence of the label sparsity problem in the dense prediction task on the performance as much as possible.

[0036] 2. The model uses an encoder-decoder architecture, wherein the encoder can be compatible with any multi-scale feature extraction network. The semantic content branch and the semantic query branch in the decoder of the model decode the parameter space features from coarse granularity to fine granularity.

[0037] 3. Meanwhile, the decoder of the model is designed and implemented by using the Transformer block as the basic structure, fully utilizes the global representation and the ability to capture long-distance dependencies of the Transformer, cooperates the bilateral region attention and the Gaussian prior cross attention to strengthen the local and global feature semantic context, and promotes the prediction of the globally optimal semantic line set. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the structure diagram of the dual-branch complementary semantic line detection model of the present application based on the Transformer.

[0039] Figure 2 is the structure diagram of the bilateral region attention module in the model of the application;

[0040] Figure 3 is the structure diagram of the Gaussian prior cross-attention module in the semantic query module of the application;

[0041] Figure 4 is the schematic diagram of the semantic line detection process based on the heat map prediction task of the application;

[0042] Figure 5 is the non-maximum suppression algorithm used by the application. DETAILED DESCRIPTION

[0043] The application will be explained in more detail below with reference to the complete model structure diagram.

[0044] The application uses a Transformer model as the main structure, and designs and implements a dual-branch complementary semantic line detection model and a complementary detection method based on the Transformer. Overall, the detection process of the method includes the extraction of multi-scale image features and the spatial transformation using a deep Hough transform; dual-branch decoding from coarse granularity to fine granularity; and semantic line complementary detection using a heat map prediction method and a parameter regression method. The specific modules are implemented and described as follows:

[0045] Step 1, use the model encoder to extract multi-scale image features, and use the deep Hough transform to convert the deep feature representation from image space to Hough parameter space. Since the coupling degree of these two steps is low, the backbone network of the encoder can use any multi-scale feature extraction network. This step specifically includes:

[0046] 1.1 The input size of the backbone network is an original image I with a size of HxWx3, where H and W are the height and width of the original image, respectively. After four-stage feature encoding, a set of multi-scale feature maps is obtained where where C i is the number of feature channels, and the multi-scale feature maps obtained from the high layer of the encoder to the low layer of the encoder are set to times the size of the original image.

[0047] 1.2 After completing the image feature extraction step, the deep Hough transform operation is obtained by extending the Hough transform method in the traditional image processing field. The straight line in the image space can be parameterized as a pair of parameters (θ, r). Since it is desired that the transformed feature map can still be represented by a three-dimensional tensor, it is necessary to set the quantization intervals Δθ and Δr to quantize the continuous line parameters to the discrete space. The quantization process is where and are quantized line parameters.

[0048] 1.3 Spatial transform of image features using the deep Hough transform operation to represent line structure semantic information more compactly in the subsequent decoding process.

[0049] 1.4 Calculate the quantization step size simultaneously with the quantization process and The smaller the quantization interval, the greater the quantization step size means the smaller the quantization error, but also leads to the larger size of the transformed feature map. To balance the performance and efficiency of the model, and match the original feature map size, the deep Hough transform adopts a fixed angle quantization step size Θ = 96 and a layer-by-layer increasing distance quantization step size R = [12, 24, 48, 96] for each layer of feature map.

[0050] 1.5 For any straight line l, its corresponding image feature is aggregated to the point in the parameter space feature map H after the deep Hough transform. where i is the position index, representing the pixel position along the straight line l in the image space, is the quantized line parameter corresponding to the straight line l.

[0051] 1.6 Finally, the output of the deep Hough transform is obtained by transforming to obtain the multi-scale feature map in the Hough parameter space where DHT(·) represents the deep Hough transform, is the multi-scale feature map, BN(·) represents batch normalization, RELU(·) represents linear rectifier unit activation function, Conv 3*3 (·) represents a convolution layer with a convolution kernel size of 3 × 3.

[0052] Step 2, dual-branch decoding from coarse to fine granularity. After obtaining the line structure feature representation in the Hough parameter space, the model decoder is used to start decoding from the highest level of coarse-grained feature map and iteratively receive more fine-grained semantic information layer by layer. Overall, the decoder has a dual-branch structure, respectively called semantic content branch and semantic query branch; from the local point of view, these two branches are respectively built by stacking semantic content modules and semantic query modules at each level of the decoder. This step specifically includes:

[0053] 2.1 The semantic content module takes the Transformer block as the main body to mine valuable semantic content information contained in the features. On the basis of the line structure feature provided by the Hough parameter space feature map output by the model encoder, the semantic content module continues to strengthen the feature expression of the region position with large semantic difference in the parameter space. Since the information that the parameter space feature map can represent is the basic unit of line structure, that is, each pixel in the parameter space feature map aggregates the feature information contained in a group of adjacent straight lines in the original image space feature map, therefore, applying local region operation to the parameter space feature map is actually equivalent to applying linear region operation to the image space feature map. This module specifically includes:

[0054] 2.1.1 A bilateral region attention mechanism is designed in the semantic content module to replace the multi-head self-attention mechanism originally used by the Transformer block to improve its performance and efficiency in processing dense feature representation. The bilateral regions separated by the semantic line should exhibit a certain heterogeneity, and the bilateral region attention focuses on the semantic information of the regions on both sides of all potential line structures, and adjusts the attention degree from near to far, measures the heterogeneity of the bilateral region semantics through similarity comparison, and thus calculates the attention map for weighting for the entire parameter space feature map. As shown in Figure 2 is the structure diagram of the bilateral region attention module. The higher the attention weight of a region, the stronger the heterogeneity of the bilateral regions of the line structure corresponding to the region, that is, the greater the probability that the line structure serves as a semantic line.

[0055] 2.1.2 The bilateral region attention module directly receives the parameter space feature map as input, and performs bilateral region feature generation according to the flow shown in (a) of Figure 2 , the purpose of which is to gather bilateral region feature information for each implicit candidate line to form a set of bilateral region feature maps that can respectively describe the left and right region features of the line structure. First, a Gaussian weighted feature map is generated. The calculation process is where H is the input parameter space feature map, c is each column of the feature map, where i represents the position index of each pixel in the feature map, represents the horizontal distance between position i and column c, and G(·) represents the monovariate Gaussian probability density function, whose standard deviation σ is set to represents the Gaussian weighted feature map generated with column c as the reference axis.

[0056] 2.1.3 Each obtained Gaussian weighted feature map is split into left and right two-part region features according to the reference axis to obtain split region feature maps and For the case where the reference axis is located at the boundary of the feature map, there are Where c is the column number, R is the number of columns in the input parameter space feature map, D is the number of channels in the input parameter space feature map, and w l and w r These are the feature indices of the left and right regions, respectively, divided along the reference axis. The calculation process will follow. and The feature maps of the split regions are summed column-wise, and the features of each region are aggregated into a single pixel representation. and These are the left and right side region clustering features generated with column c as the reference axis, respectively. Finally, the region clustering features calculated with different columns as reference axes are concatenated along the column dimension to form a bilateral region feature map at the same scale as the parameter space feature map. This process can be represented as follows: and Concat(·) represents the feature concatenation operation. and This is a set of bilateral region feature maps obtained at the end of the bilateral region feature generation process. The meaning of each pixel feature is the semantic information of the bilateral region of its corresponding line structure.

[0057] 2.1.4 Using cosine similarity This measure assesses the semantic similarity between the two regions, thereby comparing the degree of heterogeneity in their semantic information. Here, i represents the location index of the feature map of the two regions, and ||·|| represents the magnitude of the vector. and This is the final set of bilateral region feature maps obtained from the bilateral region feature generation process. The meaning of each pixel's feature is the semantic information of the bilateral region of its corresponding line structure. This process calculates the semantic similarity of the bilateral region of the corresponding line structure for each pixel position in the bilateral region feature map. The higher the similarity, the smaller the semantic difference between the left and right parts of the region.

[0058] 2.1.5 Pass Calculate bilateral region attention map A c This strengthens the representation of line structure features with strong semantic heterogeneity in both sides, among which and This is the final set of bilateral region feature maps obtained from the bilateral region feature generation process. After obtaining the bilateral region attention maps, a simple and efficient attention weighting method H is used. a =(1+A) c )⊙H calculates the attention-weighted fusion result H on the parameter space feature map. a , where ⊙ represents element-wise multiplication.

[0059] 2.1.6 In addition to using a bilateral region attention mechanism instead of a multi-head self-attention mechanism, the semantic content module also adds a deep convolution to the feedforward neural network built by the full connection layer in the standard Transformer block. Specifically, for the deep convolution feedforward neural network, the input is the attention-weighted parameter space feature Ha, and the calculation process is In the specific implementation process represents the intermediate feature generated in each step, DWConv 3×3 (·) represents a deep convolution layer with a convolution kernel size of 3x3, GELU(·) represents a Gaussian error linear unit activation function, Conv 1×1 (·) represents a convolution layer with a convolution kernel size of 1x1.

[0060] 2.1.7 The last part of the semantic content module is consistent with the Transformer block, which consists of a skip connection and layer normalization, that is, where LN(·) represents layer normalization, H c represents the intermediate feature representation output by the semantic content module.

[0061] 2.2 Compared with the semantic content module, which uses local feature modeling to mine semantic content information for all implicit candidate lines and activate the corresponding region of the heat map to obtain semantic line parameters, the semantic query module realizes line structure feature extraction on the parameter space feature map in another decoding manner. It mainly learns a set of line query embeddings as candidate line proxies, queries and aggregates the region features of interest in the feature map with the help of cross-attention mechanism, and finally directly predicts the semantic line parameters through parameter regression. The present application adopts an improved scheme named Conditional DETR in the paper “Conditional detr for fast training convergence” (Meng D, Chen X, Fan Z, et al. Conditional detr for fast training convergence [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 3651-3660) as the basic structure of the semantic query module, and designs and proposes a Gaussian prior cross-attention mechanism to replace the multi-head cross-attention mechanism in the original model. By constraining and narrowing the attention space range of the line query, the dependence of the line query on the semantic content feature is reduced, so that it can focus more on the features near the predicted line structure position, forming a more efficient detection mode.

[0062] 2.2.1 The semantic query module receives three parts of input, which are the semantic content intermediate representation H c , the iteratively updated line query decoding feature S q and the line query embedding Q l . First, the line query decoding feature and the line query embedding are fed into the multi-head self-attention module to realize the global-level line semantic feature interaction by calculating the attention between each line query. This part of the decoding process can be represented as wherein represents the line query decoding feature output by the multi-head self-attention module, Q, K, V are parameters of the self-attention module, Softmax(·) represents the Softmax function.

[0063] 2.2.2 After the multi-head self-attention module, a layer of skip connection and layer normalization operation should be provided according to the standard Transformer block structure

[0064] 2.2.3 Subsequently, before extracting the semantic content feature by using the cross-attention module, it is necessary to generate a set of position encodings suitable for the current decoding stage for the line query, following the model setting of ConditionalDETR. The first step to generate the line query position encoding is to predict a set of reference points from the line query embedding The purpose is to decouple the spatial position information from the line query embedding, so as to provide a reference position for the generation of the line query position encoding. The calculation process is and wherein represents the learnable weight parameter, represents the unnormalized reference point coordinates. The reference point coordinates are also embedded into the position encoding representation, and the embedding method adopts the sine-cosine absolute position encoding and wherein pos represents the normalized coordinate value, i represents the index of the element in the position encoding, d represents the dimension of the position encoding, which is set to Since the reference point is represented as a two-dimensional coordinate, it is necessary to calculate the position encoding for the two coordinates separately, and then obtain the overall position encoding Q pe = PE(Q p ) = Concat(PE(x p ), PE(y p )), (x p , y p) represents the two-dimensional coordinates of the reference point. After obtaining the position encoding of the reference point, in the same way, the position information contained in the line query decoding feature is decoupled and mapped to the same position encoding space by a three-layer feedforward neural network, and the calculation process is and wherein all represent learnable weight parameters, represents the extracted decoding feature position information.

[0065] 2.2.4 Convert the reference point position encoding to the line query position encoding through a simple and efficient linear mapping At this point, all the inputs required by the multi-head cross attention module are prepared, which are the semantic content intermediate feature representation H c , the line query decoding feature and the line query position encoding S pe .

[0066] 2.2.5 Attention mechanism-based calculation process, the decoding process is H pe represents the position encoding of the semantic content intermediate feature, and the generation method is the same as Q pe , and the position encoding of all position indexes in the feature map is calculated respectively.

[0067] 2.2.6 In addition, the present application proposes a Gaussian prior cross attention module to improve the multi-head cross attention module. The Gaussian prior cross attention module generates a Gaussian prior space attention weight dynamically using the line query in the decoding process, modulates the attention map obtained by the original cross attention calculation, and guides the line query to pay more attention to the line structure features in the area near the predicted parameter coordinates. The module structure is as follows Figure 3 . The line query parameter regression step predicts a focus point coordinate for each line query using the input query Q and the unnormalized reference point coordinate , and the calculation process is wherein is an intermediate quantity, W ap represents a learnable weight parameter, and Q ap represents the line query focus point coordinate.

[0068] 2.2.7 Using the line query focus point coordinate, combined with the Gaussian distribution probability density function, a Gaussian prior attention map can be generated for each line query, and the calculation process is wherein A g represents the generated Gaussian prior attention map, (i, j) represents the position index of the attention map, and (p θ , p r) denotes the line query attention focus coordinates, and σ denotes the standard deviation of the Gaussian distribution, and is set from large to small

[0069] 2.2.8Next, the present application will improve the multi-head cross attention mechanism calculation process of the Gaussian prior attention map to where Attention GP (·) denotes the Gaussian prior cross attention mechanism, Q, K, and V are parameters.

[0070] 2.2.9In addition to improvements in the calculation method, the Gaussian prior cross attention module also returns the attention weighted features and the modulated attention map A ca According to the standard Transformer block structure design, both of the above should be optimized through residual connection and feedforward neural network to express the features, and the calculation process is A q = LN(DWFFN(A ca ) + A ca ), where A ca is the modulated attention map of the Gaussian prior cross attention module, S q and A q are the final outputs of the semantic query module, S q is the iteratively updated line query decoding feature, A q participates in the attention fusion operation of the current decoding layer as input, and FFN(·) is calculated in the same way as

[0071] 2.3In order to promote the exchange of information between the two branches of the network and make full use of the supervised signal to improve the generalization ability of the model, the model decoder uses the attention fusion method to introduce complementary information to the semantic content intermediate feature representation in the decoding process. The attention fusion method uses the concatenation operation in the channel dimension to concatenate the semantic content intermediate feature representation and the attention map The meaning is to supplement the line query attention region information on the feature map to the semantic content intermediate feature representation, and the calculation process is where denotes the semantic content decoding feature.

[0072] 2.4Finally, the method based on the feature pyramid network is used to fuse the semantic content decoding features output by different decoding layers to form multi-scale decoding feature representations with rich high-level semantic and spatial detail information, and the fusion process of adjacent decoding layers is where Upsample(·) denotes an up-sampling operation using bilinear interpolation method to align the scale of the previous layer feature map to the current layer feature map, denote semantic content decoding features representing different semantic contents of the decoded layer output, denote semantic content decoding features. The output of the final model decoder contains the dual-branch decoding results, i.e., semantic content decoding features obtained by four decoding layers and line query decoding features

[0073] Step 3, semantic line complementary detection as the last stage of the overall model detection process, mainly undertakes the post-processing steps such as result prediction, result complement, and line parameter inverse mapping. Based on the semantic content decoding features S c and line query decoding features S q with different semantic properties and meanings, the heat map prediction method and the parameter regression method are used for prediction in the prediction stage, and the heat map prediction head F h and the parameter regression head F p are designed correspondingly. This step specifically includes:

[0074] 3.1 In the result prediction process, the model is equipped with a heat map prediction head and a parameter regression head for heat map prediction and parameter regression tasks, respectively. For the heat map prediction task, it can be regarded as performing a binary classification task for each pixel in the feature map. The prediction head adopts a two-layer perceptron composed of convolutional layers, and the calculation process is S' c = ReLU(BN(Conv 3×3 (S c ))). where P h denotes the predicted parameter space heat map, and the higher the value of a point in the heat map, the greater the probability that the straight line corresponding to the coordinate of the point is a semantic line.

[0075] 3.2 By selecting a heat map prediction threshold τ h as the division standard for predicting positive and negative samples, and connecting all adjacent sample points predicted as positive to form several connected regions, the centroid coordinates and corresponding probabilities of each connected region are calculated as the parameter and confidence results of the predicted semantic line. The above semantic line detection process based on the heat map prediction task is as follows Figure 4 , where p(θ, r) denotes the probability value predicted by the heat map at point (θ, r), denotes the centroid coordinates of the positive sample connected region, and c c denote the predicted semantic line parameter and confidence, respectively. By forming a set and C cThe final detection result output as semantic content branch.

[0076] 3.3 For the parameter regression task, two three-layer perceptrons are used to predict the line parameters and confidence for all line queries, respectively. In particular, the parameter coordinates directly predicted from the line query decoding features are relative position coordinates based on the reference point, so they need to be added to the reference point coordinates to obtain the absolute position coordinates. The calculation process is C q = FFN(S q ), where and C q are the semantic line parameter and confidence detection results output by the semantic query branch.

[0077] 3.4 Both branches of the network independently complete the semantic line detection task using the features obtained by decoding. Therefore, there will be some differences in the detection results. After the dual-branch network completes the semantic line detection task, the detection results are first merged into the same set, C u = C c ∪ C q , where and C u represent the union set of the dual-branch semantic line detection results.

[0078] 3.5 Referring to the idea of using the non-maximum suppression algorithm to remove redundant bounding boxes in the target detection task, by modifying the suppression judgment condition in the algorithm, the algorithm can also be used to suppress the nearby semantic line parameter coordinates in the neighborhood, thereby removing the repeated semantic line detection results. The steps of the non-maximum suppression algorithm are as follows Figure 5 , the input is the semantic line parameter detection result set the confidence detection result set C u and the suppression threshold λ, and the output is the de-duplicated semantic line parameter detection result set First, assign to empty, if is not empty, repeat the following operations: assign the position number of the maximum value in C u to i, assign to l, calculate the distance between all lines in and l, remove the lines with a distance less than the threshold λ from , remove the corresponding confidence from C u , and move l into the set Repeat the above operations. At this time, the obtained parameter space semantic line detection result complements the detection results output by the semantic content branch and the semantic query branch.

[0079] Step 4, the semantic line parameters are converted back to the original image space by applying the inverse mapping of the parameterized mapping function, and the final semantic line detection result L is obtained, and thus the entire detection task is completed.

[0080] In view of the network structure and task setting of the semantic line detection model proposed in the application, the design of the loss function is generally divided into two parts, one is a loss term for supervising the semantic content branch heat map prediction task, and the other is a loss term for supervising the semantic query branch parameter regression task.

[0081] In the heat map prediction task, given the real heat map label G h , the classic binary cross-entropy loss can be taken, where n = Θ x R represents the number of elements in the heat map, i represents the position index, P h represents the predicted parameter space heat map, and G h is the given real heat map label. In addition, considering that the positive and negative sample quantity ratio in the real heat map label is greatly different and the positive samples have certain aggregation, and the cross-entropy loss is calculated independently for each pixel prediction result and is a region-independent loss, a region-related Dice loss is also added as a supplement, where |·| represents the sum of all element values, P h represents the predicted parameter space heat map, and G h is the given real heat map label.

[0082] In the parameter regression task, since all semantic lines in the image need to be detected, the number of line queries N set by the model must be greater than the number of semantic lines M in the real semantic line label , which leads to the inability to match a unique real label for all line queries to calculate the loss. To solve this problem, the application seeks a best match for the bipartite graph composed of line query prediction results and real semantic line labels, which contains M line queries with the highest overall relevance to the real labels. Specifically, in the process of constructing the bipartite graph, all line queries and real labels are taken as vertices, and a weightless edge with weight w(i, j) is established between each line query vertex and all real label vertices in turn, where d(·) represents the L1 distance between two coordinates, and represents the parameter coordinates and confidence of the i-th line query prediction, represents the parameter coordinates of the j-th real semantic line label, and λ c represents the confidence weight, which is specifically set to 0.1. After the bipartite graph is constructed, the maximum weight matching of the graph is calculated, and the calculation process is denotes an indicator function, and σ(·) denotes a permutation function defined on the set of natural numbers. Based on the optimal permutation function, the semantic line parameters and the confidence of the line query prediction can be calculated L1 distance loss and Focal loss, respectively, where α and γ are hyperparameters of the Focal loss function, used to adjust the loss weights of positive and negative samples and easy and difficult samples, and M is the number of selected semantic lines.

[0083] Finally, the total loss function of the model can be represented as the weighted sum of the above four loss functions, L total = λ bce L bce + λ dice L dice + λ dist L dist + λ focal L focal , where λ bce , λ dice , λ dist and λ focal are the weights of the four losses in the total loss.

Claims

1. A dual-branch complementary semantic line detection method based on Transformer, characterized in that The line semantic context can be locally modeled and globally interacted, the semantic lines can be more comprehensively captured, and the optimization process for end-to-end supervised learning training can be facilitated, including the following steps: Step 1, using a model encoder to extract multi-scale image features, and using a deep Hough transform to convert the depth feature representation from image space to Hough parameter space, this step specifically includes: Step 1.

1. Backbone network input original image , a set of multi-scale feature maps is obtained after four-stage feature encoding ; Step 1.2 After completing the image feature extraction step, the deep Hough transform operation obtained by extending the Hough transform method in the traditional image processing field is used, at this time it is necessary to set the quantization interval, and the continuous valued line parameter is quantized to the discrete space; Step 1.3 The deep Hough transform operation is used to transform the image features in space, so as to more compactly represent the line structure semantic information in the subsequent decoding process; Step 1.4 Calculate the quantization level in the quantization process, wherein the deep Hough transform adopts a fixed angle quantization level and a distance quantization level that increases layer by layer for each layer feature map; Step 1.5 For any straight line its corresponding image feature is aggregated to a point in the parameter space feature map after the depth Hough transform where is the position index, is the straight line corresponding quantized line parameter; finally, the output of the depth Hough transform is obtained by transforming the multi-scale feature map in the Hough parameter space where , denotes the depth Hough transform, is the multi-scale feature map;​ Step 2, from coarse to fine-grained double-branch decoding; after obtaining the multi-scale feature map in the Hough parameter space, the model decoder is used to start decoding from the highest level coarse-grained feature map and iteratively receive more fine-grained semantic information layer by layer; Overall, the decoder has a double-branch structure, which is called semantic content branch and semantic query branch respectively; From the local point of view, the two branches are respectively stacked by semantic content modules and semantic query modules at each level of the decoder; this step specifically includes: Step 2.1 The semantic content module takes the Transformer block as the main body, which is used to mine valuable semantic content information contained in the features, and the module specifically includes: Step 2.1.1 A bilateral region attention mechanism is designed in the semantic content module to replace the multi-head self-attention mechanism originally used in the Transformer block, to improve its performance and efficiency in processing dense feature representations; Step 2.1.2 The bilateral region attention module directly receives the parameter space feature map as input and performs bilateral region feature generation according to the process, which aims to gather bilateral region feature information for each implicit candidate line to form a set of bilateral region feature maps that can describe the left and right region features of the line structure. First, generate a Gaussian weighted feature map; Step 2.1.3 For each Gaussian-weighted feature map obtained... The region features are split into left and right parts according to the reference axis to obtain the split region feature map; then the features of each region are clustered into a single pixel; then the clustered features of the regions calculated with different columns as reference axes are spliced ​​together along the column dimension to form a bilateral region feature map of the same scale as the parameter space feature map. Step 2.1.4 Cosine similarity is adopted wherein represents the bilateral region feature map position index, represents the modulus of the vector, and A set of bilateral region feature maps is finally obtained by the bilateral region feature generation process, and the meaning of the feature of each pixel point is the bilateral region semantic information of the corresponding line structure; the similarity is used to measure the semantic similarity of the bilateral region, so as to compare the heterogeneity degree of the bilateral region semantic information; the process calculates the semantic similarity of the bilateral region of the corresponding line structure for each pixel position in the bilateral region feature map, and the higher the similarity is, the smaller the semantic difference degree of the left and right parts is. Step 2.1.5 is obtained by Computing bilateral region attention map wherein and is a set of bilateral region feature maps finally obtained by the bilateral region feature generation process; after obtaining the bilateral region attention map, a simple and efficient attention weighting method is adopted Attention weighted fusion result is calculated on the parameter space feature map wherein represents element-wise multiplication operation; Step 2.1.6 In addition to using the bilateral region attention mechanism to replace the multi-head self-attention mechanism, the semantic content module also adds a deep convolution to the feedforward neural network constructed by the full connection layer in the standard Transformer block; Step 2.1.7 The last part of the semantic content module is consistent with the Transformer block, which is composed of a skip connection and a layer normalization; Step 2.2 The semantic query module realizes the line structure feature extraction of the parameter space feature map in another decoding way; it mainly learns a set of line query embeddings as candidate line agents, queries and gathers the region features they are interested in in the feature map through cross-attention mechanism, and finally directly predicts the semantic line parameters through parameter regression; Step 2.2.1 The semantic content module first receives the Hough parameter space feature map output from the encoder of the corresponding level model. After decoding, the intermediate feature representation of semantic content is obtained. And query the decoding features with the line output of the previous layer decoder. Line query embedding Together they serve as input to the semantic query module, in which Indicates the total number of line queries; Step 2.2.2 After decoding by the semantic query module, the updated line query decoding features are obtained and the attention map generated by the Gaussian prior cross attention ; Step 2.2.3 Then, in order to promote the information exchange between the two branches of the network and make full use of the supervision signal, the model decoder adopts an attention fusion method to introduce complementary information into the semantic content intermediate feature representation in the decoding process; the attention fusion method adopts a splicing operation in the channel dimension to splice the semantic content intermediate feature representation with the attention map , the meaning is to supplement the attention area information of the line query on the feature map to the semantic content intermediate feature representation; Step 2.2.

4. The semantic content decoding features output by different decoding layers are fused together using a feature pyramid network based method to form multi-scale decoding feature representations with rich high-level semantic and spatial detail information. The fusion process of adjacent decoding layers is wherein denotes an up-sampling operation, using a bilinear interpolation method to up-sample the feature map of the previous layer to align with the feature map scale of the current layer, denotes the semantic content decoding features output by different decoding layers, denotes the semantic content decoding features; the output of the final model decoder contains double-branch decoding results, i.e., the semantic content decoding features obtained through four decoding layers and line query decoding features ; Step 3, the semantic line complementary detection as the last stage of the whole model detection process, undertakes the post-processing steps of result prediction, result complement, line parameter inverse mapping, and the semantic content decoding features output by the model double-branch decoder with line query decoding features Different semantic properties and meanings possessed, in the prediction stage, respectively using heat map prediction method and parameter regression method for prediction, and corresponding designed heat map prediction head and parameter regression head , this step specifically includes: Step 3.1 In the result prediction process, the model is equipped with a heat map prediction head and a parameter regression head for heat map prediction and parameter regression tasks, respectively; for the heat map prediction task, it can be regarded as performing a binary classification task on each pixel in the feature map, and the prediction head adopts a two-layer perceptron composed of convolutional layers; Step 3.2 Predicting threshold by selecting a heat map As a division standard for predicting positive and negative samples, and connecting all adjacent sample points predicted as positive to form several connected regions, the centroid coordinates and corresponding probabilities of each connected region are calculated as the parameters and confidence results of the predicted semantic lines; by forming a parameter set of the predicted semantic lines through all the prediction results in the heat map And a confidence result set , which can be used as the final detection result of the semantic content branch output; at the same time, in the heat map prediction task, given the real heat map label , the classic binary cross-entropy loss can be used, in addition, considering that the proportion of the number of positive and negative samples in the real heat map label is greatly different and the positive samples have a certain aggregation, and the cross-entropy loss is independently calculated for each pixel prediction result, which is a region-independent loss, therefore, the region-related Dice loss is also added as a supplement. Step 3.3 For the parameter regression task, two three-layer perceptrons are used to predict the line parameters and the confidence of all line queries respectively. The parameter coordinates predicted by the line query decoding features are the relative position coordinates based on the reference point, so they need to be added to the reference point coordinates to obtain the absolute position coordinates. In the parameter regression task, the bipartite graph matching algorithm is used to find the best match for the bipartite graph composed of line query prediction results and real semantic line labels. In this matching, the line query with the highest overall relevance to the real label is included After the bipartite graph is constructed, the Hungarian algorithm is used to calculate the maximum weight matching of the graph. Then, based on the optimal permutation function, the L1 distance loss and the Focal loss can be calculated for the semantic line parameters and the confidence of the line query prediction respectively. Step 3.4 Both branches of the network independently complete the semantic line detection task using the features obtained by decoding, so there will be some differences in the results obtained; after the double-branch network completes the semantic line detection task, the detection results are first combined into the same set, , wherein and represent the union of the double-branch semantic line detection results, , , , respectively represent the semantic line parameters and the confidence obtained by predicting the heat map prediction method and the parameter regression method. Step 3.5 Referring to the idea of using the non-maximum suppression algorithm to remove redundant bounding boxes in the target detection task, by modifying the suppression judgment condition in the algorithm, the algorithm can also be used to suppress the coordinates of the semantic line parameters close to each other in the neighborhood, thereby removing the repeated semantic line detection results; Step 4, the semantic line parameters are converted back to the original image space by applying the inverse mapping of the parameterized mapping function, and the final semantic line detection result is obtained , and thus the entire detection task is completed.

2. The method of claim 1, wherein The model supports efficient end-to-end training optimization and prediction; at the same time, the model adopts an encoder-decoder architecture, in which the encoder can be compatible with any multi-scale feature extraction network, and the decoder is implemented as a double-branch structure containing a semantic content branch and a semantic query branch to perform a feature decoding strategy from coarse to fine, and through attention fusion method to couple the double-branch output to form complementary feature representation, combined with two prediction heads to undertake heat map prediction and parameter regression tasks respectively, and using complementary detection method to weaken the influence of label sparsity problem in dense prediction task on performance as much as possible.

3. The method of claim 1, wherein The model decoder is designed and implemented with a Transformer block as the basic structure, which fully utilizes the global representation and the ability to capture long-distance dependencies of the Transformer, and cooperates with bilateral regional attention and Gaussian prior cross attention to strengthen local and global feature semantic context, promoting the prediction of the global optimal semantic line set.

Images