A method of attention scheme and strip convolution semantic line detection based on deep hough network

Abstract

The application relates to an attention scheme and a strip convolution semantic line detection method based on a deep Hough network, and belongs to the field of image target recognition. The method comprises the following steps: adjusting the picture size of a data set, inputting a pretreated picture into a convolution model to perform feature extraction, attention mechanism and multi-scale feature fusion, and obtaining an output feature result. Different scale features are subjected to Hough transformation and fused feature regression prediction to obtain a recognition result; a strip convolution replaces a previous spatial convolution layer to obtain remote related semantic information of an input image recognition target; a strip-shaped pooling layer and a spatial pooling layer mixed pooling layer are used for pooling of a picture of a forward network; a channel attention module is added to a feature pyramid network to help modeling the importance of picture partitioning, so that the influence of redundant information of an input image on a recognition result is reduced; and a Ghost convolution light-weight network is designed to reduce the calculation amount of the network. Compared with a conventional semantic line detection method, the model parameter amount is smaller, and the recognition accuracy is higher.

Description

Technical Field

[0001] This invention belongs to the field of object detection technology, and more specifically, it designs an attention scheme based on deep Hough networks and a method for detecting strip convolutional semantic lines. Background Technology

[0002] Line detection is a classic problem in computer vision research. Semantic lines, as a special type of line in image distribution, are specifically defined as lines that separate different semantic backgrounds within an image, given the presence of varying backgrounds. They can clearly represent the distribution of different backgrounds in an image. Adjusting the distribution of semantic lines in an image can optimize a photographer's composition.

[0003] Semantic line detection has practical significance. For example, it can be used to estimate the level of an image and optimize the composition of photographic images. Furthermore, it can be used for preliminary operations in some downstream visual tasks: lane detection, horizon detection, and sea level detection. All of these can be optimized using semantic line detection as a preliminary operation. Since semantic lines serve as boundaries between different scenes in an image, they can be used as boundaries for preliminary semantic background segmentation. This simplification of the image's semantic background can serve as a priori feature for many visual applications.

[0004] While semantic lines have wide applications, their detection remains a challenge due to the diversity of line segments and semantic background in images. On one hand, most methods for semantic line detection focus on specific cases such as lane lines and sea level lines; few methods consider overall semantic line detection. On the other hand, the irregularity of semantic background information makes it difficult for traditional convolutional neural networks to handle both low-level details and high-level semantic information. Therefore, early deep learning-based research focused on specific cases, such as lane lines and sea level lines, proposing edge detection filtering methods to segment the target line region before applying traditional Hough transform or RANSAC algorithms to extract and detect the target line within the image. However, these methods are often affected by factors such as input image brightness and occlusion, reducing their robustness. Therefore, these methods are not well-suited for detecting and recognizing semantic lines. Until 2020, Kai et al. proposed a Deep Hough Transform (DHT) network structure for detecting semantic lines, combining a feature pyramid network and a traditional Hough transform (Zhao, K.; Han, Q.; Zhang, CB; Xu, J.; Cheng, MMD, Deep Hough Transform for Semantic Line Detection, in IEEE Transactions on Pattern Analysis and Machine Intelligence, 1 Sept. 2022, vol. 44, no. 9, pp. 4793-4806). Its basic idea is to combine the high-level semantic information and low-level detail information of the input image through a multi-scale Hough transform of the feature pyramid network, and finally predict the semantic line by channel-dimensional regression prediction, thus combining the information from both. However, this network neglects the long-range semantic background information of pixels within the image, so the output semantic line cannot accurately distinguish the background information in the image, and the prediction accuracy still needs improvement. Summary of the Invention

[0005] This invention addresses the shortcomings of existing semantic line recognition methods, which neglect global contextual information and suffer from significant interference from irrelevant redundant information. It proposes a semantic line detection method based on deep Hough convolution and attention mechanisms, helping the network to better output multi-scale images of the input image and reducing computational load. Specifically, the improved semantic line detection network first enhances the accuracy of network recognition by adding a feature pyramid network with strip convolutional layers, strip hybrid pooling layers, and a feature attention selection module. A lightweight structure incorporating Ghost convolution is also designed. Finally, the feature extraction network outputs multi-scale feature maps, which are then subjected to Hough transform and aggregated along the channel dimension to predict the final parameter space points. Finally, an inverse Hough transform converts the points in the parameter space into semantic lines in the image space.

[0006] The present invention adopts the following technical solution and implementation steps:

[0007] An improved semantic line detection method based on deep Hough networks, the method comprising:

[0008] Step 1: Image data preprocessing;

[0009] Step 2: Construct a strip convolutional neural network for detecting semantic lines;

[0010] Step 3: Train the constructed network;

[0011] Step 4: Perform semantic line detection testing using the trained strip convolutional neural network.

[0012] The data preprocessing specifically includes the following steps:

[0013] The data preprocessing refers to preprocessing the photo data acquired by devices such as cameras or existing public datasets. The specific process is as follows: modify the size of the image data in the acquired photos or datasets to H×W, where H and W represent the height and width of the input image size, respectively. In addition, in order to improve the generalization ability of the network, this invention adopts the method of flipping data to realize data augmentation.

[0014] The deep Hough attention scheme and strip convolutional network for semantic line detection specifically include the following modules:

[0015] Module 1: Striped Convolution Module

[0016] The strip convolution module refers to obtaining a multi-scale output matrix that facilitates subsequent Hough transform of the network by acquiring long-range semantic background information between pixels. Its input is preprocessed image information from the network, and its output is intermediate multi-channel image information obtained through convolution. The parameters involved in the strip convolution module are learned through training. This module uses strip convolution operations to downsample the input image information three times, while maintaining the output dimension of the image information unchanged. The specific calculation steps of the strip convolution module are as follows:

[0017] z(i,j)=f s (z1(i,j)+z2(i,j)) (1)

[0018]

[0019]

[0020] Among them: z1 (i，j) z2 (i，j) It is a strip convolution output in both vertical and horizontal directions. These are the weights in the vertical and horizontal strip convolutions, where x(i,j) is the input at position (i,j), and f... s (·) is a 1×1 convolutional layer, and z(i,j) is the output of a combination of vertical and horizontal strip convolutions. Finally, following the residual network structure, it is fused with the initial x input to output the final result.

[0021] Module 2: Strip and Spatial Hybrid Pooling Layer Module

[0022] The strip and spatial hybrid pooling layer module is used to compress pixel features and simplify network parameters in image information. Its input is a two-dimensional image matrix after convolution transformation, and its output is a two-dimensional grayscale matrix composed of pixel values ​​aggregated from n neighboring pixels. The specific operation steps are as follows: The pixels obtained by fusing the short-range information of the spatial pooling layer and the long-range information of the strip pooling layer contain global context information and channel context information. The specific calculation steps are as follows:

[0023]

[0024]

[0025] in: It is a two-dimensional input image information tensor for the strip pooling layer, where H×W is the spatial height and width, and h and w are the spatial ranges to be pooled. This is the output after horizontal strip pooling. This is the output after vertical strip pooling, w i These are the weights of different pooling results, y C，i，j This is the final pooling output.

[0026] Module 3: Feature Selection Module

[0027] The feature selection module is a channel attention network module used to emphasize important spatial details of the input features before channel dimensionality reduction between the feature pyramid network. Its input is the image pixel information in the upper layer network, and its output is the image pixel output with corresponding weights assigned to the channels in the image. The specific operation steps are as follows: using the channel attention mechanism in the SeNet network, D channel weights u = [u1, u2, ..., u] are output through a 1×1 convolution operation. D The channel-dimensional weights are multiplied by the input to assign channel-dimensional weights to the original input. Finally, the final image output is represented by adding the channel-dimensional weights to the original input and then performing a 1×1 convolution operation. The definition is as follows:

[0028]

[0029] u = f m(z) (7)

[0030] in: This is the final output, C i It is the module's raw input, f m (·) consists of a 1×1 convolutional layer and a sigmoid activation layer, responsible for assigning corresponding weights to the channels, where z is C. i The result obtained by global pooling, f s (·) is composed of 1×1 convolutional layers and is responsible for integrating and outputting the channel dimension.

[0031] Module 4: Lightweight Module for Ghost Convolution

[0032] The Ghost convolution lightweight module consists of classic convolution and low-cost convolution operations, used to replace the original convolutional layers to reduce the overall network computation. Its input is the same as the original classic convolution input—image pixel information—and its output is the features of the entire image. The process is as follows: First, the input image pixel information is divided into two parts along the channel dimension. The first part is used for feature extraction through classic convolution to obtain an intrinsic feature map. Then, the second part, with the intrinsic feature map of the first part as input, is used for feature extraction through low-cost convolution. Finally, the two feature maps are integrated along the channel dimension to output the final feature-extracted image information.

[0033] Module 5: Output Module

[0034] The output module mainly consists of a connection function and a 1×1 convolutional layer. It is used to integrate parameter space graphs of Hough transforms at different scales and to perform threshold binarization on the integrated parameter space graph for regression prediction.

[0035] The training of the network model specifically includes the following steps:

[0036] Step 1: Input the preprocessed training dataset into the improved detection semantic line network based on deep Hough for backpropagation training. The learning parameters of the model include weights and biases, and cosine annealing is used to adjust the learning rate to train the network model.

[0037] Step 2: Introduce a binary cross-entropy loss function into the parameter space.

[0038]

[0039] Where: y′ (x，y) The value of the matrix at coordinates (x, y) in the truth graph of the parameter space, where y (x，y)The value of the matrix coordinates (x, y) in the predicted label in the parameter space is used to backpropagate the model's learning parameters, including weights and biases, based on the loss function using batch gradient descent.

[0040] Step 3: Repeat steps 1 and 2 to iteratively train the network model parameters to obtain the optimal detection semantic line network model.

[0041] The trained model was used to perform semantic line detection tests.

[0042] Beneficial effects:

[0043] This invention provides a semantic line detection method based on deep Hough networks. Through the design of striped convolutional layers, striped and spatial hybrid pooling layer modules, feature selection modules between networks, and a lightweight Ghost convolution module, the semantic line detection task can be implemented end-to-end. The semantic line detection network of this invention integrates not only local spatial features but also global and deep contextual information during feature extraction, improving the network's detection accuracy while reducing computational load and improving response time. On a given set of 1700 semantic lines for testing, this invention achieves an accuracy of 69.7%, reaching state-of-the-art (SOTA) performance on such datasets. Attached Figure Description

[0044] Figure 1 This is a flowchart of the attention scheme and strip convolutional semantic line detection method based on deep Hough networks provided by the present invention;

[0045] Figure 2 This is a network structure diagram of the attention scheme and strip convolutional semantic line detection method based on deep Hough networks provided by the present invention.

[0046] Figure 3 This is a recognition result image of the attention scheme based on deep Hough networks and the strip convolutional semantic line detection method provided by this invention. The first column is the original image, and the second column is the recognition result image. Detailed Implementation

[0047] The purpose of this invention is to provide a semantic line detection method based on a deep Hough transform network, which can complete the network training end-to-end without any post-processing. The semantic line detection method of the deep Hough network of this invention can not only extract the spatial features around the pixel, but also integrate global contextual semantic information and reduce pixel redundancy features between networks, thereby improving detection accuracy.

[0048] The present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the described embodiments are intended only to facilitate the understanding of the present invention and are not intended to limit it in any way.

[0049] Figure 1 This is a network flowchart for semantic line detection provided by the present invention; Figure 2 This is a network structure diagram of semantic line detection provided by the present invention; Figure 3 This is a result diagram of semantic line detection provided by the present invention.

[0050] The semantic line detection method based on deep Hough networks provided by this invention specifically includes:

[0051] Step 1: Data Preprocessing

[0052] The data preprocessing includes a dataset preprocessing component. This involves classifying the public dataset into training and testing sets and resizing all images in the dataset to a resolution of 120×160.

[0053] Step 2: Construct a semantic line detection network

[0054] The deep Hough improved network for semantic line detection constructed in this invention specifically includes a strip convolution module, a strip and spatial hybrid pooling layer module, a feature selection module, a GhostNet module, and an output module.

[0055] The strip convolution module refers to a 1×n and n×1 strip convolution transformation designed with similar computational cost to the spatial convolution in the original network. Its input is a pre-processed input image, and its output is the intermediate output image information obtained through the convolution transformation. The parameters involved in the strip convolution module are learned through training. This module uses convolution kernels of 1×1, 1×5, and 5×1, with a stride of 1, maintaining the same number of output and input channels, and padding = 2. The final output image information has a size of c×H×W, where c represents the number of output image channels. The specific calculation of the strip convolution is shown in formula (1).

[0056] The strip and spatial hybrid pooling layer module is a pooling layer module used for downsampling and fusing features from the global context information of pixels in an image. Its input is the intermediate output image information after convolutional transformation, and the output is feature information that aggregates the pixel's surrounding information and its global context information. The specific operation steps are as follows: The input is divided into three paths: a spatial pooling layer, a horizontal strip pooling layer, and a vertical strip pooling layer. The spatial pooling layer uses an average pooling layer. The horizontal and vertical strip pooling layers perform intermediate processing along their respective directions, averaging the global pixel values ​​to 1×n or n×1, then upsampling it to an n×n size through 1D convolution, and finally fusing the features with the intermediate output of the spatial pooling layer through 2D convolution and a sigmoid function to output a final hybrid pooling result of size c×H×W, as shown below. Figure 2 As shown.

[0057] The feature selection module consists of a Squeeze layer and an Excitation layer, used to improve the network's feature expressiveness and reduce the impact of redundant information. Its input is the upper-layer image features extracted through convolutional features in the feedforward network, and its output is an intermediate global feature output with channel weights. The process by which the feature selection module improves the network's feature expressiveness is as follows: First, a channel importance vector u is obtained through global pooling, 1×1 convolution, and sigmoid activation. Then, this vector is multiplied by the input to obtain the intermediate output u×C. i Then input C i After performing the addition operation, a 1×1 convolution is used to integrate the upper-layer semantic image features with a size of c×H×W.

[0058] The Ghost lightweight convolutional module consists of two convolutional layers: a basic convolutional layer and a low-cost convolutional operation. Specifically, its input is a pre-processed input image, and its output is intermediate output image information obtained by concatenating the two convolutional layers along the channel dimension. The GhostNet module extracts image features as follows: First, a basic convolutional layer is used to generate classic features at a predetermined ratio (here, the ratio of classic convolution to low-cost convolution is 1:1). Then, the classic features generated in the first part are fed into the low-cost convolutional part to generate low-cost features. Finally, the features generated in both parts are concatenated along the channel dimension to obtain c×H×W features, where n is the channel dimension of the original classic convolution. The kernel size is [1, 3, 1], and the stride is 1.

[0059] The output module uses an aggregation function to merge the multi-scale parameter space features obtained from the deep Hough stage, and outputs a three-dimensional matrix of size 1×120×160 through a 1×1 convolutional layer module with 512 input channels and 1 output channel. Thresholding can be used to obtain the final regression prediction result in the parameter space.

[0060] The specific process of detecting semantic line networks is as follows:

[0061] To fuse semantic and detail features of the input image across multiple scales, the preprocessed input image is processed through a feedforward network consisting of striped convolutional modules, striped and spatial hybrid pooling layers, and Ghostnet convolutional layers for feature extraction. Each pixel in the output multi-scale image information comprises its n×n spatial information and b×1 and 1×b distant background information, where n=3 and b=5. The definition of each semantic pixel is as follows:

[0062]

[0063] Then, a feature selection module between the inverse network and the network is used to fuse the upper-level semantic features extracted by the forward network with the lower-level detailed features of the inverse network. Compared with low-dimensional features, high-dimensional features have the characteristic of channel information redundancy. The feature selection module can avoid the error caused by redundant information. Finally, the multi-scale features obtained in the feature extraction stage are subjected to Hough transforms at different scales to the parameter space. After bilinear interpolation to a unified scale, the features in the parameter space at different scales are fused and regressed along the channel dimension to predict the semantic line prediction probability in the parameter space.

[0064] Step 3: Train the network model:

[0065] The hardware environment of this invention is an Intel Platinum 8255C processor, 11GB of memory, and a 2080Ti GPU; the operating environment is Ubuntu 18.04 operating system, CUDA 11.0, PyTorch 1.7.0 and Python 3.8.

[0066] First, the preprocessed training dataset of photos with a batch size of 8 is used as input to train the semantic line detection network model by forward propagation to learn the network parameters. The learning rate is 0.02, the batch normalization momentum is 0.9, the maximum number of iterations is 30, and the batch size is 8. The Nesterov Momentum Algorithm (Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models, or Adan for short) optimizer is used for backpropagation training to update the model's weights, biases, and other learning parameters. Adan allows each parameter to obtain an adaptive learning rate during the model training and optimization process, thereby achieving a dual improvement in optimization quality and speed. Finally, the network model parameters are trained by iterating 30 times to obtain the optimal semantic line detection network model. Then, the predicted values ​​of the output semantic line parameter space and the true semantic line parameter space values ​​are input into the binary cross-entropy loss function (9), and backpropagation is performed using the batch gradient descent method. Finally, cosine annealing is used to update the learning rate to update the learning parameters of the iterative model to obtain the optimal detection network model.

[0067] Step 4: Perform detection testing using the trained detection semantic network.

[0068] The trained semantic line detection network is used for semantic line detection testing. The difference between the testing and training phases is that in the testing phase, only the already trained network model needs to be loaded; there is no need to train the network model again.

[0069] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications and substitutions that can be conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for semantic line detection based on attention and strip convolution using deep Hough networks, characterized in that, Includes the following steps: Step 1: Preprocess the dataset and input it into the attention scheme based on deep Hough network and the model for detecting semantic lines by strip convolution. The dataset uses publicly available online datasets and image information obtained from cameras. Human experts are used to annotate the ground truth lines of semantic lines. Step 2: Construct a semantic line detection network, which includes a strip convolution module, a strip and spatial hybrid pooling module, a feature selection module, a GhostNet convolution module, and an output module; Step 3: Model Training: Input the preprocessed training dataset photo information into the semantic line detection network model for forward propagation calculation to obtain the final prediction result; then, transform the output prediction result and ground truth map into the parameter space and input them into the loss function, and use batch gradient descent for direction propagation; use the Adan optimizer to update the various parameters of the model. The learning parameters of this model include weights and bias terms. Step 4: Perform semantic line detection using the trained semantic line detection network model; The specific calculation steps of the strip convolution module are as follows: ； ； ； in: It is a strip convolution output in both vertical and horizontal directions. These are the corresponding weights in vertical and horizontal strip convolutions. It is the input corresponding to position (i, j) It is a 1×1 convolutional layer The output of the strip convolutions is combined with the output of the vertical and horizontal strip convolutions; finally, the residual network structure is used to fuse it with the initial x input to output the final result. The strip and spatial hybrid pooling layer module refers to the pixel obtained by fusing the short-range information of the spatial pooling layer and the long-range information of the strip pooling layer, which contains both global context information and channel context information; the specific calculation steps are as follows: ； ； in: It is a two-dimensional input image information tensor of a strip pooling layer. It is the height and width of the space The space range that needs to be pooled This is the output after horizontal strip pooling. It is the output after vertical strip pooling. The weights of different pooling results This is the final pooling output.

2. The method according to claim 1, characterized in that, The preprocessing mentioned in step 1 refers to scaling the input image information of the network to the size required by the network during the preprocessing process.

3. The method according to claim 1, characterized in that, The feature selection module mentioned in step 1 refers to a channel attention network module used to emphasize important spatial details before performing channel dimensionality reduction on the input features between the feature pyramid network. Its input is the image pixel information in the upper layer network, and its output is the image pixel output that assigns corresponding weights to the channels in the image. The specific operation steps are as follows: using the channel attention mechanism in the Senet network, D channel weights are output through a 1×1 convolution operation. The channel-dimensional weights are multiplied by the input to assign channel-dimensional weights to the original input. Finally, the final image output is represented by adding the channel-dimensional weights to the original input and then performing a 1×1 convolution operation, as defined below: ； ； in: This is the final output result. It is the module's raw input. It consists of 1×1 convolutional layers and sigmoid activation layers, responsible for assigning corresponding weights to the channels. yes Results obtained from global pooling It consists of 1×1 convolutional layers and is responsible for integrating and outputting the channel dimension.

4. The method according to claim 1, characterized in that, The Ghost convolution lightweight module refers to the following: In the strip convolution module, the 1×1, 1×5 and 5×1, 1×1 convolutions are first decomposed, and the 1×1 convolutions before and after the depthwise convolution are replaced. Ghost convolution is used to reduce parameters, and finally the two convolutions are concatenated.

Images