Image road extraction method based on hybrid convolution module and block circulation attention

Abstract

The invention belongs to the technical field of image segmentation, and discloses an image road extraction method based on a hybrid convolution module and block circulation attention. A deep convolutional neural network ResNet is used to carry out feature extraction on an original image, the size of a convolution kernel is enlarged to 31 * 31 through improvement of the convolution kernel, the convolution process is divided into two parallel paths including a large convolution branch and a small convolution branch, the improved convolution is used to carry out feature extraction on the original image, and the convolution kernel size is enlarged to 31 * 31. And fusing the features extracted by the stage feature extraction and the features extracted by the hybrid convolution module by using a block circulation module to obtain hybrid road features, decoding the hybrid road features by using deconvolution and short-circuit operations, recovering the size of the decoded image to obtain a predicted road map, and carrying out prediction on the predicted road map. The method can effectively improve the extraction performance of large-span and small roads in road extraction, and improves the calculation efficiency of road extraction.

Description

Technical Field

[0001] The invention belongs to the technical field of image segmentation and relates to an image road extraction method based on a hybrid convolution module and block cyclic attention. Background Art

[0002] Roads play a vital role in cities, connecting various parts of the city. At the same time, roads contain basic and important geographic information, which is often used in many fields, including urban design and planning, vehicle navigation, disaster warning, etc. Road shapes include straight segments, curved segments, slopes, intersection shapes, lane widths and numbers, road edge shapes, and special design shapes. Due to the importance of road information, image segmentation has become a key focus of researchers.

[0003] Most traditional road extraction methods use pixel-based and object-oriented ideas to obtain road information from high-resolution remote sensing images. Pixel-based methods mainly rely on remote sensing images of different spectra. By constructing features such as the geometric shape of the road, they can only perform road extraction tasks under simple backgrounds. However, when facing remote sensing images with complex background information, they often cause serious salt and pepper noise in the extraction results. Object-oriented methods divide remote sensing images into classification units of different sizes and then extract road information. They have good noise resistance and applicability. However, due to the influence of building shadows and confusion of similar objects, this method often causes breakage or adhesion in the road extraction results. At the same time, for remote sensing images with rich spectral information, traditional road extraction methods are difficult to take into account the diversity and high complexity of road features, and cannot adapt to large-scale, high-precision road extraction tasks. The extraction effect needs to be further improved.

[0004] In existing road extraction, various road forms such as common buildings and small roads affect feature extraction. How to extract more accurate roads has become one of the key research directions. Summary of the invention In view of the shortcomings of the prior art, the purpose of the present invention is to provide an image road extraction method based on a hybrid convolution module and block recurrent attention, so as to improve the extraction performance of large-span and small roads and improve the computational efficiency of road extraction.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: The present invention provides an image road extraction method based on a hybrid convolution module and block cyclic attention, comprising: extracting features from the original image to obtain features to be processed; and stage characteristics ; For stage characteristics Perform feature extraction to obtain a hybrid convolution module; for stage features The extracted features are fused with the features extracted by the hybrid convolution module to obtain hybrid road features. ; For mixed road characteristics Decode and get the predicted road map .

[0006] Furthermore, the feature to be processed The resolution is 256×256; the stage characteristics The resolution is 128×128, 64×64, 32×32, 16×16, .

[0007] Furthermore, the feature extraction of the original image obtains the feature to be processed and stage characteristics , including: using the ResNet network as the backbone, and obtaining the features to be processed by stacking basic blocks and stage characteristics , the basic block consists of stacked ordinary convolutional layers, normalization layers, activation functions, and residual structures.

[0008] Furthermore, the mixed road features Decode and get the predicted road map , including: using stacked deconvolution layers and short-circuit operations to extract mixed road features Decode the image and restore it to its original size to get the predicted road map. .

[0009] Furthermore, the phase characteristics The hybrid convolution module is obtained by feature extraction, including: using a 31×31 convolution kernel to extract the stage features As the input of each module, multiple modules form a parallel structure, and feature extraction is performed on the features of each stage to obtain a hybrid convolution module.

[0010] Furthermore, the hybrid convolution module is composed of a large kernel module and a convolution forward propagation module in series. Furthermore, the large core module includes: stage characteristics As the input of the large convolution kernel and the small convolution kernel, further feature extraction is performed through the two branches of the large convolution kernel and the small convolution kernel, and the results are fused. The large kernel convolution layer module is obtained by combining the residual structure and the random discarding strategy.

[0011] Furthermore, the convolution forward propagation module includes: using stacked ordinary convolutional layers combined with a residual structure and a random discarding strategy.

[0012] Furthermore, the phase characteristics The extracted features are fused with the features extracted by the hybrid convolution module to obtain hybrid road features. , including: Use phase characteristics The extracted features and the features extracted by the mixed convolution module are used as the input of the block circulation module. Q, K, and V are divided into multiple groups respectively. Score calculations are performed between different groups. Masks are used to hide groups that are not involved in the calculation. Multiple group outputs are obtained. The multiple group outputs are added to obtain mixed road features. . Furthermore, stacked deconvolution layers and short-circuit operations are used to convolutionally transform the mixed road features. Decode the image and restore it to its original size to get the predicted road map. , including: a basic structure consisting of a deconvolution layer, a convolution layer, and a leaky rectified linear unit, and stacking three layers of basic structures to perform mixed road feature Decode it and restore its size to the original image size, and obtain the road probability map through the S-type activation function , according to the road probability map The internal pixel road probability value is used to obtain the predicted road map .

[0013] Compared with the prior art, the present invention has the following beneficial technical effects: The invention discloses an image road extraction method based on a hybrid convolution module and block cyclic attention. A deep convolution neural network ResNet is used to extract features from an original image. The convolution kernel size is enlarged to 31×31 by improving the convolution kernel, and the convolution process is divided into two parallel paths including a large convolution branch and a small convolution branch. A hybrid convolution module is used to extract features from the original image. A block cyclic module is used to fuse features extracted from stage features and features extracted from the hybrid convolution module to obtain hybrid road features. Deconvolution and short-circuit operations are used to decode the hybrid road features. The size of the decoded image is restored to obtain a predicted road map. The method can effectively improve the extraction performance of large-span and small roads in road extraction, and simultaneously improve the computational efficiency of road extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of a process of an image road extraction method based on a hybrid convolution module and block-by-block cyclic attention according to the present invention; Figure 2 A schematic diagram of a flow chart of an image road extraction method based on a hybrid convolution module and block-by-block cyclic attention according to the present invention; Figure 3 Schematic diagram of the process of the hybrid convolution module in an embodiment of the present invention; Figure 4 Schematic diagram of the process of the block-based cyclic attention module in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0016] Example 1 The present invention provides an image road extraction method based on a hybrid convolution module and block-by-block cyclic attention. Figure 1 As shown, feature extraction is performed on the original image to obtain the features to be processed. and stage characteristics ; For stage characteristics Perform feature extraction to obtain a hybrid convolution module; for stage features The extracted features are fused with the features extracted by the hybrid convolution module to obtain hybrid road features. ; For mixed road characteristics Decode and get the predicted road map .

[0017] Use the ResNet network to extract features from the original image to obtain features to be processed and stage characteristics ; Use 31×31 convolution kernel stage features Perform feature extraction to obtain a hybrid convolution module; use a block cycle module to extract the stage features The extracted features are fused with the features extracted by the hybrid convolution module to obtain hybrid road features. ; Use the decoder to analyze mixed road features Decode and get the predicted road map .

[0018] Specifically, we first use a convolution layer with a convolution kernel size of 7×7 and a stride of 2, a maximum pooling layer, and a convolution layer with a stride of 2. In the first stage, the original image is used to extract the features to be processed with a resolution of 256×256. , loop through the features to be processed , set different stacking layers and use basic blocks to further extract features from the second to fifth stages, and obtain stage features with resolutions of 128×128, 64×64, 32×32, and 16×16 , .

[0019] Parallel processing stage characteristics , the stage characteristics As the input of the hybrid convolution module, the hybrid convolution module and the stage features are processed in parallel , using multiple block recurrent attention to combine the mixed convolutional module with the stage features As the input of the block loop module, feature fusion is performed to obtain mixed road features , using stacked deconvolution layers and feature extraction to short-circuit each stage to extract mixed road features Decode the image and restore the decoded image size to the original image size. Then, the predicted road map is obtained by processing it with the S-type activation function. .

[0020] The sigmoid activation function is also called a squeezing function. The input with a larger range of values ​​can be compressed by the sigmoid activation function, converting the X value to a value close to 0 or 1, so that the output value is in the range of (0,1).

[0021] Example 2 The present invention provides an image road extraction method based on a hybrid convolution module and block-by-block cyclic attention. Figure 1 He Ru Figure 2 As shown, the ResNet network is used to extract features from the original image to obtain the features to be processed. and stage characteristics ; Use 31×31 convolution kernel stage features Perform feature extraction to obtain a hybrid convolution module; use a block cycle module to extract the stage features The extracted features are fused with the features extracted by the hybrid convolution module to obtain hybrid road features. ; Use the decoder to analyze mixed road features Decode and get the predicted road map .

[0022] Specifically, the convolution kernel is improved, the size of the convolution kernel is expanded to 31×31 hybrid convolution module, the convolution process is divided into two parallel branches including a large convolution branch and a small convolution branch, and a 31×31 convolution kernel is used to extract features of the processed image to obtain a hybrid convolution module. The block cycle module is used to calculate the attention score. The output of each stage is compared with the stage feature. After the block cycle module is processed, the mixed road feature map is obtained, and the mixed road feature map is processed by deconvolution and short-circuiting operations. Decode and restore the image size to obtain the final predicted road map .

[0023] The ResNet network is used as the encoder structure of the overall network design. The basic blocks are formed by stacking ordinary convolutional layers, normalization layers and activation functions, combined with the residual structure. The basic blocks are stacked to form stages 1, 2, 3, 4 and 5. Stage 1 is the feature to be processed F, and the rest are stage features. , , each stage forms a serial structure to extract features in sequence, and each stage obtains its own output results.

[0024] The hybrid convolution module is composed of hybrid convolution module 2, hybrid convolution module 3, hybrid convolution module 4 and hybrid convolution module 5 in parallel, which are respectively connected to the stage features , Stage Characteristics , Stage Characteristics , Stage Characteristics The output connection of As the input of each module, multiple modules form a parallel structure, and the features of each stage are further processed to obtain a hybrid convolution module.

[0025] In summary, the original image is subjected to feature extraction through basic blocks, including ordinary convolutional layers, residual structures, and maximum pooling operations, to obtain four-stage feature maps with different spatial resolutions and abstraction levels.

[0026] like Figure 3 As shown, the stage characteristics The extracted features are divided into two paths. One path uses various convolution and regularization operations to perform multiple small convolutions and depth-wise separable convolutions on the extracted features. The results of the small convolutions and depth-wise separable convolutions are convolved and regularized multiple times again. The other path performs a random dropout layer, and the two results are added together. The sum is used as the input of the convolution feedforward neural network. Similarly, one path performs multiple convolution regularizations, and the other path performs a random dropout layer. The two results are added together, and the calculation is repeated to obtain the final output.

[0027] Stage characteristics The resolution is 128×128, 64×64, 32×32, 16×16, where the stacking strategy is set to , the specific network structure is as follows: Table 1

[0028] The four-stage feature maps use a hybrid convolution module for feature extraction. The extraction process is carried out simultaneously through two branches, and the two results are fused to obtain the hybrid road feature. Specifically: Use a 31×31 convolution kernel to transform the stage features As the input of each module, multiple hybrid convolution modules form a parallel structure, and the features of each stage are further processed to obtain a hybrid convolution module. The block cycle module is used to combine the features extracted by the hybrid convolution module with the stage features. The extracted features are fused to obtain mixed road features .

[0029] The hybrid convolution module is composed of a large kernel module and a convolution forward propagation module in series. As the input of large convolution kernel and small convolution kernel, further feature extraction is performed through the two branches of large convolution kernel and small convolution kernel, and the results are fused. The large kernel convolution layer module is obtained by combining the residual structure and random discarding strategy. The convolution forward propagation module is composed of stacked ordinary convolution layers combined with the residual structure and random discarding strategy. Large convolution kernels usually refer to convolution kernels with a size greater than 5x5, such as 7x7, 13x13 or even 31x31. These convolution kernels have a larger receptive field and can capture more global information. Large convolution kernels can directly cover a larger image area, thereby capturing more global information at one time, which helps the model better understand the image content; using large convolution kernels can reduce the number of network layers to a certain extent, thereby simplifying the network structure and reducing the complexity of the model; when processing high-resolution images, large-scale features or tasks that require refined pixel-level cutting such as image segmentation, large convolution kernels can play their advantages and improve the performance of the model. Small convolution kernels refer to smaller convolution kernels relative to large convolution kernels, such as 1x1, 3x3 or 5x5. Although 5x5 is also considered medium size in some cases, it is still considered small compared to large convolution kernels such as 7x7 and 11x11. Small convolution kernels significantly reduce the number of parameters and computational complexity in the network, help speed up the training process, and reduce the risk of overfitting. By stacking multiple layers of small convolution kernels, the network can capture more complex feature patterns than a single large convolution kernel. This is because each small convolution kernel is usually followed by a nonlinear activation function ReLU, which increases the nonlinear ability of the network. Using small convolution kernels can make the network have a deeper decision function while maintaining the same receptive field, which helps the network learn higher-level abstract features.

[0030] like Figure 4 As shown in the figure, the features extracted by the hybrid convolution module are used as input to Q, K, and V, which are divided into blocks. The transpose of Q and K are first multiplied correspondingly. During the multiplication process, only the corresponding multiplied parts can be seen, covering the unmultiplied parts. Then, a normalized exponential function is performed, and a random discard layer is performed. The results are then divided into blocks and multiplied correspondingly with V, and the blocks are accumulated to obtain the output. The normalized exponential function compresses a K-dimensional vector containing arbitrary real numbers into another K-dimensional real vector, so that each element ranges between (0,1), and the sum of all elements is 1, and the output is a probability distribution that can be explained.

[0031] The extracted features are mapped to three different spaces to obtain three vectors Q, K, and V. This is usually achieved through linear transformations such as fully connected layers or 1x1 convolutions. The dimensions of Q, K, and V are usually the same. The transpose of Q and K are correspondingly multiplied, and the dot product of the transpose of Q and K is calculated to obtain a matrix representing the attention score of each element in the sequence to other elements. In this process, if certain positions such as filling positions or invisible positions need to be masked, the scores of the corresponding positions can be set to very small negative numbers, so that when softmax is applied later, the weights of these positions will be close to 0, the exponential function is normalized, and the softmax function is applied so that the sum of the elements in each row is 1, thereby obtaining the attention weight.

[0032] Use stacked deconvolution Tconv and short-circuit operation to mix road features Restore to the size of the original image to get the predicted road map Specifically: the deconvolution layer, convolution layer and leaky rectified linear unit constitute the basic structure, and the stacked three-layer basic structure is used to improve the stage features. The extracted features are decoded and restored to the original image size to obtain the output of each stage , , each stage restores the spatial resolution to , , , and The road probability map is obtained through the S-type activation function , the predicted road map is obtained according to the internal pixel road probability value of the road probability map FR .

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] Among them, Tconv is deconvolution and concat is the feature map connection operation.

[0039]

[0040] in, is the road probability map, is the S-type activation function.

[0041] Set the probability threshold to , the pixel probability value is changed to 1 if it is greater than or equal to the threshold, otherwise it is changed to 0. A pixel value of 1 means that the pixel is a road, otherwise it is a background, and the final road map is obtained. , the specific formula is as follows:

[0042] in Representative The value of pixels.

[0043] By using images with labeled information, the convolutional neural network model is trained and the model parameters are optimized, specifically:

[0044]

[0045]

[0046]

[0047] in, is the fine-grained segmentation loss, is the binary cross entropy loss function, is the weight coefficient, is the soft dice coefficient loss function, N is the number of samples, For the The true value of the sample number, For the The network prediction value of sample number, is the smoothing value.

[0048] To comprehensively evaluate our method, we conducted experiments on a public dataset with different image resolutions and information content. On each dataset, we compared our proposed road extraction framework with the latest state-of-the-art methods.

[0049] Table 2

[0050] Among them, P is Represents precision, R represents recall, OA represents average accuracy, F1 represents F1-Score, which is the harmonic mean of precision and recall, and I0U represents union.

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] Among them, TP represents the number of pixels accurately identified as roads, Indicates the number of pixels of other objects extracted as roads, Indicates the number of pixels correctly extracted as background, Indicates the number of road pixels extracted as background.

[0058] As can be seen from Table 2, although the other six methods show certain effects in road extraction, there is still a lot of room for optimization and improvement. For example, U-Net adopts an encoder-decoder structure, which enables the model to capture features of different scales and performs well in image segmentation. However, since it contains a large number of convolutional layers and very high parameters, it takes longer to train large-resolution images. Due to the addition of non-global modules, NL-LinkNet is able to provide more accurate segmentation results, resulting in its R exceeding our model. However, it is precisely because of the introduction of non-global modules and its excellent performance in capturing global information that local details may be lost when processing small objects, thereby reducing the P, OA, and IoU of the segmentation effect. RCFSNet adopts multiple feature fusion strategies such as residual connections and multi-scale feature extraction, which helps the model capture richer contextual information and improve the segmentation effect of complex scenes. It is the introduction of these complex modules that increases the computational complexity of the model. In contrast, our model effectively addresses these challenges without increasing the amount of computation and brings excellent performance in most cases.

[0059] The invention discloses an image road extraction method based on a hybrid convolution module and block cyclic attention. A deep convolution neural network ResNet is used to extract features from an original image. The convolution kernel is improved to expand the convolution kernel size to 31×31, and the convolution process is divided into two parallel paths including a large convolution branch and a small convolution branch. The improved convolution is used to extract features from the original image. The block cyclic module is used to fuse features extracted from stage features and features extracted from the hybrid convolution module to obtain hybrid road features. Deconvolution and short-circuit operations are used to decode the hybrid road features. The size of the decoded image is restored to obtain a predicted road map. The method can effectively improve the extraction performance of large-span and small roads in road extraction, and at the same time improve the computational efficiency of road extraction.

[0060] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "axial", "radial", "circumferential", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

Claims

1. An image road extraction method based on hybrid convolutional module and block recurrent attention, characterized in that: The following steps are involved: Extract features from the original image to obtain features to be processed and stage characteristics ; Characteristics of the stage Perform feature extraction to obtain a hybrid convolution module; Characteristics of the stage The extracted features are fused with the features extracted by the hybrid convolution module to obtain hybrid road features. ; Mixed road features Decode and get the predicted road map .

2. The image road extraction method based on hybrid convolution module and block recurrent attention according to claim 1 is characterized in that: The features to be processed The resolution is 256×256; Characteristics of the stage The resolution is 128×128, 64×64, 32×32, 16×16, .

3. The image road extraction method based on hybrid convolution module and block recurrent attention according to claim 1 is characterized in that: The feature extraction of the original image obtains the feature to be processed and stage characteristics ,include: Use the ResNet network as the backbone and obtain the features to be processed by stacking basic blocks and stage characteristics , the basic block consists of stacked ordinary convolutional layers, normalization layers, activation functions, and residual structures.

4. The image road extraction method based on hybrid convolution module and block recurrent attention according to claim 1 is characterized in that: The mixed road characteristics Decode and get the predicted road map ,include: Use stacked deconvolution layers and short-circuit operations to extract mixed road features Decode the image and restore it to its original size to get the predicted road map. .

5. The image road extraction method based on hybrid convolution module and block recurrent attention according to claim 1 is characterized in that: The phase characteristics Feature extraction is performed to obtain a hybrid convolution module, including: Use a 31×31 convolution kernel to transform the stage features As the input of each module, multiple modules form a parallel structure, and feature extraction is performed on the features of each stage to obtain a hybrid convolution module.

6. The image road extraction method based on hybrid convolution module and block recurrent attention according to claim 1 is characterized in that: The hybrid convolution module is composed of a large kernel module and a convolution forward propagation module in series.

7. The image road extraction method based on hybrid convolution module and block recurrent attention according to claim 6 is characterized in that: The large core module includes: Stage characteristics As the input of the large convolution kernel and the small convolution kernel, further feature extraction is performed through the two branches of the large convolution kernel and the small convolution kernel, and the results are fused. The large kernel module is obtained by combining the residual structure and the random discarding strategy.

8. The image road extraction method based on hybrid convolution and block recurrent attention according to claim 6 is characterized in that: The convolution forward propagation module comprises: It is composed of stacked ordinary convolutional layers combined with residual structure and random dropout strategy.

9. The image road extraction method based on hybrid convolution and block recurrent attention according to claim 1 is characterized in that: The phase characteristics The extracted features are fused with the features extracted by the hybrid convolution module to obtain hybrid road features. ,include: Use phase features The extracted features and the features extracted by the mixed convolution module are used as the input of the block circulation module. Q, K, and V are divided into multiple groups respectively. Score calculations are performed between different groups. Masks are used to hide groups that are not involved in the calculation. Multiple group outputs are obtained. The multiple group outputs are added to obtain mixed road features. .

10. The image road extraction method based on hybrid convolution module and block recurrent attention according to claim 4 is characterized in that: Use stacked deconvolution layers and short-circuit operations to extract mixed road features Decode the image and restore it to its original size to get the predicted road map. ,include: The basic structure is composed of a deconvolution layer, a convolution layer and a leaky rectified linear unit. The three-layer basic structure is stacked to improve the mixed road features. Decode it and restore its size to the original image size, and obtain the road probability map through the S-type activation function , according to the road probability map The internal pixel road probability value is used to obtain the predicted road map .

Images