A method for tactile paving segmentation based on a spatial attention downsampling module

Abstract

This invention designs a tactile paving segmentation method based on a spatial attention downsampling module. This method uses actual photographs of tactile paving in the dataset, which can better improve the generalization and stability of the subsequent model. The encoder integrates a spatial attention downsampling module, which can better improve the model's ability to extract information during feature downsampling. The network further adds an edge-assisted decoder, controlling the parameter updates of the auxiliary decoder through an edge loss function. Feature fusion with the auxiliary decoder improves the decoding capability of the main decoder. This invention improves the segmentation capability of tactile paving in simulated blind person travel scenarios, enhances contextual information extraction in pixel-level prediction tasks using a spatial attention downsampling module, and further improves the model's segmentation effect by setting up both a main and auxiliary decoder, making this invention significantly practical in the field of tactile paving segmentation.

Description

Technical Field

[0001] This invention relates to the field of image target detection based on deep learning, specifically to the construction of a blind path obstacle detection network using a convolutional neural network, and to the invention of a blind path segmentation method based on a spatial attention downsampling module. Background Technology

[0002] Blind people face numerous challenges when traveling. Most blind individuals rely on tactile paving, but this paving is subject to problems such as damage, obstruction, and improper maintenance. To safeguard the right of blind people to independent travel, research into assistive devices for their mobility has been ongoing. These devices are based on machine vision and image processing principles, and accurate segmentation of tactile paving areas is fundamental to such guiding devices. Currently, image segmentation techniques are broadly categorized into traditional methods and those based on neural network learning. Traditional image segmentation divides an image into several non-overlapping regions based on features such as grayscale, color, and spatial texture. These features exhibit consistency or similarity within the same region, but significant differences between different regions. The second category is semantic segmentation using end-to-end deep learning frameworks, comprising an encoder and a decoder. The encoder converts the input information into a low-dimensional, dense implicit vector, and the decoder converts this implicit vector into the target output. In recent years, due to the development of deep learning, semantic segmentation has been widely applied in image segmentation. Unlike traditional algorithms, these algorithms do not require manual feature design.

[0003] Reference 1 (Cao Z, Xu X, Hu B, et al. Rapid detection of blind roads and crosswalks by using a lightweight semantic segmentation network[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 22(10): 6188-6197.) uses a lightweight semantic segmentation network based on depthwise separable convolution to segment blind paths and sidewalks. Reference 2 (Ma Wenjie, Zhang Xuanxiong. Research on blind path and blind path obstacle recognition algorithm based on deep learning[J / OL]. Electronic Science and Technology, 1-10[2023-12-06]https: / / doi.org / 10.16180 / j.cnki.issn1007-7820.2024.03.010.) proposes a multi-task recognition algorithm based on deep learning to simultaneously solve the blind path localization and obstacle detection in the travel process of blind people. This algorithm extracts common features through a backbone network (Transformer Encoder), fuses the extracted features through SPP and FPN networks, and then feeds them into the segmentation network (AllMLP Decoder) and detection network (FCN DetectionHead) to complete the tasks of tactile paving segmentation and tactile paving obstacle detection. However, the algorithm's segmentation effect is somewhat insufficient when the color and texture distinction in the vicinity of the tactile paving is not significant. Reference 3 (Xu K, Ba J, Kiros R, et al. Show,attend and tell: Neural image caption generation with visual attention[C] / / International conference on machine learning. PMLR, 2015: 2048-2057.) proposes a thermal infrared tactile paving segmentation neural network (TINN) to overcome the challenges of tactile paving recognition under low light conditions. This method models two key factors in thermal infrared imaging—thermal light atmospheric transmission and thermal inertia effects—and uses an encoder-decoder architecture to fuse the features extracted by the two modules.

[0004] Patent 1 (Li Gang, Zhang Cheng, Zhou Mingle, et al. A method for tactile paving direction recognition and defect detection based on IoT and deep learning [P]. Shandong Province: CN117036937A, 2023-11-10.) uses a convolutional neural network to identify cracks and potholes on the surface of tactile paving and can identify the direction of the tactile paving in real time. Patent 2 (Wei Tong, Zhou Yinhe. A tactile paving recognition and localization algorithm based on machine learning recognition and image segmentation [P]. Beijing: CN109726681A, 2019-05-07.) proposes a method for locating and detecting tactile paving of different colors based on machine learning and image processing, but this method is relatively complex. Patent 3 (Li Zhenwei, Zhao Lei, Yang Xiaoli, et al. A method for identifying tactile paving based on image processing [P]. Henan Province: CN110334692B, 2021-07-23.) proposes a method to convert RGB tactile paving images into HSV space and use an adaptive threshold segmentation method to segment the step image, thereby segmenting the tactile paving. However, this method requires manual feature design and has poor generalization. This patent uses a multi-scale feature extraction method and designs a tactile paving segmentation method based on a spatial attention downsampling module to achieve the segmentation of tactile paving. Summary of the Invention

[0005] This invention discloses a tactile paving segmentation method based on a spatial attention downsampling module. The method employs a spatial attention downsampling module in the encoder section to enhance the encoder's multi-scale feature extraction capability. This module aims to improve contextual information extraction in pixel-level prediction tasks. In the decoder section, an auxiliary decoding method is used to extract edge information of the tactile paving, and finally, feature fusion is performed to obtain more edge information. The overall loss function is improved by incorporating the idea of ​​an auxiliary loss function before training the network.

[0006] A method for tactile paving segmentation based on a spatial attention downsampling module includes the following steps:

[0007] S1. Data Augmentation Preprocessing: Increase the original image size of the dataset to 2048. And randomly cut into The image is 3, and the cropped image is then randomly flipped to increase the data volume;

[0008] S2, Encoder Part: The constructed Spatial Attention Downsampling Block (SADB) inputs the data obtained in S2 into the SADB-improved encoder to extract multi-scale features from the data;

[0009] S3. Decoder section: The decoder section consists of an auxiliary decoder and a main decoder. The auxiliary decoder extracts edge information from the feature layer obtained in S2, and then performs feature fusion in the main decoder. The auxiliary decoder updates parameters by backpropagation of the edge loss function. The main decoder further extracts the input features, upsamples them to the mask image size, and finally trains the network model by backpropagation of the loss function to obtain the prediction result of the blind path area.

[0010] Furthermore, step S2, the encoder portion, includes the following steps:

[0011] The feature extraction backbone network is mainly composed of a convolutional neural network, a TransFormer, and the proposed spatial attention downsampling module. It is divided into two parts, front and back. The specific structure of the network is as follows:

[0012] S21. The first part of the encoder extracts features from the feature pyramid, resulting in four feature layers with sizes

[128] . 128 16],

[64] 64 32],[32 32 64],

[16] 16

[96] , the feature layer of each layer is obtained from the previous layer through the downsampling module, and each downsampling module consists of a depthwise separable convolution, a BN layer, and a ReLU6 activation function;

[0013] S22. Construct a spatial attention downsampling module, which consists of three paths: horizontal pooling, vertical pooling, and depth downsampling. The horizontal pooling path includes average pooling and max pooling layers, generating two layers of size [missing information]. After analyzing the feature vectors, feature fusion is performed. The vertical pooling path also includes average pooling and max pooling layers, resulting in two layers of size [value missing]. The feature vectors are fused, and the fused horizontal and vertical feature vectors are expanded into vectors of length W. H Two one-dimensional vectors are passed through a convolution kernel of size 1. Step size is One-dimensional convolution yields a length of The one-dimensional vector is then re-splitted and concatenated into two vectors of size 1. The two-dimensional feature vectors are then recombined into a two-dimensional feature layer using Python's broadcasting mechanism. The recombination process is as follows:

[0014]

[0015] The depthwise downsampling path contains a depthwise separable convolution that downsamples the feature layers before the input pooling path to a size of [value missing]. It is then fused with the feature layers obtained from the parallel pooling path;

[0016] S23. Based on S31 and S32, apply the spatial attention downsampling module to downsample the four feature layers obtained in S31 to... , For the first The number of channels in the layer is calculated, and the concatenation operation is performed along the channel dimension. Finally, the resulting feature vector is input into the attention module, which consists of a feedback multi-head attention mechanism and a TransFormer connected in parallel.

[0017] S24. In the latter half of the encoder, the feature layer from S31 is fused with the next layer after passing through the spatial attention downsampling module to obtain... The feature layer in S33 is segmented along the channel dimension, and then upsampled using bilinear interpolation. Perform feature fusion.

[0018] Furthermore, the decoding step S3 includes the following steps:

[0019] S31, the auxiliary decoder consists of two convolutional kernels of size [missing information]. It consists of a convolutional layer with a stride of 1, two MLP layers, a batch normalization layer and a ReLU6 activation layer. The features extracted in S2 are passed through the auxiliary decoder part and upsampled to the size of the mask image through bilinear interpolation to complete the segmentation prediction of the blind path boundary.

[0020] S32. Construct the loss function for the auxiliary decoder to perform backpropagation for the sample. , For training input data, For input The corresponding mask label real value, The true value is , To assist the decoder in predicting pixel values, the loss value of the boundary prediction result obtained in S41 is... , The definition formula is shown in the following equation:

[0021]

[0022] in Total number of pixels For the real mask image at position The gradient vector at that point, To assist the decoder in predicting the image at location The gradient vector at that point;

[0023] S33, the main decoder consists of three convolutional kernels of size [missing information]. The system consists of a convolutional layer with a stride of 1, an MLP layer, a batch normalization layer, and a sigmoid activation layer. The feature layer obtained by the decoder first passes through a convolutional layer, then is fused with the output layer in the auxiliary decoder. The fused feature layer then passes through a convolutional layer, an MLP layer, a batch normalization layer, and a sigmoid activation layer to further extract features. Finally, it is upsampled to the same size as the mask image to complete the prediction of the blind path area.

[0024] S34. The main decoder uses a fusion loss function for backpropagation to update the decoder training parameters. The fusion loss function is as follows:

[0025]

[0026] in The two-way cross-entropy loss function is given by the following formula:

[0027]

[0028] in Total number of pixels These are the actual mask pixel values. The main decoder predicts the image at the location The predicted pixel value at that location. Attached Figure Description

[0029] Figure 1 This is an overall flowchart of the lightweight tactile paving segmentation network based on the spatial attention downsampling module used in the embodiments of the present invention.

[0030] Figure 2 This is a partial structural diagram of the lightweight tactile paving segmentation network encoder based on a spatial attention downsampling module used in an embodiment of the present invention.

[0031] Figure 3 This is a partial structural diagram of the lightweight blind path segmentation network decoder based on the spatial attention downsampling module used in an embodiment of the present invention;

[0032] Figure 4 The spatial attention downsampling module SADB used in this embodiment of the invention is a lightweight tactile paving segmentation network based on a spatial attention downsampling module.

[0033] Figure 5 The image shows the implementation effect of the tactile paving segmentation network based on the spatial attention downsampling module used in the embodiments of the present invention, and a comparison with the original label. Detailed Implementation

[0034] To provide a clearer understanding of the technical features, objectives, and beneficial effects of the present invention, an embodiment of the invention will be further described in conjunction with the accompanying drawings. This embodiment is only for further illustration of the invention and should not be construed as limiting the scope of protection of the invention. Non-essential improvements and adjustments made by those skilled in the art based on the content of the invention also fall within the scope of protection of the present invention.

[0035] like Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown, a method for tactile paving segmentation based on a spatial attention downsampling module includes the following steps:

[0036] S1. Data Augmentation Preprocessing: Increase the original image size of the dataset to 2048. And randomly cut into The image is 3, and the cropped image is then randomly flipped to increase the data volume;

[0037] S2, Encoder Part: The constructed Spatial Attention Downsampling Block (SADB) inputs the data obtained in S2 into the SADB-improved encoder to extract multi-scale features from the data;

[0038] S3. Decoder section: The decoder section consists of an auxiliary decoder and a main decoder. The auxiliary decoder extracts edge information from the feature layer obtained in S2, and then performs feature fusion in the main decoder. The auxiliary decoder updates parameters by backpropagation of the edge loss function. The main decoder further extracts the input features, upsamples them to the mask image size, and finally trains the network model by backpropagation of the loss function to obtain the prediction result of the blind path area.

[0039] The encoder part of step S2 includes the following steps:

[0040] The feature extraction backbone network is mainly composed of a convolutional neural network, a TransFormer, and the proposed spatial attention downsampling module. It is divided into two parts, front and back. The specific structure of the network is as follows:

[0041] S21. The first part of the encoder extracts features from the feature pyramid, resulting in four feature layers with sizes

[128] . 128 16],

[64] 64 32],[32 32 64],

[16] 16

[96] , the feature layer of each layer is obtained from the previous layer through the downsampling module, and each downsampling module consists of a depthwise separable convolution, a BN layer, and a ReLU6 activation function;

[0042] S22. Construct a spatial attention downsampling module, which consists of three paths: horizontal pooling, vertical pooling, and depth downsampling. The horizontal pooling path includes average pooling and max pooling layers, generating two layers of size [missing information]. After analyzing the feature vectors, feature fusion is performed. The vertical pooling path also includes average pooling and max pooling layers, resulting in two layers of size [value missing]. The feature vectors are fused, and the fused horizontal and vertical feature vectors are expanded into vectors of length W. H Two one-dimensional vectors are passed through a convolution kernel of size 1. Step size is One-dimensional convolution yields a length of The one-dimensional vector is then re-splitted and concatenated into two vectors of size 1. The two-dimensional feature vectors are then recombined into a two-dimensional feature layer using Python's broadcasting mechanism. The recombination process is as follows:

[0043]

[0044] The depthwise downsampling path contains a depthwise separable convolution that downsamples the feature layers before the input pooling path to a size of [value missing]. It is then fused with the feature layers obtained from the parallel pooling path;

[0045] S23. Based on S31 and S32, apply the spatial attention downsampling module to downsample the four feature layers obtained in S31 to... , For the first The number of channels in the layer is calculated, and the concatenation operation is performed along the channel dimension. Finally, the resulting feature vector is input into the attention module, which consists of a feedback multi-head attention mechanism and a TransFormer connected in parallel.

[0046] S24. In the latter half of the encoder, the feature layer from S31 is fused with the next layer after passing through the spatial attention downsampling module to obtain... The feature layer in S33 is segmented along the channel dimension, and then upsampled using bilinear interpolation. Perform feature fusion.

[0047] The decoding step S3 includes the following steps:

[0048] S31, the auxiliary decoder consists of two convolutional kernels of size [missing information]. It consists of a convolutional layer with a stride of 1, two MLP layers, a batch normalization layer and a ReLU6 activation layer. The features extracted in S2 are passed through the auxiliary decoder part and upsampled to the size of the mask image through bilinear interpolation to complete the segmentation prediction of the blind path boundary.

[0049] S32. Construct the loss function for the auxiliary decoder to perform backpropagation for the sample. , For training input data, For input The corresponding mask label real value, The true value is , To assist the decoder in predicting pixel values, the loss value of the boundary prediction result obtained in S41 is... , The definition formula is shown in the following equation:

[0050]

[0051] in Total number of pixels For the real mask image at position The gradient vector at that point, To assist the decoder in predicting the image at location The gradient vector at that point;

[0052] S33, the main decoder consists of three convolutional kernels of size [missing information]. The system consists of a convolutional layer with a stride of 1, an MLP layer, a batch normalization layer, and a sigmoid activation layer. The feature layer obtained by the decoder first passes through a convolutional layer, then is fused with the output layer in the auxiliary decoder. The fused feature layer then passes through a convolutional layer, an MLP layer, a batch normalization layer, and a sigmoid activation layer to further extract features. Finally, it is upsampled to the same size as the mask image to complete the prediction of the blind path area.

[0053] S34. The main decoder uses a fusion loss function for backpropagation to update the decoder training parameters. The fusion loss function is as follows:

[0054]

[0055] in The two-way cross-entropy loss function is given by the following formula:

[0056]

[0057] in Total number of pixels These are the actual mask pixel values. The main decoder predicts the image at the location The predicted pixel value at that location.

[0058] Simulation Experiment

[0059] To better evaluate the segmentation capability of the network constructed in this invention, similarity coefficient (Dice), accuracy (Accuracy), foreground intersection-over-union ratio (IoU), and mean foreground intersection-over-union ratio (mIoU) were used in the experiment. The present invention achieved segmentation results of 94.2%, 97.49%, 94.51%, and 96.78% in terms of Dice, accuracy, IoU, mIoU, and FPS, respectively, which is quite good and has certain application prospects.

[0060] The method of the present invention has been described above. Those skilled in the art can implement the method of the present invention based on this description. Other embodiments obtained by those skilled in the art based on the above description of the present invention without inventive effort should all fall within the scope of protection of the present invention.

Claims

1. A method for tactile paving segmentation based on a spatial attention downsampling module, characterized in that, Includes the following steps: S1. Data augmentation preprocessing: Increase the size of the original images in the dataset to 2048×2048×3, and randomly crop them to 512×512×3. Then, randomly flip the cropped images to increase the data volume. S2, Encoder section: The constructed Spatial Attention Downsampling Module (SADB) inputs the data obtained in S1 into the SADB-improved encoder to perform multi-scale feature extraction on the data; S3, Decoder Part: The decoder part consists of an auxiliary decoder and a main decoder. The auxiliary decoder extracts edge information from the feature layer obtained in S2, and then performs feature fusion in the main decoder. The auxiliary decoder updates parameters by backpropagation of the edge loss function. The main decoder further extracts the input features, upsamples them to the mask size, and finally trains the network model by backpropagation of the loss function to obtain the prediction result of the blind path area. The encoder part of step S2 includes the following steps: The feature extraction backbone network comprises a convolutional neural network, a TransFormer, and the proposed spatial attention downsampling module, and is divided into two parts: a front-end and a back-end. The specific structure of the network is as follows: S21. The first half of the encoder extracts features from the feature pyramid, resulting in four feature layers with sizes of [128,128,16], [64,64,32], [32,32,64], and [16,16,96]. Each feature layer is obtained from the previous layer through a downsampling module. Each downsampling module consists of a depthwise separable convolution, a BN layer, and a ReLU6 activation function. S22. Construct a spatial attention downsampling module. This module consists of three paths: horizontal pooling, vertical pooling, and depth downsampling. The horizontal pooling path includes average pooling and max pooling layers, generating two feature vectors of size [1, W, C], which are then fused. The vertical pooling path also includes average pooling and max pooling layers, obtaining two feature vectors of size [H, 1, C], which are then fused. The fused horizontal and vertical feature vectors are unfolded into one-dimensional vectors of length W×C and H×C, respectively. These two one-dimensional vectors are then passed through a one-dimensional convolution with kernel size k and stride k to obtain a length of... The one-dimensional vector is then re-splitted and concatenated into two vectors of size 1. The two-dimensional feature vectors are then recombined into a two-dimensional feature layer using Python's broadcasting mechanism. The recombination process is as follows: The depthwise downsampling path contains a depthwise separable convolution that downsamples the feature layers before the input pooling path to a size of [value missing]. It is then fused with the feature layers obtained from the parallel pooling path; S23. Based on S21 and S22, apply the spatial attention downsampling module to downsample the four feature layers obtained in S21 to [8,8,C]. i ], C i Let be the number of channels in the i-th layer, and perform concatenation operations along the channel dimension. Finally, input the resulting feature vector into the attention module, which consists of a feedback multi-head attention mechanism and a TransFormer connected in parallel. S24. In the latter half of the encoder, the feature layer from S21 is fused with the next layer after passing through the spatial attention downsampling module to obtain F. i The feature layer output by the attention module in S23 is segmented along the channel dimension, and then upsampled using bilinear interpolation and F. i Perform feature fusion.

2. The tactile paving segmentation method based on a spatial attention downsampling module according to claim 1, characterized in that, The decoding step S3 includes the following steps: S31, the auxiliary decoder consists of two convolutional layers with a kernel size of 1×1 and a stride of 1, two MLP layers, a batch normalization layer and a ReLU6 activation layer. The features extracted in S2 are passed through the auxiliary decoder part and upsampled to the size of the mask image through bilinear interpolation to complete the segmentation and prediction of the blind path boundary. S32. Construct the loss function for the auxiliary decoder to perform backpropagation for the sample. For the training input data, y = {y i Let {i = 1, 2, 3, ..., n} be the actual mask label values ​​corresponding to input x, and y be the actual values ​​of input x. i The actual value is {0, 1}. To assist the decoder in predicting pixel values, the loss value of the boundary prediction result obtained in S31 is The definition formula is shown in the following equation: Where N is the total number of pixels. Let i be the gradient vector of the real masked image at position i. To assist the decoder in predicting the gradient vector of the image at position i; S33. The main decoder consists of three convolutional layers with a kernel size of 1×1 and a stride of 1, an MLP layer, a batch normalization layer, and a sigmoid activation layer. The feature layer obtained by the decoder first passes through a convolutional layer, and then is fused with the output layer in the auxiliary decoder. The fused feature layer then passes through a convolutional layer, an MLP layer, a batch normalization layer, and a sigmoid activation layer to further extract features. Finally, it is upsampled to the same size as the mask image to complete the prediction of the blind path area. S34. The main decoder uses a fusion loss function for backpropagation to update the decoder training parameters. The fusion loss function is as follows: L<0.8L c +0.2L b Where L c Let be the binary cross-entropy loss function, and its formula is as follows: Where N is the total number of pixels, y i These are the actual mask pixel values. The master decoder predicts the pixel value at position i in the image.

Images