An image semantic segmentation method for blind road scenes

Abstract

The application discloses an image semantic segmentation method for blind road scenes, which is characterized by a semantic segmentation model constructed by a Laplacian pyramid texture module, a Laplacian texture self-attention module and a decoder for segmenting blind road scene images, wherein the model comprises an LPTT block, a Segformer block and a decoder; the LPTT block extracts clear low-level features through a texture self-attention extraction layer; the Segformer block extracts accurate high-level semantic features; and the decoder fuses multi-level features. Compared with the prior art, the application has lower complexity than other semantic segmentation networks, solves the problem of slow segmentation speed of widely used semantic segmentation networks for blind road scene images, and has a segmentation accuracy comparable to other semantic segmentation networks in a large number of experiments in different environments.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation technology, and in particular to an image semantic segmentation method for tactile paving scenarios. Background Technology

[0002] Outdoors, visually impaired individuals often rely on tactile paving to navigate. They typically identify tactile paving through the tactile feedback of their shoes or using a white cane. These methods only allow for perception within a limited area and do not provide rich environmental information. Therefore, visually impaired individuals need the assistance of tactile guidance systems to walk more safely. A crucial input to these systems is information from the environment. However, acquiring this information is challenging due to complex environmental factors, such as varying lighting conditions and paving stones of similar color to the tactile paving. Therefore, in tactile guidance systems, tactile paving identification is often based on computer vision to accurately perceive environmental information.

[0003] Currently, computer vision-based methods for identifying tactile paving can be divided into statistical methods and deep learning methods. Statistical methods often employ gray-level co-occurrence matrices and color histograms to identify tactile paving, but these methods are computationally complex. Furthermore, statistical information alone is easily affected by environmental factors, failing to accurately identify tactile paving in harsh environments. With the rapid development of deep learning, current tactile paving identification methods typically rely on an encoder-decoder architecture. The encoder generates image features, and the decoder fuses these features into a segmentation map with pixel-wise classification scores. Compared to previous statistical methods, the encoder-decoder architecture extracts semantic features of tactile paving at a deeper level, achieving better recognition results. However, existing deep learning-based methods exhibit poor boundary prediction when there are tiles of similar color around the tactile paving. In digital image processing, texture is a region descriptor that provides a measure of both local structural properties and global statistical properties of an image. Deep learning-based methods for identifying blind paths underestimate the learning of texture features. The underlying features of textureless enhancement models are mainly color, with weak texture information, resulting in poor differentiation of underlying feature levels. Various studies have found that the texture information in the underlying features generated by textureless enhancement models is insufficient.

[0004] Existing semantic segmentation networks are not only highly complex, but also fail to effectively and accurately transfer rich texture information from texture features to lower-level features. As a result, the texture information is poorly reflected and the differentiation of lower-level features is poor, which greatly affects the segmentation speed and boundary prediction accuracy of tactile paving scene images. Summary of the Invention

[0005] The purpose of this invention is to provide a semantic segmentation method for tactile paving scenes to address the shortcomings of existing technologies. It employs the Laplacian Texture Self-Attention (LTSS) method to construct a semantic segmentation model for tactile paving scenes, consisting of a Laplacian pyramid texture module, a Laplacian texture self-attention module, and a decoder. This method accurately transfers texture information to low-level features. The segmentation model learns to "accept" texture information during the segmentation process, mainly including three steps: 1) compressing the original low-level features and obtaining texture features through the Laplacian pyramid; subsequently, through a self-attention mechanism, effectively transferring the rich texture information from the texture features to the original low-level features, thereby obtaining new low-level features with clear boundaries and distinct layers; 2) extracting high-level semantic features; 3) finally, using a simple and efficient decoder to integrate multi-level features and output a segmentation mask. This invention yields a semantic segmentation network with lower complexity, effectively solving the problem of slow segmentation speed for tactile paving scenes in widely used semantic segmentation networks. Furthermore, it demonstrates segmentation accuracy comparable to other semantic segmentation networks in numerous experiments under different environments. The method is simple, effective, and has high practical value and promising development prospects.

[0006] The specific technical solution to achieve the purpose of this invention is: an image semantic segmentation method for tactile paving scenes, characterized by using a semantic segmentation model constructed from a Laplacian pyramid texture module, a Laplacian texture self-attention module, and a decoder to segment tactile paving scene images. The semantic segmentation model annotates the tactile paving scene image dataset to obtain labeled data. The Laplacian pyramid texture module uses two LPTT blocks to output two low-level semantic feature maps respectively, and uses a self-attention mechanism to learn the image data. It extracts well-defined low-level features through texture self-attention to supplement texture information. The Laplacian texture self-attention module extracts accurate high-level semantic features. The decoder fuses multi-level features and outputs a semantic segmentation mask.

[0007] Specifically, this invention includes: collecting image datasets of various types of tactile paving and complex street scenes, and labeling the tactile paving scene image datasets to obtain labeled data; using a self-attention mechanism to learn image data and supplement texture information in the LPTT block, with the two LPTT blocks outputting two low-level feature maps respectively; based on the low-level features in the LPTT block, the two Segformer blocks 3 and 4 generate two semantic feature maps respectively; and the decoder fuses the information from the four feature maps to output a segmentation mask map.

[0008] Furthermore, obtaining labeled data includes: marking the edges of the tactile paving area; marking the area of ​​the road surface other than the tactile paving; treating the rest of the image as obstacles; and dividing the image into first labeled data, second labeled data, and third labeled data according to the above classification.

[0009] Furthermore, for each of the two LPTT blocks, two low-level feature maps are output. The specific method is as follows: In any LPTT block, the Segformer block first generates the original low-level features, and then the Laplacian texture self-attention accurately transfers richer texture information to the original low-level features, generating low-level features with clear hierarchy.

[0010] Furthermore, in any LPTT block, the Segformer block first generates the original low-level features, wherein the Segformer block includes:

[0011] 1) Overlapping slice embedding: Given an input image X, LPTT block 1 will... Reconstructing the embedded sequence by overlapping Where P is the downsampling rate, and D is the X... e The order of . We set P=4, D=32 in LPTT block 1;

[0012] 2) Efficient self-attention mechanism: For a given X e The self-attention parameters [q, k, v] are derived from the following equation (a):

[0013] [q, k, v] = [W] q W k W v ]⊙X e (a).

[0014] Among them, [W q W k W v [These are three learnable weight matrices, q, k, and v, all with the same shape.] To reduce computational complexity, a reduction ratio R is introduced to reduce the lengths of q, k, and v. Taking q as an example, q will first be reshaped into... Next, a linear layer reduces the number of channels in q from DR to D; therefore, the shape of q involved in the self-attention computation is... In this invention, R = 64 is set in LPTT block 1. Then, multi-head self-attention calculation is performed on [q, k, v], and the self-attention calculation result X is output. a Where H is the image height; W is the image width; P is the downsampling rate; and D is the X value. e The order of ; R is the reduction ratio; [q, k, v] are the self-attention parameters.

[0015] 3) Hybrid feedforward network: The hybrid feedforward network directly incorporates a 3×3 convolution into the feedforward network to provide positional information. The hybrid feedforward network is represented by the following (b):

[0016] X m =MLP(GELU(Conv 3×3 (MLP(X a )))+X a (b)

[0017] Among them, X m The shape is GELU stands for Activation Function, MLP stands for Multilayer Perceptron, and Conv... 3×3 The kernel is a convolution kernel; the hybrid feedforward network uses depthwise convolution.

[0018] Furthermore, Laplacian texture self-attention accurately transfers richer texture information to the original low-level features, generating low-level features with clear layers. The description of Laplacian texture self-attention is as follows:

[0019] 1) Search for the most relevant texture information: First, calculate the correlation coefficient matrix R using the following equation (c):

[0020]

[0021] Where Q = X m X m The original feature map without texture learning; Q i K is the i-th feature vector of Q; K is L after overlapping patch embedding; L is the Laplacian texture map; K j Let Q be the j-th eigenvector of K; the shapes of Q and K are both...

[0022] The relevant vector index matrix I in K is then calculated by the following equation (d):

[0023]

[0024] Among them, I i Indicate which vector in K is related to Q i Most relevantly, argmax is a function that takes the index of the maximum value within a range.

[0025] 2) Use index collection operations to extract texture information in V that is strongly correlated with Q: i.e., T i =V I Where V = K is L after overlapping patch embedding; L is the Laplacian texture map; Q represents the texture features associated with Q; Q is the raw feature map without texture learning.

[0026] 3) Texture Information Transfer: To prevent over-correction of texture features and preserve original feature information, Laplacian texture self-attention will transfer X... m The fused feature map G is obtained by fusing the searched texture feature t with the fused feature map t using the following equation (e):

[0027] G = Conv(Concat(Fold(X) m ),F old(T))) (e).

[0028] Where Fold(·) refers to the X to be expanded. m and T in resolution Fold it.

[0029] 4) Use the equation expressed in equation (f) below to strengthen the strongly correlated late-stage texture features and weaken the weakly correlated texture features:

[0030]

[0031] F1 = X m +G⊙Fold(W) (f).

[0032] F1 is the final output feature map of LPTT block 1.

[0033] Furthermore, for each of the two LPTT blocks, two low-level feature maps are output. The algorithm flow of LPTT block 2 is the same as that of LPTT block 1, but the input of LPTT block 2 is F1, and P=2, D=64, R=16 is set. The output of LPTT block 2 is F2.

[0034] Furthermore, for the low-level features based on the LPTT block, the two Segformer blocks 3 and 4 generate two semantic feature maps respectively. Compared with the LTSS block, Segformer blocks 3 and 4 remove the Laplacian texture self-attention and set D=[160, 256], R=[16, 1], P=2, outputting two high-level semantic feature maps F3 and F4.

[0035] Furthermore, the decoder fuses the information from the four feature maps to output a segmentation mask. The decoding process of the decoder is described by the following equations (g) to (j):

[0036] F i =Conv 1×1 (C i ,C)(F i ), i∈[1,4] (g);

[0037]

[0038] F′=Conv 1×1 (4C,C)(Concat([F1,F2,F3,F4])) (i);

[0039] F out =Conv 1×1 (C, N) cls )(F′) (i).

[0040] Among them, F i Let F be the i-th feature map, i = 1 to 4; F′ is the feature map after multi-level information fusion; N cls It is the number of semantic categories; F out This is the final semantic segmentation mask image; The F outputs of the encoder are respectively i The resolution and the F after upsampling by the decoder i The resolution.

[0041] Furthermore, after obtaining the semantic segmentation mask, the process also includes: evaluating the accuracy of the segmentation model by mIOU and evaluating the complexity of the model by FLOPs, wherein the mIOU evaluation is 91.25% and the FLOPs evaluation is 6.54.

[0042] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0043] (1) Innovatively, Laplacian texture was introduced and a new blind path recognition model was proposed: Laplacian Texture Self-Attention Segformer (LTSS).

[0044] (2) Innovatively utilize texture information to form low-level features that positively influence the tactile paving segmentation results, so that the tactile paving segmentation results have highly accurate boundaries.

[0045] (3) Extensive experiments have demonstrated that the model of this invention achieves state-of-the-art results on various datasets and has higher segmentation efficiency than other general segmentation networks due to its lower model complexity. Attached Figure Description

[0046] Figure 1 This is a diagram of the overall framework of Laplacian texture self-attention Segformer (LTSS) in Example 1;

[0047] Figure 2 The images show the segmentation masks obtained from different network frameworks in Example 1. Detailed Implementation

[0048] This invention uses the Laplacian Texture Self-Attention Segformer (LTSS) method to segment images of tactile paving scenes. This method consists of three key components: an LPTT block, a Segformer block, and a decoder. The LPTT block extracts well-defined low-level features through texture self-attention, the Segformer block extracts precise high-level semantic features, and the decoder fuses multi-level features. The resulting model has lower complexity than other semantic segmentation networks, solving the problem of slow segmentation speed for tactile paving scenes in widely used semantic segmentation networks. Furthermore, it demonstrates segmentation accuracy comparable to other semantic segmentation networks in numerous experiments under various environments, thus addressing the issue of slow segmentation speed for tactile paving scenes in widely used semantic segmentation networks.

[0049] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to the accompanying drawings and preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0050] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0051] Example 1

[0052] See Figure 1 The present invention achieves image semantic segmentation of tactile paving scenes according to the following steps:

[0053] Step S101: Collect a rich variety of tactile paving and complex street scene image datasets, and label the tactile paving scene image datasets to obtain labeled data;

[0054] In step S102, the LPTT block uses a self-attention mechanism to learn image data and supplement texture information, and the two LPTT blocks output two low-level feature maps respectively.

[0055] Step S103: Based on the low-level features in the LPTT block, two Segformer blocks 3 and 4 generate two semantic feature maps respectively;

[0056] In step S104, the decoder fuses the information from the four feature maps and outputs a segmentation mask map.

[0057] See Figure 2 The specific implementation plan is as follows:

[0058] 1. Collect a rich variety of images of tactile paving and complex street scenes, and annotate the images of tactile paving to obtain labeled data.

[0059] 1) Collect images of tactile paving scenes

[0060] The tactile paving in different scenarios has different textures and colors, and the types of obstacles and background complexity in the scenarios are also different. Therefore, the images collected must contain a variety of tactile paving types and complex street scenes.

[0061] 2) Dataset labeling

[0062] Use annotation tools to annotate all samples in the above images: mark the edges of the tactile paving area; mark the area of ​​the road surface other than the tactile paving; and treat the rest of the image as obstacles.

[0063] Based on the above classification, a single image is divided into first-label data, second-label data, and third-label data.

[0064] 3) Dataset partitioning

[0065] All data were divided into training, testing, and validation sets in a ratio of 7:2:1.

[0066] 2. The LPTT block uses a self-attention mechanism to learn image data and supplement texture information. The two LPTT blocks output two low-level feature maps respectively.

[0067] 1) In any LPTT block, the Segformer block first generates the original low-level features, wherein the Segformer block includes:

[0068] Overlapping slice embedding: Given an input image X, LPTT block 1 will... Reconstructing the embedded sequence by overlapping Where P is the downsampling rate, and D is the X... e The order of . We set P=4, D=32 in LPTT block 1;

[0069] Efficient self-attention mechanism: for a given X e The self-attention parameters [q, k, v] are derived from the following equation (a):

[0070] [q, k, v] = [W] q W k W v ]⊙X e (a).

[0071] Among them, [W q W k W v [These are three learnable weight matrices, q, k, and v, all with the same shape.] To reduce computational complexity, a reduction ratio R is introduced to reduce the lengths of q, k, and v. Taking q as an example, q will first be reshaped into... Next, a linear layer reduces the number of channels in q from DR to D; therefore, the shape of q involved in the self-attention computation is... In LPTT block 1, we set R = 64. Then, we perform multi-head self-attention calculation on [q, k, v] and output the self-attention calculation result X. a .

[0072] Hybrid feedforward network: The hybrid feedforward network directly incorporates a 3×3 convolution into the feedforward network to provide positional information. The hybrid feedforward network can be represented by the following (b):

[0073] X m =MLP(GELU(Conv 3×3 (MLP(X a )))+X a (b)

[0074] Among them, X m The shape is GELU stands for Activation Function, MLP stands for Multilayer Perceptron, and Hybrid Feedforward Network uses Deep Convolution.

[0075] 2) Laplacian texture self-attention accurately transfers richer texture information to the original low-level features, generating low-level features with clear layers. The description of Laplacian texture self-attention is as follows:

[0076] Search for the most relevant texture information: First, calculate the correlation coefficient matrix R using the following equation (c):

[0077]

[0078] Where Q = X m K is the L after the overlapping patch is embedded. The shapes of Q and K are both Subsequently, the relevant vector index matrix I in K is calculated by the following equation (d):

[0079]

[0080] Among them, I i Indicate which vector in K is related to Q i Most relevantly, next, we use index collection operations to extract the texture information in V that is strongly correlated with Q, i.e., T. i =V I Where V = K is L after overlapping patch embedding; L is the Laplacian texture map; Q represents the texture features associated with Q; Q is the raw feature map without texture learning.

[0081] Texture information transfer: To prevent over-correction of texture features and preserve original feature information, Laplacian texture self-attention will transfer X...m The fused feature map G is obtained by fusing the searched texture feature t with the fused feature map t using the following equation (e):

[0082] G = Conv(Concat(Fold(X) m ),F old(T))) (e).

[0083] Where Fold(·) refers to the X to be expanded. m and T in resolution Fold the material. Then, use the equation expressed in equation (f) below to strengthen the strongly correlated late-stage texture features and weaken the weakly correlated texture features:

[0084]

[0085] F1 = X m +G⊙Fold(W) (f).

[0086] F1 is the final output feature map of LPTT block 1.

[0087] 3) For the two LPTT blocks, two low-level feature maps are output respectively. The algorithm flow of LPTT block 2 is the same as that of LPTT block 1, but the input of LPTT block 2 is F1, and P=2, D=64, R=16 is set. The output of LPTT block 2 is F2.

[0088] 3. Based on the low-level features in the LPTT block, two Segformer blocks 3 and 4 generate two semantic feature maps respectively.

[0089] 1) Compared with the LTSS block, Segformer blocks 3 and 4 remove the Laplacian texture self-attention and set D=[160, 256], R=[16, 1], P=2, outputting two high-level semantic feature maps F3 and F4.

[0090] 4. The decoder will fuse the information from the four feature maps and output a segmentation mask.

[0091] 1) The decoder is described in detail in equations (g) to (j) below:

[0092] F i =Conv 1×1 (C i ,C)(F i ), i∈[1,4] (g);

[0093]

[0094] F′=Conv 1×1(4C,C)(Concat([F1,F2,F3,F4])) (i);

[0095] F out =Conv 1×1 (C, N) cls )(F′) (i).

[0096] Where, N cls It is the number of semantic categories, F out This is the final semantic segmentation mask. After obtaining the semantic segmentation mask, the following steps are also taken: evaluating the accuracy of the segmentation model using mIOU and evaluating the model's complexity using FLOPs. The evaluation results are detailed in Tables 1 and 2 below:

[0097] Table 1. Accuracy performance of different segmentation networks on the BRPS dataset.

[0098]

[0099] Table 2 shows the complexity performance of different segmentation networks on the BRPS dataset.

[0100]

[0101] By comparing the accuracy and complexity of different segmentation networks on the BRPS dataset, it can be seen that the model obtained in this invention has lower complexity than other semantic segmentation networks, solves the problem that widely used semantic segmentation networks are too slow in segmenting images of blind paths, and demonstrates segmentation accuracy comparable to other semantic segmentation networks in a large number of experiments in different environments.

[0102] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image semantic segmentation method for a blind path scene, characterized in that, The semantic segmentation model constructed by using a Laplacian pyramid texture module, a Laplacian texture self-attention module and a decoder is used to segment a blind sidewalk scene image, the semantic segmentation model is used to label a blind sidewalk scene image dataset to obtain labeled data, the Laplacian pyramid texture module uses two LPTT blocks to respectively output two low-level semantic feature maps, uses a self-attention mechanism to learn image data, extracts hierarchical clear bottom features through a texture self-attention extraction layer to supplement texture information, the Laplacian texture self-attention module extracts accurate high-level semantic features, and the decoder fuses multi-level features to output a semantic segmentation mask image. The two LPTT blocks respectively output two low-level semantic feature maps, and the specific method is that in any one LPTT block, a Segformer block first generates original low-level features, then a Laplacian texture self-attention accurately transmits texture information to the original bottom features to generate hierarchical clear bottom features. The Segformer block comprises: 1) Overlapping patch embedding: Given an input image X, LPTT block 1 will reconstruct into an embedding sequence in an overlapping manner where P is the down-sampling rate; D is the order of X e P = 4, D = 32 are set in LPTT block 1; 2) Self-attention mechanism: for a given embedding sequence X e , the self-attention parameters [q, k, v] are computed by multi-head self-attention as described by (a) below, outputting a self-attention result X a : [q, k, v] = [W q ,W k ,W v ] ⊙ X e (a); where [W q ,W k ,W v ] are three learned weight matrices; 3) Hybrid feedforward network: Hybrid feedforward network X m Mixing a 3x3 convolution directly in the feedforward network provides positional information, the hybrid feedforward network X m With deep convolution, and is expressed by the following (b) formula: X m = MLP(GELU(Conv 3×3 (MLP(X a )) + X a (b); wherein GELU represents an activation function; MLP is a multi-layer perceptron; Conv 3×3 is a convolution kernel; The Laplacian texture self-attention accurately transmits texture information to the original bottom features to generate hierarchical clear bottom features, and the generation process is as follows: 1) Texture information search: The correlation coefficient matrix R is calculated using the following equation (c) i,j : wherein Q = X m , X m is the original feature map without texture learning; Q . is the i-th feature vector of Q; K is L after the overlapping patch embedding; L is the Laplacian texture map; K / is the j-th feature vector of K; the relevant vector index matrix I in K is calculated by the following equation (d): where I . represents which vector in K is most relevant to Q . ; K is the embedding of L after overlap patching; 2) Extract texture information in V that is strongly associated with Q using an index collection operation, i.e., T . = V C where V = K is the L after overlapping patch embedding; is the texture feature associated with Q; Q is the original feature map without texture learning; 3) Texture information transmission: Laplacian texture self-attention will X m The searched texture feature t is fused by the following formula (e) to obtain a fused feature map G: （e）； where Fold(·) denotes folding the unfolded X m and T at resolution ; H is the image height; W is the image width; P is the down-sampling rate; 4) The following equation group (f) is used to strengthen the late texture feature part with strong correlation and weaken the texture feature part with weak correlation: F V = X m + G o Fold(W) (f); where F V is the final output feature map of LPTT block 1 ; W . is the texture affinity matrix.

2. The method of claim 1, wherein, The labeled data of the blind sidewalk scene image dataset comprises the following steps: labeling the edge of the blind sidewalk area, labeling the area of the road surface except the blind sidewalk, regarding the remaining part of the image as an obstacle, and dividing the image into first labeled data, second labeled data and third labeled data according to the above classification.

3. The method of claim 1, wherein, The two LPTT blocks respectively output two low-level semantic feature maps using the same algorithm process, the input of the LPTT block 1 is the low-level semantic feature map F V , and the output of the LPTT block 2 is the low-level semantic feature map F X .

4. The method of claim 3, wherein, The self-attention mechanism is used to learn image data, and the texture self-attention is used to extract hierarchical clear bottom features to supplement texture information, which is based on the low-level features in the LPTT block, and Segformer block 3 and Segformer block 4 are used to respectively generate two high-level semantic feature maps F3 and F4 without Laplacian texture self-attention.

5. The method of claim 4, wherein, The decoder fuses the information in the feature maps F1, F2, F3 and F4 to output a semantic segmentation mask image, and the process is described by the following equations (g)-(j): F . = Conv V×V (C . , C)(F . ), i e [1, 4] (g); F' = Conv V×V (4C,C) LConcat([F V ,F X ,F3,F ` ]) P (i); F bcd = Conv V×V (C,N Ffg )(F′) (i); Wherein, F . is the i-th feature map, i = 1 ~ 4; F' is the feature map after multi-level information fusion; N Ffg is the number of semantic categories; F bcd is the final semantic segmentation mask map obtained; respectively, the resolution of F . output by the encoder and the resolution of F . after decoder up-sampling.

6. The method of claim 1 or claim 5, wherein, After the semantic segmentation mask image is obtained, the precision of the segmentation model is evaluated by mIOU, and the complexity of the model is evaluated by FLOPs, the mIOU evaluation is 91.25%, and the FLOPs evaluation is 6.54.

Images