An efficient semantic segmentation method for rail transit

Abstract

An efficient semantic segmentation method for rail transit relates to the field of computer vision and image technology. This method utilizes a hybrid track segmentation dataset and a city landscape segmentation dataset. The network comprises a feature extraction module, a pyramid pooling module, an attention optimization module, an object attention module, a feature fusion module, and a segmentation head module. By training the network, it can accurately segment both tracks and backgrounds, as well as urban roads and backgrounds, at a speed fast enough to meet real-time requirements.

Description

Technical Field

[0001] The present invention relates to the field of computer vision image technology, and in particular to a real-time semantic segmentation method based on deep learning. Background Art

[0002] With the rapid economic development in recent years, autonomous driving technology has made tremendous progress. Its application has significantly improved vehicle safety and efficiency. Currently, autonomous driving primarily uses a combination of LiDAR and industrial cameras to collect environmental data. Image semantic segmentation and object detection algorithms process this data to generate results for driver assistance. Therefore, image semantic segmentation algorithms are a crucial component of autonomous driving. In recent years, autonomous driving technologies for rail transit have seen significant development and application, significantly improving rail transit safety. In the field of autonomous rail transit, efficient and accurate image semantic segmentation algorithms are a core technology. Using image semantic segmentation algorithms to efficiently and accurately demarcate track and non-track areas is a prerequisite for subsequent object detection algorithms to detect and identify obstacles in these areas. Furthermore, the relatively high speeds of rail transit necessitate real-time performance. Therefore, an efficient and accurate image semantic segmentation algorithm is crucial for the implementation of autonomous rail transit technology.

[0003] Image semantic segmentation requires identifying every location in the scene and recognizing the semantic category of each pixel in the image. Image semantic segmentation algorithms can be divided into traditional algorithms and deep learning-based algorithms. Traditional image segmentation algorithms divide the image into two regions, namely foreground and background, according to predetermined rules. The foreground usually represents the target object to be detected, and the image segmentation algorithm's task is to separate the target object from the image. Traditional image segmentation algorithms, such as pixel threshold segmentation algorithms and edge detection-based segmentation algorithms, mainly rely on physical information such as color or texture of each pixel in the image to perform feature segmentation on specific areas of the image according to pre-set judgment rules. They have the advantages of simple algorithms and easy implementation. However, when the image complexity increases and the scene contains a large number of objects, the segmentation accuracy is greatly affected, making them unsuitable for application in rail transit environments. In recent years, neural networks have been widely used in various visual processing tasks, and deep learning-based image semantic segmentation methods have also emerged.

[0004] In recent years, image semantic segmentation algorithms based on deep learning have developed rapidly. Image semantic segmentation methods based on deep learning use a large number of rich image data sets as data-driven to train image semantic segmentation models. Image semantic segmentation methods based on deep learning can significantly improve image segmentation results compared to traditional image segmentation methods. The original intention of the design of this invention is to cooperate with the point cloud segmentation of the lidar to realize the perception segmentation of the environment. Considering the acquisition speed of the lidar data, most of the current real-time semantic segmentation methods have overflows in speed, but cannot achieve good results in terms of accuracy. The operation of deep learning image semantic segmentation algorithms with higher accuracy requires a lot of time overhead and cannot achieve real-time. It is stretched in the application of perception of traffic environment. In order to cooperate with the lidar to achieve real-time perception of the rail transit environment, designing a semantic segmentation method that has both speed and accuracy is still a problem that needs to be solved urgently. Summary of the Invention

[0005] The purpose of this invention is to design an efficient semantic segmentation method for rail transit in response to the shortcomings of the existing technology. The semantic segmentation model trained by this method can achieve real-time inference speed while taking into account high-precision segmentation results.

[0006] The present invention is implemented by adopting the following technical solutions: mainly including data set establishment and efficient image segmentation network design.

[0007] Dataset establishment: A mixture of track segmentation dataset and urban landscape segmentation dataset is used.

[0008] Efficient image segmentation network: The network includes feature extraction module, pyramid pooling module, attention optimization module, object attention module, feature fusion module, and segmentation head module.

[0009] By training the network, it can accurately segment the track from the background and the urban road from the background, and achieve a fast enough speed to meet real-time requirements.

[0010] The specific steps for making the dataset are as follows:

[0011] For the collected rail transit operation video data, each image is segmented into four categories: track, hazard, and background. The background class is labeled 0, the track class is labeled 1, and the hazard class is labeled 2. To enhance the robustness and applicability of the semantic segmentation network, a public urban landscape dataset is used as an extension.

[0012] The specific steps of the efficient image segmentation network are as follows:

[0013] The efficient image segmentation network of the present invention is shown in the attached figure. Figure 1As shown in Figure 2, the efficient image segmentation network consists of the following modules, including feature extraction module, pyramid pooling module, object attention module, attention optimization module, feature fusion module, segmentation head module, etc.

[0014] As attached Figure 1 As shown in Figure 2, the feature extraction network of the segmentation network is divided into five stages, where stage one and stage two are each composed of a "Conv+BN+Relu" operation block, and stages three to five are each composed of two feature extraction modules. Figure 2 As shown in the figure, the feature extraction module consists of four "Conv+BN+Relu" operation blocks. The convolution kernel size of the first operation block is 1×1, and the convolution kernel size of the remaining blocks is 3×3. Assuming that the final output channel number of the feature extraction module is N, except for the fourth operation block, the output channel number of the i-th operation block in the feature extraction module is N / 2. i , the number of output channels of the fourth operation block is consistent with the number of output channels of the third operation block. The output of the feature extraction module is the result of the fusion of the corresponding outputs of these four blocks. In order to ensure that the size of the feature maps involved in the cascade is consistent, before the cascade, the feature maps of different sizes are downsampled to the same size through the average pooling operation. Since the shallow layer requires more channels for feature encoding detail information and the deep layer pays more attention to high-level semantic information, too many feature channels will lead to information redundancy. Therefore, the feature extraction module in the present invention has fewer deep feature channels and more shallow feature channels. Stages one to five constitute a feature extraction network, which downsamples the feature map with a step size of two. In order to reduce the amount of calculation, only one convolutional layer is used in stage one and stage two. Each stage from stage three to stage five contains two feature extraction modules, of which the first feature extraction module contains a downsampling operation, and the remaining feature extraction modules keep the feature map size unchanged.

[0015] Pyramid pooling module. The structure diagram of the pyramid pooling module is as shown in the attached figure. Figure 3 As shown. In order to increase the multi-scale contextual information in the feature map extracted by the feature extraction module, the present invention adds a pyramid pooling module to the segmentation network structure. This module can effectively improve the segmentation accuracy of the network without increasing the calculation parameters. First, the input feature map is averaged through four pooling kernels of different scales to output four feature maps of different sizes, and then the four feature maps of different sizes are reduced in dimension through the "ConvBNRelu" operation. Next, the four feature maps of different sizes are upsampled to the same size as the input feature map. Finally, the four features are cascaded as global features and then subjected to a "ConvBNRelu" operation to output the final result.

[0016] Object attention module. The structure diagram of the object attention module is as shown in the attached figure. Figure 4As shown. This module calculates the relationship matrix between the pixel features output by the pyramid pooling module, which can significantly enhance the contextual information representation capability of the features. In the present invention, the two input features are referred to as input feature 1 and input feature 2 respectively. First, input feature 1 is subjected to convolution, reshape, and transpose operations to obtain feature a, and input feature 2 is subjected to convolution and reshape operations to obtain feature b. Feature a is then multiplied by feature b to obtain feature d, which is then multiplied by feature c obtained by performing a 1×1 convolution and reshape operation on input feature 2, and finally the final output is obtained by performing transpose, reshape, and convolution operations.

[0017] Attention optimization module. The structure diagram of the attention optimization module is shown in the attached figure. Figure 5 As shown in Figure 2. This module is used to optimize the features of stages four and five. First, the positional attention mechanism is used after the first "ConvBNRelu" operation to capture the spatial dependency between any two positions in the feature map. For the features of a specific position, the features are updated by aggregating the features at all positions in a weighted sum, where the weight is determined by the feature similarity between the corresponding two positions. Therefore, any two positions with similar features can promote mutual improvement regardless of their distance in the spatial dimension, and then an average pooling operation is applied to obtain global semantic information. After that, the channel attention mechanism is used to integrate relevant features in all channel maps, selectively emphasizing mutually correlated channel maps. Finally, the features output by the first "ConvBNRelu" operation are multiplied by the features after the channel attention mechanism to guide feature learning. This structure can refine the results of each stage in the feature extraction network. It can integrate global semantic information without upsampling, with low computational cost.

[0018] Feature fusion module. The structure diagram of the feature fusion module is shown in the attached figure. Figure 6 As shown in the figure, the feature fusion module combines contextual and spatial feature information with extremely low computational overhead. The input contextual and spatial feature information are first concatenated (Concat). The concatenated features are then subjected to convolutional pooling and a nonlinear activation function to generate a weight vector. This weight vector is multiplied by the feature vector generated by the "ConvBNRelu" operation, and the resulting result is then added to the feature vector generated by the "ConvBNRelu" operation to obtain the final result. This fusion method achieves both feature selection and integration.

[0019] Split head module. The structure diagram of the split head module is as shown in the attached Figure 7As shown in Figure 2, the segmentation head includes a 3×3 convolution, BN and ReLU operations, and a 1×1 convolution. The segmentation head is inserted after stage 3 in the feature extraction network to allow the feature map output by stage 3 to contain more detailed features for fusion with high-level semantic features containing contextual information.

[0020] Through the mutual cooperation of various modules, the trained model not only achieves effective segmentation of the input image, but also avoids the problems of long processing time and poor segmentation effect.

[0021] Features of the present invention:

[0022] Based on the practical application requirements, this paper designs an efficient semantic segmentation method for rail transit. First of all, this method is universal and applicable not only to rail transit but also to ordinary road traffic. Regardless of the type of traffic scene the input image belongs to, it can be segmented effectively and quickly. For rail transit scenes, only four categories of labeling are required: rails and hazards, warnings, and background. The labeling work is simple and consumes little human resources. The network inference speed can reach real time. It can be deployed on embedded devices for real-time segmentation perception of the traffic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a structural framework diagram of an efficient semantic segmentation method for rail transit described in the present invention.

[0024] Figure 2 Schematic diagram of the feature extraction module structure in the efficient semantic segmentation network of the present invention

[0025] Figure 3 Schematic diagram of the pyramid pooling module structure in the efficient semantic segmentation network of the present invention

[0026] Figure 4 Schematic diagram of the object attention module structure in the efficient semantic segmentation network of the present invention

[0027] Figure 5 Schematic diagram of the attention optimization module structure in the efficient semantic segmentation network of the present invention

[0028] Figure 6 Schematic diagram of the feature fusion module structure in the efficient semantic segmentation network of the present invention

[0029] Figure 7 Schematic diagram of the segmentation head module structure in the efficient semantic segmentation network of the present invention DETAILED DESCRIPTION

[0030] The present invention will be described in further detail below with reference to the embodiments and accompanying drawings.

[0031] The specific steps for making the dataset are as follows:

[0032] The dataset uses collected rail transit operation monitoring video data with a resolution of 1980×1080. The monitoring videos should cover as many time periods and track conditions as possible. The images in the dataset are annotated into three categories: background areas, labeled 0 and represented by the RGB value (0, 0, 0); track areas, labeled 1 and represented by the RGB value (255, 0, 0); and dangerous areas, labeled 2 and represented by the RGB value (0, 255, 0). After data annotation, the dataset is augmented using methods such as random horizontal flipping, vertical and horizontal translation, color perturbation, and angular rotation to improve the generalization ability of the segmentation network. The dataset is divided into training, test, and validation sets in a ratio of 7:2:1.

[0033] The specific steps of the efficient image segmentation network are as follows:

[0034] An efficient semantic segmentation method for rail transit, as shown in the attached Figure 1As shown, the following steps are included: Assuming the initial size of the image input to the network is 3×1024×512 (C×W×H), after passing through the feature extraction network, feature maps of different sizes are obtained for subsequent processing. The specific operating parameters are shown in Tables 1 to 4. First, after the "ConvBNRelu" function in stage 1, i.e., convolution with a kernel size of 3 and a stride of 2, batch normalization, and Relu activation function, an output feature map of size 32×512×256 (C×W×H) is obtained. Then, after the "ConvBNRelu" function in stage 2, i.e., convolution with a kernel size of 3 and a stride of 2, batch normalization, and Relu activation function, an output feature map of size 64×256×128 (C×W×H) is obtained. After the convolution operation in stage 3, the "ConvBNRelu" function (i.e., convolution with a kernel size of 3 and a stride of 1), batch normalization, Relu activation function, and the "Conv2D" function (i.e., convolution with a kernel size of 1 and a stride of 2) and batch normalization are combined to obtain a feature map of size 256 × 128 × 64 (C × W × H). This feature map is used as the input of the segmentation head module. After the segmentation head module, the output will serve as the spatial information input of the feature fusion module. In this invention, this feature map is defined as feature map A. At the same time, the feature map is further downsampled through stage four, and a feature map of size 512×64×32 (C×W×H) is obtained through the joint action of the "ConvBNRelu" function, i.e., convolution with a kernel size of 3 and a step length of 1, batch normalization, Relu activation function, and "Conv2D" function, i.e., convolution with a kernel size of 1 and a step length of 2, and batch normalization. The feature map is then passed through the attention optimization module to output features of size 128×64×32 (C×W×H) as part of the context information, which is defined as feature map B in the present invention. Next, the 512 × 64 × 32 (C × W × H) feature map is further downsampled, and a feature map of size 1024 × 32 × 16 (C × W × H) is obtained through the combined effects of the “ConvBNRelu” function in stage five, i.e., convolution with a kernel size of 3 and a stride of 1, batch normalization, Relu activation function, and the “Conv2D” function, i.e., convolution with a kernel size of 1 and a stride of 2, and batch normalization. The feature map is passed through the pyramid pooling module to output a feature map of size 1024×32×16 (C×W×H), which is then input into the object attention module and outputs a feature map of size 1024×32×16 (C×W×H). The feature map is then downsampled by the average pooling operation, and the number of feature channels is changed to 128 by the "ConvBNRelu" operation. The feature map is upsampled using bilinear interpolation to obtain a feature map of size 128×32×16 (C×W×H), which is defined as feature map C.At the same time, the features output from stage 5 are processed through the attention optimization module to produce a feature map of size 128×32×16 (C×W×H), defined as feature map D. Feature maps C and D are superimposed (Add) to obtain a feature map of size 128×32×16 (C×W×H). This is then upsampled to obtain a feature map of size 128×64×32 (C×W×H), defined as feature map E. Feature maps B and E are then superimposed (Add) to obtain a feature map of size 128×64×32 (C×W×H). After upsampling, a feature map of size 128×128×64 (C×W×H) is obtained, defined as feature map F. Feature map F provides contextual information to the feature fusion module, while feature map A provides spatial information. The feature fusion module outputs a feature map of size 256×128×64 (C×W×H). Finally, it is upsampled by 4× to produce the final result P. d ∈R H×W , the present invention will eventually output P d ∈R H×W and the truth value g d ∈R H×W Calculate Loss.

[0035] The overall Loss function of the present invention is a detail loss function, which is composed of a binary cross-entropy loss function and a dice loss function.

[0036] The dice loss function expression is:

[0037]

[0038] Where i represents the i-th pixel, ∈ is the Laplace smoothing term. In the present invention, ∈ exists to prevent the denominator from being 0, ∈=1.

[0039] The binary cross entropy loss function is expressed as:

[0040]

[0041] Among them, P d Indicates the final output, g d Represents the groundtruth, b represents the bth sample, and N represents the number of N samples.

[0042] Training settings:

[0043] The number of training iterations was set to 320,000, with a batch size of 12 trained on one sample at a time. Mini-batch stochastic gradient descent optimization was used with momentum of 0.9 and weight decay of 4.0e-5. The number of training set samples (data loader) was equal to the number of images in the training set of the corresponding dataset, which was 700 for the track training set and 2975 for the cityscape dataset. An exponentially adjusted (Poly) learning rate strategy was used with a power of 0.9 and an initial learning rate of 0.01. A warmup strategy was used for the first 1,000 iterations. Data augmentation included color jittering, random horizontal flipping, random cropping, and random resizing. The crop resolution was 1024×512 for training the cityscape dataset. For training the track segmentation dataset, the crop resolution was 480×320. All experiments were conducted using paddlepaddle-gpu 2.3.0. We conducted all experiments on an NVIDIA RTX 2080Ti GPU under CUDA 10.1, CUDNN 7.6.5, and TensorRT 7.1.3.4 to test the computational capabilities of our method.

[0044] The present invention only requires inputting marked labels and original images for training. After evaluation, the obtained training model can achieve good results in both accuracy and speed, achieving a good balance between accuracy and speed.

[0045] Table 1. Stage 1 & Stage 2 of the feature extraction network

[0046]

[0047]

[0048] Table 2 Stage 3 of the feature extraction network

[0049] name ConvBNRelu Conv2D Input Channel 64 64 Output Channel 64 256 Kernel size 3×3 1×1 step length 1 2 filling 1 1 Does it contain BN? Y Y Activation Function Relu N

[0050] Table 3 Stage 4 of the feature extraction network

[0051] name ConvBNRelu Conv2D Input Channel 256 256 Output Channel 256 512 Kernel size 3×3 1×1 step length 1 2 filling 1 1 Does it contain BN? Y Y Activation Function Relu N

[0052] Table 4 Stage 5 of the feature extraction network

[0053]

[0054]

Claims

1. An efficient semantic segmentation method for rail transit, characterized by: Including dataset establishment and efficient image segmentation network design; Dataset establishment: A mixture of track segmentation dataset and urban landscape segmentation dataset is used; Efficient image segmentation network: The network includes a feature extraction module, a pyramid pooling module, an attention optimization module, an object attention module, a feature fusion module, and a segmentation head module; By training the network, the track and background are accurately segmented, as well as the urban road and background are segmented; The specific steps for making the dataset are as follows: For the collected rail transit operation video data, each image is divided into four categories: track class, hazard class, and background class, where the background class label is 0, the track class label is 1, and the hazard class label is 2. The public urban landscape dataset is used as an extension. The efficient image segmentation network consists of the following modules, including feature extraction module, pyramid pooling module, object attention module, attention optimization module, feature fusion module, and segmentation head module; The initial size of the image input to the network is 3×1024×512. After passing through the feature extraction network, feature maps of different sizes are obtained for subsequent processing. First, after the "ConvBNRelu" function in stage one, that is, the convolution kernel size is 3 and the step length is 2, as well as batch normalization and Relu activation function, the output size of the feature map is 32×512×256. Then, after the "ConvBNRelu" function in stage two, that is, the convolution kernel size is 3 and the step length is 2, as well as batch normalization and Relu activation function, the output size of the feature map is 64×256×128. Then, after the "ConvBNRelu" function in stage three, the output size of the feature map is 64×256×128. The "function, i.e., convolution with kernel size 3 and stride 1, batch normalization, Relu activation function and "Conv2D" function, i.e., convolution with kernel size 1 and stride 2, and batch normalization, is used to obtain a feature map of size 256×128×64; the feature is used as the input of the segmentation head module, and the output after the segmentation head module is used as the spatial information input of the feature fusion module, and the feature map is defined as feature map A; at the same time, the feature map is further downsampled through stage 4, and the "ConvBNRelu" function, i.e., convolution with kernel size 3 and stride 1, batch normalization, Relu activation function and "Conv2D" is used to obtain the ... "The function, that is, the convolution with a kernel size of 1 and a stride of 2, and batch normalization are used together to obtain a feature map of size 512×64×32. Then, the feature map is passed through the attention optimization module to output a feature of size 128×64×32 as part of the context information, which is defined as feature map B. Next, the 512×64×32 feature map is further downsampled and the "ConvBNRelu" function in stage five, that is, the convolution with a kernel size of 3 and a stride of 1, and batch normalization, Relu activation function, and the "Conv2D" function, that is, the convolution with a kernel size of 1 and a stride of 2, and batch normalization are used together to obtain a feature map of size 10 24×32×16 feature map; after passing the feature map through the pyramid pooling module, the output is a feature map of size 1024×32×16, which is then input into the object attention module, and then the output is a feature map of size 1024×32×16. The feature map is then downsampled through the average pooling operation, and the number of feature channels is changed to 128 through the "ConvBNRelu" operation. The feature map is upsampled using bilinear interpolation to obtain a feature map of size 128×32×16, which is defined as feature map C; at the same time, the features output from stage 5 are passed through the attention optimization module to output a feature map of size 128×32×16, which is defined as feature map D;The feature map C and the feature map D are superimposed to obtain a feature map of size 128×32×16. Then, a feature map of size 128×64×32 is obtained by upsampling, which is defined as feature map E. The feature map B and the feature map E are superimposed to obtain a feature map of size 128×64×32. After upsampling, a feature map of size 128×128×64 is obtained, which is defined as feature map F. Feature map F provides context information to the feature fusion module as input, while feature map A provides spatial information to the feature fusion module as input. After the feature fusion module is used, the feature map with a size of 256×128×64 is output, and the final result P is output after 4× upsampling. d ∈R H×W , the final output P d ∈R H×W and the truth value g d ∈R H×W Calculate Loss; The total loss function is the detail loss function, which is composed of the binary cross entropy loss function and the diceloss function; The dice loss function expression is: Where i represents the i-th pixel, ∈ is the Laplace smoothing term; ∈ exists to prevent the denominator from being 0; The binary cross entropy loss function is expressed as: Among them, P d Indicates the final output, g d Represents the true value, b represents the bth sample, and N represents N samples; Training settings: The number of training iterations is set to 320,000, and one sample is trained each time, that is, Batch Size = 12. The small batch stochastic gradient descent optimization algorithm with momentum of 0.9 and weight decay of 4.0e-5 is used.

Images