Real-time semantic segmentation method for complex urban street scenes

Abstract

This invention discloses a real-time semantic segmentation method for complex urban street scenes, comprising: acquiring images of complex urban street scenes; creating a training set and labeling it to obtain a training set label file; building an adaptive attention module; building a real-time semantic segmentation network based on adaptive attention; training the real-time semantic segmentation network and obtaining network weight values; creating a test set and testing the real-time semantic segmentation network using the network weight values ​​to obtain semantic segmentation results for complex urban street scenes. This invention offers high segmentation accuracy and low computational burden, achieving understanding of complex urban street scenes and solving the problem of low real-time performance in traditional segmentation methods. It requires only relatively small computational resources to meet the real-time performance requirements of resource-constrained platforms such as autonomous vehicles and mobile robots in practical applications, thereby helping mobile devices understand their surrounding environment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence computer vision technology, specifically involving a real-time semantic segmentation method for complex urban street scenes based on adaptive attention. Background Technology

[0002] In recent years, convolutional neural networks (CNNs) have become the main method for solving computer vision tasks. Their goal is to assign a corresponding semantic category to each pixel in an image, and they are widely used in fields such as autonomous vehicles, robot vision, and augmented reality. However, mobile devices such as mobile robots and autonomous vehicles are limited by energy consumption, memory, and computing power, often requiring networks with low computational burden and high segmentation accuracy for real-time online estimation and decision-making. While networks like DeepLab have achieved good semantic segmentation results, their computational burden is too high to meet the real-time requirements of these mobile devices for understanding their surroundings.

[0003] To address these issues, lightweight convolutional neural networks have been developed, utilizing network compression and convolution decomposition to improve the real-time performance of semantic segmentation networks. Network compression includes hashing, pruning, and quantization. Wen et al. used hashing to achieve random weight sharing to reduce network parameters; Han et al. used pruning to remove training redundancy and simplify the network to improve inference latency; Wu et al. used quantization to develop a smaller number of bits to encode network parameters, improving the network's computational speed. Convolution decomposition, such as in ShuffleNet and MobileNet, significantly improves network computational speed and segmentation accuracy by utilizing grouped convolutions and depthwise separable convolutions; ERFNet reduces network parameters by decomposing a 3×3 convolution into 1×3 and 3×1 convolutions. Other lightweight semantic segmentation networks focus on designing a compact network to reduce computational cost and memory consumption. Liu et al. further eliminated redundancy using sparse coding theory, with sparse weights and inference computation involving only a small number of parameters; ICNet gradually restored and refined the segmentation results with low computational cost by designing an image concatenation network; ESPNet used an encoder-decoder structure to restore object details, maintaining segmentation accuracy while reducing the number of parameters; Huang et al. proposed a real-time semantic segmentation network with block adaptive feature fusion, achieving a certain balance between the speed and accuracy of semantic segmentation; Wang et al. introduced dilated separable convolution for semantic segmentation, which had a low network computational burden but the segmentation effect was not ideal.

[0004] Attention mechanisms aim to highlight regions that require attention. Zhao et al. applied attention mechanisms to semantic segmentation and proposed the Point Spatial Attention Network (PSANet), which adaptively aggregates semantic information, fully demonstrating the effectiveness of attention mechanisms. Fu et al. proposed a dual attention mechanism, which models semantic dependencies from spatial and channel dimensions, helping to obtain more accurate semantic segmentation results.

[0005] Mohammed et al. improved the dual attention mechanism by introducing dilated convolution in the spatial dimension to increase the receptive field. Although they achieved better semantic segmentation results, they also brought a large computational burden.

[0006] Based on the above analysis, although many networks have made preliminary progress in designing real-time semantic segmentation, pursuing optimal segmentation accuracy within a very limited computational budget remains an open research problem in image semantic segmentation in computer vision. This invention studies a real-time semantic segmentation method for complex urban street scenes, achieving high segmentation accuracy with low computational burden. This method can meet the real-time requirements of resource-constrained platforms such as autonomous vehicles and mobile robots, helping them to analyze their surrounding environment. Summary of the Invention

[0007] The purpose of this invention is to provide a real-time semantic segmentation method for complex urban street scenes, which solves the problem that existing methods cannot meet the real-time requirements of mobile devices for understanding their surrounding environment.

[0008] The technical solution adopted in this invention is a real-time semantic segmentation method for complex urban street scenes, comprising the following steps:

[0009] Step 1: Acquire videos of complex city street scenes and process them into images;

[0010] Step 2: Use the images obtained in Step 1 to create a training set for complex city street scenes;

[0011] Step 3: Label the training set obtained in Step 2 to obtain the training set label file;

[0012] Step 4: Build an adaptive attention module, including a spatial attention module and a channel attention module;

[0013] Step 5: Based on the adaptive attention module obtained in Step 4, build a real-time semantic segmentation network based on adaptive attention;

[0014] Step 6: Use the training set obtained in Step 2 and the label file obtained in Step 3 to train the real-time semantic segmentation network obtained in Step 5 to obtain the network weight values.

[0015] Step 7: Create a test set from the image obtained in Step 1, and use the network weight values ​​obtained in Step 6 to test the real-time semantic segmentation network obtained in Step 5 to obtain the semantic segmentation results of complex urban street scenes.

[0016] The invention is further characterized in that,

[0017] The training set in step 2 consists of images in ".png" format, including images under two different lighting conditions: dusk and bright sunshine, totaling 367 training images and 101 validation images.

[0018] In step 3, the training set label files were labeled manually pixel by pixel using the InteractLabeler tool, and the label format was ".png".

[0019] The adaptive attention module built in step 4 is as follows:

[0020] The spatial attention module first performs 1×1 convolution to reduce the dimensionality of the input feature map, and then performs two parallel adaptive max pooling operations. The output results are fused by multiplication to obtain the attention map, as shown in Equation (1). The output feature map of the spatial attention module is shown in Equation (2).

[0021]

[0022]

[0023] In the formula, Amap represents the attention graph, amp represents adaptive max pooling, conv represents 1×1 convolution, and F in SA represents the input feature map, and SA represents the output feature map of the spatial attention module.

[0024] The channel attention module uses adaptive max pooling and adaptive average pooling to obtain features at different scales, and then reshapes and expands them to obtain the output map of the channel attention module, which is represented by equation (3):

[0025]

[0026] In the formula, CA represents the output feature map of the channel attention module, avp represents adaptive average pooling, and F in Indicates the input feature map;

[0027] Finally, the spatial attention features and channel attention features are fused by addition, and the final output feature map of the adaptive attention module is shown in Equation (4):

[0028]

[0029] In the formula, F out Output feature maps for the adaptive attention module.

[0030] The real-time semantic segmentation network in step 5 consists of two branches: a spatial feature extraction network and a semantic feature extraction network, which are fused together by a feature fusion network; among them,

[0031] The spatial feature extraction network aims to obtain the location information of pixels in the segmentation category and extract spatial detail information from high-resolution images. The spatial feature extraction network is divided into three identical stages, each consisting of two 3×3 convolutional layers with different strides. In each stage, the stride of the first layer in the two convolutional layers is 2 and the stride of the second layer is 1. Each convolutional layer has batch normalization and parameter correction linear units.

[0032] The semantic feature extraction network is based on the FDSS-nbt module and aims to obtain semantic information of category pixels and extract semantic features from low-resolution images. The semantic feature extraction network consists of four steps. Steps I and II use convolutional layers and pooling layers to extract and filter features from the input image. Steps III and IV introduce an asymmetric split-bottleneck module based on downsampling and use dilated convolutions with dilation rates of [1, 3, 6, 12] and [3, 6, 12, 24] to extract multi-scale image features.

[0033] Step 6, the training process, uses the Adam optimizer with an initial learning rate set to 1e. -3 The batch size was set to 4, the maximum training epochs were 1000, and data augmentation techniques were used: mean subtraction, horizontal flipping, and random scaling of image sizes during training, with random scaling values ​​of 0.75, 1.0, 1.25, 1.5, 1.75, and 2.0. A learning rate decay strategy and a cross-entropy loss function were used to compensate for classes with a small percentage of pixels in the dataset. The learning rate was expressed by equation (5), and the loss was calculated by equation (6).

[0034]

[0035] In the formula, lr i The initial learning rate is set to `cur_iter`, the current iteration number is set to `max_iter`, the maximum iteration number is set to `max_iter`, and `power` is the learning rate decay parameter, which is set to 0.9.

[0036]

[0037] In the formula, L t It is the total loss, L f L represents the loss of the output layer. i Let i represent the loss at stage i after applying attention and feature refinement, where i = 1, 2, 3, ..., n;

[0038] The training set was processed using the class weight calculation method in ENet, and the network weight values ​​were obtained as shown in equation (7):

[0039]

[0040] In the formula, i represents the category number, H i This indicates the proportion of category pixels to the total number of image pixels, and V represents the hyperparameter with a value of 1.10.

[0041] The test set in step 7 contains images under two different lighting conditions: dusk and bright sunshine, totaling 233 test images.

[0042] The beneficial effects of this invention are: the real-time semantic segmentation method for complex urban street scenes of this invention has high segmentation accuracy and low computational burden, realizing the understanding of complex urban street scenes and solving the problem of low real-time performance of traditional segmentation methods. It requires only relatively small computational resources to meet the real-time performance requirements of resource-constrained platforms such as autonomous vehicles and mobile robots in practical applications, thereby helping mobile devices to understand their surrounding environment. Attached Figure Description

[0043] Figure 1 This is an architecture diagram of the Adaptive Attention Module (AAM) built in the real-time semantic segmentation method for complex urban street scenes of the present invention;

[0044] Figure 2 This is a diagram of the architecture of the real-time semantic segmentation network (AANet) based on adaptive attention, built in the real-time semantic segmentation method for complex urban street scenes of the present invention.

[0045] Figure 3 This is a graph showing the training loss and validation loss curves of the AANet network built in the real-time semantic segmentation method for complex urban street scenes of the present invention during the training process.

[0046] Figure 4 This is a segmentation result image of the AANet network built in the real-time semantic segmentation method for complex urban street scenes of the present invention on a test dataset.

[0047] Figure 5 This is a comparison of the segmentation results of the AANet network built in the real-time semantic segmentation method for complex urban street scenes of this invention with other classic semantic segmentation networks on the test dataset. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0049] Example 1

[0050] This invention provides a real-time semantic segmentation method for complex urban street scenes, comprising the following steps:

[0051] Step 1: Use the vehicle-mounted camera to capture video of complex urban street scenes and process it into images;

[0052] Step 2: Use the images obtained in Step 1 to create a training set for complex city street scenes;

[0053] Step 3: Label the training dataset obtained in Step 2 to obtain the training set label file;

[0054] Step 4: Build an Adaptive Attention Module (AAM), including a Spatial Attention Module (SAM) and a Channel Attention Module (CAM), to enhance pixel prediction at each stage of the semantic segmentation path;

[0055] Step 5: Based on the adaptive attention module obtained in Step 4, build a real-time semantic segmentation network based on adaptive attention (AANet, Adaptive Attention Network);

[0056] Step 6: Use the training set obtained in Step 2 and the label file obtained in Step 3 to train the AANet designed in Step 5 to obtain the network weight values;

[0057] Step 7: Create a test set from the images acquired in Step 1, and use the weight values ​​obtained in Step 6 to test the AANet obtained in Step 5 to obtain the semantic segmentation results of the complex urban street scene.

[0058] Through the above methods, the real-time semantic segmentation method for complex urban street scenes of the present invention achieves high segmentation accuracy and low computational burden, realizing the understanding of complex urban street scenes and solving the problem of low real-time performance of traditional segmentation methods. It requires only relatively small computational resources to meet the real-time performance requirements of resource-constrained platforms such as autonomous vehicles and mobile robots in practical applications, thereby helping mobile devices to understand their surrounding environment.

[0059] Example 2

[0060] The preferred steps of the real-time semantic segmentation method for complex urban street scenes provided in Example 1 are as follows:

[0061] The images in step 1 are from a 22-minute and 14-second video, capturing 480×360 pixel frames at a speed of 30 frames / s, for a total of 701 frames. The complexity is reflected in the large number of semantic categories involved in the images, the different scales of the segmentation targets, and the different lighting conditions of the images.

[0062] In step 2, the training set images are all in ".png" format and include images under two different lighting conditions: dusk and bright sunshine, totaling 367 training images and 101 validation images.

[0063] In step 3, the training set labels were manually annotated pixel by pixel using the InteractLabeler tool, and the label format was also ".png". There are a total of 11 semantic labels.

[0064] In step 4, the Adaptive Attention Module (AAM) is divided into two branches: Spatial Attention Module (SAM) and Channel Attention Module (CAM). The Spatial Attention Module branch uses adaptive max pooling to preserve the main features of the image, making the network focus more on semantically relevant local regions related to the segmentation category, rather than the entire image, thereby enhancing the feature map's ability to represent the spatial location of key features. The Channel Attention Module branch models the correlation between each feature channel to obtain the inter-channel dependencies. Channel attention fuses the features of adaptive max pooling and adaptive average pooling through multiplication, making the network focus more on global channel dependencies and maximally compensating for the shortcomings of convolution in obtaining channel dependencies. The specific design of the real-time semantic segmentation network (AANet) based on adaptive attention built in this invention is as follows, and the architecture of the Adaptive Attention Module (AAM) is as follows: Figure 1 As shown, the AANet architecture is as follows: Figure 2 As shown. The specific implementation is as follows:

[0065] The Spatial Attention Module (SAM) first performs 1×1 convolution to reduce the dimension of the input feature map, and then performs two parallel adaptive max pooling operations. The output results are fused by multiplication to obtain the attention map, as shown in Equation (1). The output feature map of the Spatial Attention Module is shown in Equation (2).

[0066]

[0067]

[0068] Where Amap represents the attention map, amp (adaptive max pooling) represents adaptive max pooling, conv represents a 1×1 convolution, and F in SA represents the input feature map, and SA represents the output feature map of the spatial attention module.

[0069] The Channel Attention Module (CAM) uses adaptive max pooling and adaptive average pooling to obtain features at different scales, and then reshapes and expands them to obtain the output map of the module, which can be represented by Equation (3).

[0070]

[0071] Where CA represents the output feature map of the channel attention module, avp (adaptive average pooling) represents adaptive average pooling, and F in This represents the input feature map.

[0072] Finally, the spatial attention features and channel attention features are fused by addition to obtain the output feature map of the adaptive attention module as shown in Equation (4).

[0073]

[0074] Among them, F out Output feature maps for the adaptive attention module.

[0075] In step 5, the real-time semantic segmentation network based on adaptive attention is divided into two branches: Spatial Feature Extraction Network (SFEN) and Semantic Feature Extraction Network (SeFEN). These branches acquire spatial detail information and semantic information from the input image, respectively. Finally, a Feature Fusion Network (FFN) is used to fuse the spatial detail information and semantic information to obtain the final segmentation result.

[0076] Spatial feature extraction networks aim to obtain the location information of pixels for segmentation categories, and therefore consist of fewer convolutional layers. They extract spatial detail information from high-resolution images, and are implemented as follows:

[0077] The Spatial Feature Extraction Network (SFEN) consists of three identical stages, totaling six convolutional layers. Each stage comprises two 3×3 convolutional layers with different strides (stride = 2 in the first layer and stride = 1 in the second layer). These convolutional layers also incorporate batch normalization (BN) and parametrically corrected linear units (PreLU) to accelerate network convergence. The structural parameters of the Spatial Feature Extraction Network are shown in Table 1.

[0078] Table 1. Network structure parameters for spatial feature extraction

[0079]

[0080]

[0081] The semantic feature extraction network is based on the FDSS-nbt module and aims to obtain semantic information of category pixels. Therefore, it consists of a large number of convolutional layers to extract semantic features from low-resolution images. The specific implementation is as follows:

[0082] The Semantic Feature Extraction Network (SeFEN) is designed based on the Asymmetric-Split-Bottledown (FDSS-nbt) module, utilizing channel segmentation and shuffling operations to enhance information exchange within semantic feature channels. The network consists of four steps. Steps I and II use convolutional and pooling layers to extract and filter features from the input image. Steps III and IV, based on downsampling, introduce the Asymmetric-Split-Bottledown module, employing atrous convolutions with dilation rates of [1, 3, 6, 12] and [3, 6, 12, 24] to extract multi-scale image features. The structural parameters of the Semantic Feature Extraction Network are shown in Table 2.

[0083] Table 2. Semantic Feature Extraction Network Structure Parameters

[0084]

[0085] The specific implementation details of the training experiment in step 6 are as follows:

[0086] The graphics card used in the experiment was an NVIDIA GeForce GTX 1660Ti with 6GB of video memory. The deep learning framework was PyTorch 1.0. CUDA 10.2 provides a general-purpose parallel computing platform and programming model. The cuDNNV8 platform provides an acceleration library for convolutional neural networks. The CPU model was an Intel(R) Core(TM) i7-10700F. The experimental memory was 32GB.

[0087] The training process uses the Adam optimizer, with an initial learning rate set to 1e. -3 The batch size was set to 4, the maximum training epochs were 1000, and data augmentation techniques were used: mean subtraction, horizontal flipping, and random scaling of image sizes during training, with random scaling values ​​of 0.75, 1.0, 1.25, 1.5, 1.75, and 2.0. A learning rate decay strategy and a cross-entropy loss function were used to compensate for categories with small pixel proportions in the dataset. The learning rate can be represented by equation (5), and the loss calculation is represented by equation (6). The loss function curve during network training is shown in the figure. Figure 3 As shown.

[0088]

[0089] Among them, lr iis the initial learning rate, cur_iter is the current iteration number, max_iter is the maximum iteration number, and power is the learning rate decay parameter, which is set to 0.9 in this invention with reference to CGNet.

[0090]

[0091] Among them, L t It is the total loss, L f L represents the loss of the output layer. i This represents the loss in stage i after applying attention and feature refinement.

[0092] Furthermore, to address the problem of imbalanced pixel distribution across categories, this invention employs the category weight calculation method from ENet (efficient neural network) to process the dataset. The category weight calculation is shown in equation (7):

[0093]

[0094] In the formula, i represents the category number, H i V represents the proportion of category pixels to image pixels, and V represents the hyperparameter, which is set to 1.10 in the semantic segmentation experiment of this invention.

[0095] The test dataset in step 7 was also collected under conditions of dusk and sunshine, consisting of 233 images.

[0096] Example 3

[0097] The real-time semantic segmentation method for complex urban street scenes of the present invention is implemented according to the following steps:

[0098] Step 1: Use the vehicle-mounted camera to capture video of complex urban street scenes and process it into images;

[0099] Step 2: Use the images obtained in Step 1 to create a training set for complex city street scenes;

[0100] Step 3: Label the training dataset obtained in Step 2 to obtain the training set label file;

[0101] Step 4: Construct an Adaptive Attention Module (AAM), including a Spatial Attention Module (SAM) and a Channel Attention Module (CAM), to enhance pixel prediction at each stage of the semantic segmentation path, such as... Figure 1As shown, the spatial attention module branch uses adaptive max pooling to preserve the main features of the image, making the network focus more on semantically related local regions of the segmentation category, rather than the entire image, thereby enhancing the feature map's ability to represent the spatial location of key features. The channel attention module branch obtains the inter-channel dependencies by modeling the correlation between each feature channel. Channel attention fuses the features of adaptive max pooling and adaptive average pooling through multiplication, making the network focus more on global channel dependencies and maximally compensating for the lack of channel dependency acquisition in convolution.

[0102] Step 5: Based on the adaptive attention module obtained in Step 4, construct a real-time semantic segmentation network based on adaptive attention (AANet, Adaptive Attention Network), such as... Figure 2 As shown, the Spatial Feature Extraction Network (SFEN) consists of shallower network layers, extracting high-resolution spatial detail information; the Semantic Feature Extraction Network (SeFEN) consists of deeper network layers, extracting low-resolution semantic information; the Feature Fusion Network (FFN) uses a concat fusion method to combine high-resolution spatial detail information and low-resolution semantic detail information to obtain the final segmentation result; the Attention Feature Refinement Module (AFRM) uses a bilinear interpolation upsampling method to highlight useful features in the channel and spatial dimensions, and uses feature interpolation to reconstruct the feature map pixel by pixel, improving the segmentation accuracy of small-class objects.

[0103] To verify the effectiveness of the constructed adaptive attention module (AAM), an ablation study was conducted on it, using the spatial feature extraction network (SFEN) and semantic feature extraction network (SeFEN) as baselines. The ablation experiment results are shown in Table 3. AANet is the real-time semantic segmentation network based on adaptive attention constructed in this invention.

[0104] Table 3 Ablation Experiment Results of Adaptive Attention Module

[0105]

[0106] The ablation experiment data in Table 3 fully demonstrates the effectiveness of the adaptive attention module proposed in this invention. The network containing the adaptive attention module has 10.5% more parameters, 28.8% fewer frames per second, and 7.4% more floating-point operations than the baseline network, but achieves a 1.9% improvement in average segmentation accuracy. High segmentation accuracy is achieved with minimal sacrifice.

[0107] Step 6: Train the AANet constructed in Step 5 using the training set obtained in Step 2 and the label file obtained in Step 3 to obtain the network weight values. The training set loss and validation set loss during training are as follows: Figure 3As shown in the figure, `loss` represents the loss value, `epoch` represents the training epoch, the training loss curve is a curve formed by connecting squares, and the validation loss curve is a curve formed by connecting triangles. Initially, a larger learning rate is used to train the network. Figure 3 As can be seen, the training loss and validation loss values ​​tend to converge to the optimal values ​​relatively quickly. After 100 rounds, the rate of decrease in the training loss value becomes smaller. Subsequently, the learning rate is gradually reduced, and the cross-entropy loss function is used to continue training. After 1000 rounds of training, the loss reaches the optimal convergence value.

[0108] Step 7: Create a test set from the images acquired in Step 1. Use the weights obtained in Step 6 to test the AANet obtained in Step 5, and obtain the semantic segmentation results for the complex urban street scene as follows: Figure 4 As shown, GT (Ground Truth) represents the true label value. The segmentation results demonstrate that the AANet network constructed in this invention achieves good overall segmentation performance. Even with diverse image segmentation categories, irregular shapes, and varying lighting intensities in the dataset, the network can still provide correct segmentation results.

[0109] To verify the superiority of the AANet constructed in this invention, AANet was compared with other classic semantic segmentation networks in four aspects: average segmentation accuracy (mIoU), number of parameters, frames per second (FPS), and number of floating-point operations (FLOPs). The comparison results of the segmentation accuracy (IoU) for a single class with the average segmentation accuracy (mIoU) for all classes are shown in Table 4, and the comparison results of computational burden are shown in Table 5. The segmentation results are as follows: Figure 5 As shown in the figure, different rows represent different network segmentation effects, and the last column is a magnified view of the segmentation result of the person riding a bicycle in the fourth column box.

[0110] Table 4 Comparison of IoU (%) and mIoU (%) for each network category

[0111]

[0112] As can be seen from the data in Table 4, the AANet network achieves the highest average segmentation accuracy compared to other networks, and achieves higher segmentation accuracy than previous networks in individual categories such as building, road, sidewalk, traffic sign, and car.

[0113] Table 5 Comparison of network parameter count, inference speed, and computational complexity

[0114]

[0115] As can be seen from the data in Table 5, the AANet network of this invention has 2.42M parameters, which is a medium-sized network compared to small networks such as ENet and CGNet, and large networks such as SegNet and UNet. The inference speed of the network of this invention is 52 frames / second, which is lower than the inference speed of small networks such as ENet, but higher than the inference speed of large networks such as SegNet. The network of this invention has a relatively small computational complexity, with 9.98G floating-point operations.

[0116] Figure 5 In the image, different rows represent different network segmentation results, and the last column is a magnified view of the segmentation result for the person riding a bicycle in the fourth column. From Figure 5 It can be seen that the network of this invention has the best overall segmentation effect compared with other networks, and performs well in detail. For example, the edge contour segmentation of the person riding a bicycle in the black box in the first column on the left is more detailed than that of other networks. The edge refinement effect of semantic segmentation between categories is better than other networks. For example, in the second column on the left, the edge segmentation effect of the network of this invention is the best for the fence in the black box, and the edge boundary between categories is neatly segmented. Secondly, the segmentation effect of some small-scale objects is significantly improved compared with other semantic segmentation networks. For example, in the segmentation results in the third and fourth columns on the left, the small cylindrical pillar on the roadside in the black box and the person riding a bicycle in the distance (as shown in the enlarged image next to it) have a small pixel ratio. Other networks except the network of this invention failed to segment them. It can be seen that the network of this invention greatly reduces the omission rate of small-scale objects.

Claims

1. A real-time semantic segmentation method for complex urban street scenes, characterized in that, Includes the following steps: Step 1: Acquire videos of complex city street scenes and process them into images; Step 2: Use the images obtained in Step 1 to create a training set for complex city street scenes; Step 3: Label the training set obtained in Step 2 to obtain the training set label file; Step 4: Build the adaptive attention module, including the spatial attention module and the channel attention module, specifically as follows: The spatial attention module first performs 1×1 convolution to reduce the dimensionality of the input feature map, and then performs two parallel adaptive max pooling operations. The output results are fused by multiplication to obtain the attention map, as shown in Equation (1). The output feature map of the spatial attention module is shown in Equation (2). （1） （2） In the formula, Indicate attention to the diagram. This indicates adaptive max pooling. Represents a 1×1 convolution. Indicates the input feature map, The spatial attention module outputs a feature map; The channel attention module uses adaptive max pooling and adaptive average pooling to obtain features at different scales, and then reshapes and expands them to obtain the output map of the channel attention module, which is represented by equation (3): （3） In the formula, This indicates the feature map output by the channel attention module. Indicates adaptive average pooling. Indicates the input feature map; Finally, the spatial attention features and channel attention features are fused by addition, and the final output feature map of the adaptive attention module is shown in Equation (4): （4） In the formula, Output feature maps for the adaptive attention module; Step 5: Based on the adaptive attention module obtained in Step 4, construct a real-time semantic segmentation network based on adaptive attention, consisting of two branches: a spatial feature extraction network and a semantic feature extraction network, which are fused through a feature fusion network; wherein, The spatial feature extraction network aims to obtain the location information of pixels in the segmentation category and extract spatial detail information from high-resolution images. The spatial feature extraction network is divided into three identical stages, each consisting of two 3×3 convolutional layers with different strides. In each stage, the stride of the first layer in the two convolutional layers is 2 and the stride of the second layer is 1. Each convolutional layer has batch normalization and parameter correction linear units. The semantic feature extraction network is based on the FDSS-nbt module and aims to obtain semantic information of category pixels and extract semantic features from low-resolution images. The semantic feature extraction network consists of four steps. Steps I and II use convolutional layers and pooling layers to extract and filter features from the input image. Steps III and IV introduce an asymmetric-split-bottleneck module based on downsampling and use dilated convolutions with dilation rates of [1, 3, 6, 12] and [3, 6, 12, 24] to extract multi-scale image features. Step 6: Use the training set obtained in Step 2 and the label file obtained in Step 3 to train the real-time semantic segmentation network obtained in Step 5 to obtain the network weight values. Step 7: Create a test set from the image obtained in Step 1, and use the network weight values ​​obtained in Step 6 to test the real-time semantic segmentation network obtained in Step 5 to obtain the semantic segmentation results of complex urban street scenes.

2. The real-time semantic segmentation method for complex urban street scenes as described in claim 1, characterized in that, The training set in step 2 consists of images in ".png" format, including images under two different lighting conditions: dusk and bright sunshine, totaling 367 training images and 101 verification images.

3. The real-time semantic segmentation method for complex urban street scenes as described in claim 1, characterized in that, In step 3, the training set label files are labeled manually pixel by pixel using the InteractLabeler tool, and the label format is ".png".

4. The real-time semantic segmentation method for complex urban street scenes as described in claim 1, characterized in that, The training process in step 6 uses the Adam optimizer, with an initial learning rate set to 1e. -3 The batch size was set to 4, the maximum training epochs were 1000, and data augmentation techniques were used: mean subtraction, horizontal flipping, and random scaling of image sizes during training, with random scaling values ​​of 0.75, 1.0, 1.25, 1.5, 1.75, and 2.

0. A learning rate decay strategy and a cross-entropy loss function were used to compensate for categories with small pixel proportions in the dataset. The learning rate was expressed by equation (5), and the loss was calculated by equation (6). （5） In the formula, The initial learning rate, This represents the current iteration number. The maximum number of iterations, The learning rate decay parameter is set to 0.9; （6） In the formula, It is the total loss. Indicates the loss of the output layer. This indicates the first step after applying attention and feature refinement. Phase losses, ; The training set is processed using the class weight calculation method in ENet, and the network weight values ​​are shown in Equation (7): （7） In the formula, Indicates the category number. This indicates the proportion of category pixels to the total number of image pixels. This indicates a hyperparameter with a value of 1.

10.

5. The real-time semantic segmentation method for complex urban street scenes as described in claim 1, characterized in that, The test set in step 7 contains images under two different lighting conditions: dusk and bright sunshine, totaling 233 test images.

Images