A real-time semantic segmentation method based on an improved double-flow feature extraction network

Abstract

The application provides a real-time semantic segmentation method based on an improved double-flow feature extraction network, which constructs a real-time semantic segmentation network to perform semantic segmentation on image data to be processed. The real-time semantic segmentation network is a double-flow feature extraction network, and the feature maps at the input end of the double-flow feature extraction network are derived from three feature maps with different resolutions in a semantic path. In this way, repeated calculation can be avoided, the number of model parameters can be reduced, the relevance of feature information between paths can be deepened, and the network model has better multi-scale feature information fusion effect. By designing a spatial feature information enhancement module, the input feature information of the spatial path is changed, the optimization of the traditional double-flow feature extraction network is completed, sufficient spatial feature detail information is ensured to be restored for a prediction task, and meanwhile, the application technical scheme can be used for real-time segmentation in an experimental environment, so that the semantic segmentation algorithm can be more flexibly applied to different application scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a real-time semantic segmentation method based on an improved two-stream feature extraction network. Background Technology

[0002] In AI applications such as image recognition, target identification is often based on semantic segmentation results. In dynamic visual SLAM algorithms based on semantic segmentation, the overall performance of the semantic segmentation network directly impacts the real-time performance and stability of the visual SLAM algorithm. Traditional semantic segmentation algorithms utilize features such as grayscale values, colors, and geometric textures in the original image, employing manually designed segmentation methods to separate the target from the original image. Common manually designed segmentation methods include thresholding, region segmentation, and graph theory segmentation. However, traditional semantic segmentation algorithms require manual adjustment of algorithm parameters for different scenarios, resulting in poor robustness and difficulty in achieving the required segmentation accuracy and speed in practical applications.

[0003] In recent years, deep learning has been widely used in semantic segmentation, with implementations mainly falling into two categories: one is the TWO-Stage model based on region convolutional networks, primarily represented by the R-CNN series; the other is the ONE-Stage model based on fully convolutional neural networks, primarily represented by the YOLO series. Because the TWO-Stage model focuses primarily on segmentation accuracy, it is relatively complex and suffers from speed limitations, restricting its application scenarios. In scenarios requiring real-time computation, the ONE-Stage model has become the mainstream method for semantic segmentation tasks. However, while the ONE-Stage model exhibits good performance, its real-time semantic segmentation network still falls short in terms of segmentation accuracy. Summary of the Invention

[0004] To address the issue that existing ONE-Stage real-time semantic segmentation networks cannot simultaneously achieve both efficiency and segmentation accuracy, this invention provides a real-time semantic segmentation method based on an improved dual-stream feature extraction network. This method can improve segmentation efficiency and achieve real-time semantic segmentation while capturing richer feature information to enhance segmentation accuracy, ensuring greater flexibility for application scenarios.

[0005] The technical solution of this invention is as follows: a real-time semantic segmentation method based on an improved dual-stream feature extraction network, characterized by comprising the following steps:

[0006] S1: Construct a semantic path feature extraction network;

[0007] The semantic path feature extraction network includes: a convolutional processing module, a C3 module, a Concat module, an SPPF module, and an upsampling module;

[0008] Among them, one of the convolution processing modules and one of the C3 modules constitute a downsampling module, which performs downsampling operations on the input image feature data and outputs a feature map of a specified resolution.

[0009] The original input image is fed into a convolutional processing module for processing, and then into a downsampling module to output a first-level shallow feature map; the first-level shallow feature map is input into a downsampling module to output a second-level shallow feature map; the second-level shallow feature map is then input into a downsampling module to output a third-level shallow feature map.

[0010] The convolutional processing module, C3 module, Concat module, SPPF module, and upsampling module are used to perform subsequent processing on the first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map to obtain the output feature map of the semantic path feature extraction network;

[0011] S2: Construct a spatial feature enhancement module;

[0012] The spatial feature enhancement module is based on the RFB module framework, replacing the original 5×5 convolutional layer with a 3×3 convolutional layer. It also inherits from the HDC module by using cascaded multiple dilated convolutional layers with different dilation coefficients. Its specific processing flow includes the following steps:

[0013] a1: A 1×1 convolutional layer is used to process the original input feature map by increasing the number of channels, resulting in two types of output feature maps: Output Feature Maps Figure 1 and output features Figure 2 Output features Figure 1 and output features Figure 2 The number of channels is 1 / 4 of the number of channels in the original input feature map;

[0014] a2: The output features Figure 1 Processing with a 1×1 convolutional layer yields a set of output feature maps with 1 / 4 of the channels, denoted as: Output Feature Map. Figure 3 ;

[0015] a3: The output features Figure 2 The process utilizes consecutive convolutional layers with 3×3 kernels and dilation factors of 1, 2, and 3. The feature maps obtained after each convolution are retained, resulting in three sets of output feature maps with 1 / 4 of the channels, denoted as: Output Feature Maps. Figure 4 ;

[0016] a4: The output features Figure 3and the output features Figure 4 The four sets of feature maps are concatenated, and the resulting feature map is used as the output feature map of the spatial feature enhancement module.

[0017] S3: Construct a spatial path feature extraction network;

[0018] The spatial path feature extraction network includes: a spatial feature enhancement module, a Concat module, and an upsampling module;

[0019] The input to the spatial path feature extraction network is the first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map extracted by the semantic path feature extraction network;

[0020] The specific processing steps of the spatial path feature extraction network include the following steps:

[0021] b1: The third-level shallow feature map is fed into the upsampling module, and upsampling is completed through bilinear interpolation to obtain intermediate features. Figure 1 ;

[0022] b2: The secondary shallow feature map is processed by the spatial feature enhancement module to obtain intermediate features. Figure 2 ;

[0023] b3: The intermediate features Figure 1 and the intermediate features Figure 2 Together, they are fed into the Concat module for concatenation to obtain: intermediate features Figure 3 ;

[0024] b4: The intermediate features Figure 3 The data is fed into the upsampling module, where it is upsampled using bilinear interpolation to obtain intermediate features. Figure 4 ;

[0025] b5: Obtain intermediate features from the first-level shallow feature map through the spatial feature enhancement module. Figure 5 ;

[0026] b6: The intermediate feature Figure 4 and the intermediate features Figure 5 The features are all fed into the Concat module for concatenation, and the resulting feature map is used as the output feature map of the spatial path feature extraction network.

[0027] S4: Construct the feature fusion module;

[0028] The feature fusion module includes: a Concat module, an FFM module, a convolution processing module, and an upsampling module connected in sequence;

[0029] The feature maps output by the semantic path feature extraction network and the spatial path feature extraction network are used as inputs to the feature fusion module.

[0030] The two input feature maps first enter the Concat module for concat stitching, then the FFM module and CBS module are used to fuse the features of the two feature maps, and finally they are sent to the upsampling module. The feature map is upsampled through the Up operation, and the feature map size is enlarged to the original input image size before classification.

[0031] S5: Construct a real-time semantic segmentation network;

[0032] The real-time semantic segmentation network includes: a spatial path feature extraction network, a semantic path feature extraction network, and a feature fusion module. The spatial path feature extraction network is responsible for extracting shallow spatial feature information from the input image, the semantic path feature extraction network is responsible for extracting deep semantic feature information from the input image, and the feature fusion module is responsible for realizing the information fusion of deep semantic features and shallow spatial features.

[0033] The classified feature map output by the feature fusion module is used as the output result of the real-time semantic segmentation network.

[0034] S6: Train the real-time semantic segmentation network based on historical data to obtain the trained real-time semantic segmentation network;

[0035] S7: The image data to be processed is fed into the trained real-time semantic segmentation network as the original input image to perform semantic segmentation and obtain the segmented feature map.

[0036] Its further features are:

[0037] The first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map are feature maps of the original input image at 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution, respectively.

[0038] The subsequent processing of the first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map in the semantic path feature extraction network specifically includes:

[0039] The third-level shallow feature map is fed into the SPPF module for pooling, and then into the convolution module to obtain the fourth-level shallow feature map. The fourth-level shallow feature map is then fed into the upsampling module for upsampling, and then combined with the second-level shallow feature map. Figure 1 The first layer of features is fed into the Concat module for concatenation. The resulting feature maps are then processed sequentially by the C3 module and the convolutional processing module to obtain a fifth-level shallow feature map. This fifth-level shallow feature map is then processed by the upsampling module and combined with the first-level shallow feature map. Figure 1 The data is fed into the Concat module for concatenation to obtain a six-level shallow feature map.

[0040] The sixth-level shallow feature map is sent to the C3 module for processing to obtain the seventh-level feature map;

[0041] The seventh-level feature map is fed into a convolutional processing module and processed to obtain an eighth-level feature map.

[0042] The eighth-level feature map and the fifth-level shallow feature map Figure 1 The feature map obtained by splicing the data into the Concat module is then processed by the C3 module to obtain a nine-level feature map.

[0043] The ninth-level feature map is processed by the convolutional processing module and then concatenated with the fourth-level shallow feature map by the Concat module. The resulting feature map is then processed by the C3 module to obtain the tenth-level feature map.

[0044] The tenth-level feature map is sent to the upsampling module for processing and then compared with the ninth-level feature map. Figure 1 The data is fed into the Concat module for concatenation to obtain an eleven-level feature map.

[0045] After being processed by the upsampling module, the eleventh-level feature map is compared with the seventh-level feature map. Figure 1 The feature map obtained by concatenating the inputs into the Concat module is used as the output feature map of the semantic path feature extraction network.

[0046] The real-time semantic segmentation network also includes an auxiliary loss function;

[0047] Multiple auxiliary loss functions are introduced into the semantic path to supervise the training of the network model. The specific locations of these functions include the tenth-level feature map and the eleventh-level feature map.

[0048] The formula for calculating the total loss is:

[0049] loss = loss1 + αloss2 + βloss3

[0050] In the formula, loss1 is the main loss, loss2 and loss3 are auxiliary losses, and α and β are the weight coefficients of loss2 and loss3, respectively. Each loss value is calculated using the cross-entropy loss function nn.CrossEntropyLoss(), as follows:

[0051]

[0052] In the formula, x is the prediction result of each category, class is the sample label of each category, weight is the weight coefficient of the loss value of each category, and j is the number of label categories in the dataset used in semantic segmentation;

[0053] The real-time semantic segmentation network also includes: a Coordinate Attention mechanism and an SPD-Conv module;

[0054] The Coordinate Attention mechanism and the SPD-Conv module are introduced in the following locations:

[0055] In the semantic path feature extraction network: an SPD-Conv module is added after the convolutional processing module and before the C3 module; a Coordinate Attention mechanism is added after the SPPF module and before the C3 module.

[0056] In the spatial path feature extraction network: a Coordinate Attention mechanism is added after concatting feature maps at different scales;

[0057] In step S6, OHEM online hard sample mining is introduced to train on hard samples whose total loss value is greater than a predetermined threshold.

[0058] In step S6, Mosaic data augmentation and Label Smoothing are used to process the training sample data. At the same time, the optimizer uses the Adam optimizer to accelerate network convergence in the early stage of network model training, and switches to the SGD optimizer in the later stage of network model training.

[0059] This invention provides a real-time semantic segmentation method based on an improved dual-stream feature extraction network. It constructs a real-time semantic segmentation network to perform semantic segmentation on the image data to be processed. This real-time semantic segmentation network is a dual-stream feature extraction network, whose input feature maps are derived from three different resolution feature maps within the semantic path. This avoids redundant computation, reduces the number of model parameters, and deepens the correlation between feature information in each path, resulting in better multi-scale feature information fusion performance. This method optimizes the traditional dual-stream feature extraction network by designing a spatial feature information enhancement module to modify the input feature information of the spatial path. This not only effectively improves the problem of insufficient final feature information fusion performance in traditional network models but also effectively avoids the loss of spatial feature information due to an overly deep feature extraction network, ensuring sufficient spatial feature details are restored for prediction tasks. Furthermore, this invention ensures that the technical solution can be performed in real-time under experimental conditions, making the semantic segmentation algorithm more flexible and applicable to different application scenarios. Attached Figure Description

[0060] Figure 1 This is a diagram of the real-time semantic segmentation network framework of the present invention;

[0061] Figure 2 This is a schematic diagram of the FFM feature fusion module used in this invention;

[0062] Figure 3 This is a schematic diagram of the spatial path feature extraction network of the present invention;

[0063] Figure 4 This is a schematic diagram of the spatial feature enhancement module of the present invention;

[0064] Figure 5 This is a schematic diagram of the Coordinate Attention module used in this invention;

[0065] Figure 6 This is a diagram showing the experimental results of the real-time semantic network dataset of this invention;

[0066] Figure 7 This is a training curve of the loss value of the real-time semantic network model of this invention;

[0067] Figure 8 This is a training curve diagram of the evaluation index of the real-time semantic network model of the present invention;

[0068] Figure 9 The figure shows the experimental results of the real-time semantic network of this invention in a real-world scenario. Detailed Implementation

[0069] This invention includes a real-time semantic segmentation method based on an improved two-stream feature extraction network, which includes the following steps.

[0070] S1: Construct a semantic path feature extraction network.

[0071] like Figure 1 As shown, in the semantic path feature extraction network, firstly, multiple CBS and C3 modules are used to continuously downsample the original input image, and the SPPF module is used to perform multi-scale feature processing on the obtained feature maps. Then, the CBS and C3 modules are used to process the above feature maps, and the processed feature maps are concatenated with the shallow feature maps in the semantic path to obtain feature map information at three different scales. Finally, the Up operation is used to upsample the feature maps, and the feature maps at the three different scales are concatenated and used as one input to the feature fusion module.

[0072] The semantic path feature extraction network includes: a convolutional processing module, a C3 module, a Concat module, an SPPF module, and an upsampling module.

[0073] The convolutional processing module (labeled as the CBS module in the diagram) is the convolutional processing layer in this network model. It performs convolutional activation of feature maps and mainly consists of Conv layers, BN (Batch Normalization) layers, and the SiLu activation function. The SiLu activation function is a combination of the ReLU function and the Sigmoid function. The SiLu activation function calculation formula is as follows:

[0074] SiLu(x)=x*Sigmoid(x)=x(1+e -x ) -1

[0075] Compared to the traditional ReLU activation function, the SiLu activation function, while inheriting the characteristics of being unbounded at the upper limit and bounded at the lower limit, adds the features of smooth curves and non-monotonicity, which can effectively avoid the gradient saturation problem. In network models with too many layers, the gradient descent effect of the SiLu function is significantly better than that of traditional activation functions.

[0076] The C3 module consists of a residual module and a CBS module. It processes the input image features through convolutional layers and the residual module, then concatenates these features with the original input features after the convolutional layers, and finally activates the output through the CBS module. The C3 module can solve the training difficulty problem caused by overly deep network models and avoid the gradient guarantee phenomenon.

[0077] The Concat module directly concatenates the input feature maps without changing the information of the feature maps in each channel. It merges the feature maps in terms of the number of channels, thereby increasing the number of channels in the output feature map.

[0078] SPPF module: Based on Spatial Pyramid Pooling (SPP), it processes the input feature map by serially using multiple max pooling layers with the same size pooling kernels, which can solve the problem caused by the relatively fixed output dimension of traditional pooling layers.

[0079] The upsampling module (UP-Sample, hereinafter referred to as Up) performs an upsampling operation on the input feature map, which can enlarge the size of the input feature map. Here, the Up operation is implemented through bilinear interpolation.

[0080] Among them, a convolutional processing module and a C3 module constitute a downsampling module. The downsampling module performs downsampling operations on the input image feature data and outputs a feature map of a specified resolution.

[0081] The original input image is fed into a convolutional processing module, then into a downsampling module to output a first-level shallow feature map; the first-level shallow feature map is fed into a downsampling module to output a second-level shallow feature map; the second-level shallow feature map is then fed into a downsampling module to output a third-level shallow feature map.

[0082] The convolutional processing module, C3 module, Concat module, SPPF module, and upsampling module are used to perform subsequent processing on the first-level shallow feature maps, second-level shallow feature maps, and third-level shallow feature maps to obtain the output feature maps of the semantic path feature extraction network.

[0083] The subsequent processing of the first-level, second-level, and third-level shallow feature maps in the semantic path feature extraction network specifically includes:

[0084] The third-level shallow feature map is fed into the SPPF module for pooling, and then into the convolution module to obtain the fourth-level shallow feature map. The fourth-level shallow feature map is then fed into the upsampling module for upsampling, and then combined with the second-level shallow feature map... Figure 1 The first part is fed into the Concat module for concatenation. The resulting feature maps are then processed sequentially by the C3 module and the convolutional processing module to obtain a fifth-level shallow feature map. The fifth-level shallow feature map is then processed by the upsampling module and combined with the first-level shallow feature map. Figure 1 The data is fed into the Concat module for concatenation to obtain a six-level shallow feature map.

[0085] The sixth-level shallow feature map is fed into the C3 module for processing to obtain the seventh-level feature map;

[0086] The seventh-level feature map is fed into a convolutional processing module and processed to obtain an eighth-level feature map.

[0087] Level 8 feature map and Level 5 shallow feature map Figure 1 The feature map obtained by concatenating the first part is then processed by the C3 module to obtain a nine-level feature map.

[0088] The level 9 feature map is processed by a convolutional processing module and then concatenated with the level 4 shallow feature map in the Concat module. The resulting feature map is then processed by the C3 module to obtain the level 10 feature map. The level 10 feature map is then processed by the upsampling module and then concatenated with the level 9 feature map. Figure 1 The feature map obtained by concatenation in the Concat module is then processed by the upsampling module and combined with the seventh-level feature map. Figure 1 The feature map obtained by concatenating the inputs into the Concat module is used as the output feature map of the semantic path feature extraction network.

[0089] The specific positions of the first-level, second-level, and third-level shallow feature maps are set according to the network detection accuracy requirements. In this embodiment, the first-level, second-level, and third-level shallow feature maps are feature maps at 1 / 8, 1 / 16, and 1 / 32 resolutions of the original input image, respectively.

[0090] S2: Construct the Spatial Feature Enhancement Module (SFEM);

[0091] As shown in the attached diagram of the instruction manual. Figure 4 As shown, the spatial feature enhancement module is based on the RFB (Receptive Field Block) module, replacing the original 5×5 convolutional layers with 3×3 convolutional layers, which greatly reduces the computational cost. Simultaneously, inheriting from the HDC (Hybrid Dilated Convolution) module, it employs cascaded multiple dilated convolutional layers with different dilation coefficients to avoid the raster effect. In this application, the SFEM module expands the receptive field of the feature map to enhance the representation of spatial path feature information and compensate for the spatial feature information lost due to the feature extraction network.

[0092] The specific processing flow of the SFEM module includes the following steps:

[0093] a1: A 1×1 convolutional layer is used to process the original input feature map by increasing the number of channels, resulting in two types of output feature maps: Output Feature Maps Figure 1 and output features Figure 2 Output features Figure 1 and output features Figure 2 The number of channels in each feature map is 1 / 4 of the number of channels in the original input feature map, which reduces the computational load of the model.

[0094] a2: Output features Figure 1 Processing with a 1×1 convolutional layer yields a set of output feature maps with 1 / 4 of the channels, denoted as: Output Feature Map. Figure 3 ;

[0095] a3: Output features Figure 2 The process utilizes consecutive convolutional layers with 3×3 kernels and dilation factors of 1, 2, and 3. The feature maps obtained after each convolution are retained, resulting in three sets of output feature maps with 1 / 4 of the channels, denoted as: Output Feature Maps. Figure 4 ;

[0096] a4: Output features Figure 3 and output features Figure 4 The four sets of feature maps are stitched together, and the resulting feature map is used as the output feature map of the spatial feature enhancement module.

[0097] The SFEM module, through the spatial feature enhancement module, has features such as good nonlinearity and a large receptive field, which can compensate for the spatial feature information lost by the feature extraction network to a certain extent.

[0098] S3: Construct a spatial path feature extraction network;

[0099] The spatial path feature extraction network takes three different resolution feature maps from the shallow layer of the semantic path as input, then uses the SFEM module to process part of the input feature maps, and finally uses the Up operation to upsample the feature maps. The three different resolution feature maps are concat concatenated and used as another input to the feature fusion module.

[0100] As shown in the attached diagram of the instruction manual. Figure 3 As shown, the spatial path feature extraction network includes: a spatial feature enhancement module, a Concat module, and an upsampling module;

[0101] The input to the spatial path feature extraction network is the first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map extracted by the semantic path feature extraction network.

[0102] In this embodiment, feature maps at 1 / 8, 1 / 16, and 1 / 32 resolutions of the original input image are used as the first-level, second-level, and third-level shallow feature maps, respectively. The specific processing steps of the spatial path feature extraction network then include the following:

[0103] b1: The 1 / 32 resolution feature map of the original input image is fed into the upsampling module, and upsampling is performed through bilinear interpolation to obtain a 1 / 16 resolution feature map with 128 channels as intermediate features. Figure 1 ;

[0104] b2: The 1 / 16 resolution feature map of the original input image is processed by the spatial feature enhancement module to obtain a 128-channel 1 / 16 resolution feature map as an intermediate feature map. Figure 2 ;

[0105] b3: Transform intermediate features Figure 1 and intermediate features Figure 2 The features are fed into the Concat module for concatenation, resulting in a feature map with 256 channels and a resolution of 1 / 16, which serves as the intermediate features. Figure 3 ;

[0106] b4: Transform intermediate features Figure 3 The data is fed into the upsampling module, where bilinear interpolation is used to perform upsampling, resulting in a feature map with 128 channels and 1 / 8 resolution, which is then used as the intermediate feature map. Figure 4 ;

[0107] b5: The 1 / 8 resolution feature map of the original input image is processed by the spatial feature enhancement module to obtain a 128-channel 1 / 8 resolution feature map as an intermediate feature. Figure 5 ;

[0108] b6: Transform intermediate features Figure 4 and intermediate features Figure 5 The data are all fed into the Concat module for concatenation, resulting in a feature map with 256 channels and 1 / 8 resolution, which is used as the output feature map of the spatial path feature extraction network.

[0109] The input feature maps in the spatial path extraction network of this application are no longer extracted solely from the original image, but rather derived from shallow features of the semantic path. This avoids redundant computation, reduces the number of model parameters, and strengthens the correlation of feature information between different paths, enabling the network model to achieve better multi-scale feature information fusion. Secondly, the spatial path feature extraction network utilizes feature maps of three different resolutions (1 / 8, 1 / 16, and 1 / 32) of the semantic path to obtain more spatial scale feature information, and integrates a hybrid attention mechanism (Coordinate Attention, CA) to reduce the loss of spatial feature information. The technical solution of this invention expands the amount of information in the spatial path input feature map through the spatial feature enhancement module SFEM, improves the spatial feature information expression capability, and enhances the final feature information fusion effect.

[0110] S4: Construct the feature fusion module.

[0111] like Figure 1 As shown, the feature fusion module includes: a Concat module, an FFM module, a convolutional processing module, and an upsampling module connected in sequence; the feature maps output by the semantic path feature extraction network and the spatial path feature extraction network are used as inputs to the feature fusion module; the two input feature maps first enter the Concat module for concat concatenation, then the feature maps of the two paths are fused using the FFM module and the CBS module, and finally sent to the upsampling module, where the feature maps are upsampled through the Up operation to expand the feature map size to the original input image size before classification.

[0112] Among them, the FFM (Feature Fusion Module, hereinafter referred to as the FFM module) module is as follows: Figure 2 As shown, the input features are first concatenated using Concat, and then output using Conv, BN, and ReLU activation functions. Next, global pooling and convolutional layers (Conv, Reluctant, Conv, Sigmoid) are used to obtain the weighted features of each channel. Finally, the input feature map and the weighted features are multiplied by Mul and the feature maps are added to obtain the output feature map.

[0113] In this application, the output feature information extracted from the spatial path and semantic path is fused by the FFM module, the two parts of the output feature map are concatted together, and then the channel attention mechanism is used to calculate the weighted features of each channel to achieve the effect of feature selection and fusion.

[0114] S5: Construct a real-time semantic segmentation network.

[0115] The network used in this application is a two-stream network. Traditional two-stream network models, by incorporating a spatial information extraction network, become more complex for semantic segmentation, leading to a decrease in segmentation speed. Furthermore, in traditional two-stream networks, the two paths perform feature extraction relatively independently, resulting in relatively independent feature information before the feature fusion module, which may lead to less than ideal final fusion results. Therefore, traditional two-stream network models also suffer from the inability to simultaneously achieve both segmentation efficiency and accuracy.

[0116] like Figure 1 The image shows the real-time semantic segmentation network in this application, which includes: a spatial path feature extraction network, a semantic path feature extraction network, and a feature fusion module. Figure 1 The semantic path is implemented based on a semantic path feature extraction network, the spatial path is implemented based on a spatial path feature extraction network, and feature fusion is implemented based on a feature fusion module. The spatial path feature extraction network is responsible for extracting shallow spatial feature information from the input image, the semantic path feature extraction network is responsible for extracting deep semantic feature information from the input image, and the feature fusion module is responsible for fusing information from deep semantic features and shallow spatial features. The classified feature map output by the feature fusion module serves as the output result of the real-time semantic segmentation network.

[0117] In the technical solution of this application, spatial path feature information is extracted from the original image through a spatial path feature extraction network. By making full use of the spatial path feature information, sufficient spatial feature details are restored during the semantic prediction upsampling process for the prediction task, thereby improving the accuracy of semantic segmentation.

[0118] Real-time semantic segmentation networks also include auxiliary loss functions;

[0119] Multiple auxiliary loss functions are introduced into the semantic path to supervise the training of the network model. Specific locations where these functions are introduced include:

[0120] For the positions of the tenth-level feature map and the eleventh-level feature map in the semantic path, please refer to [the relevant documentation]. Figure 1 The location marked in the middle is the location of the auxiliary loss.

[0121] The formula for calculating the total loss is:

[0122] loss = loss1 + αloss2 + βloss3

[0123] In the formula, loss1 is the main loss, loss2 and loss3 are auxiliary losses, and α and β are the weight coefficients of loss2 and loss3, respectively. Each loss value is calculated using the cross-entropy loss function nn.CrossEntropyLoss(), as follows:

[0124]

[0125] In the formula, x represents the prediction result for each category, class represents the sample label for each category, weight represents the weight coefficient of the loss value for each category, and j represents the number of label categories in the dataset used in semantic segmentation.

[0126] In the technical solution of this invention, multiple auxiliary loss functions are introduced into the semantic path to supervise the training of the network model. By adding auxiliary losses after multiple outputs in the semantic path, it is ensured that information can be transmitted to the shallow layer during the gradient backpropagation stage, thereby achieving the purpose of optimizing the network model learning process and improving the training effect of the network model.

[0127] The real-time semantic segmentation network also includes: the Coordinate Attention (labeled CA in the figure) attention mechanism and the SPD-Conv module;

[0128] The Coordinate Attention mechanism and the SPD-Conv module are introduced in the following locations:

[0129] In the semantic path feature extraction network: an SPD-Conv module is added after the convolutional processing module and before the C3 module; a Coordinate Attention mechanism is added after the SPPF module and before the C3 module.

[0130] In the spatial path feature extraction network: after concatenating feature maps at different scales, a CoordinateAttention mechanism is added.

[0131] This method improves model performance by introducing a Coordinate Attention mechanism and an SPD-Conv module into the real-time semantic segmentation network. Specifically, the Coordinate Attention mechanism uses average pooling layers to extract spatial feature information from each direction of the image and integrates it into channel attention through convolution, thereby preserving spatial feature information and enhancing its expressive power. The attention module is shown in the attached figure in the specification. Figure 5 As shown.

[0132] exist Figure 5 In this process, the input feature map Residual is an image of size CxHxW. First, average pooling (AvgPool) is performed on the input feature map in both X and Y dimensions, and the resulting feature maps are concatenated (Concat). Then, convolutional layers (Conv2d), batch normalization (BN) layers, and non-linear optimization are used to refine the feature maps. Figure 2 The next step involves processing the feature maps split into their original input size using a convolutional layer (Conv2d). The processing method is as follows: first, the feature maps are split using the split function, and then convolution is performed. Finally, the weighted features are output through the activation function (Sigmoid) and multiplied with the original input feature map to obtain the weighted output feature map Re-weight.

[0133] S6: Train the real-time semantic segmentation network based on historical data to obtain a trained real-time semantic segmentation network;

[0134] To improve training performance, the technical solution of this invention introduces OHEM (Online Hard Negative Example Mining) on ​​the basis of the cross-entropy loss function. It trains on hard samples whose total loss value is greater than a predetermined threshold, which can significantly improve the network training speed and segmentation performance even when positive and negative samples are imbalanced.

[0135] During training, Mosaic data augmentation and Label Smoothing are used to process the training sample data, improving the training speed and effectiveness of the network model. Meanwhile, the optimizer uses the Adam optimizer to accelerate network convergence in the early stages of training, and switches to the SGD (Stochastic Gradient Descent) optimizer in the later stages. This combination effectively avoids the persistent oscillation problem caused by SGD getting stuck in local optima.

[0136] S7: The image data to be processed is fed into the trained real-time semantic segmentation network as the original input image to perform semantic segmentation and obtain the segmented feature map.

[0137] In this embodiment, the CityScapes dataset was selected for ablation experiments to train the real-time semantic segmentation network, and quantitative analysis was performed on the model parameters, frame rate, and segmentation accuracy.

[0138] Network model training environment: The hardware configuration for training the network model of this invention is an i5-10400F CPU, 32GB of RAM, and an NVIDIA GeForce RTX 3060 graphics card. The network model framework is built and trained using PyTorch 1.12.1 and Python 3.8. The environment configuration is CUDA 11.3 and cuDNN 8.2.1. The training parameters are set as follows: initial learning rate of 0.0015, number of epochs of 200, batch size of 12, momentum of both Adam and SGD optimizers set to 0.9, and weight decay rate set to 0.0005.

[0139] In the ablation experiments on the CityScapes dataset, the basic network framework and training strategy used in the ablation experiments were the same. Based on the experimental results, we compared and analyzed how each improved module in the network framework affected the performance of the algorithm in various aspects.

[0140] (1) Testing of the two-stream feature extraction network

[0141] based on Figure 1 The improved dual-stream feature extraction network provided by this method enhances the network's spatial feature capture capability by adding spatial paths. Ablation experiments were conducted on the improved network using the CityScapes dataset, and the model metrics are shown in Table 1.

[0142] Table 1 Ablation Experiments Using Dual-Stream Feature Extraction Network

[0143]

[0144] As shown in Table 1, regarding segmentation accuracy, the addition of a complete spatial multi-scale module to the dual-stream feature extraction network provided in this application significantly improves the segmentation accuracy of the network model. Furthermore, the detection performance of the network model fusing multi-resolution channel feature maps is significantly better than that fusing single-channel feature maps. On the CityScapes dataset, the MIoU value of the model fusing the dual-stream feature extraction network is improved by up to 2.0. Regarding segmentation speed, the addition of the dual-stream feature extraction network inevitably increases the number of network model parameters. Although this results in a certain decrease in inference speed, it still meets the real-time requirements of practical application scenarios.

[0145] (2) Testing of the spatial feature enhancement module

[0146] In the technical solution of this invention, a spatial feature enhancement module is designed through cascaded dilated convolutional layers. The dilated convolution expands the receptive field of the output feature map and enhances the expressive power of spatial feature information. An ablation experiment was conducted on the spatial feature enhancement module on the CityScapes dataset, and the model metrics are shown in Table 2.

[0147] Table 2 Ablation Experiment of Spatial Feature Enhancement Module

[0148]

[0149] As shown in Table 2, regarding segmentation accuracy, the addition of a spatial feature enhancement module to the dual-stream feature extraction network provided in this application improves the segmentation accuracy of the network model to a certain extent. On the CityScapes dataset, the MIoU value of the network model incorporating the spatial feature enhancement module increased by at most 0.3. Regarding segmentation speed, although the addition of the spatial feature enhancement module slightly increases the number of network model parameters, the inference speed of the network model is only slightly affected and can still meet the real-time requirements of practical application scenarios.

[0150] (3) Testing of the auxiliary loss function

[0151] The technical solution of this invention supervises the training process of the network model by adding auxiliary loss functions at multiple output ends of the semantic path, so as to optimize the model learning process and improve the training effect of the network model. An ablation experiment was conducted on the auxiliary loss function on the CityScapes dataset.

[0152] To determine the appropriate weight coefficients for each auxiliary loss, the technical solution of this invention first fixes the weight coefficient β of auxiliary loss 3 to 0.05 by controlling variables in the experiment, and then adjusts the value of the weight coefficient α of auxiliary loss 2 for training. The model indicators are shown in Table 3.

[0153] Table 3 Auxiliary loss function Ablation Experiment 1

[0154]

[0155] As shown in Table 3, under the condition that the weight coefficient β of the auxiliary loss 3 is fixed at 0.05 and the weight coefficient α of loss 2 is fixed at 0.3, the segmentation accuracy of the dual-stream feature extraction network provided in this application is the highest. Therefore, the weight coefficient α of the auxiliary loss 2 is fixed at 0.3, and the model is trained again by adjusting the value of the weight coefficient β of the auxiliary loss 3. The model indicators are shown in Table 4.

[0156] Table 4 Auxiliary loss function Ablation Experiment 2

[0157]

[0158] As shown in Table 4, the network model achieves the highest segmentation accuracy when the weight coefficient α of the fixed auxiliary loss loss2 is 0.3 and the weight coefficient β of loss3 is 0.05. Therefore, the weight coefficients α and β of the two auxiliary losses are finally determined to be 0.3 and 0.05, respectively, and an ablation experiment of the auxiliary loss function is conducted. The model indices are shown in Table 5.

[0159] Table 5 Auxiliary loss function ablation experiment

[0160]

[0161] As shown in Table 5, regarding segmentation accuracy, the addition of an auxiliary loss function to the dual-stream feature extraction network provided in this application improves the segmentation accuracy of the network model to a certain extent. On the CityScapes dataset, the MIoU value of the network model with the auxiliary loss function increased by at most 0.6. Regarding segmentation speed, the addition of the auxiliary loss function has almost no impact on the number of network model parameters, and the inference speed of the network model is essentially unaffected, meeting the real-time requirements of practical application scenarios.

[0162] As can be seen from the experimental data in Tables 1 to 5, the various methods proposed in this scheme have a significant effect on improving the performance of the semantic segmentation network model, which fully demonstrates the effectiveness of each functional module of this method.

[0163] To verify the feasibility of the technical solution of this invention, experiments were conducted using the CityScaps dataset and real-world laboratory scenarios. The results were compared with mainstream real-time semantic segmentation methods, and quantitative analysis was performed in terms of model parameter count, frame rate, and segmentation accuracy.

[0164] Table 6 Auxiliary Loss Function Ablation Experiment

[0165]

[0166] As shown in Table 6, regarding segmentation accuracy, the technical solution of this invention (marked as Ours in the figure) incorporates an improved dual-stream feature extraction network, a spatial feature enhancement module, an auxiliary loss function, and an attention mechanism into the network model, resulting in a significant improvement in model accuracy. On the CityScapes dataset, the improved model's MIoU value reaches 75.3, only slightly lower than the SwiftNet model. Regarding segmentation speed, the technical solution of this invention uses the lightweight YOLOv5 as its network framework, with a designed network model parameter count of 10.9M, which is a significant reduction compared to mainstream algorithm models. Although the inference speed is somewhat limited by hardware limitations, it still meets the real-time requirements of the semantic segmentation-based visual SLAM algorithm. The dataset detection results are as follows: Figure 6 As shown.

[0167] exist Figure 6 In the image, each row contains three sub-images: the original image, the original image label, and the network segmentation result of the technical solution of this invention. It can be seen that the network designed by the technical solution of this invention has good overall performance on the CityScapes dataset.

[0168] To verify the comprehensive performance of the real-time semantic segmentation network designed in this invention in the actual running scenario of the visual SLAM algorithm, this invention trains the semantic segmentation network model using a self-made dataset, and then verifies whether the performance of the improved algorithm meets the requirements based on the detection and segmentation results in the actual running scenario.

[0169] The self-made dataset consists of 3500 images selected from the PASCAL VOC 2007 and 2012 datasets, 1000 images selected from the network, and the last 500 images taken in common laboratory scenes. The self-made dataset is divided into training, validation, and test sets in a 6:1:3 ratio. The model training curves of the network model on the self-made dataset are shown below. Figure 7 As shown.

[0170] Figure 7 The image shows the training curve trends of the DetectLoss object detection loss and the SegLoss semantic segmentation loss. Figure 8 The image shows the training curve trends of MAP0.5, an evaluation metric for object detection, and MIou, an evaluation metric for semantic segmentation. (Source: [Insert Source Here]) Figure 7 and Figure 8 It can be seen that due to the difference in input image size between the self-made dataset and the CityScapes dataset, the change in input image size during model training caused fluctuations in the evaluation index curve. However, after 200 iterations of training, the various evaluation indices of the model gradually converged, and the network model segmentation accuracy MIoU value was 73.5, indicating that the model has good generalization performance.

[0171] After the model is trained, the technical solution of this invention uses an Astra camera to collect RGB image information of the experimental scene and conducts experiments in the actual operating scenario of a mobile robot. The trained semantic segmentation network model achieves target detection and semantic segmentation of dynamic objects. In the mobile robot experiment, the network model's operating configuration was: i7-8550U CPU, 8GB RAM, and NVIDIA GeForce MX 150 graphics card. The dynamic object detection results are as follows: Figure 9 As shown.

[0172] This application provides a real-time semantic segmentation method based on an improved dual-stream feature extraction network. The input feature maps of the real-time semantic segmentation network are sourced from three different resolution feature maps along the semantic path. This avoids redundant computation, reduces the number of model parameters, and deepens the correlation of feature information between different paths, enabling the network model to achieve better multi-scale feature information fusion. Simultaneously, a spatial feature enhancement module is designed by incorporating hybrid dilated convolutions and expanding the receptive field of the feature maps using SFEM, thereby improving the expressive power of spatial path feature information and compensating for spatial feature information lost due to the feature extraction network. Multiple auxiliary loss functions are introduced at the end of the semantic path, and attention mechanisms are added to each module. The network model training is supervised through auxiliary loss functions and other methods to optimize the learning process and improve the training effect. The real-time semantic segmentation network designed in this invention can achieve real-time object detection and semantic segmentation of acquired RGB images, accurately and efficiently completing dynamic object detection tasks and meeting the accuracy and efficiency requirements of applications such as visual SLAM.

Claims

1. A real-time semantic segmentation method based on an improved two-stream feature extraction network, characterized in that, It includes the following steps: S1: Construct a semantic path feature extraction network; The semantic path feature extraction network includes: a convolutional processing module, a C3 module, a Concat module, an SPPF module, and an upsampling module; Among them, one of the convolution processing modules and one of the C3 modules constitute a downsampling module, which performs downsampling operations on the input image feature data and outputs a feature map of a specified resolution. The original input image is fed into a convolutional processing module for processing, and then into a downsampling module to output a first-level shallow feature map; the first-level shallow feature map is input into a downsampling module to output a second-level shallow feature map; the second-level shallow feature map is then input into a downsampling module to output a third-level shallow feature map. The convolutional processing module, C3 module, Concat module, SPPF module, and upsampling module are used to perform subsequent processing on the first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map to obtain the output feature map of the semantic path feature extraction network; S2: Construct a spatial feature enhancement module; The spatial feature enhancement module is based on the RFB module framework, replacing the original 5×5 convolutional layer with a 3×3 convolutional layer. It also inherits from the HDC module by using cascaded multiple dilated convolutional layers with different dilation coefficients. Its specific processing flow includes the following steps: a1: The original input feature map is processed by a 1×1 convolutional layer to obtain two output feature maps: output feature map 1 and output feature map 2. The number of channels in output feature map 1 and output feature map 2 is 1 / 4 of the number of channels in the original input feature map. a2: The output feature map 1 is processed using a 1×1 convolutional layer to obtain a set of output feature maps with 1 / 4 channels, denoted as: output feature map 3; a3: The output feature map 2 is processed by a series of convolutional layers with a kernel of 3×3 and dilation factors of 1, 2 and 3 respectively. The feature map obtained after each convolution process is retained to obtain 3 sets of output feature maps with 1 / 4 channel number, denoted as: output feature map 4. a4: The four sets of feature maps, namely the output feature map 3 and the output feature map 4, are concatenated, and the resulting feature map is used as the output feature map of the spatial feature enhancement module; S3: Construct a spatial path feature extraction network; The spatial path feature extraction network includes: a spatial feature enhancement module, a Concat module, and an upsampling module; The input to the spatial path feature extraction network is the first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map extracted by the semantic path feature extraction network; The specific processing steps of the spatial path feature extraction network include the following steps: b1: The third-level shallow feature map is sent to the upsampling module, and upsampling is completed by bilinear interpolation to obtain intermediate feature map 1; b2: The secondary shallow feature map is passed through the spatial feature enhancement module to obtain intermediate feature map 2; b3: The intermediate feature map 1 and the intermediate feature map 2 are fed into the Concat module for concatenation to obtain: intermediate feature map 3; b4: The intermediate feature map 3 is fed into the upsampling module, and upsampling is completed by bilinear interpolation to obtain: intermediate feature map 4; b5: The first-level shallow feature map is processed by the spatial feature enhancement module to obtain: intermediate feature map 5; b6: The intermediate feature map 4 and the intermediate feature map 5 are fed into the Concat module for concatenation, and the resulting feature map is used as the output feature map of the spatial path feature extraction network. S4: Construct the feature fusion module; The feature fusion module includes: a Concat module, an FFM module, a convolution processing module, and an upsampling module connected in sequence; The feature maps output by the semantic path feature extraction network and the spatial path feature extraction network are used as inputs to the feature fusion module. The two input feature maps first enter the Concat module for concat stitching, then the FFM module and CBS module are used to fuse the features of the two feature maps, and finally they are sent to the upsampling module. The feature map is upsampled through the Up operation, and the feature map size is enlarged to the original input image size before classification. S5: Construct a real-time semantic segmentation network; The real-time semantic segmentation network includes: a spatial path feature extraction network, a semantic path feature extraction network, and a feature fusion module. The spatial path feature extraction network is responsible for extracting shallow spatial feature information from the input image, the semantic path feature extraction network is responsible for extracting deep semantic feature information from the input image, and the feature fusion module is responsible for realizing the information fusion of deep semantic features and shallow spatial features. The classified feature map output by the feature fusion module is used as the output result of the real-time semantic segmentation network. S6: Train the real-time semantic segmentation network based on historical data to obtain the trained real-time semantic segmentation network; S7: The image data to be processed is fed into the trained real-time semantic segmentation network as the original input image to perform semantic segmentation and obtain the segmented feature map.

2. The real-time semantic segmentation method based on an improved two-stream feature extraction network according to claim 1, characterized in that: The first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map are feature maps of the original input image at 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution, respectively.

3. The real-time semantic segmentation method based on an improved two-stream feature extraction network according to claim 1, characterized in that: The subsequent processing of the first-level shallow feature map, the second-level shallow feature map, and the third-level shallow feature map in the semantic path feature extraction network specifically includes: The third-level shallow feature map is fed into the SPPF module for pooling, and then into the convolution module to obtain the fourth-level shallow feature map. The fourth-level shallow feature map is fed into the upsampling module for upsampling, and then together with the second-level shallow feature map, it is fed into the Concat module for concatenation. The resulting feature map is then processed by the C3 module and the convolution module to obtain the fifth-level shallow feature map. The fifth-level shallow feature map is processed by the upsampling module and then together with the first-level shallow feature map, it is fed into the Concat module for concatenation to obtain the sixth-level shallow feature map. The sixth-level shallow feature map is sent to the C3 module for processing to obtain the seventh-level feature map; The seventh-level feature map is fed into a convolutional processing module and processed to obtain an eighth-level feature map. The eighth-level feature map and the fifth-level shallow feature map are fed into the Concat module for splicing. The resulting feature map is then processed by the C3 module to obtain the ninth-level feature map. The ninth-level feature map is processed by the convolutional processing module and then concatenated with the fourth-level shallow feature map by the Concat module. The resulting feature map is then processed by the C3 module to obtain the tenth-level feature map. The tenth-level feature map is processed by the upsampling module and then sent together with the ninth-level feature map into the Concat module to be concatenated to obtain the eleventh-level feature map. After being processed by the upsampling module, the eleventh-level feature map is sent together with the seventh-level feature map to the Concat module for concatenation, and the resulting feature map is used as the output feature map of the semantic path feature extraction network.

4. The real-time semantic segmentation method based on an improved two-stream feature extraction network according to claim 3, characterized in that: The real-time semantic segmentation network also includes an auxiliary loss function; Multiple auxiliary loss functions are introduced into the semantic path to supervise the training of the network model. The specific locations of these functions include the tenth-level feature map and the eleventh-level feature map. The formula for calculating the total loss is: loss = loss1 + αloss2 + βloss3 In the formula, loss1 is the main loss, loss2 and loss3 are auxiliary losses, and α and β are the weight coefficients of loss2 and loss3, respectively. Each loss value is calculated using the cross-entropy loss function nn.CrossEntropyLoss(), as follows: In the formula, x represents the prediction result for each category, class represents the sample label for each category, weight represents the weight coefficient of the loss value for each category, and j represents the number of label categories in the dataset used in semantic segmentation.

5. The real-time semantic segmentation method based on an improved two-stream feature extraction network according to claim 1, characterized in that: The real-time semantic segmentation network also includes: a Coordinate Attention mechanism and an SPD-Conv module; The Coordinate Attention mechanism and the SPD-Conv module are introduced in the following locations: In the semantic path feature extraction network: an SPD-Conv module is added after the convolutional processing module and before the C3 module; a Coordinate Attention mechanism is added after the SPPF module and before the C3 module. In the spatial path feature extraction network, a CoordinateAttention mechanism is added after concatting feature maps at different scales.

6. The real-time semantic segmentation method based on an improved two-stream feature extraction network according to claim 1, characterized in that: In step S6, OHEM online hard sample mining is introduced to train on hard samples whose total loss value is greater than a predetermined threshold.

7. The real-time semantic segmentation method based on an improved two-stream feature extraction network according to claim 1, characterized in that: In step S6, Mosaic data augmentation and Label Smoothing are used to process the training sample data. At the same time, the optimizer uses the Adam optimizer to accelerate network convergence in the early stage of network model training, and switches to the SGD optimizer in the later stage of network model training.

Images