A method and device for real-time semantic segmentation of images based on STDC networks

Abstract

This disclosure discloses a real-time semantic segmentation method for images based on an STDC network, comprising the following steps: S100: acquiring an input image; S200: performing data augmentation on the input image; S300: constructing and training a real-time semantic segmentation model based on a lightweight STDC network; S400: inputting the data-augmented input image into the trained model to achieve real-time semantic segmentation of the input image. This disclosure also discloses a real-time semantic segmentation device, storage medium, and electronic device based on an STDC network. By constructing a real-time semantic segmentation model with an encoder-decoder structure, this disclosure achieves real-time semantic segmentation of input images with high accuracy.

Description

Technical Field

[0001] This disclosure belongs to the field of image processing, and specifically relates to a real-time semantic segmentation method and apparatus for images based on STDC networks. Background Technology

[0002] In recent years, deep convolutional neural networks have been increasingly applied to semantic segmentation tasks, achieving high segmentation accuracy. However, most algorithms relying on deep convolutional neural networks have high computational complexity, which cannot meet the requirements of real-time segmentation tasks.

[0003] Existing deep convolutional networks (DCNNs) for semantic segmentation include SegNet, DeepLab, RefineNet, and PSPNet. These DCNN-based methods demonstrate excellent segmentation results on public benchmark datasets. However, most of these semantic segmentation methods rely on complex backbone networks containing thousands of channels and hundreds of layers. While these methods achieve high accuracy, improved accuracy translates to more complex computational operations and higher memory consumption, thus reducing segmentation speed. For example, PSPNet achieves high segmentation accuracy in semantic segmentation tasks, but it contains a large number of parameters, resulting in an inference speed of less than 1 FPS. Obviously, this is infeasible for applications requiring real-time operation, such as autonomous driving and robotics. Therefore, in real-world scenarios, maintaining high inference speed and high accuracy on high-resolution images is a key challenge facing current semantic segmentation algorithms. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this disclosure is to provide a real-time semantic segmentation method for images based on STDC networks. This method achieves real-time semantic segmentation of input images while obtaining high accuracy through the use of an encoder-decoder structure.

[0005] To achieve the above objectives, this disclosure provides the following technical solutions:

[0006] A real-time semantic segmentation method for images based on STDC networks includes the following steps:

[0007] S100: Acquire input image;

[0008] S200: Perform data augmentation on the input image;

[0009] S300: Construct and train a real-time semantic segmentation model based on the lightweight network STDC;

[0010] The model employs an encoder and decoder structure. The encoder uses the lightweight STDC network as its backbone, followed by a hybrid pooling attention module to capture contextual information from different receptive fields. The decoder includes a channel attention-based feature fusion module to select features at different levels.

[0011] S400: Input the data-enhanced input image into the trained model to achieve real-time semantic segmentation of the input image.

[0012] Preferably, in step S200, the data augmentation of the input image includes the following methods: color dithering, random horizontal flipping, random cropping, and random scaling of the input image. The resolution of random cropping is 1024×512, and the scaling ratio range is [0.125, 1.5].

[0013] Preferably, in step S300, the model is trained using the following method:

[0014] S301: Obtain the training dataset, perform data augmentation on the dataset, and divide the augmented dataset into training set, validation set, and test set;

[0015] S302: Set training parameters, train the model using the training set, and the model training is complete when the preset number of training iterations is reached;

[0016] S303: Validate the trained model using the validation set. During the validation process, the model is evaluated using frames per second (FPS) and average intersection-union ratio (mIoU). The model is validated when the mIoU is 71.2 / 74.6 and the FPS is 250.4 / 126.7.

[0017] S304: Test the trained model using the test set. During the test, the model is evaluated using frames per second and average intersection-union ratio (IoU). The model passes the test when mIoU is 71.9 / 75.3 and FPS is 250.4 / 126.7.

[0018] This disclosure also provides a real-time semantic segmentation device for images based on an STDC network, comprising:

[0019] The acquisition module is used to acquire input images;

[0020] The data augmentation module is used to augment the input image.

[0021] The model building and training module is used to build and train a real-time semantic segmentation model based on the lightweight network STDC.

[0022] The model employs an encoder and decoder structure. The encoder uses the lightweight STDC network as its backbone, followed by a hybrid pooling attention module to capture contextual information from different receptive fields. The decoder includes a channel attention-based feature fusion module to select features at different levels.

[0023] The segmentation module is used to input the data-augmented input image into the trained model to achieve real-time semantic segmentation of the input image.

[0024] Preferably, the model building and training module includes:

[0025] The `get` submodule is used to retrieve the training dataset;

[0026] The data augmentation submodule is used to augment the training dataset.

[0027] The partitioning module is used to divide the augmented training dataset into training, validation, and test sets.

[0028] The training submodule is used to train the model using the training set;

[0029] The validation submodule is used to validate the trained model using a validation set;

[0030] The test submodule is used to test the trained model using a test set.

[0031] This disclosure also provides a computer storage medium storing computer-executable instructions for performing any of the methods described above.

[0032] This disclosure also provides an electronic device, including:

[0033] Memory, processor, and computer programs stored in memory and executable on the processor, wherein,

[0034] When the processor executes the program, it implements the method described in any of the preceding methods.

[0035] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0036] 1. The real-time semantic segmentation model used in this disclosure balances inference speed and segmentation accuracy, and can generate high-quality segmentation results in real time;

[0037] 2. The real-time semantic segmentation model used in this disclosure does not rely on complex deep convolutional neural networks, but instead uses a lightweight STDC network to form an encoder to extract multi-scale feature information under different receptive fields.

[0038] 3. This disclosure introduces a hybrid pooling attention module after the encoder, which can aggregate different types of contextual information using average pooling and strip pooling, thereby generating high-quality segmentation results;

[0039] 4. By introducing a loss function based on the Tversky index, this disclosure enables the generated segmented image to have richer detail information. Attached Figure Description

[0040] Figure 1 This is a flowchart of a real-time semantic segmentation method for images based on an STDC network, provided in one embodiment of this disclosure;

[0041] Figure 2 This is a schematic diagram of the structure of a real-time semantic segmentation model (SEDNet) based on an STDC network provided in another embodiment of this disclosure;

[0042] Figure 3 This is a schematic diagram of the structure of a hybrid pool attention module provided in another embodiment of this disclosure;

[0043] Figure 4 This is a schematic diagram of the structure of a feature fusion module based on channel attention provided in another embodiment of this disclosure;

[0044] Figure 5 This is a schematic diagram comparing different α and β values ​​during network training, provided in another embodiment of this disclosure;

[0045] Figure 6 This is a comparative schematic diagram of different void ratios when the kernel size is 3 in the feature fusion module provided in another embodiment of this disclosure;

[0046] Figure 7 This is a radar chart of specific IoU values ​​for different algorithms across 19 categories on the Cityscapes test set, provided in another embodiment of this disclosure;

[0047] Figure 8 This is a heatmap of the feature fusion module provided in another embodiment of this disclosure on the Cityscapes dataset;

[0048] Figure 9 This is a schematic diagram of the visualization results of different algorithms provided in another embodiment of this disclosure on the Cityscapes validation set. Detailed Implementation

[0049] The following will refer to the appendix. Figures 1 to 9Specific embodiments of this disclosure are described in detail. While specific embodiments of this disclosure are shown in the accompanying drawings, it should be understood that this disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0050] It should be noted that certain terms are used in the specification and claims to refer to specific components. Those skilled in the art will understand that different terms may be used to refer to the same component. This specification and claims do not distinguish components based on differences in terminology, but rather on differences in function. The terms "comprising" or "including" used throughout the specification and claims are open-ended and should be interpreted as "comprising but not limited to." The following descriptions are preferred embodiments for carrying out this disclosure; however, these descriptions are for the purpose of understanding the general principles of the specification and are not intended to limit the scope of this disclosure. The scope of protection of this disclosure is determined by the appended claims.

[0051] To facilitate understanding of the embodiments of this disclosure, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. The accompanying drawings do not constitute a limitation on the embodiments of this disclosure.

[0052] In one embodiment, such as Figure 1 As shown, this disclosure provides a real-time semantic segmentation method for images based on STDC networks, including the following steps:

[0053] S100: Acquire input image;

[0054] S200: Perform data augmentation on the input image;

[0055] S300: Construct and train a real-time semantic segmentation model based on the lightweight network STDC;

[0056] S400: Input the data-enhanced input image into the trained model to achieve real-time semantic segmentation of the input image.

[0057] In another embodiment, step S200, the data augmentation of the input image includes the following methods: color dithering, random horizontal flipping, random cropping, and random scaling of the input image.

[0058] In this embodiment, the resolution of the randomly cropped input image is 1024×512, and the random scaling ratio range is [0.125, 1.5].

[0059] In another embodiment, such as Figure 2As shown, the model includes an input layer, an encoder, a hybrid pooling attention module, a decoder, and an output layer.

[0060] Below, this embodiment will provide a detailed introduction to the structure and characteristics of the encoder, hybrid pooling attention module, and decoder.

[0061] First, the encoder uses a real-time semantic segmentation network (STDC network) as its backbone network, such as... Figure 2 As shown, the encoder includes stages 1 and 2, stage 3, stage 4, and stage 5, with a step size of 2 for each stage. Stages 1 and 2, 3, 4, and 5 generate feature maps with downsampling rates of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively.

[0062] Secondly, a hybrid pooling attention module is set after the encoder. This module is used to capture contextual information under different receptive fields, making the semantic features more discriminative. After the hybrid pooling attention module, a decoder is set. The decoder can upsample the global features generated by the hybrid pooling attention module, perform pixel addition operation on the features corresponding to stage 5, and then upsample the combined features and fuse them with the features corresponding to stage 4 using a similar operation.

[0063] Specifically, such as Figure 3 As shown, the Hybrid Pooling Attention Module (MPAM) includes a strip attention module and a lightweight pyramid pooling module. The strip attention module uses strip pooling to enhance contextual consistency in the vertical direction and further collects the interrelationships between pixels in the horizontal direction, thereby capturing long-distance dependencies between different locations. The pyramid pooling module uses different average pooling to capture the dependencies between pixels, thereby capturing short-distance dependencies between different locations.

[0064] More specifically, the strip attention module comprises three branches: the first branch includes a 1×1 convolutional layer and a strip pooling layer; the second branch includes a 1×1 convolutional layer; and the third branch includes a 1×1 convolutional layer and a strip pooling layer. The strip attention module processes the input features (i.e., the encoder's output) f1∈R. c×h×w (where c represents the number of channels, h represents the height of the input feature map, and w represents the width of the input feature map) Two feature maps are obtained by passing through 1×1 convolutional layers in the first and second branches respectively. One feature map is then subjected to strip pooling with a kernel size of H×1 in the first branch and adjusted using the reshape function. c1 = 64; the other feature map is directly adjusted using the reshape function in the second branch. n = h × w. Then, perform an Affinity operation (Affinity represents batch matrix multiplication) between q and u to generate an attention map d ∈ R. w×n ,Right now

[0065] d j，i =Affinity(q) i u j )

[0066] Where, d j，i This represents the correlation between the i-th position in feature map q and the j-th position in feature map u.

[0067] In addition, the input feature map is fed into a 1×1 convolutional layer in the third branch, and after strip pooling, a feature map with spatial dimensions of c×1×w is obtained. This feature map is then reshaped using the reshape function to v∈R. c×w Finally, the feature map d is processed by the softmax function and then subjected to affinity matching with the feature map v. Figure 3 shown The operation is performed, and the result is reshaped using the reshape function to form the feature map f2∈R. c×h×w .

[0068] The lightweight pyramid pooling module consists of a 1×1 convolutional layer followed by three branches. The first branch includes a 3×3 convolutional layer, the second branch includes an average pooling layer and a 3×3 convolutional layer, and the third branch includes an average pooling layer and a 3×3 convolutional layer. First, the feature map with reduced channel count is obtained through the 1×1 convolutional layer. c2 = c / 4, then multi-scale features are captured through parallel average pooling operations and 3×3 convolutional layers. Simultaneously, a 3×3 two-dimensional convolution is performed in parallel on top of the average pooling operation to preserve the original spatial information. Finally, the three different types of feature maps generated by the three branches are added pixel-by-pixel to generate a new feature map. As shown in the following formula, semantic features from different receptive fields are further integrated:

[0069]

[0070] Among them, c k Let ρ represent a 2D convolution with kernel size k, f represent the input features after adjusting the number of channels, and ρ represent the input features after adjusting the number of channels. i The values ​​represent different average pooling operations, n represents the total number of parallel pooling branches, and the subscript i represents the ordinal number of the current pooling branch. In the experiment, the size of the feature map after pooling was set to 20×20 and 12×12, respectively.

[0071] After obtaining feature map f2 containing long-range dependencies and feature map f3 containing short-range dependencies, they need to be concatenated and fed into a 1×1 convolution to adjust the number of channels. To facilitate element-wise summation with the input features, the number of channels is adjusted to C, and the final output f4∈R c×h×w As shown in the following formula:

[0072] f4 = c k (φ(f2, f3)) + f1, k = 1

[0073] Where φ represents the concat function, c k This represents a two-dimensional convolution with a kernel size of k.

[0074] Based on the above, it is easy to conclude that the hybrid pooling attention module proposed in this embodiment can enhance the pixel-level feature representation capability. By learning from images with complex scenes through average pooling and strip pooling, it can obtain semantic features with closely distributed characteristics while strengthening the contextual consistency in a specific direction and taking advantage of the advantages of average pooling.

[0075] Finally, a decoder is set after the hybrid pooling attention module. In the decoder, the feature f4 input to the decoder is processed by two pixel addition operations to generate feature f5, which retains sufficient contextual semantic information, while the 1 / 8 downsampled feature f6 from encoder stage 3 provides rich detail information. Since these two feature maps contain feature representations at different levels, a simple fusion method may not be able to fully utilize the advantages of different features. Therefore, this embodiment specifically designs a feature fusion module (CAFF) in the decoder. The feature fusion module can select features with different levels. Finally, the feature map processed by the feature fusion module is directly upsampled 8 times to restore the image resolution, which can avoid the problem of slow inference speed caused by multiple upsampling in large-resolution images.

[0076] Channel attention-based feature fusion module (CAFF) such as Figure 4 As shown, Figure 4 In the middle, the feature map and feature map Channel concatenation is performed between the channels, where c3 = 128 and c4 = 256. Since feature map f6 is obtained by upsampling 1 / 16 of its original size, the concatenated features are then scaled using batch normalization to generate the final feature map. c5 = 256, as shown in the following formula:

[0077] f7 = g(φ(f5, f6))

[0078] Where g is a transformation function implemented by CBL (C: 1×1 conv; B: batch normalization (BN: BatchNormalization; L: ReLU activation function), and φ represents the concat operation.

[0079] Next, the connected features are aggregated into a feature vector, and a weight vector is calculated. This effectively captures cross-channel dependencies while avoiding channel dimensionality reduction. The channel attention used in CAFF is an improvement on ECA.

[0080] like Figure 4 As shown, this embodiment adds dilated convolution to enhance feature representation based on one-dimensional convolution. First, without dimensionality reduction, the feature map f7 is aggregated into convolutional features using adaptive global average pooling (GAP). Then, a fast one-dimensional dilated convolution with a kernel size of k is used to capture the interdependencies between different channels. Figure 4 As shown, a dilation rate is set based on standard one-dimensional convolution, and then a weight vector is generated using the sigmoid function. Finally, the feature map f7 is reweighted according to the following operation, which is equivalent to reselecting and combining the features to generate a feature map for predicting segmentation.

[0081]

[0082] Where σ represents the sigmoid function and ρ represents the global average pooling operation. This represents a one-dimensional convolution with kernel size k and dilation rate r.

[0083] Compared with the FFM model, the Channel Attention-based Feature Fusion Module (CAFF) proposed in this embodiment brings significant performance improvement while having lower model complexity. It captures cross-channel interaction information in a lightweight manner while avoiding channel dimensionality reduction, and can effectively fuse features from different stages, re-select features, and thus learn the feature information of the target pixel and its surrounding neighboring pixels.

[0084] Furthermore, in order to further improve model performance without increasing computational complexity, and to enhance feature representation during the training phase, this embodiment incorporates different elements in the decoder (e.g., ...). Figure 2 As shown, a Seg Head is added at the positions after the first addition operation in the decoder, after the second addition operation, and after the CAFF module. A Detail Head is added at stage 3 in the encoder. It should be noted that both the Detail Head and the Seg Head consist of a 3×3 convolutional layer, a batch normalization (BN) layer, a ReLU normalization layer, and a 1×1 convolutional layer.

[0085] Furthermore, pixel-level classification tasks are extremely difficult when training networks with weighted cross-entropy loss. The networks struggle to distinguish between true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN) pixels, resulting in thick and blurry edges in the generated segmented images. Adding additional supervision can simplify the optimization process of deep convolutional neural networks and generate high-quality segmentation results. To achieve higher efficiency, this embodiment adds a detail guidance module at a downsampling rate of 1 / 8 in the backbone network. Figure 2 In the process, the detail loss generated by the detail header added at stage 3 is part of the detail guidance module, which, along with the segmentation loss module ( Figure 2 In this model, the segmentation loss (Seg Loss) generated by the Seg Head is a portion of the total loss (Loss) calculated by adding the segments from the Seg Loss module. The detail guidance module uses binary cross-entropy loss and Dice loss to optimize detail information. Assuming P and G are the predicted and ground truth labels respectively, the Dice similarity coefficient D between the two labels is defined as:

[0086]

[0087] If used in the loss layer of the detail guidance module, the weights of FP and FN are equal. However, the number of detail pixels in this module is far less than the number of non-detail pixels, making detail prediction a highly imbalanced problem. Inspired by edge detection tasks, this embodiment generalizes the Dice coefficient, using a Tversky exponent-based loss function for the detail guidance module to control FP and FN and generate sharper edges. The Tversky exponent is defined as:

[0088]

[0089] Here, α and β represent hyperparameters, controlling the penalty magnitude for FP and FN, respectively. For semantic segmentation tasks, the following formula is used in the detailed guidance module to define Tversky's loss function:

[0090]

[0091] Where, p i g i Let P and G represent the values ​​of the i-th pixels in the predicted image P and the ground truth label G, respectively, where N represents the number of pixels and ε = 10. -6ε is used to avoid division by zero errors during calculation, ensuring the loss function is always non-zero. Furthermore, the trade-off between false positives and false negatives is controlled by adjusting the hyperparameters α and β. Specifically, when α = β = 0.5, the Tversky exponent simplifies to the same value as the Dice coefficient.

[0092] In this module, the final total loss function f detail It is expressed as follows:

[0093] f detail ×f T(α，β) +f bce

[0094] Among them, f T(α，β) It is based on Tversky's loss function, f bce It is the binary cross-entropy loss function.

[0095] In another embodiment, this disclosure utilizes the Cityscapes dataset to train the model. The Cityscapes dataset is a well-known semantic segmentation dataset focusing on urban street scene parsing, containing 5000 finely annotated images with a resolution of up to 2048×1024. The specific process of training the model using the Cityscapes dataset is as follows:

[0096] S301: Data augmentation was performed on the image data in the Cityscapes dataset (data augmentation methods are described above). The 2975 augmented images were used as the training set for model training, 500 images as the validation set for model validation, and 1525 images as the test set for model testing. The dense annotations in all these images contain 30 common categories, such as sky, road, and car, 19 of which are suitable for semantic segmentation tasks.

[0097] S302: The model is trained using the training set with an initial learning rate of 0.005 and a power of 0.9. The SGD optimizer is used, and the model is trained for 60,000 iterations on two NVIDIA GTX 1080Ti GPUs with a batch size of 24. Once the required number of training iterations is reached, the model training is complete.

[0098] S303: Validate the trained model using a validation set. During validation, frames per second (FPS) and mean class-wise intersection over union (mIoU) are used as model evaluation metrics. mIoU is the ratio of the intersection to the union of the true and predicted label sets, used to evaluate model accuracy. The calculation formula is as follows:

[0099]

[0100] Where n+1 represents the total number of segmentation categories, including the background, and P ij P represents the number of pixels whose actual category is i are predicted to be category j. ii This represents the number of pixels in category i that were correctly predicted.

[0101] The model was validated when mIoU was 71.2 / 74.6 and FPS was 250.4 / 126.7.

[0102] S304: Test the trained model using the test set. During the test, the model is evaluated using frames per second and average intersection-union ratio (IoU). The model passes the test when mIoU is 71.9 / 75.3 and FPS is 250.4 / 126.7.

[0103] Below, this disclosure, in conjunction with Tables 1 to 5 and Figures 5 to 8 The characteristics of the model described in this disclosure are described in detail.

[0104] First, since the method proposed in this disclosure includes improvements to three modules—TDGM, MPAM, and CAFF—the effectiveness of these three modules can be verified through ablation experiments. The specific experimental results are shown in Table 1.

[0105] Table 1

[0106]

[0107] As shown in Table 1, the baseline models for different resolutions are STDC1-Seg50 and STDC1-Seg75. Based on the three proposed modules, a mIoU of 73.02% was achieved at a resolution of 512×1024 with an inference speed of 206.0 FPS, and a mIoU of 76.77% was achieved at a resolution of 768×1536 with an inference speed of 105.2 FPS.

[0108] Secondly, this disclosure progressively studies and analyzes the effectiveness of the three modules TDGM, MPAM, and CAFF on the Cityscapes validation set.

[0109] 1. TDGM module

[0110] To verify the effectiveness of the Tversky function in the TDGM module, the network was trained using different α and β values. Except for optimizing the loss function, the composition of other parts of the network remained consistent with STDCSeg. Performance metrics for different α and β values ​​are as follows: Figure 5 As shown, it is easy to see that the best results are achieved when α = 0.3 and β = 0.7 for inference, regardless of whether the image resolution is 512×1024 or 768×1536. As shown in Table 1, with an image resolution of 512×1024, the benchmark algorithm STDC1-Seg50 achieves an mIoU of 72.22%, where the loss function used in the detail guidance module is the Dice loss. In Table 1, TDGM indicates that the detail guidance module using the Tversky loss function, with only the loss function optimized, produces a 0.25% accuracy improvement. Under the same conditions, STDC1-Seg75 achieves an mIoU of 74.50% with an image resolution of 768×1536, while when the detail guidance module in the network is trained based on the Tversky loss function, it produces an mIoU of 75.95%, an improvement of 1.45%. This proves that the loss function proposed in this disclosure is effective for deep supervision of the detail guidance module.

[0111] 2. MPAM module

[0112] To demonstrate the effectiveness of the MPAM module, this disclosure compares the performance of using attention and pooling (A&P) in the network structure and using only hybrid pooling attention (MPAM) in the network, with the TDGM described above. As shown in Table 1, A&P represents using channel attention (ARM) and pooling operations. In contrast, using the MPAM module to enhance feature representation after the encoder brings accuracy improvements of 0.03% and 0.58% at resolutions of 512×1024 and 768×1536, respectively. Although this module incurs some speed loss, it still enables the network to achieve real-time inference quite well.

[0113] Furthermore, this disclosure also evaluates the performance of different connection methods when fusing long-range and short-range dependencies in the MPAM module, as shown in Table 2:

[0114] Table 2

[0115]

[0116] In Table 2, `concat` indicates channel concatenation, and `add` indicates pixel-by-pixel addition. As can be seen from Table 2, pixel addition is slightly faster than channel concatenation, but channel concatenation yields a significantly higher mIoU value. Therefore, it can be concluded that using channel concatenation in the MPAM module is the most effective method for feature enhancement.

[0117] 3. CAFF module

[0118] This disclosure employs a CAFF module in addition to the TDGM and MPAM modules. As shown in Table 1, at a resolution of 512×1024, the mIoU after adopting the CAFF module reaches 73.02%, which is 0.52% higher than using only TDGM and MPAM, and 0.8% higher than the baseline network STDC1-Seg50. At a resolution of 768×1536, the mIoU after adopting the CAFF module reaches 76.77%, which is 0.24% higher than using only TDGM and MPAM, and 2.27% higher than the baseline network STDC1-Seg75.

[0119] Furthermore, to verify the effectiveness of the dilated convolution-based channel attention used in the CAFF module, we first applied ECA channel attention using TDGM and MPAM, following the ECA settings of k=3 or k=5, as shown in Table 3:

[0120] Table 3

[0121]

[0122] Table 3 shows that the best mIoU is achieved with k=3 at all resolutions. However, this is still lower than the mIoU of 72.50% and 76.53% achieved using TDGM and MPAM, respectively. This is because when the feature map has 256 channels, the kernel size of the one-dimensional convolution is 3, resulting in a smaller coverage area for information interaction. Increasing the kernel size, on the other hand, leads to a decrease in segmentation accuracy. Based on this, dilated convolution is used to expand the receptive field while maintaining a kernel size of 3, such as... Figure 6 As shown, the impact of different hole ratios on network performance was investigated. Hole ratios of 2, 6, 8, 10, and 12 all improved accuracy, with the optimal mIoU of 73.02% and 76.77% achieved at a hole ratio of 8. In comparison, the CAFF module proposed in this disclosure can more effectively fuse features from different stages.

[0123] To further illustrate the characteristics of the model described in this disclosure, this disclosure also compares it with other semantic segmentation methods, including DeepLabv2, PSPNet, and Enet. The comparison results are shown in Table 4.

[0124] Table 4

[0125]

[0126] As shown in Table 4, at resolutions of 512×1024 and 768×1536, the method described in this disclosure achieved mIoU of 73.1% and 76.4%, respectively, with speeds as high as 206.0 FPS and 105.2 FPS. It is evident that compared to several other advanced algorithms, this method is faster and more accurate in terms of both accuracy and inference speed. Furthermore, as shown in Table 4, compared to the classic real-time semantic segmentation algorithm ENet, the method described in this disclosure achieves a 14.8% improvement in mIoU and is nearly three times faster. Compared to the BiSeNet series, the method described in this disclosure also achieves competitive results. Although this method is only slower than STDCSeg in terms of inference speed, it maintains the same real-time performance and achieves a higher mIoU. Therefore, among all competing algorithms, this method achieves a good balance between mIoU and FPS.

[0127] To more intuitively illustrate the differences between different algorithms, this disclosure compares the IoU values ​​for different categories on the Cityscapes test set. A radar chart is used to represent the specific IoU values ​​of known algorithms, as shown below. Figure 7 As shown, by Figure 7 It can be observed that the model described in this disclosure is located on the periphery of the image, and the IoU obtained is significantly better than other algorithms in most categories. While most algorithms perform well in categories such as Rode, Car, and Sky, the accuracy obtained by different algorithms varies significantly in categories such as Trafficlight, Rider, and Motorcycle. However, this method still performs excellently, thus demonstrating its superiority.

[0128] This disclosure also analyzes the CAFF module using illustrations, visually comparing the feature maps of different stages before and after CAFF module fusion. Specifically, as shown below... Figure 8 As shown, Figure 8In the diagram, the second column (b) contains feature f5 from the corresponding decoding part, and its heatmap is more discernible overall. The third column (c) contains feature f6 from the encoding part, and its heatmap contains more spatial details, such as road lines and building outlines. The heatmap of feature map f8, generated by the feature fusion module, is located in the fourth column (d). It can be seen that the features in this column, while possessing semantic features, also incorporate detailed information from the encoding part.

[0129] Figure 9 The visualization results of the model described in this disclosure are shown. Figure 9 In the visualization of the baseline network in the third column (c), the model described in this method in the fourth column (d) corrects its erroneous predictions, such as the motorcycle predicted as a bicycle in the first row and the over-segmented truck in the third row. Furthermore, in the second row, it can be observed that the segmentation at the sign outline in the fourth column is better than the results in the third column; in the third row, the segmentation results in the fourth column are more complete in the road direction compared to the third column; and in the fourth row, the model described in this method can still correctly assign labels to buses obscured by cars.

[0130] Finally, the robustness of the proposed model is evaluated using the publicly available dataset CamVid. CamVid, a road scene dataset, consists of images extracted from high-resolution video sequences with a resolution of 960×720. It contains 701 labeled images, divided into training, validation, and test sets, containing 367, 101, and 233 images respectively. To facilitate comparison with previous research, the proposed network model is evaluated on the test set using 11 classes from CamVid. The specific evaluation results are shown in Table 5.

[0131] Table 5

[0132]

[0133] As shown in Table 5, the model (SEDNet) described in this disclosure achieves a mIoU of 72.80% with an FPS of 165.6. This demonstrates superior accuracy and faster speed compared to the state-of-the-art BiSeNet series. Although its inference speed is slightly slower than BiSeNetV1-X and STDC1-Seg, it achieves higher accuracy. Therefore, the model described in this method achieves a good balance between accuracy and speed.

[0134] In another embodiment, this disclosure also provides a real-time semantic segmentation apparatus for images based on an STDC network, comprising:

[0135] The acquisition module is used to acquire input images;

[0136] The data augmentation module is used to augment the input image.

[0137] The model building and training module is used to build and train a real-time semantic segmentation model based on the lightweight network STDC.

[0138] The model employs an encoder and decoder structure. The encoder uses the lightweight STDC network as its backbone, followed by a hybrid pooling attention module to capture contextual information from different receptive fields. The decoder includes a channel attention-based feature fusion module to select features at different levels.

[0139] The segmentation module is used to input the data-augmented input image into the trained model to achieve real-time semantic segmentation of the input image.

[0140] In another embodiment, the model building and training module includes:

[0141] The `get` submodule is used to retrieve the training dataset;

[0142] The data augmentation submodule is used to augment the training dataset.

[0143] The partitioning module is used to divide the augmented training dataset into training, validation, and test sets.

[0144] The training submodule is used to train the model using the training set;

[0145] The validation submodule is used to validate the trained model using a validation set;

[0146] The test submodule is used to test the trained model using a test set.

[0147] In another embodiment, this disclosure also provides a computer storage medium, characterized in that the computer storage medium stores computer-executable instructions for performing any of the methods described above.

[0148] In another embodiment, this disclosure also provides an electronic device, characterized in that it includes:

[0149] Memory, processor, and computer programs stored in memory and executable on the processor, wherein,

[0150] When the processor executes the program, it implements the method described in any of the preceding methods.

[0151] The applicant has provided a detailed description of the embodiments of this disclosure in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred examples of this disclosure and are not limited to the specific embodiments described above. The detailed description is intended to help readers better understand the spirit of this disclosure and is not intended to limit the scope of protection of this disclosure. On the contrary, any improvements or modifications made based on the inventive spirit of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A real-time semantic segmentation method for images based on STDC networks, characterized in that, The method includes the following steps: S100: Acquire input image; S200: Perform data augmentation on the input image; S300: Construct and train a real-time semantic segmentation model based on the lightweight network STDC; The model employs an encoder and decoder structure. The encoder uses the lightweight STDC network as its backbone, followed by a hybrid pooling attention module (MPAM) to capture contextual information from different receptive fields. The decoder includes a channel attention-based feature fusion module (CAFF) to select features at different levels. S400: Input the data-enhanced input image into the trained model to achieve real-time semantic segmentation of the input image; The Hybrid Pooling Attention Module (MPAM) includes a parallel strip attention module and a lightweight pyramid pooling module. The strip attention module comprises three branches: the first branch includes a 1×1 convolutional layer and a strip pooling layer; the second branch includes a 1×1 convolutional layer; and the third branch includes a 1×1 convolutional layer and a strip pooling layer. The feature maps output from the first and second branches are then multiplied in batches to generate the attention map. d , feature map d The softmax function performs batch matrix multiplication with the feature map output from the third branch. The lightweight pyramid pooling module includes a 1×1 convolutional layer followed by three branches: the first branch includes a 3×3 convolutional layer, the second branch includes an average pooling layer and a 3×3 convolutional layer, and the third branch includes an average pooling layer and a 3×3 convolutional layer. The strip attention module and the pyramid pooling module are concatenated and fed into a 1×1 convolutional layer to adjust the number of channels. The channel attention used in the feature fusion module (CAFF) is an improvement on ECA. Without dimensionality reduction, the feature map is aggregated into convolutional features through adaptive global average pooling (GAP), and then the interdependencies between different channels are captured through fast one-dimensional dilated convolution.

2. The method according to claim 1, characterized in that, In step S200, the data augmentation of the input image includes the following methods: color dithering, random horizontal flipping, random cropping, and random scaling of the input image.

3. The method according to claim 1, characterized in that, In step S300, the model is trained using the following method: S301: Obtain the training dataset, perform data augmentation on the dataset, and divide the augmented dataset into training set, validation set, and test set; S302: Set training parameters, train the model using the training set, and the model training is complete when the preset number of training iterations is reached; S303: Validate the trained model using the validation set. During the validation process, the model is evaluated using frames per second (FPS) and average intersection-union ratio (mIoU). The model is validated when the mIoU is 71.2 / 74.6 and the FPS is 250.4 / 126.

7. S304: Test the trained model using the test set. During the test, the model is evaluated using frames per second and average intersection-union ratio (IoU). The model passes the test when mIoU is 71.9 / 75.3 and FPS is 250.4 / 126.

7.

4. A real-time semantic segmentation device for images based on STDC networks, characterized in that, The device includes: The acquisition module is used to acquire input images; The data augmentation module is used to augment the input image. The model building and training module is used to build and train a real-time semantic segmentation model based on the lightweight network STDC. The model employs an encoder and decoder structure. The encoder uses the lightweight STDC network as its backbone, followed by a hybrid pooling attention module (MPAM) to capture contextual information from different receptive fields. The decoder includes a channel attention-based feature fusion module (CAFF) to select features at different levels. The segmentation module is used to input the data-enhanced input image into the trained model to achieve real-time semantic segmentation of the input image; The Hybrid Pooling Attention Module (MPAM) includes a parallel strip attention module and a lightweight pyramid pooling module. The strip attention module comprises three branches: the first branch includes a 1×1 convolutional layer and a strip pooling layer; the second branch includes a 1×1 convolutional layer; and the third branch includes a 1×1 convolutional layer and a strip pooling layer. The feature maps output from the first and second branches are then multiplied in batches to generate the attention map. d , feature map d The softmax function performs batch matrix multiplication with the feature map output from the third branch. The lightweight pyramid pooling module includes a 1×1 convolutional layer followed by three branches: the first branch includes a 3×3 convolutional layer, the second branch includes an average pooling layer and a 3×3 convolutional layer, and the third branch includes an average pooling layer and a 3×3 convolutional layer. The strip attention module and the pyramid pooling module are concatenated and fed into a 1×1 convolutional layer to adjust the number of channels. The channel attention used in the feature fusion module (CAFF) is an improvement on ECA. Without dimensionality reduction, the feature map is aggregated into convolutional features through adaptive global average pooling (GAP), and then the interdependencies between different channels are captured through fast one-dimensional dilated convolution.

5. The apparatus according to claim 4, characterized in that, The model building and training module includes: The `get` submodule is used to retrieve the training dataset; The data augmentation submodule is used to augment the training dataset. The partitioning module is used to divide the augmented training dataset into training, validation, and test sets. The training submodule is used to train the model using the training set; The validation submodule is used to validate the trained model using a validation set; The test submodule is used to test the trained model using a test set.

6. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions for performing the method according to any one of claims 1 to 3.

7. An electronic device, characterized in that, include: Memory, processor, and computer programs stored in memory and executable on the processor, wherein, When the processor executes the program, it implements the method according to any one of claims 1 to 3.

Images