A lightweight semantic segmentation method for high-precision stereo vision perception in complex scenes

Abstract

This invention belongs to the field of computer semantic segmentation technology, and relates to a lightweight semantic segmentation method for high-precision stereo vision perception in complex scenes. The proposed network structure uses an ultra-lightweight semantic segmentation network as a baseline, employs a CLIP text encoder as a category prototype generator, and generates multiple category representations as class centers to guide network learning, thereby improving the network's segmentation performance among similar categories. Simultaneously, because the network's segmentation performance is poor at edges, this invention adds an edge optimization module to the network decoder side to improve the segmentation accuracy at object edges, and proposes a ternary loss function for edge extraction. The network of this invention effectively improves generalization performance and simultaneously enhances the segmentation effect at category edges.

Description

Technical Field

[0001] This invention belongs to the field of deep learning semantic segmentation technology, and relates to a lightweight semantic segmentation method for high-precision stereo vision perception in complex scenes. Background Technology

[0002] With the development of automation and computer technology, unmanned operation and intelligence have become the themes of modern technology. Unmanned application platforms can perceive their surrounding environment using devices such as cameras and infrared detectors, thereby making control and decisions. The visual perception module is the only module for unmanned application platforms to interact with their environment. Therefore, the accuracy and robustness of visual perception algorithms are important sources of the mobility and intelligence capabilities of unmanned application platforms, determining the technical level of their core functions.

[0003] Semantic segmentation is a crucial visual task for the visual perception modules of existing unmanned application platforms. It involves assigning a category label to each pixel in an image to facilitate scene understanding and object detection, providing reliable intelligence for target localization. Key requirements for the visual perception modules of unmanned application platforms are low computational resource consumption and low prediction latency. Many high-performance semantic segmentation models are computationally expensive and slow in prediction, making them unsuitable for deployment. Lightweight semantic segmentation models aim to be deployed on low-memory embedded systems with a more compact and efficient approach, while simultaneously meeting the requirements for real-time and accurate inference.

[0004] Currently, there are studies on lightweight frameworks in the field of semantic segmentation. The encoder-decoder structure is the paradigm of semantic segmentation networks. The computational cost of applying convolution to images or feature maps is proportional to their resolution. Downsampling the input image can significantly reduce the computational resource consumption of the network. In lightweight segmentation networks, interpolation and a minimal amount of convolution are typically used to upsample feature maps. A. Paszke et al. used downsampling and convolutional encoding in the shallow layers of the network to generate a more compact model (A. Paszke, A. Chaurasia, S. Kim and E. Culurciello, "Enet: A deep neural network architecture for real-time semantic segmentation," arXiv preprint arXiv:1606.02147, 2016). In lightweight semantic segmentation, convolution strategies that reduce parameters and computational cost are frequently used. F. Chollet et al. used depthwise separable convolution to divide convolution into depthwise convolution and pointwise convolution (F. Chollet, "Xception: Deep Learning With Depthwise Separable Convolutions," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017). G. Papandreou et al. used dilated convolution, which sparsely applies kernel weights over a larger input window to achieve a larger receptive field without increasing the kernel size (G. Papandreou, I. Kokkinos and P.-A. Savalle, "Modeling Local and Global Deformations in Deep Learning: Epitomic Convolution Multiple Instance Learning, and Sliding Window Detection," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015).X. Zhang et al. used grouped convolution and channel shuffling to obtain satisfactory classification results from lightweight networks for operation on embedded systems (X. Zhang, X. Zhou, M. Lin and J. Sun, "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.). However, in the application scenarios of unmanned platforms, target objects are often hidden, requiring algorithms to have more accurate category discrimination and clearer discrimination of object texture edges to obtain accurate results; existing lightweight semantic segmentation algorithms are often poorly adapted to the complex environment of unmanned application platforms. Summary of the Invention

[0005] The purpose of this invention is to provide a lightweight semantic segmentation method for high-precision stereo vision perception in complex scenes, for use in unmanned application platforms.

[0006] This invention presents a lightweight semantic segmentation method that proposes a semantic segmentation network based on a category prototype generation module and an edge optimization module. The edge optimization module utilizes the edge optimization ternary loss proposed in this invention to improve the problem of imperfect boundaries in the semantic segmentation results.

[0007] The specific technical solution of the present invention is as follows:

[0008] A lightweight semantic segmentation method for high-precision stereo vision perception in complex scenes is described below:

[0009] Step 1) Constructing the Category Prototype Generation Module: The category prototype generation module uses a pre-trained CLIP text encoder (A. Radford, JW Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” Image, vol. 2, p. T2, 2021..) as robust category prototype features to guide the semantic segmentation network in learning category features. The category prototype generation module is structured as a text input Transformer network, which maps the text input to the feature space to generate category prototypes. Before inputting the category name text into the category generation module, the category name is first constructed into a text prompt, such as “a photo with {label}”, to reduce the network’s misclassification of ambiguous text. Then, the text prompt is input into the category prototype generation module. Construct N text hints for a specific category text name, thereby constructing N category prototypes for a category, representing the different manifestations of instances within each category, and also representing the differences between different categories.

[0010] Step 2) Constructing the edge optimization module: The input to the edge optimization module is the features from each layer of the decoder in the backbone of the semantic segmentation network. After the input feature map, it passes through two convolutional layers and activation layers to obtain a one-channel object edge prediction map. The class edge map obtained from the ground truth annotation is used to supervise the multi-level class edge prediction map, and the edge map supervision is optimized using a supervision method that combines edge ternary loss with classification loss. The supervision method for the joint classification loss of edge ternary loss is as follows: In the ternary loss function for optimizing the edge map, the set of pixels with confidence scores higher than the threshold α and true values ​​being edges are considered true examples, the set of pixels with confidence scores lower than β and true values ​​being within the category are considered false negatives, and the remaining set of pixels with true values ​​being edges are considered the target to be optimized; in the ternary loss, the features of the true examples, false negatives, and target to be optimized are calculated separately, the cosine metric is used to describe the distance between the target to be optimized and the true examples and false negatives, and the cross-entropy function is used to construct and calculate the ternary loss; based on the statistics of edge pixels in the dataset, the number of feature points of the true examples, false negatives, and target to be optimized is offset by γ% of the total number of pixels to achieve faster convergence. In this context, α, β, and γ are all parameters. The principle for determining the values ​​of α and β is to ensure that α > β, and that the number of true positive pixels with a confidence level higher than α is less than the number of false negative pixels with a confidence level lower than β, so as to ensure the representativeness of the features of true positive and false negative pixels. The value of γ is slightly less than the number of class edge pixels in the ground truth mask to ensure its smoothing effect and ensure that the calculated gradient value is relatively stable.

[0011] Step 3) Constructing the semantic segmentation network: The overall structure of the semantic segmentation network is constructed using the category prototype generation module and edge optimization module built in Steps 1) and 2), together with the backbone network. The backbone network consists of two parts: encoding and decoding. For the encoder, the original image is extracted to... The encoder generates a C1-dimensional feature map, which is twice the size of the original input. The decoder then decodes the feature map generated by the encoder into a C2-dimensional feature map of the original input size. Each upsampled feature from the decoder is used as input to the edge optimization module to obtain edge prediction maps at different resolutions, and edge triplets are used for supervision.

[0012] Step 4) Semantic Segmentation Network Training and Inference Process: The final C2-dimensional vector output by the decoder is compared with the category prototypes generated by the category prototype generation module to obtain the final segmentation result. During training, the category prototype generation module randomly selects the mean of M category prototypes and performs a pixel-by-pixel cosine similarity measurement with the final output features of the decoder. The category with the highest cosine similarity is the predicted category of that pixel. During inference, the edge optimization module does not participate in the calculation. The decoder features are compared pixel-by-pixel with all category prototypes for similarity. For each category, the mean of the M scores with the highest similarity scores is calculated as the predicted score of that pixel belonging to that category. The category with the highest predicted score for each pixel across all categories is taken as the final predicted category. The prediction results of all pixels constitute the segmentation result map.

[0013] The beneficial effects of this invention are:

[0014] This invention presents a lightweight semantic segmentation method for high-precision visual perception in complex scenes. The semantic segmentation network is constructed based on a category prototype generation module and an edge optimization module. The former enhances the network's ability to distinguish objects of interest and reduces missegmentation in some regions; the latter strengthens the backbone network's ability to detect object edges and constructs a joint semantic and edge feature space. The time complexity of the category prototype prediction mask is the same as that of traditional one-hot vector generation, and the edge optimization module does not participate in the calculation during prediction. Both are suitable for direct integration into a lightweight semantic segmentation framework, making it suitable for deployment on unmanned application platforms. This achieves a good balance between speed and accuracy in segmentation tasks, providing a new solution for high-precision stereo vision perception in complex scenes.

[0015] The proposed network structure uses an ultra-lightweight semantic segmentation network as a baseline and employs the CLIP text encoder as a category prototype generator to generate multiple category representations as class centers to guide network learning, thereby improving the network's segmentation performance among similar categories. Simultaneously, because the network's segmentation performance is poor at edges, this invention adds an edge optimization module to the network decoder side to improve the segmentation accuracy at object edges, and proposes a ternary loss function for edge extraction. The network of this invention effectively improves generalization performance while simultaneously enhancing segmentation performance at category edges. Attached Figure Description

[0016] Figure 1 This is a network structure diagram of the present invention.

[0017] Figure 2 The results are the detection results of this invention on a general dataset. The first column is the original image, the second column is the ground truth annotation, the third column is the baseline network effect, and the fourth column is the segmentation effect of this invention. Detailed Implementation

[0018] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0019] This embodiment is implemented using a Tesla V100 GPU and an Intel Xeon CPU E5-2680 v4 with a CUDA 11.4 backend. The semantic segmentation framework proposed in this invention is implemented in PyTorch. The image resolution is 1024×512 for both training and inference, using the Adam optimizer with an initial learning rate set to 5e. -4 The batch size was 16. Zero-mean normalization, random flipping, random scaling (between 0.8 and 1.5), and cropping were used to augment the data. The cityscapes dataset (M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Rothand B. Schiele, "The Cityscapes Dataset for Semantic Urban Scene Understanding," in 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016) was used as the training data for the network. Cityscapes is an image set containing 5000 densely labeled and 20000 coarsely labeled images, covering semantic and instance segmentation, with a resolution of 2048×1024.

[0020] like Figure 1 As shown, the specific steps of the method of the present invention are as follows:

[0021] 1) Constructing a category prototype generation module: Its structure consists of a text input Transformer (Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, PJ Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683,2019.), using a 63M parameter, 12-layer, 512-width model with 8 attention heads. For an input text, the module outputs a 768-dimensional text representation. A three-layer fully connected network is then used after the Transformer structure to reduce the dimensionality of the text representation to 64 dimensions, generating category prototypes. In this invention, the predicted category generates text prompts. For example, for the category "car", the generated text prompts are "a picture of a car", "a blurry picture of a car", "a clear picture of a car", etc. The generated text prompts are input into the category prototype generation module to build 10 category prototypes for each category, which have inter-class differences and intra-class variance.

[0022] 2) Constructing an Edge Optimization Module: The edge optimization module takes as input the features of each layer of the decoder in the backbone network. After inputting the feature map, it passes through two convolutional layers and an activation layer to obtain a one-channel object edge prediction map. The class edge map obtained from ground truth annotations is used to supervise the multi-level class edge prediction maps. This invention proposes a supervision method combining edge ternary loss and classification loss to optimize edge map supervision. In the ternary loss function for edge map optimization, the set of pixels with a confidence score higher than the threshold of 0.9 and a true value indicating an edge is considered a true example; the set of pixels with a confidence score lower than 0.5 and a true value indicating an inside-class condition is considered a false negative example; and the remaining set of pixels with a true value indicating an edge is considered the target to be optimized. In the ternary loss, the features at the locations of true examples, false negative examples, and the target to be optimized are calculated separately. A cosine metric is used to describe the distance between the target to be optimized and the true examples and false negative examples, and a cross-entropy function is used to construct and calculate the ternary loss. Based on the statistics of edge pixels in the dataset, an offset of 0.5% of the total number of pixels is added to the feature point counts of true examples, false negative examples, and the target to be optimized for faster convergence.

[0023] 3) Constructing a semantic segmentation network: The overall structure is constructed by combining the category prototype generation module and edge optimization module built in steps 1) and 2) with the backbone network. The backbone network consists of two parts: encoding and decoding. For the encoder, 10 residual blocks are used to extract a 128-dimensional feature map of 1 / 8 of the original image size. The decoder uses 6 residual blocks to decode the feature map generated by the encoder into a 64-dimensional feature map of the original input size. Each upsampled feature from the decoder and the output feature from the encoder are used as input to the edge optimization module to obtain edge prediction maps at different resolutions, and edge tripartite optimization loss is used for supervision.

[0024] 4) Network Training and Inference Process: The final 64-dimensional vector output by the decoder is used to calculate the final segmentation result with the category prototypes. During training, the category prototype generator randomly selects the mean of three category prototypes and performs a cosine similarity measurement with the pixel-by-pixel features output by the decoder. The category with the highest cosine similarity is the predicted category. During inference, the edge optimization module does not participate in the calculation. The decoder compares the pixel-by-pixel features with all category prototypes for similarity. For each pixel, the mean of the three highest similarity scores is selected as the final predicted score.

[0025] Ablation experiments were conducted on the category prototype generation module and the edge optimization module to verify their contributions to the overall structure, as shown in Table 1. For the category prototype generation module, firstly, randomized category prototypes and contrastive learning methods in semantic segmentation (Zhou, T., Wang, W., Konukoglu, E., Van Gool, L.: Rethinking semanticsegmentation: A prototype view. In: CVPR (2022)) were used for supervision, achieving better results than the baseline. Based on the category prototypes from the previous step, an edge detection head was added, and the edge detection head was supervised using traditional cross-entropy, further improving the performance. After adding the edge ternary loss function proposed in this invention to the previous step, the performance was further improved. Based on the previous step, the category prototypes were adjusted to the prototypes generated by the category prototype generator, resulting in the best segmentation performance.

[0026] Table 1 Ablation Experiment

[0027]

[0028] As shown in Table 1, the method proposed in this invention achieves 72.26 mIoU on the cityscapes validation dataset, which is 2.24 mIoU higher than the baseline network. Compared with the baseline network, the method proposed in this invention improves the prediction accuracy for most categories of interest without changing the prediction time. This means that the proposed solution is superior to existing baseline networks and can be better embedded into unmanned application platforms.

[0029] Table 3. IoU comparison between the algorithm of this invention and the baseline network.

[0030]

[0031]

[0032] Table 3 shows the segmentation metrics of the proposed method compared to the baseline network on the Cityscapes dataset validation set for all categories of interest. It is evident that the proposed segmentation method achieves significant performance improvements over the baseline in the vast majority of categories of interest. The method of this invention achieves improvements in general categories such as roads, sidewalks, buildings, vegetation, sky, and cars; and also achieves significant accuracy improvements in unique categories in intelligent driving scenarios such as walls, fences, utility poles, traffic lights, traffic signs, pedestrians, riders, trucks, buses, trains, motorcycles, and bicycles. These unique categories have less training data, more distinctive shapes, or are easily confused with other categories. For example, the proposed method achieves 72.26 mIoU on the Cityscapes validation dataset for categories easily confused with other categories, such as cars, which is 2.24 mIoU higher than the baseline network. Therefore, this invention significantly improves the handling of complex structural edges (e.g., traffic lights, traffic signs, fences) and differences between similar categories (e.g., cars and buses, trucks). Figure 2 For the first and second rows of images, the baseline network exhibited missegmentation of sub-regions for the car and wall categories, with small portions within the same object being identified as neighboring vehicle and building types. In contrast, the proposed method correctly segmented vehicle and wall categories, reducing the likelihood of misclassification of neighboring categories. In the third row of segmentation results, the proposed method, compared to the baseline network, segmented smaller categories such as utility poles and traffic signs more completely and with clearer boundaries. Therefore, visualization experiments demonstrate that the proposed method reduces missegmentation of some sub-regions compared to the baseline network, while optimizing the boundaries of smaller categories, thus improving the network's segmentation performance. Compared to the baseline network, the proposed method improves the prediction accuracy for most categories of interest without changing the prediction time. This means that the proposed solution is superior to existing baseline networks and can be better integrated into unmanned application platforms.

Claims

1. A lightweight semantic segmentation method for high-precision stereo vision perception of complex scenes, characterized in that, Specifically as follows: Step 1) constructing a category prototype generation module: the category prototype generation module uses a pre-trained CLIP text encoder as a robust category prototype feature to guide the semantic segmentation network to learn category features; the category prototype generation module structure is a text input Transformer network, which maps the text input to the feature space to generate a category prototype; before inputting the category name text into the category generation module, first construct the category name into a text prompt to reduce the network's wrong judgment of ambiguous text, and then input the text prompt into the category prototype generation module; N text prompts are constructed for the category text name, thereby constructing N category prototypes for a category, representing different forms of instances within each category, and representing the differences between different categories; Step 2) constructing an edge optimization module: the input of the edge optimization module is the decoder features of the backbone network of the semantic segmentation network, after inputting the feature map, it is sequentially passed through 2 convolution layers and activation layers to obtain an object edge prediction map of 1 channel; the class edge map obtained by the real label is used to supervise the multi-level class edge prediction map, and the edge ternary loss combined with the classification loss is used to optimize the edge map supervision; wherein, the edge ternary loss combined with the classification loss supervision method is as follows: in the optimization of the ternary loss function of the edge map, the pixel set with confidence higher than the threshold α and the true value as the edge is regarded as the true positive, the pixel set with confidence lower than β and the true value as the class inside is regarded as the false negative, and the remaining pixel set with the true value as the edge is regarded as the optimization target; in the ternary loss, the features of the true positive, false negative and optimization target positions are calculated respectively, the cosine distance is used to describe the distance between the optimization target and the true positive and false negative, and the cross entropy function is used to construct the calculation of the ternary loss; according to the statistical quantity of the edge pixels of the data set, the feature points of the true positive, false negative and optimization target are added with a bias of γ% of the total pixel number to converge more quickly; wherein, α, β and γ are parameters; the value principles of α and β are to ensure that α> β, and the number of true positive pixel points with confidence higher than α is less than that of false negative pixel points with confidence lower than β, so as to ensure the representativeness of the features of the true positive and false negative; the value of γ is less than the number of class edge pixel points in the true value mask, so as to ensure its smoothing effect and the relative stability of the calculated gradient value; Step 3) constructing a semantic segmentation network: the class prototype generation module and the edge optimization module constructed by step 1) and step 2) and the backbone network construct the overall structure of the semantic segmentation network; the backbone network is divided into two parts of encoding and decoding; for the encoder, the original image is extracted to a feature map size of 1 / 8 original size, dimensional feature map; the decoder part decodes the feature map generated by the encoder to the input original size, dimensional feature map; the feature of each upsampling of the decoder is taken as the input of the edge optimization module to obtain the edge prediction map under different resolutions, and is supervised by using the edge ternary loss. Step 4) semantic segmentation network training inference process: the final output of the decoder vector and the category prototype generated by the category prototype generation module to obtain the final segmentation result; during training, the category prototype generation module randomly selects M category prototypes and performs pixel-by-pixel cosine similarity measurement on the final output features of the decoder, and the category with the highest cosine similarity is the predicted category of the pixel; during inference, the edge optimization module does not participate in the calculation, and the decoder features are compared with all category prototypes pixel by pixel, and the highest M scores of each category are selected to calculate the mean value as the predicted score of the pixel belonging to the category; the highest category of the predicted score of each pixel point in all categories is the final predicted category, and the predicted results of all pixel points form a segmentation result image.

Images