A new fast semantic segmentation method, system and storage medium for unmanned bus systems

Abstract

The present invention relates to a novel method, system, and storage medium for rapid semantic segmentation of an unmanned public transportation system. The method comprises: M1. When an unmanned public transportation system is traveling on a road, it acquires road image data information in real time using an onboard camera, inputs the road image data information into the HRNetv2-W18 backbone network for downsampling feature extraction of different sizes, and outputs a group of downsampled feature maps of the road image; M2. Inputs the group of downsampled feature maps of the road image into an OCR network for contextual target pixel feature enhancement, and outputs the enhanced road image data information; M3. Inputs the enhanced road image data information into a CBAM network for channeling. The present invention not only achieves high-precision extraction of targets at multiple scales, utilizing a new context extraction module to enhance the representation capability of different pixel features, but also utilizes an attention mechanism to improve the feature extraction performance of features at different scales.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned public transportation, and in particular to a novel fast semantic segmentation method, system and storage medium for an unmanned public transportation system. Background Art

[0002] Semantic segmentation technology plays a crucial role in autonomous driving in urban traffic scenarios. Currently, semantic segmentation networks that achieve good segmentation results have relatively high accuracy. However, these neural network algorithms have a large number of model parameters and computational complexity, requiring significant computing power within the vehicle system. This leads to high computational resource consumption, high costs, and low energy efficiency. To ensure that vehicle systems can achieve real-time semantic object extraction in traffic environments, lightweightness and real-time performance are crucial evaluation factors, in addition to segmentation accuracy. To achieve targeted feature extraction modules for semantic segmentation, these modules are used to construct a feature extraction backbone network. The integration of multi-scale groups, as well as multi-scale and contextual features, enhances feature representation capabilities at each scale.

[0003] Prior art 1, patent (CN111275711B), discloses a real-time image semantic segmentation algorithm based on a lightweight convolutional neural network model. The algorithm involves: 1) downloading the pre-trained ESPNETv2 general network module from the open-source GitHub website; 2) designing a shared link adaptive unit based on left and right results; 3) building a lightweight convolutional neural network module; 4) training the lightweight neural network using the Cityscapes training and validation sets to obtain a well-circulated real-time image semantic segmentation model; 5) inputting the Cityscapes test set into the trained real-time image semantic segmentation model to obtain image semantic segmentation results. However, ESPNetv2 is a lightweight, energy-efficient, and general-purpose convolutional neural network. It uses grouped convolution kernels and deep dilated convolutions to learn a large effective receptive field, requiring fewer floating-point computations and parameters. Even with the shared link adaptive unit, the extraction accuracy evaluation metric (miou) remains relatively low, and object segmentation can be blurred in low-resolution videos.

[0004] Prior art 2, patent (CN115965779A) discloses a semantic segmentation method, device and equipment for vehicle-mounted cameras, 1) obtaining a first target image through a vehicle-mounted camera with a first perspective on a target vehicle, and obtaining a second target image through a vehicle-mounted camera with a second perspective on the target vehicle, wherein the first perspective and the second perspective are different; 2) performing perspective conversion on the first target image to obtain a first target image after the second perspective conversion, and performing perspective conversion on the second target image to obtain a second target image after the first perspective conversion; 3) using optical flow transformation to compensate for the difference between the first target image after the second perspective conversion and the second target image. The method comprises the following steps: 1) performing semantic fusion on the feature map of the compensated first target image, 2) performing semantic fusion on the feature map of the first target image, and 3) performing semantic fusion on the feature map of the second target image. 4) performing semantic fusion on the feature map of the compensated first target image, the feature map of the first target image, and the feature map of the second target image, and determining the semantic segmentation result of the second target image based on the obtained fusion result. 5) performing semantic fusion on the feature map of the compensated second target image, the feature map of the second target image, and the feature map of the first target image, and determining the semantic segmentation result of the first target image based on the obtained fusion result. However, although this method improves the depth of semantic sharing and fusion between different cameras, realizes the complementary advantages of semantic segmentation results of different cameras, and improves the accuracy and real-time performance of semantic segmentation results to a certain extent, the generalization ability of this method in different scenarios is still relatively weak. It requires a large amount of data for training to improve the accuracy of its results. At the same time, its real-time performance is also poor on the vehicle side with weaker computing power. Summary of the Invention

[0005] In view of the above shortcomings of the existing technology, the present invention provides a new method, system and storage medium for rapid semantic segmentation of unmanned bus systems, which not only achieves high-precision extraction of targets at multiple scales, but also utilizes a new context extraction module to enhance the representation capability of different pixel features, and utilizes the attention mechanism to improve the feature extraction performance at different scales.

[0006] In order to achieve the above-mentioned and other related purposes, the present invention provides the following technical solutions:

[0007] A new fast semantic segmentation method for unmanned bus systems, comprising:

[0008] M1. An unmanned bus, while traveling on a road, acquires road image data in real time using an onboard camera. This image data is then fed into the HRNetv2-W18 backbone network for downsampled feature extraction at different scales, outputting a set of downsampled feature maps of the road image.

[0009] M2. The downsampled feature map group of the road image is input into the OCR network for contextual target pixel feature enhancement, and the enhanced road image data information is output;

[0010] M3. The enhanced road image data information is input into the CBAM network for training in two independent dimensions of channel and space, and outputs the attention map data information of the road image;

[0011] M4. Multiplying the attention map data of the road image by the enhanced road image data and performing adaptive feature modification to output comprehensive road feature image data;

[0012] M5. Based on the comprehensive road feature image data information, use a data augmentation algorithm and a data post-processing algorithm to perform image classification processing and output road image semantic segmentation data information.

[0013] Furthermore, in step M5, the data post-processing algorithm includes:

[0014] M51. Based on the comprehensive characteristic image data information of the road, different categories of targets are separated according to different pixel values, and the target image data information is output;

[0015] M52. Based on the target image data information, remove noise using morphological open-loop and closed-loop operations, and output the target image data information after removing noise;

[0016] M53. Based on the target image data information after removing noise, the target image data information with high confidence is obtained by arranging the contour area;

[0017] M54. Arrange the target image data information with high confidence at equal intervals and output the road image semantic segmentation data information.

[0018] Furthermore, in step M5, the data augmentation algorithm sequentially performs image mirroring, color space transformation, image cropping and scaling, image linear transformation, image cropping, and image aliasing processing on the road comprehensive feature image data information.

[0019] Furthermore, in step M1, the input HRNetv2-W18 backbone network is used to extract downsampled features of different sizes, including:

[0020] M11. Input the road image data information into the first module of the HRNetv2-W18 backbone network and output a 4x downsampled feature map and an 8x downsampled feature map, i.e., the first image group;

[0021] M12. Input the first image group into the second module of the HRNetv2-W18 backbone network and output the feature maps with a scale of 4 times downsampling, 8 times downsampling, and 16 times downsampling, i.e., the second image group;

[0022] M13. Input the second image group into the third module of the HRNetv2-W18 backbone network, and output the feature maps with a scale of 4 times downsampling, 8 times downsampling, 16 times downsampling, and 32 times downsampling, i.e., the third image group;

[0023] M14. After inputting the third image group into the fourth module of the HRNetv2-W18 backbone network, the output scales are 4x downsampled feature maps, 8x downsampled feature maps, 16x downsampled feature maps, and 32x downsampled feature maps, that is, the downsampled feature map group of the road image.

[0024] Furthermore, the first module of the HRNetv2-W18 backbone network includes multiple multi-scale group convolutions, the second module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions, the third module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions, and the fourth module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions.

[0025] Furthermore, the CBAM network includes a global maximum pooling layer and a global average pooling layer.

[0026] Furthermore, in step M2, the OCR network performs context target pixel feature enhancement including:

[0027] M21. Divide the context pixels of the downsampled feature map group of the road image into a group of soft target regions, where each channel corresponds to a category segmentation result;

[0028] M22. Aggregate the set of soft target regions and output a feature representation of the target region;

[0029] M23. Based on the feature representation of the target region, the feature representation of each pixel of the target region is enhanced using the target context representation.

[0030] In order to achieve the above-mentioned and other related objectives, the present invention further provides a new unmanned bus system rapid semantic segmentation system, the system comprising:

[0031] The HRNetv2-W18 backbone network system includes the first module, the second module, the third module, and the fourth module, which are used to extract features of image data information by downsampling at different sizes;

[0032] OCR network system, used to enhance the contextual target pixel features of the downsampled features of the image;

[0033] The CBAM network system is used to train two independent dimensions, channel and space, and output attention map data information of road images;

[0034] The image data augmentation and post-processing system uses data augmentation algorithm and data post-processing algorithm to perform image classification processing.

[0035] Furthermore, the HRNetv2-W18 backbone network system is connected to the OCR network system, the OCR network system is connected to the CBAM network system, and the CBAM network system is connected to the image data augmentation and post-processing system.

[0036] In order to achieve the above-mentioned objectives and other related objectives, the present invention also provides a computer-readable storage medium, which stores a computer program programmed or configured to execute any one of the new unmanned bus system fast semantic segmentation methods.

[0037] The present invention has the following positive effects:

[0038] 1. To address the issue of ineffective balance between processing speed and extraction accuracy in semantic perception networks on the vehicle side, this paper utilizes a multi-scale convolution module to achieve high-precision extraction of targets at various scales. It also utilizes a new context extraction module to enhance the ability to represent different pixel features. Furthermore, it utilizes an attention mechanism to improve feature extraction performance at different scales.

[0039] 2. Since the present invention increases the dimension of feature extraction, the factor extraction efficiency of the present invention can be improved by a small number of network layers of the HRnetV2 network, thus solving the problem of balancing target extraction efficiency and accuracy on the vehicle side. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the process of the present invention;

[0041] Figure 2 Schematic diagram of the data post-processing algorithm flow of the present invention. DETAILED DESCRIPTION

[0042] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] Example 1: Figure 1 As shown, a new fast semantic segmentation method for unmanned bus system is provided, the method comprising:

[0044] M1. An unmanned bus, while traveling on a road, acquires road image data in real time using an onboard camera. This image data is then fed into the HRNetv2-W18 backbone network for downsampled feature extraction at different scales, outputting a set of downsampled feature maps of the road image.

[0045] M2. The downsampled feature map group of the road image is input into the OCR network for contextual target pixel feature enhancement, and the enhanced road image data information is output;

[0046] M3. The enhanced road image data information is input into the CBAM network for training in two independent dimensions of channel and space, and outputs the attention map data information of the road image;

[0047] M4. Multiplying the attention map data of the road image by the enhanced road image data and performing adaptive feature modification to output comprehensive road feature image data;

[0048] M5. Based on the comprehensive road feature image data information, use a data augmentation algorithm and a data post-processing algorithm to perform image classification processing and output road image semantic segmentation data information.

[0049] In this embodiment, if Figure 2 As shown, in step M5, the data post-processing algorithm includes:

[0050] M51. Based on the comprehensive characteristic image data information of the road, different categories of targets are separated according to different pixel values, and the target image data information is output;

[0051] M52. Based on the target image data information, remove noise using morphological open-loop and closed-loop operations, and output the target image data information after removing noise;

[0052] M53. Based on the target image data information after removing noise, the target image data information with high confidence is obtained by arranging the contour area;

[0053] M54. Arrange the target image data information with high confidence at equal intervals and output the road image semantic segmentation data information.

[0054] In this embodiment, in step M5, the data augmentation algorithm sequentially performs image mirroring, color space transformation, image cropping and scaling, image linear transformation, image cropping, and image aliasing processing on the comprehensive road feature image data information.

[0055] In this embodiment, in step M1, the input HRNetv2-W18 backbone network is used to extract downsampled features of different sizes, including:

[0056] M11. Input the road image data information into the first module of the HRNetv2-W18 backbone network and output a 4x downsampled feature map and an 8x downsampled feature map, i.e., the first image group;

[0057] M12. Input the first image group into the second module of the HRNetv2-W18 backbone network and output the feature maps with a scale of 4 times downsampling, 8 times downsampling, and 16 times downsampling, i.e., the second image group;

[0058] M13. Input the second image group into the third module of the HRNetv2-W18 backbone network, and output the feature maps with a scale of 4 times downsampling, 8 times downsampling, 16 times downsampling, and 32 times downsampling, i.e., the third image group;

[0059] M14. After inputting the third image group into the fourth module of the HRNetv2-W18 backbone network, the output scales are 4x downsampled feature maps, 8x downsampled feature maps, 16x downsampled feature maps, and 32x downsampled feature maps, that is, the downsampled feature map group of the road image.

[0060] In this embodiment, the first module of the HRNetv2-W18 backbone network includes multiple multi-scale group convolutions, the second module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions, the third module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions, and the fourth module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions.

[0061] Example 2: Based on the novel fast semantic segmentation method for unmanned public transportation system in Example 1, the present invention is further illustrated and described below.

[0062] The CBAM network includes a global maximum pooling layer and a global average pooling layer.

[0063] In this embodiment, the OCR network performs context target pixel feature enhancement including:

[0064] M21. Divide the context pixels of the downsampled feature map group of the road image into a group of soft target regions, where each channel corresponds to a category segmentation result;

[0065] M22. Aggregate the set of soft target regions and output a feature representation of the target region;

[0066] M23. Based on the feature representation of the target region, the feature representation of each pixel of the target region is enhanced using the target context representation.

[0067] The present invention also provides a new unmanned bus system rapid semantic segmentation system, the system comprising:

[0068] The HRNetv2-W18 backbone network system includes the first module, the second module, the third module, and the fourth module, which are used to extract features of image data information by downsampling at different sizes;

[0069] OCR network system, used to enhance the contextual target pixel features of the downsampled features of the image;

[0070] The CBAM network system is used to train two independent dimensions, channel and space, and output attention map data information of road images;

[0071] The image data augmentation and post-processing system uses data augmentation algorithm and data post-processing algorithm to perform image classification processing.

[0072] In this embodiment, the HRNetv2-W18 backbone network system is connected to the OCR network system, the OCR network system is connected to the CBAM network system, and the CBAM network system is connected to the image data augmentation and post-processing system.

[0073] In order to achieve the above-mentioned objectives and other related objectives, the present invention also provides a computer-readable storage medium, which stores a computer program programmed or configured to execute any one of the new unmanned bus system fast semantic segmentation methods.

[0074] Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0075] In summary, the present invention not only achieves high-precision extraction of targets at various scales, utilizes a new context extraction module to enhance the representation capability of different pixel features, but also utilizes the attention mechanism to improve the feature extraction performance at different scales.

[0076] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A new fast semantic segmentation method for unmanned bus systems, characterized by: The method comprises: M1. An unmanned bus, while traveling on a road, acquires road image data in real time using an onboard camera. This image data is then fed into the HRNetv2-W18 backbone network for downsampled feature extraction at different scales, outputting a set of downsampled feature maps of the road image. M2. The downsampled feature map group of the road image is input into the OCR network for contextual target pixel feature enhancement, and the enhanced road image data information is output; M3. The enhanced road image data information is input into the CBAM network for training in two independent dimensions of channel and space, and outputs the attention map data information of the road image; M4. Multiplying the attention map data of the road image by the enhanced road image data and performing adaptive feature modification to output comprehensive road feature image data; M5. Based on the comprehensive feature image data information of the road, the image classification processing is performed using the data augmentation algorithm and the data post-processing algorithm to output the semantic segmentation data information of the road image; In step M5, the data post-processing algorithm includes: M51. Based on the comprehensive characteristic image data information of the road, different categories of targets are separated according to different pixel values, and the target image data information is output; M52. Based on the target image data information, remove noise using morphological open-loop and closed-loop operations, and output the target image data information after removing noise; M53. Based on the target image data information after removing noise, the target image data information with high confidence is obtained by arranging the contour area; M54. Arrange the target image data information with high confidence at equal intervals and output the semantic segmentation data information of the road image; In step M1, the input HRNetv2-W18 backbone network is used to extract downsampled features of different sizes, including: M11. Input the road image data information into the first module of the HRNetv2-W18 backbone network and output a 4x downsampled feature map and an 8x downsampled feature map, i.e., the first image group; M12. Input the first image group into the second module of the HRNetv2-W18 backbone network and output the feature maps with a scale of 4 times downsampling, 8 times downsampling, and 16 times downsampling, i.e., the second image group; M13. Input the second image group into the third module of the HRNetv2-W18 backbone network, and output the feature maps with a scale of 4 times downsampling, 8 times downsampling, 16 times downsampling, and 32 times downsampling, i.e., the third image group; M14. After inputting the third image group into the fourth module of the HRNetv2-W18 backbone network, the output scales are 4x down-sampled feature maps, 8x down-sampled feature maps, 16x down-sampled feature maps, and 32x down-sampled feature maps, that is, a down-sampled feature map group of the road image; the first module of the HRNetv2-W18 backbone network includes multiple multi-scale group convolutions, the second module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions, the third module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions, and the fourth module of the HRNetv2-W18 backbone network includes multi-scale group convolutions and multi-scale convolutions.

2. The novel fast semantic segmentation method for unmanned public transportation system according to claim 1 is characterized in that: In step M5, the data augmentation algorithm sequentially processes the road comprehensive feature image data information using image mirroring, color space transformation, image cropping and scaling, image linear transformation, image cropping, and image aliasing.

3. The novel fast semantic segmentation method for unmanned public transportation system according to claim 1 is characterized by: The CBAM network includes a global maximum pooling layer and a global average pooling layer.

4. The novel fast semantic segmentation method for unmanned public transportation system according to claim 1 is characterized in that: In step M2, the OCR network performs context target pixel feature enhancement, including: M21. Divide the context pixels of the downsampled feature map group of the road image into a group of soft target regions, where each channel corresponds to a category segmentation result; M22. Aggregate the set of soft target regions and output a feature representation of the target region; M23. Based on the feature representation of the target region, the feature representation of each pixel of the target region is enhanced using the target context representation.

5. A new fast semantic segmentation system for unmanned bus systems, characterized by: A method for implementing the new unmanned public transportation system rapid semantic segmentation method according to any one of claims 1 to 4, the system comprising: The HRNetv2-W18 backbone network system includes the first module, the second module, the third module, and the fourth module, which are used to extract features of image data information by downsampling at different sizes; OCR network system, used to enhance the contextual target pixel features of the downsampled features of the image; The CBAM network system is used to train two independent dimensions, channel and space, and output attention map data information of road images; The image data augmentation and post-processing system uses data augmentation algorithm and data post-processing algorithm to perform image classification processing.

6. The novel fast semantic segmentation system for unmanned public transportation system according to claim 5 is characterized by: The HRNetv2-W18 backbone network system is connected to the OCR network system, the OCR network system is connected to the CBAM network system, and the CBAM network system is connected to the image data augmentation and post-processing system.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program that is programmed or configured to execute the new unmanned public transportation system fast semantic segmentation method according to any one of claims 1 to 4.

Images