Real-time semantic segmentation method and system for multi-shape pyramid in traffic scene

Abstract

The invention relates to the technical field of scene recognition, in particular to a real-time semantic segmentation method and system for a multi-shape pyramid in a traffic scene, and the method comprises the steps: initializing an original to-be-segmented image, and obtaining an initial segmentation feature map; channel splicing is carried out on the initial segmentation feature map and an output feature map obtained after first downsampling is carried out on the original to-be-segmented image, downsampling is carried out after a fused feature map is obtained, detail information is extracted, and a detail information feature map is obtained; down-sampling the detail information feature map and extracting semantic information; the method comprises the following steps: extracting transverse and longitudinal receptive field information of a semantic information feature map, performing expansion convolution to obtain feature maps of different receptive fields, obtaining a multi-scale and multi-shape context information feature map, performing up-sampling, performing point-by-point convolution dimension reduction on a detail information feature map, and performing channel splicing on the two features to obtain a fusion feature map containing semantic and detail information. And adjusting the channel number into a prediction category number, and normalizing to obtain a prediction segmentation result. The method has good segmentation precision and reasoning speed.

Description

Technical Field

[0001] The present invention relates to the technical field of scene recognition, and in particular to a real-time semantic segmentation method and system for multi-shape pyramids in a traffic scene. Background Art

[0002] The semantic segmentation task aims to classify each pixel in an image and assign it to a specific semantic category. Due to the high requirements for real-time performance and accuracy in traffic scenarios, traditional semantic segmentation models are difficult to meet the needs of actual engineering applications due to their huge number of parameters and slow inference speed. Therefore, it is crucial to develop a real-time semantic segmentation network that is both lightweight and highly accurate. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides a real-time semantic segmentation method of multi-shape pyramids in traffic scenarios. The semantic segmentation network of the method adopts an asymmetric encoder-decoder architecture, uses an efficient feature extraction module for detail information and semantic information, and proposes a multi-shape pyramid module in the decoder of the network to obtain semantic information of different scales and shapes to improve the network segmentation accuracy. Finally, a fusion module is used to fuse the feature maps and output the final segmentation result, thereby achieving good segmentation accuracy and reasoning speed.

[0004] Based on the same inventive concept, the present invention also provides a real-time semantic segmentation system for multi-shape pyramids in traffic scenarios.

[0005] In an embodiment of the present invention, a real-time semantic segmentation method of multi-shape pyramids in a traffic scene includes the following steps:

[0006] S1, preprocessing the original image to be segmented, initializing the preprocessed image to be segmented, and obtaining an initial segmentation feature map;

[0007] S2, down-sampling the original image to be segmented to obtain the output feature map of the first down-sampling; performing channel splicing on the initial segmentation feature map and the output feature map of the first down-sampling, and outputting the processed fusion feature map;

[0008] S3, downsampling the fused feature map obtained in step S2 to obtain a second downsampling output feature map; extracting detail information from the second downsampling output feature map to obtain a detail information feature map;

[0009] S4, downsampling the detail information feature map obtained in step S3 to obtain a third downsampling output feature map; extracting semantic information from the third downsampling output feature map to obtain a semantic information feature map;

[0010] S5, extracting the horizontal receptive field information and the vertical receptive field information in the semantic information feature map output by step S4, and obtaining feature maps containing different receptive field information through dilated convolution with different dilation coefficients, so as to obtain a multi-scale and multi-shape context information feature map;

[0011] S6, upsampling the multi-scale and multi-shape context information feature map outputted in step S5 to obtain an upsampled context information feature map; reducing the dimension of the detail information feature map outputted in step S3 by point-by-point convolution to obtain a reduced-dimensional detail information feature map; performing channel splicing on the upsampled context information feature map and the reduced-dimensional detail information feature map to obtain a fused feature map containing rich semantic information and detail information;

[0012] S7, upsample the fused feature map obtained in step S6 to the size of the original image to be segmented, use convolution operation to adjust the number of channels to the number of predicted categories, and obtain the predicted segmentation result through normalization; then use the loss function to calculate the loss value between the predicted probability and the true label, and gradually obtain the predicted segmentation result close to the true label through repeated iterative updates of the neural network.

[0013] In an embodiment of the present invention, a real-time semantic segmentation system for multi-shape pyramids in traffic scenarios includes the following modules:

[0014] An initialization module preprocesses the original image to be segmented, and initializes the preprocessed image to be segmented to obtain an initial segmentation feature map;

[0015] The first fusion module downsamples the original image to be segmented to obtain an output feature map of the first downsampling; performs channel splicing on the initial segmentation feature map and the output feature map of the first downsampling, and outputs a processed fusion feature map;

[0016] A detail information extraction module downsamples the fused feature map to obtain a second downsampled output feature map; extracts detail information from the second downsampled output feature map to obtain a detail information feature map;

[0017] The semantic information extraction module downsamples the detail information feature map to obtain a third downsampled output feature map; extracts semantic information from the third downsampled output feature map to obtain a semantic information feature map;

[0018] The multi-shape pyramid module extracts the horizontal and vertical receptive field information in the semantic information feature map, obtains feature maps containing different receptive field information through dilated convolution with different dilation coefficients, and obtains multi-scale and multi-shape contextual information feature maps;

[0019] The second fusion module upsamples the multi-scale and multi-shape context information feature map to obtain an upsampled context information feature map; reduces the dimension of the detail information feature map by point-by-point convolution to obtain a reduced-dimensional detail information feature map; and performs channel splicing on the upsampled context information feature map and the reduced-dimensional detail information feature map to obtain a fused feature map containing rich semantic information and detail information.

[0020] The semantic segmentation module upsamples the fused feature map obtained by the second fusion module to the size of the original image to be segmented, uses convolution operation to adjust the number of channels to the number of predicted categories, and obtains the predicted segmentation result through normalization; then uses the loss function to calculate the loss value between the predicted probability and the true label, and gradually obtains the predicted segmentation result close to the true label through repeated iterative updates of the neural network.

[0021] Compared with the prior art, the beneficial effects achieved by the present invention specifically include:

[0022] 1. By analyzing the characteristics of shallow detail information and deep semantic information, the present invention proposes two efficient residual feature extraction modules, which can efficiently extract detail information and semantic information respectively. Both modules use deep separable convolution to reduce calculation and improve reasoning speed. In addition, the present invention extracts detail information and semantic information separately using different residual structures, which can achieve a trade-off between segmentation accuracy and reasoning speed.

[0023] 2. The present invention adds a spatial attention module to the shallow detail extraction module to obtain the weight values ​​of different pixels, so that the model can better learn the characteristics of boundary information; and the spatial attention module only designs a small amount of parameters and calculations, which can extract more effective detail boundary information for the shallow detail extraction module.

[0024] 3. The present invention uses a multi-shape pyramid module to extract and fuse multi-scale and multi-shape features, so that the model can effectively segment objects of different scales and shapes, significantly improving the segmentation accuracy of objects of different categories. The multi-shape pyramid module uses factored dilated convolution, which only adds a small amount of computational burden to the model, and completes the extraction and fusion of multi-scale and multi-shape context information.

[0025] 4. The present invention can segment traffic road scene images more accurately and quickly, and has a wide range of application scenarios in scene recognition fields such as autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flow chart of the real-time semantic segmentation method of multi-shape pyramids in an embodiment of the present invention.

[0027] Figure 2 Schematic diagram of the detail extraction module framework in an embodiment of the present invention.

[0028] Figure 3 Schematic diagram of the framework of the semantic extraction module in an embodiment of the present invention.

[0029] Figure 4 Schematic diagram of a multi-shape pyramid module framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0031] Example 1

[0032] This embodiment provides a real-time semantic segmentation method for multi-shape pyramids in a traffic scene, and the specific implementation includes the following steps:

[0033] S1. Preprocess the original image to be segmented, and initialize the preprocessed image to be segmented to obtain an initial segmentation feature map.

[0034] In this step, preprocessing is mainly data enhancement operation, and initialization processing mainly includes dimensionality reduction and preliminary feature extraction. The initialization module can be used to reduce the resolution of the original image to be segmented and reduce the feature dimension of the original image to be segmented. Then, preliminary feature extraction is performed on the image to be segmented, and the number of channels of the feature map is expanded to obtain the initial feature map.

[0035] S2. In order to prevent the forward propagation from losing shallow information, the original image to be segmented is downsampled to 1 / 2 of the size of the original image to be segmented to obtain the output feature map of the first downsampling; the initial segmentation feature map obtained in step S1 and the output feature map of the first downsampling are channel-joined, the joined feature map is normalized, and the PReLU activation function is used to process it, and the processed fusion feature map is output.

[0036] S3: downsample the fusion feature map obtained in step S2 to 1 / 4 of the size of the original image to be segmented to obtain the second downsampled output feature map F input The output feature map of the second downsampling is input into a multi-layer cascade detail extraction module to extract detail information and obtain a detail information feature map.

[0037] The downsampling module used in this step consists of a maximum pooling with a kernel size of 2×2 and a 3×3 convolution in parallel, increasing the number of channels to 64.

[0038] Specifically, each layer of detail extraction module adopts an inverted residual structure design to efficiently extract detail boundary information; rich detail boundary information can effectively segment the boundaries between different categories in the image. Each layer of detail extraction module includes two parallel branches, that is, the detail extraction module adopts a dual-branch structure; one branch uses depthwise separable convolution; the other branch uses depthwise separable dilated convolution with a dilation rate of 2 to obtain a richer receptive field, thereby extracting detail information of multi-scale targets.

[0039] In this embodiment, taking a three-layer cascaded detail extraction module as an example, the steps included in the detail information extraction process are described as follows:

[0040] S31, the output feature map F of the second downsampling input As the input of the first-layer detail extraction module, the first-layer detail extraction module performs point-by-point convolution on the second down-sampled output feature map F input Perform channel dimension increase to obtain the dimension-added feature map F x :

[0041] F x =C 1×1 (ρ(F input )) (1)

[0042] Among them, C represents the convolution operation, ρ represents the PReLU activation function; C 1×1 It is a 1×1 convolution operation, which increases the channel dimension to 128.

[0043] S32, the dimension-upgraded feature map F x Perform channel separation to obtain two separation feature maps F of the same size x1 、F x2 ; Then the two separated feature maps are input into different parallel branches of the detail extraction module to obtain two feature maps F with different scale information y1 、F y2 :

[0044] F y1 =C 3×3 (F x1 ) (2)

[0045] F y2 =C 3×3,d (F x2 ) (3)

[0046] Where d represents the convolution expansion coefficient, C 3×3 It is a 3×3 convolution operation.

[0047] S33, the feature maps F of two different scale information y1 、F y2Channel splicing is performed, and the spatial attention mechanism is used to extract more effective detail information. The number of channels is then restored through 1×1 point-by-point convolution, and then the input of the detail extraction module (i.e., the output feature map F of the second downsampling) is fused using a jump connection. input ) to recover some of the information lost in the detail extraction process and obtain the detail information feature map F output by the first-layer detail extraction module output :

[0048] F output =C 1×1 (SA(Cat(F y1 ,F y2 )))+F input (4)

[0049] Among them, SA is the spatial attention module, which processes the feature map to enhance the feature representation capability; Cat is the channel splicing operation.

[0050] S34, the last two layers of detail extraction modules (i.e., the remaining detail extraction modules) are connected to the output of the first layer of detail extraction modules in a cascade manner, and their inputs are all detail information feature maps output by the detail extraction modules of the previous layer, i.e., the input of the second layer of detail extraction modules is the detail information feature map output by the first layer of detail extraction modules, and the input of the third layer of detail extraction modules is the detail information feature map output by the second layer of detail extraction modules, and the process of detail information extraction by each layer of detail extraction modules is the same. Through the detail extraction processing of the three-layer cascaded detail extraction modules, the detail information feature map output by the last layer of detail extraction modules is obtained.

[0051] S35, downsampling the original image to be segmented to 1 / 4 of the size of the original image to be segmented, and obtaining the original image to be segmented after four times downsampling; performing channel splicing on the detail information feature map output by the last layer detail extraction module, the output feature map of the second downsampling, and the original image to be segmented after four times downsampling, and processing the spliced ​​feature map using normalization and PReLU activation function to obtain the final detail information feature map.

[0052] S4, downsample the detail information feature map obtained in step S3 to 1 / 8 of the size of the original image to be segmented, and obtain the output feature map F of the third downsampling i ′ nput The output feature map of the third downsampling is input into a multi-layer cascade semantic extraction module to extract semantic information and obtain a semantic information feature map.

[0053] The downsampling module structure used in this step is the same as the downsampling module in step S3, including maximum pooling with a kernel size of 2×2 and 3×3 convolution in parallel, and the number of channels is increased to 128.

[0054] Specifically, each layer of the semantic extraction module adopts a residual structure design to extract rich semantic information and help the segmentation network to effectively classify pixels within the class. Rich semantic information helps to correctly classify pixels in different areas. The semantic extraction module has the advantages of small parameters and fast reasoning speed. In this embodiment, the semantic extraction module adopts a dual-branch structure, that is, it includes two parallel branches; one branch uses depthwise separable convolution, and the other branch uses dilated convolution to obtain semantic information with a larger receptive field, thereby extracting rich semantic information at multiple scales; the convolution expansion coefficient (i.e., expansion rate) of the semantic extraction modules at different layers gradually expands, which can prevent the loss of important information during the forward propagation process while obtaining semantic information with a larger receptive field.

[0055] In this embodiment, taking an eleven-layer cascaded semantic extraction module as an example, the steps included in the semantic information extraction process are described as follows:

[0056] S41, the output feature map of the third downsampling is used as the input of the first-layer semantic extraction module, and the first-layer semantic extraction module uses point-by-point convolution to obtain the convolution feature map F′ x :

[0057] F′ x =C 1×1 (ρ(F′ input )) (5)

[0058] S42, convolution feature map F' x Perform channel segmentation to obtain two feature maps F′ with the same size and half the channel size x1 , F′ x2 ; Then, the two feature maps with the same channel size halved are processed by different parallel branches of the semantic extraction module to obtain two feature maps F′ with different receptive field sizes y1 , F′ y2 :

[0059] F′ y1 =C 3×3 (F′ x1 ) (6)

[0060] F′ y2 =C 3×3,d (F′ x2 ) (7)

[0061] S43, two feature maps F′ with different receptive field sizes y1 , F′ y2 Channel splicing is performed and point-by-point convolution is used for fusion. Then, the input of the semantic extraction module is skipped to supplement the information lost during the forward propagation process, and the semantic information feature map F′ output by the first-layer semantic extraction module is obtained. output :

[0062] F′ output =C 1×1 (Cat(F′ y1 ,F′ y2 )+F′ input (8)

[0063] S44, the remaining (i.e., the last ten layers) semantic extraction modules are connected to the output of the first layer semantic extraction module in a cascade manner, and their inputs are all semantic information feature maps output by the semantic extraction modules of the previous layer, and the process of semantic information extraction by each layer of semantic extraction modules is the same. Through the semantic information extraction of the eleven layers of cascaded semantic extraction modules, the semantic information feature map output by the last layer of semantic extraction modules is obtained.

[0064] S45, downsampling the original image to be segmented to 1 / 8 of the size of the original image to be segmented, and obtaining the original image to be segmented after eight-fold downsampling; then performing channel splicing on the semantic information feature map output by the last layer of semantic extraction module, the output feature map of the third downsampling, and the original image to be segmented after eight-fold downsampling, and processing the spliced ​​feature map using normalization and PReLU activation function to obtain the final semantic information feature map.

[0065] S5. Input the semantic information feature map outputted from step S4 into a multi-shape pyramid module to obtain a multi-scale multi-shape context information feature map.

[0066] The multi-shape pyramid module can obtain feature maps of receptive fields of different scales and shapes to effectively segment objects of different scales and shapes in the image. The multi-shape pyramid module includes two cascaded parallel modules, which are used to extract horizontal receptive field information and vertical receptive field information respectively. Each parallel module has four parallel branches, which use dilated convolution to obtain feature maps containing different receptive field information, and fuse the feature maps containing different receptive field information.

[0067] In this embodiment, the process of acquiring the multi-scale and multi-shape context information feature map includes the following steps:

[0068] S51, inputting the semantic information feature map into the first parallel module, the first parallel module performs point-by-point convolution dimensionality reduction through four parallel branches, reduces the number of channels of the feature map to reduce the amount of calculation, and obtains four dimension-reduced feature maps F of the same size i :

[0069] F i =C 1×1 (F′ output ),i=1,2,3,4 (9)

[0070] S52: The reduced dimension feature map output by the first parallel branch obtained in step S51 is not processed in any way to retain the original information; the reduced dimension feature maps output by the last three parallel branches are respectively subjected to cascaded lateral factor convolutions with different expansion coefficients, for example, two cascaded 3×1 factor convolutions with input expansion coefficients of 2, 4, and 6, respectively, to obtain lateral receptive field information feature maps F′ of different scales. i :

[0071] F′ i =C 3×1,d (C 3×1,d (F i )),i=2,3,4 (10)

[0072] Finally, the multi-scale information feature maps of four parallel branches are obtained.

[0073] S53, performing channel splicing on the multi-scale information feature maps of the four parallel branches of the first parallel module processed in step S52, and then performing channel dimension reduction and feature fusion through point-by-point convolution to obtain a fused feature map F of the first parallel module fuse :

[0074] F fuse =C 1×1 (Cat(F i,i＝1 ,F′ i,i＝2 ,F′ i,i＝3 ,F′ i,i＝4 )) (11)

[0075] S54, taking the fused feature map of the first parallel module as the input of the second parallel module; in the second parallel module, the fused feature map of the first parallel module is subjected to point-by-point convolution dimensionality reduction of four parallel branches to obtain four reduced-dimensional feature maps F of the same size j :

[0076] F j =C 1×1 (F fuse ),j=1,2,3,4 (12)

[0077] S55. The dimension reduction feature map output by the first parallel branch obtained in step S54 is not processed in any way, and the original information is retained; the dimension reduction feature maps output by the last three parallel branches are respectively subjected to cascaded longitudinal factor convolutions with different expansion coefficients, for example, two cascaded 1×3 factor convolutions with input expansion rates of 2, 6, and 8, to obtain longitudinal receptive field information feature maps F′ of different scales. j :

[0078] F′ j =C 1×3,d (C 1×3,d (F j)),j=2,3,4 (13)

[0079] Finally, the multi-scale information feature maps of four parallel branches are obtained.

[0080] S56, channel-joining the multi-scale information feature maps of the four parallel branches of the second parallel module after processing in step S55, and then performing point-by-point convolution dimensionality reduction and fusion to finally obtain a feature map F with rich multi-scale and multi-shape context information. context :

[0081] F context =C 1×1 (Cat(F j,j＝1 ,F′ j,j＝2 ,F′ j,j＝3 ,F′ j,j＝4 )) (14)

[0082] S6. Upsample the multi-scale and multi-shape context information feature map output by step S5 to 1 / 4 of the size of the original image to be segmented by bilinear interpolation to obtain an upsampled context information feature map; reduce the dimension of the detail information feature map output by step S3 by point-by-point convolution to obtain a reduced-dimensional detail information feature map; then perform channel splicing on the upsampled context information feature map and the reduced-dimensional detail information feature map to finally obtain a fused feature map containing rich semantic information and detail information.

[0083] S7. Upsample the fused feature map obtained in step S6 to the size of the original image to be segmented by bilinear interpolation, adjust the number of channels to the number of predicted categories by convolution operation, and obtain the predicted segmentation result by normalization; then use the loss function to calculate the loss value between the predicted probability and the true label, and gradually obtain the predicted segmentation result close to the true label through repeated iterative updates of the neural network.

[0084] This embodiment also provides a real-time semantic segmentation system for multi-shape pyramids in traffic scenarios, including the following modules:

[0085] An initialization module preprocesses the original image to be segmented, and initializes the preprocessed image to be segmented to obtain an initial segmentation feature map;

[0086] The first fusion module downsamples the original image to be segmented to obtain an output feature map of the first downsampling; performs channel splicing on the initial segmentation feature map and the output feature map of the first downsampling, and outputs a processed fusion feature map;

[0087] A detail information extraction module downsamples the fused feature map to obtain a second downsampled output feature map; extracts detail information from the second downsampled output feature map to obtain a detail information feature map;

[0088] The semantic information extraction module downsamples the detail information feature map to obtain a third downsampled output feature map; extracts semantic information from the third downsampled output feature map to obtain a semantic information feature map;

[0089] The multi-shape pyramid module extracts the horizontal and vertical receptive field information in the semantic information feature map, obtains feature maps containing different receptive field information through dilated convolution with different dilation coefficients, and obtains multi-scale and multi-shape contextual information feature maps;

[0090] The second fusion module upsamples the multi-scale and multi-shape context information feature map to obtain an upsampled context information feature map; reduces the dimension of the detail information feature map by point-by-point convolution to obtain a reduced-dimensional detail information feature map; and performs channel splicing on the upsampled context information feature map and the reduced-dimensional detail information feature map to obtain a fused feature map containing rich semantic information and detail information.

[0091] The semantic segmentation module upsamples the fused feature map obtained by the second fusion module to the size of the original image to be segmented, uses convolution operation to adjust the number of channels to the number of predicted categories, and obtains the predicted segmentation result through normalization; then uses the loss function to calculate the loss value between the predicted probability and the true label, and gradually obtains the predicted segmentation result close to the true label through repeated iterative updates of the neural network.

[0092] In this embodiment, the output feature map of the second downsampling is passed through a continuous detail extraction module to obtain a high-resolution detail information feature map; then the high-resolution detail information feature map is passed through a continuous semantic extraction module to obtain a low-resolution semantic information feature map; and then the low-resolution semantic information feature map is passed through a multi-scale and multi-shape pyramid module to further extract a semantic information feature map with a larger and richer receptive field.

[0093] The present invention can not only achieve high-precision pixel-level classification in traffic scenarios, but also significantly improve the reasoning speed to meet real-time requirements; it greatly reduces the amount of calculation and the size of the network while improving its accuracy, thereby improving the performance of real-time semantic segmentation. Compared with the prior art, the present invention effectively processes objects of different scales and shapes by introducing a multi-shape feature pyramid module, thereby improving the ability to extract and fuse multi-scale information features. In addition, the present invention uses dilated factor convolution to design a multi-scale context module, so that the network can perceive information of different sizes, thereby improving the recognition accuracy of targets of different categories. By optimizing the network structure, the present invention achieves a good balance between reasoning speed and accuracy while reducing the amount of calculation and the amount of parameters, and is very suitable for high-real-time and high-precision application scenarios such as autonomous driving and traffic monitoring.

[0094] The above is only the better / preferred implementation method of the invention of the present invention, but the protection scope of the invention patent is not limited to this. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principle of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A real-time semantic segmentation method for multi-shape pyramids in traffic scenes, characterized in that: The following steps are involved: S1, preprocessing the original image to be segmented, initializing the preprocessed image to be segmented, and obtaining an initial segmentation feature map; S2, down-sampling the original image to be segmented to obtain the output feature map of the first down-sampling; performing channel splicing on the initial segmentation feature map and the output feature map of the first down-sampling, and outputting the processed fusion feature map; S3, downsampling the fused feature map obtained in step S2 to obtain a second downsampling output feature map; extracting detail information from the second downsampling output feature map to obtain a detail information feature map; S4, downsampling the detail information feature map obtained in step S3 to obtain a third downsampling output feature map; extracting semantic information from the third downsampling output feature map to obtain a semantic information feature map; S5, extracting the horizontal receptive field information and the vertical receptive field information in the semantic information feature map output by step S4, and obtaining feature maps containing different receptive field information through dilated convolution with different dilation coefficients, so as to obtain a multi-scale and multi-shape context information feature map; S6, upsampling the multi-scale and multi-shape context information feature map outputted in step S5 to obtain an upsampled context information feature map; reducing the dimension of the detail information feature map outputted in step S3 by point-by-point convolution to obtain a reduced-dimensional detail information feature map; performing channel splicing on the upsampled context information feature map and the reduced-dimensional detail information feature map to obtain a fused feature map containing rich semantic information and detail information; S7, upsample the fused feature map obtained in step S6 to the size of the original image to be segmented, use convolution operation to adjust the number of channels to the number of predicted categories, and obtain the predicted segmentation result through normalization; then use the loss function to calculate the loss value between the predicted probability and the true label, and gradually obtain the predicted segmentation result close to the true label through repeated iterative updates of the neural network.

2. The real-time semantic segmentation method according to claim 1, characterized in that: In step S3, a multi-layer detail extraction module is used to extract detail information. Each layer of detail extraction module is designed with an inverted residual structure to extract detail boundary information so as to effectively segment the boundaries between different categories in the image according to the detail boundary information.

3. The real-time semantic segmentation method according to claim 1 or 2, characterized in that: Step S3 uses a multi-layer detail extraction module to extract detail information. Each layer of the detail extraction module includes two parallel branches, one branch uses depthwise separable convolution; the other branch uses depthwise separable dilated convolution to obtain a richer receptive field.

4. The real-time semantic segmentation method according to claim 3, characterized in that: The detailed information extraction process in step S3 includes: S31, using the output feature map of the second downsampling as the input of the first-layer detail extraction module, and the first-layer detail extraction module performs channel dimension increase on the output feature map of the second downsampling through point-by-point convolution to obtain a dimension increase feature map; S32, performing channel separation on the dimension-increased feature map to obtain two separated feature maps of the same size; and then inputting the two separated feature maps into different parallel branches of the detail extraction module respectively to obtain two feature maps of different scale information; S33, concatenating the channels of the feature maps of two different scale information, and using the spatial attention mechanism to extract more effective detail information, then restoring the number of channels through point-by-point convolution, and then using the jump connection to fuse the input of the detail extraction module to restore part of the information lost during the detail extraction process, and obtaining the detail information feature map output by the first-layer detail extraction module; S34, the remaining detail extraction modules are connected to the output of the first layer detail extraction module in a cascade manner, and their inputs are all detail information feature maps output by the detail extraction modules of the previous layer, and the detail extraction process of each layer of detail extraction modules is the same; through the detail extraction processing of the multi-layer cascaded detail extraction modules, the detail information feature map output by the last layer of detail extraction modules is obtained; S35, downsampling the original image to be segmented to obtain the original image to be segmented after four times downsampling; channel-joining the detail information feature map output by the last layer detail extraction module, the output feature map of the second downsampling, and the original image to be segmented after four times downsampling to obtain the final detail information feature map.

5. The real-time semantic segmentation method according to claim 1, characterized in that: Step S4 uses multi-layer cascaded semantic extraction modules to extract semantic information; each layer of the semantic extraction module adopts a residual structure design, including two parallel branches, one branch uses depthwise separable convolution, and the other branch uses dilated convolution to obtain semantic information with a larger receptive field; the convolution dilation coefficients of semantic extraction modules in different layers are gradually expanded.

6. The real-time semantic segmentation method according to claim 5, characterized in that: The process of extracting semantic information in step S4 includes: S41, using the output feature map of the third downsampling as the input of the first-layer semantic extraction module, and the first-layer semantic extraction module uses point-by-point convolution to obtain a convolution feature map; S42, performing channel segmentation on the convolution feature map to obtain two feature maps with the same size and the channel halved; then processing the two feature maps with the same size and the channel halved through different parallel branches of the semantic extraction module to obtain two feature maps with different receptive field sizes; S43, channel-joining the two feature maps with different receptive field sizes, fusing them using point-by-point convolution, and then fusing the input of the semantic extraction module through a jump connection to supplement the information lost during the forward propagation process, thereby obtaining a semantic information feature map output by the first-layer semantic extraction module; S44, the remaining semantic extraction modules are connected to the output of the first layer semantic extraction module in a cascade manner, and their inputs are all semantic information feature maps output by the semantic extraction modules of the previous layer, and the process of semantic information extraction of each layer of semantic extraction modules is the same; through the semantic information extraction of multiple layers of cascaded semantic extraction modules, the semantic information feature map output by the semantic extraction module of the last layer is obtained; S45, downsampling the original image to be segmented to obtain the original image to be segmented after eight-fold downsampling; then channel-joining the semantic information feature map output by the last layer of semantic extraction module, the output feature map of the third downsampling, and the original image to be segmented after eight-fold downsampling to obtain the final semantic information feature map.

7. The real-time semantic segmentation method according to claim 1, characterized in that: Step S5 obtains a multi-scale and multi-shape context information feature map through a multi-shape pyramid module; the multi-shape pyramid module includes two cascaded parallel modules, which are used to extract horizontal receptive field information and vertical receptive field information respectively; each parallel module has four parallel branches, and feature maps of different receptive fields are obtained by using dilated convolution, and the feature maps of different receptive fields are fused.

8. The real-time semantic segmentation method according to claim 7, characterized in that: The process of acquiring the multi-scale and multi-shape context information feature map in step S5 includes the following steps: S51, inputting the semantic information feature map into the first parallel module, and the first parallel module performs point-by-point convolution dimensionality reduction through four parallel branches to obtain four reduced-dimensional feature maps of the same size; S52: The dimension reduction feature map output by the first parallel branch obtained in step S51 is not processed in any way, and the dimension reduction feature maps output by the last three parallel branches are respectively subjected to cascaded lateral factor convolutions with different expansion coefficients to obtain lateral receptive field information feature maps of different scales; S53, performing channel splicing on the multi-scale information feature maps of the four parallel branches of the first parallel module processed in step S52, and then performing channel dimension reduction and feature fusion through point-by-point convolution to obtain a fused feature map of the first parallel module; S54, using the fused feature map of the first parallel module as the input of the second parallel module; in the second parallel module, the fused feature map of the first parallel module is subjected to point-by-point convolution dimensionality reduction of four parallel branches to obtain four reduced-dimensional feature maps of the same size; S55: The dimension reduction feature map output by the first parallel branch obtained in step S54 is not processed in any way, and the dimension reduction feature maps output by the last three parallel branches are respectively subjected to cascaded longitudinal factor convolutions with different expansion coefficients to obtain longitudinal receptive field information feature maps of different scales; and finally, multi-scale information feature maps of four parallel branches are obtained; S56, performing channel splicing on the multi-scale information feature maps of the four parallel branches of the second parallel module processed in step S55, and then performing dimensionality reduction and fusion through point-by-point convolution, so as to finally obtain a feature map with rich multi-scale and multi-shape contextual information.

9. A real-time semantic segmentation system for multi-shape pyramids in traffic scenes, characterized in that: Includes the following modules: An initialization module preprocesses the original image to be segmented, and initializes the preprocessed image to be segmented to obtain an initial segmentation feature map; The first fusion module downsamples the original image to be segmented to obtain an output feature map of the first downsampling; performs channel splicing on the initial segmentation feature map and the output feature map of the first downsampling, and outputs a processed fusion feature map; A detail information extraction module downsamples the fused feature map to obtain a second downsampled output feature map; extracts detail information from the second downsampled output feature map to obtain a detail information feature map; The semantic information extraction module downsamples the detail information feature map to obtain a third downsampled output feature map; extracts semantic information from the third downsampled output feature map to obtain a semantic information feature map; The multi-shape pyramid module extracts the horizontal and vertical receptive field information in the semantic information feature map, obtains feature maps containing different receptive field information through dilated convolution with different dilation coefficients, and obtains multi-scale and multi-shape contextual information feature maps; The second fusion module upsamples the multi-scale and multi-shape context information feature map to obtain an upsampled context information feature map; The detail information feature map is reduced in dimension by point-by-point convolution to obtain a reduced-dimensional detail information feature map; the upsampled context information feature map and the reduced-dimensional detail information feature map are channel-joined to obtain a fused feature map containing rich semantic information and detail information; The semantic segmentation module upsamples the fused feature map obtained by the second fusion module to the size of the original image to be segmented, uses convolution operation to adjust the number of channels to the number of predicted categories, and obtains the predicted segmentation result through normalization; then uses the loss function to calculate the loss value between the predicted probability and the true label, and gradually obtains the predicted segmentation result close to the true label through repeated iterative updates of the neural network.

10. The real-time semantic segmentation system according to claim 9, characterized in that: The multi-shape pyramid module includes two cascaded parallel modules, which are used to extract horizontal receptive field information and vertical receptive field information respectively; each parallel module has four parallel branches, which use dilated convolution to obtain feature maps of different receptive fields and fuse the feature maps of different receptive fields.

Images