A real-time semantic segmentation method and system for multi-shape pyramids in traffic scenarios

Abstract

The present invention relates to the field of scene recognition technology, and is a real-time semantic segmentation method and system for multi-shape pyramids in traffic scenarios. The method comprises the following steps: initializing an original image to be segmented to obtain an initial segmentation feature map; performing channel splicing on the initial segmentation feature map and an output feature map obtained by first downsampling the original image to be segmented to obtain a fused feature map, then downsampling and extracting detail information to obtain a detail information feature map; downsampling the detail information feature map and extracting semantic information; extracting horizontal and vertical receptive field information from the semantic information feature map, performing dilated convolution to obtain feature maps with different receptive fields, obtaining multi-scale and multi-shape contextual information feature maps, then upsampling them, performing point-by-point convolution on the detail information feature map to reduce dimensionality, performing channel splicing on the two to obtain a fused feature map containing semantic and detail information, adjusting the number of channels to the number of predicted categories, and normalizing to obtain a predicted segmentation result. The method has good segmentation accuracy and inference speed.

Description

Technical Field

[0001] The present invention relates to the field of scene recognition technology, and in particular to a real-time semantic segmentation method and system for multi-shape pyramids in traffic scenes. Background Art

[0002] Semantic segmentation aims to classify each pixel in an image and assign it to a specific semantic category. Due to the high real-time and precision requirements in traffic scenarios, traditional semantic segmentation models, with their large number of parameters and slow inference speed, are unable to meet the demands of practical engineering applications. Therefore, developing a lightweight and highly accurate real-time semantic segmentation network is crucial. Summary of the Invention

[0003] In response to the problems existing in the prior art, the present invention provides a real-time semantic segmentation method for multi-shape pyramids in traffic scenarios. The semantic segmentation network of this method adopts an asymmetric encoder-decoder architecture and uses an efficient feature extraction module for detail information and semantic information. A multi-shape pyramid module is proposed in the decoder of the network to obtain semantic information of different scales and shapes to improve the network segmentation accuracy. Finally, a fusion module is used to fuse the feature maps and output the final segmentation result, achieving good segmentation accuracy and inference speed.

[0004] Based on the same inventive concept, the present invention also provides a real-time semantic segmentation system for multi-shape pyramids in traffic scenarios.

[0005] In an embodiment of the present invention, a real-time semantic segmentation method for multi-shape pyramids in a traffic scene includes the following steps:

[0006] S1. Preprocess the original image to be segmented, and initialize the preprocessed image to be segmented to obtain an initial segmentation feature map;

[0007] S2. Downsample the original image to be segmented to obtain the output feature map of the first downsampling; perform channel splicing on the initial segmentation feature map and the output feature map of the first downsampling, and output the processed fusion feature map;

[0008] S3, downsampling the fusion feature map obtained in step S2 to obtain a second downsampling output feature map; extracting detail information from the second downsampling output feature map to obtain a detail information feature map;

[0009] S4, downsampling the detail information feature map obtained in step S3 to obtain a third downsampling output feature map; extracting semantic information from the third downsampling output feature map to obtain a semantic information feature map;

[0010] S5. Extracting the horizontal receptive field information and the vertical receptive field information from the semantic information feature map output in step S4, and obtaining feature maps containing different receptive field information through dilated convolution with different dilation coefficients to obtain a multi-scale and multi-shape contextual information feature map;

[0011] S6. Upsampling the multi-scale and multi-shape context information feature map output in step S5 to obtain an upsampled context information feature map; performing point-by-point convolution on the detail information feature map output in step S3 to obtain a reduced-dimensional detail information feature map; performing channel concatenation on the upsampled context information feature map and the reduced-dimensional detail information feature map to obtain a fused feature map containing rich semantic information and detail information;

[0012] S7. Upsample the fused feature map obtained in step S6 to the size of the original image to be segmented, use convolution operation to adjust the number of channels to the number of predicted categories, and obtain the predicted segmentation result through normalization; then use the loss function to calculate the loss value between the predicted probability and the true label, and through repeated iterative updates of the neural network, gradually obtain the predicted segmentation result close to the true label.

[0013] In an embodiment of the present invention, a real-time semantic segmentation system for multi-shape pyramids in traffic scenarios includes the following modules:

[0014] The initialization module preprocesses the original image to be segmented, and initializes the preprocessed image to be segmented to obtain an initial segmentation feature map;

[0015] The first fusion module downsamples the original image to be segmented to obtain the output feature map of the first downsampling; performs channel splicing on the initial segmentation feature map and the output feature map of the first downsampling, and outputs the processed fusion feature map;

[0016] A detail information extraction module downsamples the fused feature map to obtain a second downsampled output feature map; extracts detail information from the second downsampled output feature map to obtain a detail information feature map;

[0017] A semantic information extraction module downsamples the detail information feature map to obtain a third downsampled output feature map; and extracts semantic information from the third downsampled output feature map to obtain a semantic information feature map.

[0018] The multi-shape pyramid module extracts the horizontal and vertical receptive field information from the semantic information feature map, and obtains feature maps containing different receptive field information through dilated convolution with different dilation coefficients to obtain multi-scale and multi-shape contextual information feature maps.

[0019] The second fusion module upsamples the multi-scale and multi-shape context information feature map to obtain an upsampled context information feature map; reduces the dimension of the detail information feature map through point-by-point convolution to obtain a reduced dimension detail information feature map; and performs channel splicing on the upsampled context information feature map and the reduced dimension detail information feature map to obtain a fused feature map containing rich semantic information and detail information.

[0020] The semantic segmentation module upsamples the fused feature map obtained by the second fusion module to the size of the original image to be segmented, uses convolution operations to adjust the number of channels to the number of predicted categories, and obtains the predicted segmentation result through normalization; then uses the loss function to calculate the loss value between the predicted probability and the true label, and through repeated iterative updates of the neural network, gradually obtains the predicted segmentation result close to the true label.

[0021] Compared with the prior art, the beneficial effects achieved by the present invention specifically include:

[0022] 1. By analyzing the characteristics of shallow detail information and deep semantic information, this paper proposes two efficient residual feature extraction modules, which can effectively extract detail information and semantic information respectively. Both modules use depthwise separable convolution to reduce computation and improve inference speed. In addition, this paper extracts detail information and semantic information using different residual structures, achieving a trade-off between segmentation accuracy and inference speed.

[0023] 2. The present invention adds a spatial attention module to the shallow detail extraction module to obtain the weight values ​​of different pixels, so that the model can better learn the characteristics of boundary information; and the spatial attention module only designs a small amount of parameters and calculations, which can extract more effective detail boundary information for the shallow detail extraction module.

[0024] 3. This paper uses a multi-shape pyramid module to extract and fuse multi-scale and multi-shape features, enabling the model to effectively segment objects of different scales and shapes, significantly improving the segmentation accuracy of objects of different categories. The multi-shape pyramid module uses factored dilated convolutions, adding only a small computational burden to the model while extracting and fusing multi-scale and multi-shape contextual information.

[0025] 4. The present invention can segment traffic road scene images more accurately and quickly, and has great application scenarios in scene recognition fields such as autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of the real-time semantic segmentation method of multi-shape pyramids in an embodiment of the present invention.

[0027] Figure 2 Schematic diagram of the detail extraction module framework in an embodiment of the present invention.

[0028] Figure 3 Schematic diagram of the semantic extraction module framework in an embodiment of the present invention.

[0029] Figure 4 Schematic diagram of a multi-shaped pyramid module framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0031] Example 1

[0032] like Figure 1 As shown, this embodiment provides a real-time semantic segmentation method for multi-shape pyramids in traffic scenarios, and the specific implementation includes the following steps:

[0033] S1. Preprocess the original image to be segmented, and initialize the preprocessed image to be segmented to obtain an initial segmentation feature map.

[0034] In this step, preprocessing is mainly data enhancement operation, and initialization processing mainly includes dimensionality reduction and preliminary feature extraction. The initialization module can be used to reduce the resolution of the original image to be segmented and reduce the feature dimension of the original image to be segmented. Then, preliminary feature extraction is performed on the image to be segmented, and the number of channels of the feature map is expanded to obtain the initial feature map.

[0035] S2. In order to prevent the forward propagation from losing shallow information, the original image to be segmented is downsampled to 1 / 2 the size of the original image to be segmented to obtain the output feature map of the first downsampling; the initial segmentation feature map obtained in step S1 and the output feature map of the first downsampling are channel-joined, the joined feature map is normalized, and the PReLU activation function is used to process it, and the processed fusion feature map is output.

[0036] S3, downsample the fusion feature map obtained in step S2 to 1 / 4 of the size of the original image to be segmented, and obtain the second downsampled output feature map F input The output feature map of the second downsampling is input into the multi-layer cascade detail extraction module to extract detail information and obtain a detail information feature map.

[0037] The downsampling module used in this step consists of a 2×2 maximum pooling and a 3×3 convolution in parallel, increasing the number of channels to 64.

[0038] Specifically, if Figure 2As shown in the figure, each layer's detail extraction module uses an inverted residual structure to efficiently extract detail boundary information. This rich detail boundary information effectively segments the boundaries between different image classes. Each layer's detail extraction module consists of two parallel branches, a dual-branch structure. One branch uses depthwise separable convolutions, while the other uses depthwise separable dilated convolutions with a dilation rate of 2 to obtain a richer receptive field, thereby extracting detail information for multi-scale objects.

[0039] In this embodiment, taking a three-layer cascaded detail extraction module as an example, the steps included in the detail information extraction process are described as follows:

[0040] S31, the second down-sampled output feature map F input As the input of the first-layer detail extraction module, the first-layer detail extraction module performs point-by-point convolution on the second down-sampled output feature map F input Perform channel dimensionality increase to obtain the dimensionality-increased feature map F x :

[0041] F x =C 1×1 (ρ(F input )) (1)

[0042] Among them, C represents the convolution operation, ρ represents the PReLU activation function; C 1×1 It is a 1×1 convolution operation, which increases the channel dimension to 128.

[0043] S32, the dimension-upgraded feature map F x Perform channel separation to obtain two separation feature maps F of the same size x1 、F x2 ; Then the two separated feature maps are input into different parallel branches of the detail extraction module to obtain two feature maps F with different scale information y1 、F y2 :

[0044] F y1 =C 3×3 (F x1 ) (2)

[0045] F y2 =C 3×3,d (F x2 ) (3)

[0046] Where d represents the convolution expansion coefficient, C 3×3 It is a 3×3 convolution operation.

[0047] S33, the feature maps F of two different scale information y1 、F y2Perform channel splicing and use the spatial attention mechanism to extract more effective detail information. Then restore the number of channels through 1×1 point-by-point convolution, and then use the jump connection to fuse the input of the detail extraction module (i.e., the output feature map F of the second downsampling). input ) to recover some of the information lost in the detail extraction process and obtain the detail information feature map F output by the first layer detail extraction module output :

[0048] F output =C 1×1 (SA(Cat(F y1 ,F y2 )))+F input (4)

[0049] SA is a spatial attention module that processes feature maps to enhance feature representation capabilities; Cat is a channel splicing operation.

[0050] S34. The last two layers of detail extraction modules (i.e., the remaining detail extraction modules) are connected to the output of the first layer of detail extraction modules in a cascaded manner. Their inputs are all the detail information feature maps output by the detail extraction modules in the previous layer. That is, the input of the second layer of detail extraction modules is the detail information feature map output by the first layer of detail extraction modules, and the input of the third layer of detail extraction modules is the detail information feature map output by the second layer of detail extraction modules. Each layer of detail extraction modules performs the same detail information extraction process. After the detail extraction processing of the three layers of cascaded detail extraction modules, the detail information feature map output by the final layer of detail extraction modules is obtained.

[0051] S35. Downsample the original image to be segmented to 1 / 4 of the size of the original image to be segmented to obtain the original image to be segmented after four times downsampling; perform channel splicing on the detail information feature map output by the last layer detail extraction module, the output feature map of the second downsampling, and the original image to be segmented after four times downsampling, and use normalization and PReLU activation function to process the spliced ​​feature map to obtain the final detail information feature map.

[0052] S4: downsample the detail information feature map obtained in step S3 to 1 / 8 of the size of the original image to be segmented, and obtain the output feature map F′ of the third downsampling. input The output feature map of the third downsampling is input into the multi-layer cascade semantic extraction module to extract semantic information and obtain a semantic information feature map.

[0053] The downsampling module structure used in this step is the same as the downsampling module in step S3, including maximum pooling with a kernel size of 2×2 and 3×3 convolution in parallel, and the number of channels is increased to 128.

[0054] Specifically, if Figure 3As shown, each layer of the semantic extraction module adopts a residual structure design to extract rich semantic information and help the segmentation network to effectively classify pixels within the class. Rich semantic information helps to correctly classify pixels in different areas. The semantic extraction module has the advantages of small number of parameters and fast inference speed. In this embodiment, the semantic extraction module adopts a dual-branch structure, that is, it includes two parallel branches; one branch uses depthwise separable convolution, and the other branch uses dilated convolution to obtain semantic information with a larger receptive field, thereby extracting rich semantic information at multiple scales; the convolution expansion coefficient (i.e., expansion rate) of the semantic extraction modules at different layers gradually expands, which can prevent the loss of important information during the forward propagation process while obtaining semantic information with a larger receptive field.

[0055] In this embodiment, taking an eleven-layer cascade semantic extraction module as an example, the steps included in the semantic information extraction process are described as follows:

[0056] S41, the output feature map of the third downsampling is used as the input of the first layer semantic extraction module, and the first layer semantic extraction module uses point-by-point convolution to obtain the convolution feature map F′ x :

[0057] F′ x =C 1×1 (ρ(F′ input )) (5)

[0058] S42, convolution feature map F' x Perform channel segmentation to obtain two feature maps F′ with the same size and half the channel size x1 , F′ x2 ; Then, the two feature maps with the same channel size halved are processed by different parallel branches of the semantic extraction module to obtain two feature maps F′ with different receptive field sizes y1 , F′ y2 :

[0059] F′ y1 =C 3×3 (F′ x1 ) (6)

[0060] F′ y2 =C 3×3,d (F′ x2 ) (7)

[0061] S43, the two feature maps F′ with different receptive field sizes y1 , F′ y2 Perform channel splicing and use point-by-point convolution for fusion, and then use jump connection to the input of the semantic extraction module to supplement the information lost in the forward propagation process to obtain the semantic information feature map F′ output by the first layer semantic extraction module output :

[0062] F′ output =C 1×1 (Cat(F′ y1 ,F′ y2 )+F′ input (8)

[0063] S44. The remaining (i.e., the last ten) semantic extraction modules are connected in a cascade fashion to the output of the first semantic extraction module. Their input is the semantic information feature map output by the semantic extraction module in the previous layer. Each semantic extraction module in each layer performs the same semantic information extraction process. The semantic information is extracted by these eleven cascaded semantic extraction modules, resulting in the output of a semantic information feature map output by the semantic extraction module in the final layer.

[0064] S45. Downsample the original image to be segmented to 1 / 8 of the size of the original image to be segmented to obtain the original image to be segmented after eight times downsampling; then perform channel splicing on the semantic information feature map output by the last layer of semantic extraction module, the output feature map of the third downsampling, and the original image to be segmented after eight times downsampling, and use normalization and PReLU activation function to process the spliced ​​feature map to obtain the final semantic information feature map.

[0065] S5. Input the semantic information feature map outputted in step S4 into the multi-shape pyramid module to obtain a multi-scale and multi-shape context information feature map.

[0066] like Figure 4 The Multi-Shape Pyramid Module can obtain feature maps with receptive fields of different scales and shapes, effectively segmenting objects of varying scales and shapes in images. The Multi-Shape Pyramid Module consists of two cascaded parallel modules, one for extracting horizontal and one for extracting vertical receptive fields. Each parallel module has four parallel branches, using dilated convolution to generate feature maps with different receptive fields and then fuse these feature maps.

[0067] In this embodiment, the process of obtaining the multi-scale and multi-shape context information feature map includes the following steps:

[0068] S51, input the semantic information feature map into the first parallel module, the first parallel module performs point-by-point convolution dimensionality reduction through four parallel branches, reduces the number of channels of the feature map to reduce the amount of calculation, and obtains four dimension-reduced feature maps F of the same size i :

[0069] F i =C 1×1 (F′ output ),i=1,2,3,4 (9)

[0070] S52: The reduced dimensionality feature map output by the first parallel branch obtained in step S51 is not processed in any way to retain the original information; the reduced dimensionality feature maps output by the next three parallel branches are subjected to cascaded horizontal factor convolutions with different expansion coefficients, for example, two cascaded 3×1 factor convolutions with input expansion coefficients of 2, 4, and 6, respectively, to obtain horizontal receptive field information feature maps F′ of different scales. i :

[0071] F′ i =C 3×1,d (C 3×1,d (F i )),i=2,3,4 (10)

[0072] Finally, the multi-scale information feature maps of four parallel branches are obtained.

[0073] S53, channel splicing is performed on the multi-scale information feature maps of the four parallel branches of the first parallel module processed in step S52, and then channel dimension reduction and feature fusion are performed through point-by-point convolution to obtain the fused feature map F of the first parallel module fuse :

[0074] F fuse =C 1×1 (Cat(F i,i＝1 ,F′ i,i＝2 ,F′ i,i＝3 ,F′ i,i＝4 )) (11)

[0075] S54, taking the fused feature map of the first parallel module as the input of the second parallel module; in the second parallel module, the fused feature map of the first parallel module is subjected to point-by-point convolution dimensionality reduction of four parallel branches to obtain four reduced-dimensional feature maps F of the same size j :

[0076] F j =C 1×1 (F fuse ),j=1,2,3,4 (12)

[0077] S55. The dimensionality reduction feature map output by the first parallel branch obtained in step S54 is not processed in any way and retains the original information; the dimensionality reduction feature maps output by the next three parallel branches are respectively subjected to cascaded longitudinal factor convolutions with different expansion coefficients, for example, two cascaded 1×3 factor convolutions with input expansion rates of 2, 6, and 8, to obtain longitudinal receptive field information feature maps F′ of different scales. j :

[0078] F′ j =C 1×3,d (C 1×3,d (F j)),j=2,3,4 (13)

[0079] Finally, the multi-scale information feature maps of four parallel branches are obtained.

[0080] S56: perform channel splicing on the multi-scale information feature maps of the four parallel branches of the second parallel module after processing in step S55, and then perform point-by-point convolution dimensionality reduction and fusion to finally obtain a feature map F with rich multi-scale and multi-shape context information. context :

[0081] F context =C 1×1 (Cat(F j,j＝1 ,F′ j,j＝2 ,F′ j,j＝3 ,F′ j,j＝4 )) (14)

[0082] S6. Upsample the multi-scale and multi-shape context information feature map output in step S5 to 1 / 4 of the size of the original image to be segmented by bilinear interpolation to obtain an upsampled context information feature map; reduce the dimension of the detail information feature map output in step S3 by point-by-point convolution to obtain a reduced-dimensional detail information feature map; then perform channel splicing on the upsampled context information feature map and the reduced-dimensional detail information feature map to finally obtain a fused feature map containing rich semantic information and detail information.

[0083] S7. Upsample the fused feature map obtained in step S6 to the size of the original image to be segmented by bilinear interpolation, adjust the number of channels to the number of predicted categories by convolution operation, and obtain the predicted segmentation result by normalization; then use the loss function to calculate the loss value between the predicted probability and the true label, and gradually obtain the predicted segmentation result close to the true label through repeated iterative updates of the neural network.

[0084] This embodiment also provides a real-time semantic segmentation system for multi-shape pyramids in traffic scenarios, including the following modules:

[0085] The initialization module preprocesses the original image to be segmented, and initializes the preprocessed image to be segmented to obtain an initial segmentation feature map;

[0086] The first fusion module downsamples the original image to be segmented to obtain the output feature map of the first downsampling; performs channel splicing on the initial segmentation feature map and the output feature map of the first downsampling, and outputs the processed fusion feature map;

[0087] A detail information extraction module downsamples the fused feature map to obtain a second downsampled output feature map; extracts detail information from the second downsampled output feature map to obtain a detail information feature map;

[0088] A semantic information extraction module downsamples the detail information feature map to obtain a third downsampled output feature map; and extracts semantic information from the third downsampled output feature map to obtain a semantic information feature map.

[0089] The multi-shape pyramid module extracts the horizontal and vertical receptive field information from the semantic information feature map, and obtains feature maps containing different receptive field information through dilated convolution with different dilation coefficients to obtain multi-scale and multi-shape contextual information feature maps.

[0090] The second fusion module upsamples the multi-scale and multi-shape context information feature map to obtain an upsampled context information feature map; reduces the dimension of the detail information feature map through point-by-point convolution to obtain a reduced dimension detail information feature map; and performs channel splicing on the upsampled context information feature map and the reduced dimension detail information feature map to obtain a fused feature map containing rich semantic information and detail information.

[0091] The semantic segmentation module upsamples the fused feature map obtained by the second fusion module to the size of the original image to be segmented, uses convolution operations to adjust the number of channels to the number of predicted categories, and obtains the predicted segmentation result through normalization; then uses the loss function to calculate the loss value between the predicted probability and the true label, and through repeated iterative updates of the neural network, gradually obtains the predicted segmentation result close to the true label.

[0092] In this embodiment, the output feature map of the second downsampling is passed through a continuous detail extraction module to obtain a high-resolution detail information feature map; then the high-resolution detail information feature map is passed through a continuous semantic extraction module to obtain a low-resolution semantic information feature map; and then the low-resolution semantic information feature map is passed through a multi-scale and multi-shape pyramid module to further extract a semantic information feature map with a larger and richer receptive field.

[0093] The present invention can not only achieve high-precision pixel-level classification in traffic scenarios, but also significantly improve the inference speed to meet real-time requirements; it greatly reduces the amount of calculation and the size of the network while improving its accuracy, thereby improving the performance of real-time semantic segmentation. Compared with the existing technology, the present invention effectively processes objects of different scales and shapes by introducing a multi-shape feature pyramid module, thereby improving the extraction and fusion capabilities of multi-scale information features. In addition, the present invention adopts dilated factor convolution to design a multi-scale context module, so that the network can perceive information of different sizes, thereby improving the recognition accuracy of targets of different categories. By optimizing the network structure, the present invention achieves a good balance between inference speed and accuracy while reducing the amount of calculation and the number of parameters, and is very suitable for high-real-time and high-precision application scenarios such as autonomous driving and traffic monitoring.

[0094] The above is only a better / preferred embodiment of the invention of the present invention, but the scope of protection of the invention patent is not limited thereto. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A real-time semantic segmentation method for multi-shape pyramids in traffic scenes, characterized by: The following steps are involved: S1. Preprocess the original image to be segmented, and initialize the preprocessed image to be segmented to obtain an initial segmentation feature map; S2. Downsample the original image to be segmented to obtain the output feature map of the first downsampling; perform channel splicing on the initial segmentation feature map and the output feature map of the first downsampling, and output the processed fusion feature map; S3, downsampling the fusion feature map obtained in step S2 to obtain a second downsampling output feature map; extracting detail information from the second downsampling output feature map to obtain a detail information feature map; S4, downsampling the detail information feature map obtained in step S3 to obtain a third downsampling output feature map; extracting semantic information from the third downsampling output feature map to obtain a semantic information feature map; S5. Extracting the horizontal receptive field information and the vertical receptive field information from the semantic information feature map output in step S4, and obtaining feature maps containing different receptive field information through dilated convolution with different dilation coefficients to obtain a multi-scale and multi-shape contextual information feature map; S6. Upsampling the multi-scale and multi-shape context information feature map output in step S5 to obtain an upsampled context information feature map; performing point-by-point convolution on the detail information feature map output in step S3 to obtain a reduced-dimensional detail information feature map; performing channel concatenation on the upsampled context information feature map and the reduced-dimensional detail information feature map to obtain a fused feature map containing rich semantic information and detail information; S7, upsampling the fused feature map obtained in step S6 to the size of the original image to be segmented, adjusting the number of channels to the number of predicted categories using a convolution operation, and obtaining a predicted segmentation result by normalization; then using a loss function to calculate the loss value between the predicted probability and the true label, and gradually obtaining a predicted segmentation result close to the true label through repeated iterative updates of the neural network; Step S5 obtains a multi-scale and multi-shape context information feature map through a multi-shape pyramid module; The multi-shape pyramid module consists of two cascaded parallel modules, which are used to extract horizontal receptive field information and vertical receptive field information respectively; each parallel module has four parallel branches, which use dilated convolution to obtain feature maps of different receptive fields and fuse the feature maps of different receptive fields.

2. The real-time semantic segmentation method according to claim 1, characterized in that In step S3, a multi-layer detail extraction module is used to extract detail information. Each layer of the detail extraction module is designed with an inverted residual structure to extract detail boundary information, so as to effectively segment the boundaries between different categories in the image according to the detail boundary information.

3. The real-time semantic segmentation method according to claim 1 or 2, characterized in that Step S3 uses a multi-layer detail extraction module to extract detail information. Each layer of the detail extraction module includes two parallel branches, one branch uses depthwise separable convolution; the other branch uses depthwise separable dilated convolution to obtain a richer receptive field.

4. The real-time semantic segmentation method according to claim 3, characterized in that The detailed information extraction process in step S3 includes: S31, using the output feature map of the second downsampling as the input of the first-layer detail extraction module, and the first-layer detail extraction module performs channel dimension increase on the output feature map of the second downsampling by point-by-point convolution to obtain a dimension-increased feature map; S32, performing channel separation on the increased-dimensional feature map to obtain two separated feature maps of the same size; then inputting the two separated feature maps into different parallel branches of the detail extraction module to obtain two feature maps of different scale information; S33, concatenate the channels of the two feature maps of information at different scales, and use the spatial attention mechanism to extract more effective detail information. Then, restore the number of channels through point-by-point convolution, and use skip connections to fuse the input of the detail extraction module to restore some of the information lost during the detail extraction process, thus obtaining the detail information feature map output by the first-layer detail extraction module. S34. The remaining detail extraction modules are connected to the output of the first layer detail extraction module in a cascade manner. Their inputs are all detail information feature maps output by the detail extraction modules of the previous layer. Each layer of detail extraction modules performs the same detail information extraction process. The detail extraction processing of the multiple layers of cascaded detail extraction modules obtains the detail information feature map output by the detail extraction module of the last layer. S35. Downsample the original image to be segmented to obtain a four-fold downsampled original image to be segmented; perform channel stitching on the detail information feature map output by the last layer detail extraction module, the output feature map of the second downsampling, and the four-fold downsampled original image to be segmented to obtain a final detail information feature map.

5. The real-time semantic segmentation method according to claim 1, characterized in that Step S4 uses a multi-layer cascade of semantic extraction modules to extract semantic information. Each layer of the semantic extraction module adopts a residual structure design, including two parallel branches, one branch uses depthwise separable convolution, and the other branch uses dilated convolution to obtain semantic information with a larger receptive field. The convolution dilation coefficient of the semantic extraction modules in different layers gradually increases.

6. The real-time semantic segmentation method according to claim 5, characterized in that The process of semantic information extraction in step S4 includes: S41, using the output feature map of the third downsampling as the input of the first-layer semantic extraction module, and the first-layer semantic extraction module uses point-by-point convolution to obtain a convolution feature map; S42, performing channel segmentation on the convolution feature map to obtain two feature maps of the same size with half the channel; then processing the two feature maps of the same size with half the channel through different parallel branches of the semantic extraction module to obtain two feature maps with different receptive field sizes; S43, channel-wise splicing of the two feature maps with different receptive field sizes, and fusing them using point-by-point convolution, and then fusing the input of the semantic extraction module through skip connection to supplement the information lost during the forward propagation process, thereby obtaining the semantic information feature map output by the first-layer semantic extraction module; S44, the remaining semantic extraction modules are connected to the output of the first layer semantic extraction module in a cascade manner, and their inputs are all the semantic information feature maps output by the semantic extraction modules of the previous layer. The semantic information extraction process of each layer of semantic extraction modules is the same; through the semantic information extraction of the multiple layers of cascaded semantic extraction modules, the semantic information feature map output by the semantic extraction module of the last layer is obtained; S45. Downsample the original image to be segmented to obtain the original image to be segmented after eight-fold downsampling; then perform channel splicing on the semantic information feature map output by the last layer of semantic extraction module, the output feature map of the third downsampling, and the original image to be segmented after eight-fold downsampling to obtain the final semantic information feature map.

7. The real-time semantic segmentation method according to claim 1, characterized in that The process of obtaining the multi-scale and multi-shape context information feature map in step S5 includes the following steps: S51, inputting the semantic information feature map into the first parallel module, and performing point-by-point convolution dimensionality reduction in the first parallel module through four parallel branches to obtain four reduced-dimensionality feature maps of the same size; S52: The dimensionality reduction feature map output by the first parallel branch obtained in step S51 is not processed in any way, and the dimensionality reduction feature maps output by the next three parallel branches are respectively subjected to cascaded horizontal factor convolutions with different expansion coefficients to obtain horizontal receptive field information feature maps of different scales; S53, performing channel splicing on the multi-scale information feature maps of the four parallel branches of the first parallel module processed in step S52, and then performing channel dimensionality reduction and feature fusion through point-by-point convolution to obtain a fused feature map of the first parallel module; S54, using the fused feature map of the first parallel module as the input of the second parallel module; in the second parallel module, the fused feature map of the first parallel module is subjected to point-by-point convolution dimensionality reduction of four parallel branches to obtain four reduced-dimensional feature maps of the same size; S55: The dimensionality reduction feature map output by the first parallel branch obtained in step S54 is not processed in any way, and the dimensionality reduction feature maps output by the next three parallel branches are respectively subjected to cascaded longitudinal factor convolutions with different expansion coefficients to obtain longitudinal receptive field information feature maps of different scales; and finally, multi-scale information feature maps of the four parallel branches are obtained; S56: perform channel splicing on the multi-scale information feature maps of the four parallel branches of the second parallel module after processing in step S55, and then perform dimensionality reduction and fusion through point-by-point convolution to finally obtain a feature map with rich multi-scale and multi-shape contextual information.

8. A real-time semantic segmentation system for multi-shape pyramids in traffic scenes, characterized by: Includes the following modules: The initialization module preprocesses the original image to be segmented, and initializes the preprocessed image to be segmented to obtain an initial segmentation feature map; The first fusion module downsamples the original image to be segmented to obtain the output feature map of the first downsampling; performs channel splicing on the initial segmentation feature map and the output feature map of the first downsampling, and outputs the processed fusion feature map; A detail information extraction module downsamples the fused feature map to obtain a second downsampled output feature map; extracts detail information from the second downsampled output feature map to obtain a detail information feature map; A semantic information extraction module downsamples the detail information feature map to obtain a third downsampled output feature map; and extracts semantic information from the third downsampled output feature map to obtain a semantic information feature map. The multi-shape pyramid module extracts the horizontal and vertical receptive field information from the semantic information feature map, and obtains feature maps containing different receptive field information through dilated convolution with different dilation coefficients to obtain multi-scale and multi-shape contextual information feature maps. The second fusion module upsamples the multi-scale and multi-shape context information feature map to obtain an upsampled context information feature map; The detail information feature map is reduced in dimension by point-by-point convolution to obtain a reduced-dimensional detail information feature map; the upsampled context information feature map and the reduced-dimensional detail information feature map are channel-joined to obtain a fused feature map containing rich semantic information and detail information; The semantic segmentation module upsamples the fused feature map obtained by the second fusion module to the size of the original image to be segmented, uses convolution operations to adjust the number of channels to the number of predicted categories, and obtains the predicted segmentation result through normalization. It then uses the loss function to calculate the loss between the predicted probability and the true label. Through repeated iterative updates of the neural network, it gradually obtains the predicted segmentation result close to the true label. The multi-shape pyramid module consists of two cascaded parallel modules, which are used to extract horizontal receptive field information and vertical receptive field information respectively; each parallel module has four parallel branches, which use dilated convolution to obtain feature maps of different receptive fields and fuse the feature maps of different receptive fields.

Images