Context aggregation network and real-time image semantic segmentation method based on the network

Abstract

The present invention relates to an efficient multi-scale context aggregation network for real-time semantic segmentation, comprising a downsampling module, an asymmetric convolution module, a multi-branch asymmetric convolution module, a spatial attention module, a channel attention module and an upsampling module, wherein the asymmetric convolution module comprises two or three asymmetric convolution units, and the multi-branch asymmetric convolution module comprises three multi-branch asymmetric convolution units. The encoder and decoder parts of the network trunk branch of the present invention are basically in a symmetrical relationship. At the end of the decoder, an upsampling operation is performed on the image to obtain the original resolution image. The last upsampling module is the classification convolution, and its output is the final prediction result of the original resolution size. The prediction result is compared with the corresponding semantic label, and the objective function is set to the cross entropy loss function to obtain a trained network model. Using this trained network model, it is convenient to predict the results of image semantic segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and pattern recognition, and relates to an efficient multi-scale context aggregation network and a real-time semantic segmentation method of an image based on the network, which is used to classify each pixel of an image. Background Art

[0002] Currently, a key issue in the field of computer vision is semantic segmentation, which involves pixel-level classification of images. This involves classifying each pixel in an image, grouping pixels of the same category together and assigning them a corresponding label. Semantic segmentation plays a key role in road scene recognition and understanding in autonomous driving systems, determining landing points for drones, and numerous wearable devices. The design of semantic segmentation models typically focuses on reducing network complexity and shortening model runtime. Continuously improving model segmentation accuracy remains a major challenge in this field.

[0003] Deep convolutional neural network-based methods are currently the mainstream approach for image semantic segmentation. They employ an encoder-decoder architecture. The encoder downsamples the image to capture abstract semantic information, while the decoder upsamples the image, requiring further upsampling to output a predicted image at the original resolution. Currently, semantic segmentation network architectures can be broadly divided into two types: deep networks and lightweight networks. Deep networks, primarily using VGGNet and ResNet architectures, primarily focus on improving segmentation accuracy, while neglecting operational efficiency. The large number of parameters places a computational burden on devices and increases runtime, making them unsuitable for edge devices with limited computing resources or applications requiring real-time semantic segmentation. Considering the reliability of real-world applications, researchers have focused on lightweight semantic segmentation. Lightweight semantic segmentation employs lightweight network architectures, significantly reducing model parameters and shortening training and prediction time. However, this also reduces segmentation accuracy to a certain extent. Therefore, how to strike a balance between model segmentation accuracy and inference speed, and design an image semantic segmentation network model that can not only ensure segmentation accuracy but also have a high inference speed and is suitable for border electronic devices with small memory capacity has become an important issue in the field of image segmentation.

[0004] With the continuous development of deep learning technology, research on deep convolution has deepened. Various convolution methods have emerged, including factorized convolution, dilated convolution, grouped convolution, point-by-point convolution, and depthwise separable convolution. These are all extensions and improvements to the standard two-dimensional convolution. Asymmetric convolution, also known as factorized convolution, decomposes a standard convolution with a kernel of k into two one-dimensional convolutions, k×1 and 1×k. This decomposes a two-dimensional calculation into the sum of two one-dimensional calculations, significantly reducing the model's computational complexity. Dilated convolution, by adding zero elements between the kernel elements to expand the kernel, can widen the receptive field and capture multi-scale contextual information.

[0005] In recent years, attention mechanisms have also gained widespread application in deep learning. They can be categorized into channel-wise and spatial-wise attention mechanisms. The channel-wise attention module calculates the weights of each channel in the input image through a network, calibrating features to retain valuable features and eliminate worthless ones, thereby improving feature representation. The spatial-wise attention module complements the channel-wise attention module by identifying locations with the most information, focusing on valuable locations and suppressing unimportant ones.

[0006] A search revealed that Chinese patent CN113486897A discloses a semantic segmentation method with upsampling decoding and convolutional attention mechanism. This method extracts features through a dilated convolutional deep neural network to increase the receptive field of feature extraction; aggregates local and global contextual information of semantic segmentation categories through a convolutional pooling pyramid, and calculates feature maps through average pooling; utilizes the spatial relationship of the feature maps to generate spatial attention feature information, performing maximum pooling of one channel and average pooling of another; passes through a two-layer neural network, adds the obtained feature maps and passes them through an activation function to introduce nonlinear information, enhance the generalization of the channel, and obtain weight coefficients; after convolutional spatial attention and convolutional channel attention, two new feature maps are obtained, which are expressed through fusion upsampling to express spatial resolution information and output the semantic segmentation result of the predicted classification. However, this semantic segmentation method with upsampling decoding and convolutional attention mechanism uses a large number of parameters, resulting in a large amount of computation, slow operation, and low segmentation accuracy.

[0007] Related references are as follows:

[0008] [1]A.Paszke,A.Chaurasia,S.Kim,and E.Culurciello,“ENet:A deep neuralnetwork architecture for real-time semantic segmentation,”2016,arXiv:1606.02147.[Online].Available:http: / / arxiv.org / abs / 1606.02147

[0009] [2]V.Badrinarayanan,A.Kendall,and R.Cipolla,“SegNet:A deepconvolutional encoder-decoder architecture for image segmentation,”IEEETrans.Pattern Anal.Mach.Intell.,vol.39,no.12,pp.2481–2495,Dec.2017.

[0010] [3]J.Long,E.Shelhamer,and T.Darrell,“Fully convolutional networks forsemantic segmentation,”in Proc.IEEE Conf.Comput.Vis.Pattern Recognit.(CVPR),Jun.2015,pp.3431–3440.

[0011] [4]Z.Yang et al.,“NDNet:Narrow while deep network for real-timesemantic segmentation,”IEEE Trans.Intell.Transp.Syst.,early access,Apr.27,2020,doi:10.1109 / TITS.2020.2987816.

[0012] [5]H.Li,P.Xiong,H.Fan,and J.Sun,“DFANet:Deep feature aggregation forreal-time semantic segmentation,”in Proc.IEEE / CVF Conf.Comput.Vis.PatternRecognit.(CVPR),Jun.2019,pp.9522–9531.

[0013] [6]F.Yu and V.Koltun,“Multi-scale context aggregation by dilatedconvolutions,”2015,arXiv:1511.07122.[Online].Available:http: / / arxiv.org / abs / 1511.07122

[0014] [7]C.Yu,J.Wang,C.Peng,C.Gao,G.Yu,and N.Sang,“BiSeNet:Bilateralsegmentation network for real-time semantic segmentation,”inProc.Eur.Conf.Comput.Vis.(ECCV),2018,pp.325–341.

[0015] [8]T.Wu,S.Tang,R.Zhang,J.Cao,and Y.Zhang,“CGNet:A light-weightcontext guided network for semantic segmentation,”IEEE Trans.Image Process.,vol.30,pp.1169–1179,2021.

[0016] [9]G.Li,I.Yun,J.Kim,and J.Kim,“DABNet:Depth-wise asymmetricbottleneck for real-time semantic segmentation,”2019,arXiv:1907.11357.[Online].Available:http: / / arxiv.org / abs / 1907.11357

[0017]

[10] Liu, Jia et al., “FDDWNet: A Lightweight Convolutional Neural Network for Real-Time Semantic Segmentation,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Process. (ICASSP), May, 2020.

[0018]

[11] H. Zhao, X. Qi, X. Shen, J. Shi, and J. Jia, “ICNet for real-time semantic segmentation on high-resolution images,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 405–420.

[0019]

[12] S. Hao, Y. Zhou, Y. Guo, and R. Hong, “Bi-direction context propagation network for real-time semantic segmentation,” arXiv preprint arXiv:2005.11034, 2020.

[0020]

[13] C. Yu, C. Gao, J. Wang, G. Yu, C. Shen, and N. Sang, “Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation,” arXiv preprint arXiv:2004.02147, 2020.

[0021]

[14] Gao, G., et al. "MSCFNet: A Lightweight Network With Multi-Scale Context Fusion for Real-Time Semantic Segmentation," IEEE Trans. Intelligent Transportation Systems. (TITS), July, 2021. Summary of the Invention

[0022] The purpose of the present invention is to address the defects of the existing technology and propose an efficient multi-scale context aggregation network and a real-time semantic segmentation method for images based on the network. The method adopts a lightweight symmetric encoding-decoding structure, replaces all convolutions with asymmetric convolutions and dilated convolutions, and adds spatial attention modules and channel attention modules, achieving a good balance between model accuracy and inference speed.

[0023] In order to achieve the above objectives, the present invention provides a context aggregation network, including a downsampling module, an asymmetric convolution module, a multi-branch asymmetric convolution module, a spatial attention module, a channel attention module and an upsampling module, wherein the asymmetric convolution module includes two or three asymmetric convolution units, and the multi-branch asymmetric convolution module includes three multi-branch asymmetric convolution units.

[0024] Compared with the semantic segmentation method of upsampling decoding of the convolutional attention mechanism, the present invention achieves a significant reduction in the number of parameters and greatly improves the running speed. While using its small amount of computation, it obtains segmentation accuracy that is difficult to obtain with this structure.

[0025] The present invention further adopts the following technical solution:

[0026] Preferably, the asymmetric convolution module is divided into a first encoder asymmetric convolution module and a second encoder asymmetric convolution module in the encoder part of the aggregation network, the first encoder asymmetric convolution module includes three asymmetric convolution units, whose convolution kernel is 3, and the second encoder asymmetric convolution module includes two asymmetric convolution units, whose convolution kernel is 5; the asymmetric convolution module is divided into a first decoder asymmetric convolution module and a second decoder asymmetric convolution module in the decoder part of the aggregation network, the first decoder asymmetric convolution module consists of two asymmetric convolution units, whose convolution kernel is 5, and the second decoder asymmetric convolution module consists of two asymmetric convolution units, whose convolution kernel is 3.

[0027] Thus, the asymmetric convolution module is based on the asymmetric convolution unit, contains two asymmetric convolutions, and adds a channel attention mechanism between the two asymmetric convolutions (see Figure 2 ).

[0028] Preferably, the multi-branch asymmetric convolution unit includes four asymmetric convolution branches, each asymmetric convolution branch adopts asymmetric hole convolution, and its convolution kernel is 3, 5, 7, and 9 respectively, and the expansion rate is 2.

[0029] The fusion of the four branches described above utilizes hierarchical feature fusion. Using four convolution kernels of varying sizes captures global context at multiple scales, resulting in richer semantic information. Dilated convolutions can expand the image's receptive field. However, a drawback of dilated convolutions is the resulting grid artifacts. Hierarchical feature fusion eliminates these artifacts with high computational efficiency. This method hierarchically fuses feature maps learned using asymmetric dilated convolutions, merging branches from the lowest receptive field with branches from the next lower receptive field at each level of the hierarchy. Following the cascade, a channel attention mechanism and 1x1 grouped convolutions are employed.

[0030] Preferably, the downsampling module includes a convolution layer and a pooling layer, the kernel of the convolution layer is 3×3, and the stride of the pooling layer is 2; the upsampling module includes a deconvolution layer with a kernel of 3×3 and a stride of 2.

[0031] The present invention also provides a method for real-time semantic segmentation of images based on the above context aggregation network, comprising the following steps:

[0032] Step 1: Build a real-time semantic segmentation network model based on a lightweight symmetric network, which includes a downsampling module, an asymmetric convolution module, a multi-branch asymmetric convolution module, a spatial attention module, a channel attention module, and an upsampling module.

[0033] Step 2: The initial input image is input to the downsampling module for the first downsampling to extract the features of the initial image and reduce its image resolution to 1 / 2.

[0034] F1=Down(F in ) (1)

[0035] Among them, F in represents the input image, Down represents the downsampling module with a downsampling rate of 2, and F1 represents the output after the initial image is downsampled for the first time with a downsampling rate of 2;

[0036] Then, the image after the first downsampling is processed using the first asymmetric convolution module of the encoder.

[0037] F 13 =C 1×3 (C 3×1 (C 1×3 (C 3×1 (C 1×3 (C 3×1 (F1)))))) (8)

[0038] Among them, F1 represents the output after the initial image is downsampled for the first time with a downsampling rate of 2, and C 3×1Indicates a convolution operation with a convolution kernel of 3×1, C 1×3 Indicates a convolution operation with a convolution kernel of 1×3, F 13 represents the output of the first asymmetric convolutional module of the encoder;

[0039] Step 3: The output F of the first asymmetric convolution module of the encoder 13 The first feature fusion is performed with the output Y1 of the 1 / 2 resolution image after being processed by the spatial attention module.

[0040] F c1 =Concat(F 13 ,Y1) (9)

[0041] Among them, Concat represents the Concatenate cascade operation, F c1 Represents the output feature map after the first feature fusion;

[0042] Step 4: Send the output feature map after the first feature fusion to the channel attention module for processing.

[0043] F CAM1 =CAM(F c1 ) (10)

[0044] Among them, CAM represents the channel attention module, F CAM1 represents the output of the first channel attention module;

[0045] The output of the first channel attention module is then sent to the downsampling module for a second downsampling to extract the features of the output feature image.

[0046] F2=Down(F CAM1 ) (11)

[0047] Where Down represents a downsampling module with a downsampling rate of 2, and F2 represents the 1 / 4 resolution output after the image is downsampled for the second time.

[0048] Then, the second downsampled image is processed using the encoder’s second asymmetric convolution module.

[0049] F 22 =C 1×5 (C 5×1 (C 1×5 (C 5×1 (F2)))) (12)

[0050] Among them, C 5×1 Indicates a convolution operation with a convolution kernel of 5×1, C 1×5 Indicates a convolution operation with a convolution kernel of 1×5, F 22represents the output of the second asymmetric convolutional module of the encoder;

[0051] Step 5: The output F of the second asymmetric convolution module of the encoder 22 The second feature fusion is performed with the output Y2 of the 1 / 4 resolution image after being processed by the spatial attention module.

[0052] F c2 =Concat(F 22 ,Y2) (13)

[0053] Among them, Y2 represents the output of the 1 / 4 resolution image after being processed by the spatial attention module, and F c2 Represents the output after the second feature fusion;

[0054] Step 6: Send the output feature map after the second feature fusion to the channel attention module for processing.

[0055] F CAM2 =CAM(F c2 ) (14)

[0056] Among them, F CAM2 represents the output of the second channel attention module;

[0057] The output of the second channel attention module is then sent to the downsampling module for a third downsampling to extract the features of the output feature image.

[0058] F3=Down(F CAM2 ) (15)

[0059] Among them, F3 represents the 1 / 8 resolution output obtained after the image is downsampled for the third time;

[0060] Then, a multi-branch asymmetric convolution module is used to process the image after the third downsampling.

[0061] F 31 =PFCU1(F3) (32)

[0062] F 32 =PFCU2(F 31 ) (33)

[0063] F 33 =PFCU3(F 32 ) (34)

[0064] Among them, PFCU1 represents the first multi-branch asymmetric convolution unit, PFCU2 represents the second multi-branch asymmetric convolution unit, PFCU3 represents the third multi-branch asymmetric convolution unit, F 31represents the output of the first multi-branch asymmetric convolutional unit, F 32 represents the output of the second multi-branch asymmetric convolutional unit, F 33 Represents the output of the third multi-branch asymmetric convolutional unit;

[0065] Step 7: The output F of the multi-branch asymmetric convolution module 33 The third feature fusion is performed with the output Y3 of the 1 / 8 resolution image after being processed by the spatial attention module.

[0066] F c3 =Concat(F 33 ,Y3) (35)

[0067] Among them, Y3 represents the output of the 1 / 8 resolution image after being processed by the spatial attention module, and F c3 Represents the output after the third feature fusion;

[0068] Step 8: The output feature map after the third feature fusion is sent to the channel attention module for processing.

[0069] F CAM3 =CAM(F c3 ) (36)

[0070] Among them, F CAM3 Represents the output of the third channel attention module;

[0071] The output of the third channel attention module is then sent to the upsampling module for the first upsampling.

[0072] F4=UP(F CAM3 ) (37)

[0073] Among them, UP represents the upsampling module with an upsampling rate of 2, and F4 represents the 1 / 4 resolution output obtained after the first upsampling module;

[0074] Then, the first asymmetric convolution module of the decoder is used to process the image after the first upsampling.

[0075] F 42 =C 1×5 (C 5×1 (C 1×5 (C 5×1 (F4)))) (38)

[0076] Among them, C 5×1 Indicates a convolution operation with a convolution kernel of 5×1, C 1×5 Indicates a convolution operation with a convolution kernel of 1×5, F 42 represents the output of the first asymmetric convolutional module of the decoder;

[0077] Step 9: Send the image after the first convolution process of the decoder to the channel attention module for processing.

[0078] F CAM4 =CAM(F 42 ) (39)

[0079] Among them, F CAM4 Represents the output of the fourth channel attention module;

[0080] The output of the fourth channel attention module is then sent to the upsampling module for a second upsampling.

[0081] F5=UP(F CAM4 ) (40)

[0082] Among them, F5 represents the 1 / 2 resolution output obtained after the second upsampling module;

[0083] Then, the decoder's second asymmetric convolution module is used to process the image after the first upsampling.

[0084] F 52 =C 1×3 (C 3×1 (C 1×3 (C 3×1 (F5)))) (41)

[0085] Among them, C 3×1 Indicates a convolution operation with a convolution kernel of 3×1, C 1×3 Indicates a convolution operation with a convolution kernel of 1×3, F 52 represents the output of the second asymmetric convolutional module of the decoder;

[0086] Step 10: The image after the second convolution process of the decoder is sent to the upsampling module for the third upsampling to obtain the prediction output of the entire semantic segmentation network.

[0087] F out =UP(F 52 ) (42)

[0088] Among them, F out Represents the full-resolution output after the third upsampling module.

[0089] Step 3 further includes the following operations: taking an initial image and performing one average pooling process to generate an image with a resolution of 1 / 2, sending the image to the spatial attention module for processing, and obtaining the spatial attention feature information Y1 under the condition of 1 / 2 resolution. The specific operations are as follows:

[0090] Y 11 =AvgP(F in ) (2)

[0091] Y1=SAM(Y 11 ) (5)

[0092] Among them, F in represents the input image, AvgP represents the average pooling operation with a convolution kernel of 3×3 and a stride of 2, and Y 11 represents the obtained 1 / 2 resolution feature map, SAM represents the spatial attention module, and Y1 represents Y 11 Output after processing by the spatial attention module.

[0093] Step 5 also includes the following operations: taking an initial image and performing two average pooling processes to generate an image with a resolution of 1 / 4, sending the image to the spatial attention module for processing, and obtaining the spatial attention feature information Y2 under the condition of 1 / 4 resolution. The specific operations are as follows:

[0094] Y 21 =2*AvgP(F in ) (3)

[0095] Y2=SAM(Y 21 ) (6)

[0096] Among them, * represents the number of average pooling operations, Y 21 Represents the obtained 1 / 4 resolution feature map, Y2 represents Y 21 Output after processing by the spatial attention module.

[0097] Step 7 also includes the following operations: taking an initial image and performing three average pooling processes to generate an image with a resolution of 1 / 8, sending the image to the spatial attention module for processing, and obtaining the spatial attention feature information Y3 under the condition of 1 / 8 resolution. The specific operations are as follows:

[0098] Y 31 =3*AvgP(F in ) (4)

[0099] Y3=SAM(Y 31 ) (7)

[0100] Among them, Y 31 Indicates the obtained 1 / 8 resolution feature map, Y3 represents Y 31 Output after processing by the spatial attention module.

[0101] The present invention provides a real-time semantic segmentation method based on a contextual attention mechanism, which adopts the basis of a lightweight network model for lightweight semantic segmentation. The overall network structure includes a downsampling module, an asymmetric convolution module, a multi-branch asymmetric convolution module, a spatial attention module, a channel attention module and an upsampling module. When pre-training the network, the images and their corresponding labels in the data set are first pre-processed and cropped to 1 / 2 the size of the original images and corresponding labels, and then put into the network for training. In the network, the input image is first downsampled to reduce its resolution to 1 / 2 of the original resolution. At the same time, three separate downsampling modules are used to downsample the input image to obtain images with 1 / 2, 1 / 4 and 1 / 8 resolutions of the original image. After the three feature maps of different resolutions are processed by the spatial attention module, they are sequentially integrated into the subsequent backbone network to complete feature fusion. After each feature fusion, the channel attention module is entered, so that richer semantic information can be extracted and the overall performance of the network can be improved.

[0102] The encoder and decoder components of the proposed network backbone are essentially symmetrical. At the end of the decoder, the image is upsampled once to obtain the original resolution. This final upsampling module, a classification convolution, outputs the final prediction at the original resolution. This prediction is compared with the corresponding semantic label, and the objective function is set as the cross-entropy loss function to obtain a trained network model. This trained network model facilitates the prediction of image semantic segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] The present invention will be further described below with reference to the accompanying drawings.

[0104] Figure 1 This is the overall network structure diagram of the present invention.

[0105] Figure 2 This is a schematic diagram of the asymmetric convolution unit in the present invention.

[0106] Figure 3 This is a schematic diagram of the multi-branch asymmetric convolution unit in the present invention.

[0107] Figure 4 Schematic diagram of the experimental results in the present invention. DETAILED DESCRIPTION

[0108] The present invention will be further described below with reference to the accompanying drawings and in combination with specific implementation methods.

[0109] Example 1

[0110] like Figure 1As shown in the figure, an efficient multi-scale context aggregation network includes a downsampling module, an asymmetric convolution module, a multi-branch asymmetric convolution module, a spatial attention module, a channel attention module and an upsampling module.

[0111] The down sampling module consists of a 3×3 convolutional layer and a pooling layer with a stride of 2. The initial input image undergoes a convolution operation with a 3×3 convolutional kernel and a pooling operation with a stride of 2 to extract the features of the initial image and reduce its image resolution to 1 / 2:

[0112] F1=Down(F in ) (1)

[0113] Among them, F in = represents the input image, Down represents the downsampling module with a downsampling rate of 2, and F1 represents the output after the initial image is downsampled to a rate of 2. Downsampling can be used to reduce feature dimensions while retaining valid information, which can help prevent overfitting to a certain extent. Pooling, on the other hand, aims to maintain distortion under rotation, translation, and scaling.

[0114] The Spatial Attention module processes images with resolutions of 1 / 2, 1 / 4, and 1 / 8 to generate spatial attention features. The three initial images are processed through one, two, and three average pooling layers with a 3×3 convolution kernel and a stride of 2, respectively, to generate images with resolutions of 1 / 2, 1 / 4, and 1 / 8. These images are then fed into the Spatial Attention module for processing, generating spatial attention features at three resolutions. These features are then fused with the subsequent backbone network.

[0115] Y 11 =AvgP(F in ) (2)

[0116] Y 21 =2*AvgP(F in ) (3)

[0117] Y 31 =3*AvgP(F in ) (4)

[0118] Y1=SAM(Y 11 ) (5)

[0119] Y2=SAM(Y 21 ) (6)

[0120] Y3=SAM(Y 31 ) (7)

[0121] Among them, F in Represents the input image, AvgP represents the average pooling operation with a convolution kernel of 3×3 and a stride of 2, * represents the number of average pooling operations, and Y 11 , Y 21 , Y 31 They represent the feature maps of 1 / 2, 1 / 4, and 1 / 8 resolution, respectively. SAM represents the spatial attention module, and Y1, Y2, and Y3 represent Y 11 , Y 21 , Y 31 The output after processing by the spatial attention module is the feature map that needs to be fused with the backbone network.

[0122] Asymmetric convolution module, including two or three asymmetric convolution units. Asymmetric convolution, also known as decomposed convolution, is to decompose a standard convolution kernel of k×k into convolution kernels of k×1 and 1×k. Decomposed convolution can greatly reduce the amount of model calculation. Figure 2 As shown in the figure, the asymmetric convolution module is based on the asymmetric convolution unit and contains two asymmetric convolutions. A channel attention mechanism is added between the two asymmetric convolutions. The encoder part of the network contains two asymmetric convolution modules. The first asymmetric convolution module contains three asymmetric convolution units with a convolution kernel of 3, and the second asymmetric convolution module contains two asymmetric convolution units with a convolution kernel of 5. The decoder part of the network also contains two asymmetric convolution modules. The first asymmetric convolution module consists of two asymmetric convolution units with a convolution kernel of 5, and the second asymmetric convolution module consists of two asymmetric convolution units with a convolution kernel of 3. For images after the first downsampling, the first asymmetric convolution module of the encoder is used.

[0123] F 13 =C 1×3 (C 3×1 (C 1×3 (C 3×1 (C 1×3 (C 3×1 (F1)))))) (8)

[0124] Among them, F1 represents the output after the initial image is downsampled for the first time with a downsampling rate of 2, and C 3×1 Indicates a convolution operation with a convolution kernel of 3×1, C1×3 Indicates a convolution operation with a convolution kernel of 1×3, F 13 It represents the asymmetric output of the image after three convolutions with a kernel of 3.

[0125] Then the first feature fusion is performed.

[0126] F c1 =Concat(F 13 ,Y1) (9)

[0127] Among them, F 13 represents the output of the image after three asymmetric convolutions with a kernel of 3, Y1 represents the output of the 1 / 2 resolution image after being processed by the spatial attention module, Concat represents the concatenation operation, and F c1 Represents the output feature map after the first feature fusion.

[0128] The channel attention module is used to adjust the feature image after fusion. After each feature fusion, the channel attention modules are placed in series, which can model the dependency between channels, adaptively adjust the feature response values ​​of each channel, give greater weights to key features, and give smaller weights to secondary features, that is, starting from global information, selectively amplify valuable feature channels and suppress useless feature channels, thereby improving the performance of the network. The channel attention module of this embodiment adopts Efficient Channel Attention (ECA), which adopts a local cross-channel interaction strategy without dimensionality reduction. While keeping the dimension unchanged, it performs channel-level global average pooling, and obtains local cross-channel interaction information through each channel and its surrounding k neighbors. Using a fast one-dimensional convolution of size k is an effective way to implement ECA, and the convolution kernel k represents the coverage of local cross-channel interactions.

[0129] After the first feature fusion, it is sent to the first channel attention module for processing.

[0130] F CAM1 =CAM(F c1 ) (10)

[0131] F2=Down(F CAM1 ) (11)

[0132] F 22 =C 1×5 (C 5×1 (C 1×5 (C 5×1 (F2)))) (12)

[0133] Fc2 =Concat(F 22 ,Y2) (13)

[0134] F CAM2 =CAM(F c2 ) (14)

[0135] F3=Down(F CAM2 ) (15)

[0136] Among them, F c1 represents the output feature map after the first feature fusion, CAM represents the channel attention module, F CAM1 represents the output of the first channel attention module, Down represents the downsampling module with a downsampling rate of 2, F2 represents the 1 / 4 resolution output after the image is downsampled for the second time, and C 5×1 Indicates a convolution operation with a convolution kernel of 5×1, C 1×5 Indicates a convolution operation with a convolution kernel of 1×5, F 22 represents the output of the image after two decomposition convolutions with a convolution kernel of 5, Y2 represents the output of the 1 / 4 resolution image after being processed by the spatial attention module, Concat represents the Concatenate cascade operation, and F c2 Represents the output after the second feature fusion, F CAM2 represents the output of the second channel attention module, and F3 represents the 1 / 8 resolution output after the image is downsampled for the third time.

[0137] The Multi-branch Asymmetric Convolution module contains three multi-branch asymmetric convolution units. Figure 3 As shown, each multi-branch asymmetric convolutional unit contains four asymmetric convolutional branches, all employing asymmetric dilated convolutions with kernel sizes of 3, 5, 7, and 9, respectively, and a dilation rate of 2. These four branches are fused using a hierarchical feature fusion approach. Using four convolutional kernels of varying sizes is equivalent to capturing global context at multiple scales, resulting in richer semantic information. Dilated convolutions can expand the image's receptive field. However, a drawback of dilated convolutions is that they can cause gridding artifacts. Hierarchical feature fusion can eliminate these artifacts with high computational efficiency. This method hierarchically fuses the feature maps learned using asymmetric dilated convolutions, merging the branches from the lowest receptive field with the branches from the next lower receptive field at each level of the hierarchy. After cascading, a channel-wise attention mechanism and 1x1 grouped convolutions are subsequently employed. The processing of the multi-branch asymmetric convolutional unit can be described as follows.

[0138] I1=C 1×1(I in ) (16)

[0139] I 31 =DC 3×1 (I1) (17)

[0140] I 32 =DC 1×3 (I 31 ) (18)

[0141] I 51 =DC 5×1 (I1) (19)

[0142] I 52 =DC 1×5 (I 51 ) (20)

[0143] I 71 =DC 7×1 (I1) (21)

[0144] I 72 =DC 1×7 (I 71 ) (22)

[0145] I 91 =DC 9×1 (I1) (23)

[0146] I 92 =DC 1×9 (I 91 ) (24)

[0147] add1=I 32 +I 52 (25)

[0148] add2=add1+I 72 (26)

[0149] add3=add2+I 92 (27)

[0150] I3=Concat(I 32 ,add1,add2,add3) (28)

[0151] I4=CAM(I3) (29)

[0152] I5=GC 1×1 (I4) (30)

[0153] I out =I5+I in (31)

[0154] Among them, I in represents the input image, C 1×1 Indicates a convolution operation with a convolution kernel of 1×1, and I1 indicates the output after a 1×1 convolution operation. 3×1 , DC 1×3 , DC 5×1 , DC 1×5 , DC 7×1 , DC 1×7 , DC 9×1 , DC 1×9 They represent convolution operations with kernels of 3×1, 1×3, 5×1, 1×5, 7×1, 1×7, 9×1, and 1×9, respectively, and expansion rates of 2. 31 , I 32 , I 51 , I 52 , I 71 , I 72 , I 91 , I 92 Represent I1, I 31 , I1, I 51 , I1, I 71 , I1, I 91 The output after the convolution operation with kernels of 3×1, 1×3, 5×1, 1×5, 7×1, 1×7, 9×1, 1×9 and dilation rate of 2. add1 means adding I 32 and I 52 The output after feature fusion, add2 means adding add1 and I 72 Output of feature fusion, add3 means finally add2 and I 92 The output of feature fusion, Concat means cascade operation, I3 means I 32 , add1, add2, add3 cascade output, I4 represents the output after the channel attention mechanism processing, GC 1×1 Represents the group convolution with a kernel of 1×1, I5 represents the output after I4 is subjected to group convolution with a kernel of 1×1, I out Represents the final output of the parallel decomposition convolution unit.

[0155] The multi-branch asymmetric convolution module consists of three multi-branch asymmetric convolution units as described above, and needs to complete the following processing on the image.

[0156] F 31 =PFCU1(F3) (32)

[0157] F 32 =PFCU2(F 31 ) (33)

[0158] F 33 =PFCU3(F 32 ) (34)

[0159] F c3 =Concat(F 33 ,Y3) (35)

[0160] F CAM3 =CAM(F c3 ) (36)

[0161] Among them, F3 represents the 1 / 8 resolution output after the third downsampling of the image, PFCU1, PFCU2, and PFCU3 represent the first, second, and third multi-branch asymmetric convolution units respectively, and F 31 , F 32 , F 33 They represent the outputs of the first, second, and third multi-branch asymmetric convolutional units, respectively. Y3 represents the output of the 1 / 8 resolution image after being processed by the spatial attention module. Concat represents the concatenation operation. F c3 represents the output after the third feature fusion, CAM represents the channel attention module, F CAM3 Represents the output of the third channel attention module.

[0162] The upsampling module contains a deconvolution layer with a kernel size of 3×3 and a stride of 2. The upsampling part uses a deconvolution layer with a kernel size of 3×3 and a stride of 2 to upsample the image by a factor of 2 so that the image can be processed as follows.

[0163] F4=UP(F CAM3 ) (37)

[0164] Among them, F CAM3 represents the output of the third channel attention module, UP represents an upsampling module with an upsampling rate of 2, and F4 represents the 1 / 4 resolution output after the first upsampling module. The rest of the decoder's image processing is described as follows.

[0165] F 42 =C 1×5 (C 5×1 (C 1×5 (C 5×1 (F4)))) (38)

[0166] F CAM4 =CAM(F 42 ) (39)

[0167] F5=UP(F CAM4) (40)

[0168] F 52 =C 1×3 (C 3×1 (C 1×3 (C 3×1 (F5)))) (41)

[0169] F out =UP(F 52 ) (42)

[0170] Among them, F4 represents the 1 / 4 resolution output obtained after the first upsampling module, C 5×1 Indicates a convolution operation with a convolution kernel of 5×1, C 1×5 Indicates a convolution operation with a convolution kernel of 1×5, F 42 represents the output of the decomposition convolution module with a kernel of 5 in the decoder part, CAM represents the channel attention module, and F CAM4 represents the output of the fourth channel attention module, UP represents the upsampling module with an upsampling rate of 2, F5 represents the 1 / 2 resolution output after the second upsampling module, and C 3×1 Indicates a convolution operation with a convolution kernel of 3×1, C 1×3 Indicates a convolution operation with a convolution kernel of 1×3, F 52 represents the output of the decomposition convolution module with a kernel of 3 in the decoder part, F out It represents the full-resolution output after the third upsampling module, which is the predicted output of the entire semantic segmentation network.

[0171] This embodiment also provides a real-time image semantic segmentation method based on a contextual attention mechanism, and the specific operations are as follows:

[0172] Step 1: Build a real-time semantic segmentation network model based on a lightweight symmetric network. The entire network contains six modules, namely, down sampling module, asymmetric convolution module, multi-branch asymmetric convolution module, spatial attention module, channel attention module, up sampling module. The overall structure is as follows: Figure 1 As shown;

[0173] Step 2: The initial input image is input to the downsampling module for the first downsampling to extract the features of the initial image and reduce its image resolution to 1 / 2.

[0174] F1=Down(F in ) (1)

[0175] Among them, F in represents the input image, Down represents the downsampling module with a downsampling rate of 2, and F1 represents the output after the initial image is downsampled for the first time with a downsampling rate of 2;

[0176] Then, the image after the first downsampling is processed using the first asymmetric convolution module of the encoder.

[0177] F 13 =C 1×3 (C 3×1 (C 1×3 (C 3×1 (C 1×3 (C 3×1 (F1)))))) (8)

[0178] Among them, F1 represents the output after the initial image is downsampled for the first time with a downsampling rate of 2, and C 3×1 Indicates a convolution operation with a convolution kernel of 3×1, C 1×3 Indicates a convolution operation with a convolution kernel of 1×3, F 13 represents the output of the first asymmetric convolutional module of the encoder;

[0179] Step 3: Take an initial image and perform one average pooling process to generate an image with a resolution of 1 / 2. Send it to the spatial attention module for processing to obtain the spatial attention feature information Y1 under the condition of 1 / 2 resolution. The specific operations are as follows:

[0180] Y 11 =AvgP(F in ) (2)

[0181] Y1=SAM(Y 11 ) (5)

[0182] Among them, F in represents the input image, AvgP represents the average pooling operation with a convolution kernel of 3×3 and a stride of 2, and Y 11 represents the obtained 1 / 2 resolution feature map, SAM represents the spatial attention module, and Y1 represents Y 11 Output after processing by the spatial attention module;

[0183] The output F of the first asymmetric convolutional module of the encoder is 13 The first feature fusion is performed with the output Y1 of the 1 / 2 resolution image after being processed by the first spatial attention module.

[0184] F c1=Concat(F 13 ,Y1) (9)

[0185] Among them, Concat represents the Concatenate cascade operation, F c1 Represents the output feature map after the first feature fusion;

[0186] Step 4: Send the output feature map after the first feature fusion to the first channel attention module for processing.

[0187] F CAM1 =CAM(F c1 ) (10)

[0188] Among them, CAM represents the channel attention module, F CAM1 represents the output of the first channel attention module;

[0189] The output of the first channel attention module is then sent to the second downsampling module for a second downsampling to extract the features of the output feature image.

[0190] F2=Down(F CAM1 ) (11)

[0191] Where Down represents a downsampling module with a downsampling rate of 2, and F2 represents the 1 / 4 resolution output after the image is downsampled for the second time.

[0192] Then, the second downsampled image is processed using the encoder’s second asymmetric convolution module.

[0193] F 22 =C 1×5 (C 5×1 (C 1×5 (C 5×1 (F2)))) (12)

[0194] Among them, C 5×1 Indicates a convolution operation with a convolution kernel of 5×1, C 1×5 Indicates a convolution operation with a convolution kernel of 1×5, F 22 represents the output of the second asymmetric convolutional module of the encoder;

[0195] Step 5: Take an initial image and perform two average pooling processes to generate an image with a resolution of 1 / 4. Send it to the second spatial attention module for processing to obtain the spatial attention feature information Y2 under the 1 / 4 resolution condition. The specific operations are as follows:

[0196] Y 21 =2*AvgP(F in ) (3)

[0197] Y2=SAM(Y 21 ) (6)

[0198] Among them, * represents the number of average pooling operations, Y 21 Represents the obtained 1 / 4 resolution feature map, Y2 represents Y 21 Output after processing by the spatial attention module;

[0199] The output F of the second asymmetric convolutional module of the encoder is 22 The second feature fusion is performed with the output Y2 of the 1 / 4 resolution image after being processed by the second spatial attention module.

[0200] F c2 =Concat(F 22 ,Y2) (13)

[0201] Among them, Y2 represents the output of the 1 / 4 resolution image after being processed by the spatial attention module, and F c2 Represents the output after the second feature fusion;

[0202] Step 6: The output feature map after the second feature fusion is sent to the second channel attention module for processing.

[0203] F CAM2 =CAM(F c2 ) (14)

[0204] Among them, F CAM2 represents the output of the second channel attention module;

[0205] The output of the second channel attention module is then sent to the third downsampling module for a third downsampling to extract the features of the output feature image.

[0206] F3=Down(F CAM2 ) (15)

[0207] Among them, F3 represents the 1 / 8 resolution output obtained after the image is downsampled for the third time;

[0208] Then, a multi-branch asymmetric convolution module is used to process the image after the third downsampling.

[0209] F 31 =PFCU1(F3) (32)

[0210] F 32 =PFCU2(F 31 ) (33)

[0211] F 33=PFCU3(F 32 ) (34)

[0212] Among them, PFCU1 represents the first multi-branch asymmetric convolution unit, PFCU2 represents the second multi-branch asymmetric convolution unit, PFCU3 represents the third multi-branch asymmetric convolution unit, F 31 represents the output of the first multi-branch asymmetric convolutional unit, F 32 represents the output of the second multi-branch asymmetric convolutional unit, F 33 Represents the output of the third multi-branch asymmetric convolutional unit;

[0213] Step 7: Take an initial image and perform three average pooling processes to generate an image with a resolution of 1 / 8. Send it to the third spatial attention module for processing to obtain the spatial attention feature information Y3 under the condition of 1 / 8 resolution. The specific operations are as follows:

[0214] Y 31 =3*AvgP(F in ) (4)

[0215] Y3=SAM(Y 31 ) (7)

[0216] Among them, Y 31 Indicates the obtained 1 / 8 resolution feature map, Y3 represents Y 31 Output after processing by the spatial attention module;

[0217] The output F of the multi-branch asymmetric convolution module 33 The third feature fusion is performed with the output Y3 of the 1 / 8 resolution image after being processed by the third spatial attention module.

[0218] F c3 =Concat(F 33 ,Y3) (35)

[0219] Among them, Y3 represents the output of the 1 / 8 resolution image after being processed by the spatial attention module, and F c3 Represents the output after the third feature fusion;

[0220] Step 8: The output feature map after the third feature fusion is sent to the third channel attention module for processing.

[0221] F CAM3 =CAM(F c3 ) (36)

[0222] Among them, F CAM3 Represents the output of the third channel attention module;

[0223] Then the output of the third channel attention module is sent to the first upsampling module for the first upsampling.

[0224] F4=UP(F CAM3 ) (37)

[0225] Among them, UP represents the upsampling module with an upsampling rate of 2, and F4 represents the 1 / 4 resolution output obtained after the first upsampling module;

[0226] Then, the first asymmetric convolution module of the decoder is used to process the image after the first upsampling.

[0227] F 42 =C 1×5 (C 5×1 (C 1×5 (C 5×1 (F4)))) (38)

[0228] Among them, C 5×1 Indicates a convolution operation with a convolution kernel of 5×1, C 1×5 Indicates a convolution operation with a convolution kernel of 1×5, F 42 represents the output of the first asymmetric convolutional module of the decoder;

[0229] Step 9: The image after the first convolution process of the decoder is sent to the fourth channel attention module for processing.

[0230] F CAM4 =CAM(F 42 ) (39)

[0231] Among them, F CAM4 Represents the output of the fourth channel attention module;

[0232] The output of the fourth channel attention module is then sent to the second upsampling module for a second upsampling.

[0233] F5=UP(F CAM4 ) (40)

[0234] Among them, F5 represents the 1 / 2 resolution output obtained after the second upsampling module;

[0235] Then, the decoder's second asymmetric convolution module is used to process the image after the first upsampling.

[0236] F 52 =C 1×3 (C 3×1 (C 1×3 (C 3×1 (F5)))) (41)

[0237] Among them, C 3×1 Indicates a convolution operation with a convolution kernel of 3×1, C 1×3 Indicates a convolution operation with a convolution kernel of 1×3, F 52 represents the output of the second asymmetric convolutional module of the decoder;

[0238] Step 10: The image after the second convolution process of the decoder is sent to the third upsampling module for the third upsampling to obtain the prediction output of the entire semantic segmentation network.

[0239] F out =UP(F 52 ) (42)

[0240] Among them, F out Represents the full-resolution output after the third upsampling module.

[0241] The effective multi-scale context aggregation network proposed in this invention is tested for segmentation results. Figure 4 As shown in the figure. From left to right are the original observed image, label value, and the segmentation results of EMCANet, CGNet[8], ENet[1], SegNet[2] and DABNet[9] in this embodiment. Figure 4 From the results of the 1st, 2nd and 5th rows, we can see that EMCANet achieves better recognition results for smaller objects, such as street light poles. Figure 4 The results in the third row show that EMCANet can also distinguish objects with similar appearances well, and its segmentation effect on roads and roadside steps is very ideal. Figure 4 As can be seen from the results in the fourth row, compared with other networks, EMCANet can obtain good boundary information and can clearly depict the boundaries of pedestrians with a smaller size in the image.

[0242] The experimental results on the Camvid test set are shown in Table 1.

[0243] Table 1 Experimental results on the Camvid dataset

[0244] Method Pretrained InputSize mIoU (%)↑ Parameters↓ ENet[1] No 360×480 51.3 0.36M SegNet[2] ImageNet 360×480 55.6 29.50M FCN-8s[3] ImageNet 360×480 57.0 134.50M NDNet[4] No 360×480 57.2 0.50M DFANet[5] ImageNet 360×480 64.7 7.80M Dilation8[6] ImageNet 360×480 65.3 140.80M BiseNet[7] ImageNet 360×480 65.6 5.80M CGNet[8] No 360×480 65.6 0.50M DABNet[9] No 360×480 66.4 0.76M FDDWNet

[10] No 360×480 66.9 0.80M ICNet

[11] ImageNet 360×480 67.1 26.50M BCPNet

[12] No 720×720 67.8 0.61M BiseNet-v2

[13] ResNet18 720×960 68.7 49.00M EMCANet(ours) No 360×480 69.4 0.71M

[0245] As shown in Table 1, although ENet, NDNet, CGNet, and BCPNet have fewer parameters than EMCANet in this example, their segmentation accuracy is 18.1%, 12.2%, 4.2%, and 1.6% lower than ours, respectively. BiseNet-v2 and ICNet, which achieve similar segmentation accuracy, have 69 and 37 times the number of parameters of EMCANet. This shows that EMCANet in this example achieves a good balance between parameter count and segmentation accuracy.

[0246] In addition to the above embodiments, the present invention may also have other implementations. Any technical solution formed by equivalent replacement or equivalent transformation falls within the scope of protection required by the present invention.

Claims

1. A real-time semantic segmentation method for images based on a context aggregation network, characterized by: The following steps are involved: Step 1. Construct a real-time semantic segmentation network model based on a lightweight symmetric network, which includes a downsampling module, an asymmetric convolution module, a multi-branch asymmetric convolution module, a spatial attention module, a channel attention module, and an upsampling module; the asymmetric convolution module includes two or three asymmetric convolution units, and the multi-branch asymmetric convolution module includes three multi-branch asymmetric convolution units; the asymmetric convolution module is divided into a first encoder asymmetric convolution module and a second encoder asymmetric convolution module in the encoder part of the aggregation network, the first encoder asymmetric convolution module includes three asymmetric convolution units, and its convolution kernel is 3, and the second encoder asymmetric convolution module includes two asymmetric convolution units, and its convolution kernel is 5; the asymmetric convolution module includes two asymmetric convolution units, and its convolution kernel is 5; The convolution module is divided into a first decoder asymmetric convolution module and a second decoder asymmetric convolution module in the decoder part of the aggregation network. The first decoder asymmetric convolution module consists of two asymmetric convolution units, and its convolution kernel is 5. The second decoder asymmetric convolution module consists of two asymmetric convolution units, and its convolution kernel is 3. The multi-branch asymmetric convolution unit contains four asymmetric convolution branches, and each asymmetric convolution branch adopts asymmetric hole convolution, and its convolution kernel is 3, 5, 7, and 9 respectively, and the expansion rate is 2. The downsampling module includes a convolution layer and a pooling layer, the kernel of the convolution layer is 3×3, and the stride of the pooling layer is 2. The upsampling module includes a deconvolution layer with a kernel of 3×3 and a stride of 2. Step 2: The initial input image is input to the downsampling module for the first downsampling to extract the features of the initial image and reduce its image resolution to 1 / 2. F1=Down(F in ) (1) Among them, F in represents the input image, Down represents the downsampling module with a downsampling rate of 2, and F1 represents the output after the initial image is downsampled for the first time with a downsampling rate of 2; Then, the image after the first downsampling is processed using the first asymmetric convolution module of the encoder. F 13 =C 1×3 (C 3×1 (C 1×3 (C 3×1 (C 1×3 (C 3×1 (F1)))))) (8) Among them, F1 represents the output after the initial image is downsampled for the first time with a downsampling rate of 2, and C 3×1 Indicates a convolution operation with a convolution kernel of 3×1, C 1×3 Indicates a convolution operation with a convolution kernel of 1×3, F 13 represents the output of the first asymmetric convolutional module of the encoder; Step 3: The output F of the first asymmetric convolution module of the encoder 13 The first feature fusion is performed with the output Y1 of the 1 / 2 resolution image after being processed by the spatial attention module. F c1 =Concat(F 13 ,Y1) (9) Among them, Concat represents the cascade operation, F c1 Represents the output feature map after the first feature fusion; Step 4: Send the output feature map after the first feature fusion to the channel attention module for processing. F CAM1 =CAM(F c1 ) (10) Among them, CAM represents the channel attention module, F CAM1 represents the output of the first channel attention module; The output of the first channel attention module is then sent to the downsampling module for a second downsampling to extract the features of the output feature image. F2=Down(F CAM1 ) (11) Where Down represents a downsampling module with a downsampling rate of 2, and F2 represents the 1 / 4 resolution output after the image is downsampled for the second time. Then, the second downsampled image is processed using the encoder’s second asymmetric convolution module. F 22 =C 1×5 (C 5×1 (C 1×5 (C 5×1 (F2)))) (12) Among them, C 5×1 Indicates a convolution operation with a convolution kernel of 5×1, C 1×5 Indicates the convolution operation with a convolution kernel of 15, F 22 represents the output of the second asymmetric convolutional module of the encoder; Step 5: The output F of the second asymmetric convolution module of the encoder 22 The second feature fusion is performed with the output Y2 of the 1 / 4 resolution image after being processed by the spatial attention module. F c2 =Concat(F 22 ,Y2) (13) Among them, Y2 represents the output of the 1 / 4 resolution image after being processed by the spatial attention module, and F c2 Represents the output after the second feature fusion; Step 6: Send the output feature map after the second feature fusion to the channel attention module for processing. F CAM2 =CAM(F c2 )(14) Among them, F CAM2 represents the output of the second channel attention module; The output of the second channel attention module is then sent to the downsampling module for a third downsampling to extract the features of the output feature image. F3=Down(F CAM2 ) (15) Among them, F3 represents the 1 / 8 resolution output obtained after the image is downsampled for the third time; Then, a multi-branch asymmetric convolution module is used to process the image after the third downsampling. F 31 =PFCU1(F3) (32) F 32 =PFCU2(F 31 ) (33) <h2 style=";text-align:left;direction:ltr">F<h2 style=";text-align:left;direction:ltr"> 33 <h2 style=";text-align:left;direction:ltr"> =PFCU3(F<h2 style=";text-align:left;direction:ltr"> 32 <h2 style=";text-align:left;direction:ltr"> ) (34) Among them, PFCU1 represents the first multi-branch asymmetric convolution unit, PFCU2 represents the second multi-branch asymmetric convolution unit, PFCU3 represents the third multi-branch asymmetric convolution unit, F 31 represents the output of the first multi-branch asymmetric convolutional unit, F 32 represents the output of the second multi-branch asymmetric convolutional unit, F 33 Represents the output of the third multi-branch asymmetric convolutional unit; Step 7: The output F of the multi-branch asymmetric convolution module 33 The third feature fusion is performed with the output Y3 of the 1 / 8 resolution image after being processed by the spatial attention module. F c3 =Concat(F 33 ,Y3) (35) Among them, Y3 represents the output of the 1 / 8 resolution image after being processed by the spatial attention module, and F c3 Represents the output after the third feature fusion; Step 8: The output feature map after the third feature fusion is sent to the channel attention module for processing. F CAM3 =CAM(F c3 ) (36) Among them, F CAM3 Represents the output of the third channel attention module; The output of the third channel attention module is then sent to the upsampling module for the first upsampling. F4=UP(F CAM3 )(37) Among them, UP represents the upsampling module with an upsampling rate of 2, and F4 represents the 1 / 4 resolution output obtained after the first upsampling module; Then, the first asymmetric convolution module of the decoder is used to process the image after the first upsampling. F 42 =C 1×5 (C 5×1 (C 1×5 (C 5×1 (F4)))) (38) Among them, C 5×1 Indicates a convolution operation with a convolution kernel of 5×1, C 1×5 Indicates the convolution operation with a convolution kernel of 15, F 42 represents the output of the first asymmetric convolutional module of the decoder; Step 9: Send the image after the first convolution process of the decoder to the channel attention module for processing. F CAM4 =CAM(F 42 )(39) Among them, F CAM4 Represents the output of the fourth channel attention module; The output of the fourth channel attention module is then sent to the upsampling module for a second upsampling. F5=UP(F CAM4 ) (40) Among them, F5 represents the 1 / 2 resolution output obtained after the second upsampling module; Then, the decoder's second asymmetric convolution module is used to process the image after the first upsampling. F 52 =C 1×3 (C 3×1 (C 1×3 (C 3×1 (F5)))) (41) Among them, C 3×1 Indicates a convolution operation with a convolution kernel of 3×1, C 1×3 Indicates the convolution operation with a convolution kernel of 13, F 52 represents the output of the second asymmetric convolutional module of the decoder; Step 10: The image after the second convolution process of the decoder is sent to the upsampling module for the third upsampling to obtain the prediction output of the entire semantic segmentation network. F out =UP(F 52 )(42) Among them, F out Represents the full-resolution output after the third upsampling module.

2. The method for real-time semantic segmentation of an image based on a context aggregation network according to claim 1, characterized in that: Step 3 further includes the following operations: taking an initial image and performing one average pooling process to generate an image with a resolution of 1 / 2, sending the image to the spatial attention module for processing, and obtaining the spatial attention feature information Y1 under the condition of 1 / 2 resolution. The specific operations are as follows: AND 11 =AvgP(F in ) (2) Y1=SAM(Y 11 ) (5) Among them, F in represents the input image, AvgP represents the average pooling operation with a convolution kernel of 3×3 and a stride of 2, and Y 11 represents the obtained 1 / 2 resolution feature map, SAM represents the spatial attention module, and Y1 represents Y 11 Output after processing by the spatial attention module.

3. The method for real-time semantic segmentation of an image based on a context aggregation network according to claim 2, characterized in that: Step 5 also includes the following operations: taking an initial image and performing two average pooling processes to generate an image with a resolution of 1 / 4, sending the image to the spatial attention module for processing, and obtaining spatial attention feature information Y2 under the condition of 1 / 4 resolution. The specific operations are as follows: AND 21 =2*AvgP(F in )(3) Y2=SAM(Y 21 ) (6) Among them, * represents the number of average pooling operations, Y 21 Represents the obtained 1 / 4 resolution feature map, Y2 represents Y 21 Output after processing by the spatial attention module.

4. The method for real-time semantic segmentation of an image based on a context aggregation network according to claim 3, characterized in that: In step 7, The following operations are included: take an initial image and perform three average pooling processes to generate an image with a resolution of 1 / 8, send it to the spatial attention module for processing, and obtain the spatial attention feature information Y3 under the condition of 1 / 8 resolution. The specific operations are as follows: AND 31 =3*AvgP(F in ) (4) <h2 style=";text-align:left;direction:ltr">Y3=SAM(Y<h2 style=";text-align:left;direction:ltr"> 31 <h2 style=";text-align:left;direction:ltr"> ) (7) Among them, Y 31 Indicates the obtained 1 / 8 resolution feature map, Y3 represents Y 31 Output after processing by the spatial attention module.

Images