A real-time semantic segmentation system and method in a road scene

Abstract

The application belongs to the technical field of scene segmentation and automatic driving, and discloses a real-time semantic segmentation model and method under a road scene, which comprises an initialization module, a channel attention module, a down-sampling module, a mixed hollow grouping module and an up-sampling module; the technical scheme of the application utilizes decomposition convolution, depth separable convolution and hollow convolution to construct the mixed hollow grouping module, extracts local and context information in a simple but effective way, then adopts the channel attention module to capture information interaction between channels, and finally adopts a skip layer connection mode to perform feature fusion on feature branches from different stages in the network, so that shallow features and deep high-level semantic information are fused, feature representation is enhanced, segmentation efficiency is improved, and the balance between segmentation accuracy and inference speed is realized.

Description

Technical Field

[0001] The present invention belongs to the field of scene segmentation and autonomous driving technology, and in particular relates to a real-time semantic segmentation system and method in road scenes. Background Art

[0002] Semantic segmentation is a fundamental research area in computer vision. It involves assigning semantic labels to each pixel, aiming to predict the semantic category labels for all pixels in a given input image. The segmentation results are presented as color-coded region masks. Image semantic segmentation is fundamental to scene understanding and requires accurate prediction of objects, locations, and detailed boundaries within an image. Compared to image classification, image semantic segmentation is generally more challenging to design complex algorithms, requiring intensive pixel-level prediction capabilities and requiring greater computational effort.

[0003] In traditional image semantic segmentation, images are segmented based on underlying features such as color and texture, and then semantically annotated. Based on different segmentation criteria, traditional image segmentation algorithms can be categorized into threshold-based segmentation, cluster-based segmentation, and graph-based segmentation. Threshold-based image segmentation methods use a threshold as the dividing line to separate image pixels into target and background regions. The threshold is typically a grayscale value. Representative methods include the Ostu segmentation method. Cluster segmentation uses basic pixel features such as grayscale to partition the image into regions according to specific rules. The regions to which pixels belong are then labeled and segmented. Mean-shift and SLIC are widely used in cluster segmentation. Graph-based image segmentation methods transform the segmentation problem into graph partitioning, with the "Normalized cut" and "Grab cut" algorithms being the most commonly used techniques based on graph-based segmentation. Traditional semantic segmentation generally uses Markov random fields (MRFs) and conditional random fields (CRFs) to construct probabilistic graphical models and solve them using graph-based methods. The main idea is to assign a random vector to each feature and pixel, and then determine the classification of each pixel by calculating the probability that the pixel belongs to each class. In the semantic segmentation process, traditional feature extraction methods mainly rely on manually designed extractors. At the same time, each method is application-specific, with poor generalization and robustness, resulting in poor segmentation results.

[0004] With the advancement of deep learning technology, semantic segmentation based on convolutional neural networks has significantly improved both accuracy and speed compared to traditional methods. This advancement has significantly enhanced the quality of semantic segmentation in real-world applications, such as autonomous driving, medical diagnosis, remote sensing image segmentation, and video surveillance. Semantic segmentation of outdoor scenes has a wide range of applications in fields requiring refined information, such as autonomous driving. It ensures the safe operation of autonomous vehicles and provides important support for computer scene understanding. Furthermore, real-time semantic segmentation is even more crucial, enabling rapid acquisition of scene information and environmental assessment. In recent years, deep learning has produced a new generation of image segmentation models with significant performance improvements. Deep convolutional neural networks have demonstrated strong capabilities in high-resolution image classification. In particular, fully convolutional networks (FCNs) are a forerunner of CNNs for semantic segmentation. Encoder-decoder networks have also become a popular architecture for solving segmentation problems. While previous networks have achieved significant results in segmentation accuracy, they have largely neglected efficiency, resulting in excessive computational and storage requirements, making them inadequate for the rapid exchange of information and the environment in real-world applications. At the same time, electronic devices such as robots, mobile phones, and telemedicine have limited storage capacity and computational costs, making them unable to support complex algorithms. Therefore, designing lightweight and efficient networks is a key trend in overcoming these challenges. Smaller networks achieve faster inference speeds and minimize redundancy. Existing improvements to semantic segmentation models primarily focus on compressing networks and reducing parameters to shorten processing time. While these models have achieved a certain level of inference speed, they also significantly sacrifice accuracy.

[0005] Therefore, how to design a semantic segmentation network with fast inference speed, small capacity and high segmentation accuracy becomes a challenging problem. Summary of the Invention

[0006] The purpose of the present invention is to provide a real-time semantic segmentation system and method in road scenarios to solve the above technical problems.

[0007] To solve the above technical problems, the specific technical solutions of the real-time semantic segmentation system and method in road scenarios of the present invention are as follows:

[0008] A real-time semantic segmentation system for road scenes includes an initialization module, a channel attention module, a downsampling module, a hybrid hole grouping module and an upsampling module; the initialization module is used to perform initialization feature extraction on the input image, while reducing the image resolution, and output the data to the channel attention module; the channel attention module is used to perform skip-layer connection on the data output by the initialization module and the data downsampled from the original image, as the input of the channel attention module, extract channel features, and output the data to the downsampling module; the downsampling module is used to output the data output by the channel attention module to a hybrid hole convolution block after downsampling processing; the hybrid hole grouping module is used to use the output of the downsampling module as the input of the hybrid hole grouping module, extract dense features, and input the data to the upsampling module; the upsampling module is used to upsample the input data to a resolution consistent with the original image, and output the final segmentation prediction result.

[0009] The present invention also discloses a real-time semantic segmentation method of a real-time semantic segmentation system in a road scene, comprising the following steps:

[0010] Step 1: Initialize feature extraction of the input image, reduce the image resolution, and output the data to the channel attention module;

[0011] Step 2: Perform a skip-layer connection between the data output from step 1 and the data downsampled from the original image, use them as the input of the channel attention module, extract channel features, and input the data into the downsampling module;

[0012] Step 3: The output data of step 2 is downsampled and output to the hybrid void convolution block;

[0013] Step 4: Use the output of step 3 as the input of the hybrid hole grouping module to extract dense features and input the data into the upsampling module.

[0014] Furthermore, the step 1 includes the following specific steps:

[0015] After obtaining image data of road scenes, three 3×3 convolutions are used to extract initial features. The stride of the first convolution is set to 2 to downsample the image. The obtained initial feature map is used as the input of the lightweight channel attention module.

[0016] Furthermore, the step 2 includes the following specific steps:

[0017] First, a 1×1×C vector is obtained through global average pooling. Then, the weight of each channel is obtained through one-dimensional convolution to complete the information interaction between channels. By adding a channel attention mechanism at different stages of the network, the feature expression ability of the model is adaptively improved, and the connection between local and contextual information is promoted. It can be expressed by the following formula:

[0018] ECA(F)=σ(f k×k (T(AvgP(F))))×F (1)

[0019] Where T represents a series of operations such as tensor compression, transposition and expansion, f k×k represents the adaptively selected convolution kernel of size k, AvgP represents the global average pooling, σ is the sigmoid activation function, and F is the input feature map;

[0020] Before each channel attention module, the input data and the data downsampled from the original image are skip-connected as the input of the channel attention module to compensate for information loss.

[0021] Furthermore, the step 3 includes the following specific steps:

[0022] The downsampling block is applied to all downsampling layers and expanded into two modes. When the number of output channels is less than the number of input channels, this module is a simple 3×3 convolution with a stride of 2. Otherwise, a 2×2 maximum pooling layer with a stride of 2 is added, and then the convolution and pooling feature maps are concatenated to form the final downsampled output.

[0023] Furthermore, step 4 includes the following specific steps:

[0024] The hybrid dilated grouping module combines dilated convolution with decomposed convolution. The input channel remains unchanged and is then fed into three branches with different dilation rates. Finally, the number of channels is restored and all channel information is fused through residual connection and 1×1 point-by-point convolution. Each branch is composed of decomposed convolution in the depth direction. The decomposed convolution in the depth direction is expressed by the following formula:

[0025]

[0026]

[0027] Among them, I w represents a 2D image with w channels, K is a 2D convolution kernel, K x and K y They are one-dimensional convolution kernels along the x and y directions respectively, M and N represent the size of the convolution kernel, (x, y) represents the position of the pixel, Map w Represents the feature map obtained using decomposed convolution in the depth direction;

[0028] The decomposed convolution is further expanded to capture complex long-distance feature information. After the branch outputs with different expansion rates, a residual connection is established with the input, and PReLU is used as the activation function to improve the nonlinear ability of the model. The entire hybrid void grouping module is described as follows:

[0029] x b =p(x HDGin ) (3)

[0030] y1=C 1×3,d1 (C 3×1,d1 (x b )) (4)

[0031] y2=C 1×3,d2 (C 3×1,d2 (x b )) (5)

[0032] y3=C 1×3,d3 (C 3×1,d3 (x b )) (6)

[0033] y HDGout =C 1×1 (p(concat(y1,y2,y3,x HDGin ))) (7)

[0034] Among them, x HDGin and y HDGout They represent the input and output of the HDG module, p is the PReLU nonlinear activation function, y1, y2 and y3 represent the outputs of the three branches, respectively. m×n represents the m×n convolution kernel operation, d represents the expansion rate, and concat represents feature fusion.

[0035] Furthermore, two hybrid hole grouping blocks are used in the entire network, which are composed of 3 and 6 hybrid hole grouping modules respectively. The dilation rates in the first block are {[1,2,5],[1,2,5],[1,2,5]}, and the dilation rates in the second block are {[3,5,7],[3,5,7],[3,5,7],[5,9,11],[5,9,11],[5,9,11]}. The receptive field is gradually expanded by setting different dilation rates in each module. In each hybrid hole grouping module, skip connection is used to connect the input with the deepest layer. In each HDG block, the input of the first HDG module is added to the output of the last HDG module.

[0036] Furthermore, in step 5, only the feature maps of the two middle layers of the network are upsampled and then a skip-layer connection is established with the final output, and then 2 times upsampling is used to restore the original input image size.

[0037] Finally, the final segmentation prediction result is output.

[0038] The real-time semantic segmentation system and method for road scenes of the present invention have the following advantages:

[0039] Advantage 1: This invention takes advantage of the advantages of decomposition convolution, depthwise separable convolution and dilated convolution, and is based on a hybrid dilated grouping module, which ensures image segmentation accuracy while taking into account the lightweight and real-time performance of the algorithm model.

[0040] Advantage 2: This paper introduces a lightweight channel attention module to capture information interactions between channels. This involves only a small number of parameters but can achieve significant performance gains.

[0041] Advantage 3: This invention uses skip-layer connections to fuse features from different layers, thus fusing shallow features with deep high-level semantic information, enhancing feature representation, greatly promoting the interaction between local and global information, and further optimizing segmentation results.

[0042] Advantage 4: This invention can effectively segment high-resolution road scene images and has a wide range of applications in fields that require refined information, such as autonomous driving.

[0043] In summary, the technical solution of the present invention utilizes decomposed convolution, depthwise separable convolution, and dilated convolution to construct a hybrid dilated grouping module, extracting local and contextual information in a simple but effective manner. A channel attention module is then employed to capture information interactions between channels. Finally, feature branches from different stages of the network are fused using skip-layer connections, allowing shallow features to be integrated with deep, high-level semantic information, enhancing feature representation, improving segmentation efficiency, and achieving a balance between segmentation accuracy and inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the overall structure of the network system of the lightweight semantic segmentation method of the present invention.

[0045] Figure 2 Schematic diagram of the structure of the hybrid hole grouping module of the present invention.

[0046] Figure 3 Schematic diagram of the structure of the downsampling block of the present invention.

[0047] Figure 4 Schematic diagram of the structure of the lightweight channel attention module of the present invention. DETAILED DESCRIPTION

[0048] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a real-time semantic segmentation method for road scenarios based on a hybrid hole packet network in conjunction with the accompanying drawings.

[0049] The present invention provides a real-time semantic segmentation system for road scenes, comprising an initialization module, a channel attention module, a downsampling module, a hybrid hole grouping module, and an upsampling module. The initialization module is used to perform initialization feature extraction on the input image, while reducing the image resolution, and outputting the data to the channel attention module; the channel attention module is used to perform a jump-layer connection between the data output by the initialization module and the data downsampled from the original image, as the input of the channel attention module, extract channel features, and output the data to the downsampling module; the downsampling module is used to output the data output by the channel attention module to a hybrid hole convolution block after downsampling processing; the hybrid hole grouping module is used to use the output of the downsampling module as the input of the hybrid hole grouping module, extract dense features, and input the data to the upsampling module; the upsampling module is used to upsample the input data to a resolution consistent with the original image, and output the final segmentation prediction result.

[0050] like Figure 1 As shown, the present invention provides a real-time semantic segmentation method in a road scene, comprising the following steps:

[0051] Step 1: Initialize feature extraction of the input image, reduce the image resolution, and output the data to the channel attention module.

[0052] After obtaining image data of road scenes, three 3×3 convolutions are used to extract initial features. The stride of the first convolution is set to 2 to downsample the image. The obtained initial feature map is used as the input of the lightweight channel attention module.

[0053] Step 2: Perform a skip-layer connection between the data output from step 1 and the data downsampled from the original image, use them as the input of the channel attention module, extract channel features, and input the data into the downsampling module.

[0054] Lightweight attention module, i.e. Figure 1 Considering that the network has used mixed dilated convolution blocks with different expansion rates, it can capture long-distance spatial information. Therefore, in order to balance spatial and channel information, the channel attention module is introduced to assign different weights to each channel to strengthen the features of important channels. The specific structure is as follows Figure 4As shown, first, a 1x1xC vector is obtained by global average pooling, and then the weight of each channel is obtained by one-dimensional convolution to complete the information interaction between channels. By adding channel attention mechanisms at different stages of the network, the feature expression ability of the model can be adaptively improved, greatly promoting the connection of local and contextual information. It can be expressed by the following formula:

[0055] ECA(F) = σ(f k×k (T(AvgP(F)))) x F (1)

[0056] Where T represents a series of operations such as tensor compression, transposition and expansion, f k×k represents an adaptive selection of a convolution kernel with a size of k, AvgP represents global average pooling, and σ is a sigmoid activation function. F is the input feature map.

[0057] Before each channel attention module, the input data and the data after down-sampling of the original image are connected by layer skipping, serving as the input of the channel attention module to compensate for information loss.

[0058] Step 3: The output data of step 2 is output to the hybrid dilated convolution block after down-sampling processing.

[0059] The down-sampling block is applied to all down-sampling layers ( Figure 1 DownSample) in the network, and is expanded into two modes. When the number of output channels is less than the number of input channels, this module is a simple 3x3 convolution with a step of 2. Otherwise, a 2x2 max-pooling layer with a step of 2 is added, and then the convolution and pooling features are connected to form the output of the last down-sampling, as shown in Figure 3 .

[0060] Step 4: The output of step 3 is input into the hybrid dilated grouping module to extract dense features, and the data is input into the up-sampling module.

[0061] The hybrid dilated grouping module (HDGModule) is shown in Figure 2 , which combines dilated convolution and decomposition convolution. The input channel remains unchanged, and then is sent into three branches with different expansion rates. Finally, the channel number is restored and all channel information is fused through residual connection and 1x1 pointwise convolution. Specifically, each branch is composed of decomposition convolution in the depth direction, which greatly reduces the computational complexity. The decomposition convolution in the depth direction can be expressed by the following formula:

[0062]

[0063]

[0064] Where, Iw represents a 2D image with w channels, K is a 2D convolution kernel, K x and K y They are one-dimensional convolution kernels along the x and y directions respectively, M and N represent the size of the convolution kernel, (x, y) represents the position of the pixel, Map w Represents the feature map obtained using decomposed convolution in the depth direction.

[0065] To expand the network's receptive field, the decomposed convolutions are further dilated to capture complex, long-range feature information. Therefore, this hybrid hole grouping block reduces computational effort while also capturing multi-scale contextual information. After branch outputs with different dilation rates, a residual connection is established with the input, and PReLU is used as the activation function to enhance the model's nonlinear capabilities. The entire hybrid hole grouping module is described as follows:

[0066] x b =p(x HDGin ) (3)

[0067] y1=C 1×3,d1 (C 3×1,d1 (x b )) (4)

[0068] y2=C 1×3,d2 (C 3×1,d2 (x b )) (5)

[0069] y3=C 1×3,d3 (C 3×1,d3 (x b )) (6)

[0070] y HDGout =C 1×1 (p(concat(y1,y2,y3,x HDGin ))) (7)

[0071] Among them, x HDGin and y HDGout They represent the input and output of the HDG module, p is the PReLU nonlinear activation function, y1, y2 and y3 represent the outputs of the three branches, respectively. m×n represents the m×n convolution kernel operation, d represents the expansion rate, and concat represents feature fusion.

[0072] In order to extract more semantic features, two hybrid hole grouping blocks are used in the entire network, namely Figure 1HDG Block1 and HDG Block2 in

[15] consist of 3 and 6 hybrid hole grouping modules, respectively. The dilation rates in the first block are {[1, 2, 5], [1, 2, 5], [1, 2, 5]}, and the dilation rates in the second block are {[3, 5, 7], [3, 5, 7], [3, 5, 7], [5, 9, 11], [5, 9, 11], [5, 9, 11]}. The receptive field is gradually expanded by setting different dilation rates in each module. To better strengthen spatial relationships and feature propagation, a skip connection is used in each hybrid hole grouping module to connect the input to the deepest layer, which means that in each HDG block, the input of the first HDG module is added to the output of the last HDG module.

[0073] Step 5: Upsample the output data of step 4 to the same resolution as the original image and output the final segmentation prediction result.

[0074] To achieve faster inference speed, this network model abandons the complex decoder structure design. Instead, it simply upsamples the feature maps of the two middle layers of the network and establishes a skip-layer connection with the final output. It then uses a 2x upsampling to restore the original input image size, and finally outputs the final segmentation prediction result. This ensures not only computational simplicity but also maximum recovery of decoded information.

[0075] The above embodiments of the present invention are for illustration only and do not represent the advantages or disadvantages of the embodiments.

[0076] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain example, reference can be made to the relevant descriptions of other embodiments.

[0077] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A real-time semantic segmentation system for road scenes, characterized by: It includes an initialization module, a channel attention module, a downsampling module, a hybrid hole grouping module and an upsampling module; the initialization module is used to perform initialization feature extraction on the input image, reduce the image resolution, and output the data to the channel attention module; The channel attention module is used to perform a skip-layer connection on the data output by the initialization module and the data downsampled from the original image, and use them as the input of the channel attention module to extract channel features and output the data to the downsampling module; the downsampling module is used to output the data output by the channel attention module to the mixed void convolution block after downsampling; the mixed void grouping module is used to use the output of the downsampling module as the input of the mixed void grouping module, extract dense features, and input the data to the upsampling module; the upsampling module is used to upsample the input data to a resolution consistent with the original image and output the final segmentation prediction result; The hybrid dilated grouping module combines dilated convolution with decomposed convolution. The input channel remains unchanged and is then fed into three branches with different dilation rates. Finally, the number of channels is restored and all channel information is fused through residual connection and 1×1 point-by-point convolution. Each branch is composed of decomposed convolution in the depth direction. The decomposed convolution in the depth direction is expressed by the following formula: Among them, I w represents a 2D image with w channels, K is a 2D convolution kernel, K x and K y They are one-dimensional convolution kernels along the x and y directions respectively, M and N represent the size of the convolution kernel, (x, y) represents the position of the pixel, Map w Represents the feature map obtained using decomposed convolution in the depth direction; The decomposed convolution is further expanded to capture complex long-distance feature information. After the branch outputs with different expansion rates, a residual connection is established with the input, and PReLU is used as the activation function to improve the nonlinear ability of the model. The entire hybrid void grouping module is described as follows: x b =p(x HDGin ) (3) y1=C 1×3,d1 (C 3×1,d1 (x b )) (4) y2=C 1×3,d2 (C 3×1,d2 (x b )) (5) y3=C 1×3,d3 (C 3×1,d3 (x b )) (6) y HDGout =C 1×1 (p(concat(y1,y2,y3,x HDGin ))) (7) Among them, x HDGin and y HDGout They represent the input and output of the HDG module, p is the PReLU nonlinear activation function, y1, y2 and y3 represent the outputs of the three branches, respectively. m×n represents the m×n convolution kernel operation, d represents the expansion rate, and concat represents feature fusion.

2. A real-time semantic segmentation method for a real-time semantic segmentation system in a road scene according to claim 1, characterized in that: The steps include: Step 1: Initialize feature extraction of the input image, reduce the image resolution, and output the data to the channel attention module; Step 2: Perform a skip-layer connection between the data output from step 1 and the data downsampled from the original image, use them as the input of the channel attention module, extract channel features, and input the data into the downsampling module; Step 3: The output data of step 2 is downsampled and output to the hybrid void convolution block; Step 4: Use the output of step 3 as the input of the hybrid hole grouping module to extract dense features and input the data into the upsampling module.

3. The real-time semantic segmentation method according to claim 2, characterized in that The step 1 includes the following specific steps: After obtaining image data of road scenes, three 3×3 convolutions are used to extract initial features. The stride of the first convolution is set to 2 to downsample the image. The obtained initial feature map is used as the input of the lightweight channel attention module.

4. The real-time semantic segmentation method according to claim 2, characterized in that The step 2 includes the following specific steps: First, a 1×1×C vector is obtained through global average pooling. Then, the weight of each channel is obtained through one-dimensional convolution to complete the information interaction between channels. By adding a channel attention mechanism at different stages of the network, the feature expression ability of the model is adaptively improved, and the connection between local and contextual information is promoted. It can be expressed by the following formula: ECA(F)=σ(f k×k (T(AvgP(F))))×F (1) Where T represents tensor compression, transposition and expansion operations, f k×k represents the adaptively selected convolution kernel of size k, AvgP represents the global average pooling, σ is the sigmoid activation function, and F is the input feature map; Before each channel attention module, the input data and the data downsampled from the original image are skip-connected as the input of the channel attention module to compensate for information loss.

5. The real-time semantic segmentation method according to claim 2, characterized in that: The step 3 includes the following specific steps: The downsampling block is applied to all downsampling layers and expanded into two modes. When the number of output channels is less than the number of input channels, this module is a simple 3×3 convolution with a stride of 2. Otherwise, a 2×2 maximum pooling layer with a stride of 2 is added, and then the convolution and pooling feature maps are concatenated to form the final downsampled output.

6. The real-time semantic segmentation method according to claim 2, characterized in that: Two hybrid hole grouping blocks are used in the entire network, which are composed of 3 and 6 hybrid hole grouping modules respectively. The dilation rates in the first block are {[1,2,5],[1,2,5],[1,2,5]}, and the dilation rates in the second block are {[3,5,7],[3,5,7],[3,5,7],[5,9,11],[5,9,11],[5,9,11]}. The receptive field is gradually expanded by setting different dilation rates in each module. In each hybrid hole grouping module, skip layer connection is used to connect the input with the deepest layer. In each HDG block, the input of the first HDG module is added to the output of the last HDG module.

7. The real-time semantic segmentation method according to claim 2, characterized in that: In step 5, only the feature maps of the two middle layers of the network are upsampled and then a skip-layer connection is established with the final output. Then, 2x upsampling is used to restore the original input image size, and finally the final segmentation prediction result is output.

Images