Real-time semantic segmentation method combining hierarchical decoding and multi-branch structures

Abstract

The present invention relates to a real-time semantic segmentation method combining hierarchical decoding and multi-branch structures, the method involving a data set construction module, which divides a data set into a training set and a verification set; a real-time semantic segmentation model construction module, which mainly combines a multi-branch structure and a hierarchical decoding structure to enable a model to acquire multi-scale semantic information and detail information, thereby enhancing the effect of segmenting small objects and multi-scale objects; and a real-time segmentation model training and evaluation module, which trains a real-time segmentation model on the constructed data set, and evaluates a model effect on the verification set. In the present invention, a real-time semantic segmentation method combining hierarchical decoding and multi-branch structures is established, by means of which high-precision segmentation can be performed at a relatively high inference speed, and a relatively high stability is achieved.

Description

A real-time semantic segmentation method combining step-by-step decoding and multi-branching Technical Field

[0001] The present invention relates to the field of scene understanding for autonomous driving vehicles, and in particular to a method for constructing a real-time semantic segmentation model for autonomous driving vehicles. Background Art

[0002] Semantic segmentation is a key task in computer vision. Its goal is to classify each pixel in an image to achieve a more comprehensive understanding of the image. In autonomous vehicle applications, fully convolutional networks based on deep learning have achieved high segmentation accuracy. However, these models suffer from slow inference speed, which is a challenge in autonomous driving. Therefore, building a semantic segmentation model with high segmentation accuracy and high real-time performance is crucial for analyzing traffic scenes, thereby reducing the computational burden.

[0003] With the rapid development of deep learning technology, convolutional neural networks have been widely used in image segmentation, replacing traditional manual feature extraction methods. Since the introduction of fully convolutional networks (FCNs), a series of novel networks have been proposed, such as DeepLabv3 and SFNet, aiming to improve segmentation accuracy. The continuous development and improvement of these networks have brought more accurate and robust solutions to image segmentation. Although these models achieve impressive segmentation accuracy, their high computational cost poses a significant obstacle to their application in autonomous driving scenarios. To meet real-time requirements, several real-time semantic segmentation methods have recently been proposed. These methods are typically designed as progressive decoding or multi-branch networks, which can effectively improve segmentation accuracy. However, using only progressive decoding structures may not effectively extract detailed image information. Conversely, relying solely on multi-branch structures may lack the integration of semantic information at different stages. Therefore, further exploration is needed in the design of real-time segmentation methods.

[0004] Summary of the Invention

[0005] Based on the technical problems raised above, the present invention proposes a real-time semantic segmentation method combining step-by-step decoding and multi-branching to solve the problem that existing real-time semantic segmentation methods cannot effectively extract detailed information of images or lack the integration of semantic information at different stages.

[0006] The technical means adopted by the present invention are as follows: a real-time semantic segmentation method combining step-by-step decoding and multi-branching, comprising the following steps:

[0007] Obtain the original images of the autonomous driving scene and annotate them to obtain semantically segmented labeled images. The original images and labeled images constitute the dataset;

[0008] The original image is segmented in real time to obtain the scene segmentation results for autonomous driving.

[0009] The real-time segmentation of the original image comprises the following steps:

[0010] The detail information extraction branch extracts the features extracted from the original image through convolution, performs a first convolution on them, and fuses them with the features output by the first residual network module in the semantic information extraction branch through a first feature fusion module. The features are then subjected to a second convolution to obtain detail information including the boundaries and textures of the original image. The features output by the first feature fusion module are passed through an edge detection head to obtain an edge prediction map of the original image. The cross entropy loss is calculated with the true edge label map obtained by the Canny operator on the label image to optimize the features output by the feature fusion module.

[0011] The semantic information extraction branch extracts the features extracted from the original image through convolution, passes them through the first residual network module, the second residual network module, the third residual network module, and the multi-scale information extraction module in sequence, and fuses them with the features output by the second residual network module through the second feature fusion module. The fused features are then fused with the features output by the first residual network module through the third feature fusion module to obtain semantic information.

[0012] The multi-branch information fusion module uses the fourth feature fusion module to fuse the detail information of the detail information extraction branch with the semantic information of the semantic information extraction branch to obtain features containing detail information and semantic information. This is used as the feature for predicting the segmentation result, and the scene segmentation result for autonomous driving is obtained through the segmentation head.

[0013] A real-time semantic segmentation system combining step-by-step decoding and multi-branching, including:

[0014] The dataset construction module is used to obtain and annotate original images of autonomous driving scenes to obtain semantically segmented labeled images. The original images and labeled images constitute the dataset.

[0015] The real-time semantic segmentation model building module is used to segment the original image in real time to obtain the scene segmentation results for autonomous driving.

[0016] The real-time semantic segmentation model construction module includes:

[0017] The detail information extraction branch is used to perform feature fusion on the features extracted from the original image by convolution, after the first convolution, and the features output by the first residual network module in the semantic information extraction branch, and then perform feature fusion on the first feature fusion module, and then obtain detail information including the boundaries and textures in the original image by the second convolution; the features output by the first feature fusion module are passed through the edge detection head to obtain an edge prediction map of the original image, and the cross entropy loss is calculated with the true edge label map obtained by the Canny operator on the label image to optimize the features output by the feature fusion module;

[0018] The semantic information extraction branch is used to sequentially pass the features extracted from the original image through the first residual network module, the second residual network module, the third residual network module, and the multi-scale information extraction module, and then fuse the features output by the second residual network module through the second feature fusion module. The fused features are then fused with the features output by the first residual network module through the third feature fusion module to obtain semantic information.

[0019] The multi-branch information fusion module is used to use the fourth feature fusion module to fuse the detail information of the detail information extraction branch with the semantic information of the semantic information extraction branch to obtain features containing detail information and semantic information. This is used as a feature for predicting the segmentation result, and the scene segmentation result for autonomous driving is obtained through the segmentation head.

[0020] The first feature fusion module, the second feature fusion module, the third feature fusion module, and the fourth feature fusion module all adopt the UAFM module.

[0021] The edge detection head is composed of convolution, activation function and bilinear interpolation in sequence.

[0022] The segmentation head is composed of convolution, activation function, and bilinear interpolation in sequence.

[0023] A real-time semantic segmentation device combining step-by-step decoding and multiple branches comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the real-time semantic segmentation method combining step-by-step decoding and multiple branches when executing the computer program.

[0024] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the real-time semantic segmentation method combining step-by-step decoding and multi-branching is implemented.

[0025] The beneficial effects of the present invention are:

[0026] 1. This paper constructs a new real-time semantic segmentation model, which combines the advantages of multi-branch structure and step-by-step decoding structure.

[0027] 2. The network structure proposed in this invention achieves a good balance between speed and accuracy, has strong robustness, and is suitable for the field of autonomous driving vehicle traffic scenarios. It can also be extended to semantic segmentation problems and has good practicality.

[0028] 3. The present invention establishes a real-time semantic segmentation method that combines step-by-step decoding and multi-branching, which can perform high-precision segmentation at a faster inference speed and has strong stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is an overall flow chart of the construction of a real-time semantic segmentation model of the present invention;

[0030] FIG2 is a diagram showing the overall network structure of the real-time segmentation model of the present invention. DETAILED DESCRIPTION

[0031] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, the specific implementation methods of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the invention. Therefore, the present invention is not limited to the specific implementation methods disclosed below.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of the art to which the present invention pertains. The terms used in the specification of the invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention.

[0033] The present invention relates to a real-time semantic segmentation method combining step-by-step decoding and multiple branches, comprising a data set construction module, a real-time semantic segmentation model construction module, and a real-time segmentation model training and evaluation module. The data set construction module includes pixel-by-pixel annotation of the collected images, and division into training sets and verification sets; the real-time semantic segmentation model construction module includes a detail information extraction network and a semantic information extraction network. The detail information extraction network retains the detail information of the original image by retaining the high-resolution features of the original image and introducing boundary loss guidance, and the semantic information extraction network extracts semantic information by increasing the depth of the network and the number of channels and constructing a step-by-step decoding structure; the real-time segmentation model training and evaluation module includes training and verifying the model through the constructed data set. The present invention establishes a real-time semantic segmentation method combining step-by-step decoding and multiple branches, which can perform high-precision segmentation of different street scenes at a faster inference speed and has strong stability.

[0034] A real-time semantic segmentation method combining step-by-step decoding and multi-branching includes the following steps:

[0035] The dataset construction module divides the dataset into training set and validation set;

[0036] Real-time semantic segmentation model building module to achieve real-time segmentation of original images;

[0037] Training and evaluating real-time segmentation model modules, used to train and evaluate the model;

[0038] The real-time semantic segmentation model construction module includes:

[0039] Detail information extraction network, which extracts detail information by retaining the high-resolution features of the original image and guiding the network through edge recognition loss;

[0040] The semantic information extraction network consists of three residual basic blocks, a step-by-step decoding structure and a feature fusion method.

[0041] The multi-branch information fusion module adopts UAFM as the information fusion method and uses the fused features as the features of the final predicted segmentation results.

[0042] The detail information extraction network extracts the detail information of the original image through the following two methods:

[0043] Preserving high-resolution features: In the design of the detail information extraction network, no downsampling method is adopted, and high-resolution features are always maintained to reduce the damage to the original image spatial structure.

[0044] Introducing edge recognition loss: By adding an edge detection head to the detail information extraction network, an additional edge detection task is constructed to guide the network to learn detail features. The label image for edge detection is obtained by applying the Canny operator to the segmentation label image.

[0045] The edge detection head consists of basic convolution, activation function, and bilinear interpolation.

[0046] The edge detection loss is a binary cross entropy loss.

[0047] The semantic information extraction network extracts the semantic information of the original image. Specifically, the features obtained by the original image through the downsampling layer are sequentially passed through multiple residual network modules and multi-scale receptive field extraction modules to obtain features of different stages and with different receptive fields. Then, features containing multi-scale semantic information are obtained through a step-by-step decoding method.

[0048] The semantic information extraction network adopts a U-Net structure, including three residual network modules, a multi-scale information extraction module, a first feature fusion module, and a second feature fusion module;

[0049] The residual network module is composed of a convolutional layer, a batch normalization layer, and an activation layer connected in sequence to achieve feature extraction;

[0050] The first feature fusion module performs feature fusion on the features obtained by the multi-scale receptive field extraction module and the features output by the second residual network module stage;

[0051] The second feature fusion module performs feature fusion on the features output by the first feature fusion module and the features output by the first residual network module stage as the output of the semantic information extraction network.

[0052] A real-time semantic segmentation method combining step-by-step decoding and multi-branching, including:

[0053] The dataset construction module annotates the collected autonomous driving scene images pixel by pixel and divides them into training and validation sets;

[0054] Real-time semantic segmentation model building module, used to build real-time segmentation models that achieve a good balance between speed and accuracy;

[0055] Train and evaluate the real-time segmentation model module, and train and verify the model using the constructed dataset.

[0056] The real-time semantic segmentation method combining step-by-step decoding and multi-branching consists of two branches, namely a detail information extraction network and a semantic information extraction network. It extracts details and semantic information from the original image and fuses them together through a feature fusion network to achieve real-time semantic segmentation of the image.

[0057] The detail information extraction network extracts detail information of the original image by retaining high-resolution features and adopting a boundary loss-guided method.

[0058] The boundary loss is obtained by adding an edge detection head to the detail branch to output the edge detection result, and then calculating the binary cross entropy loss on the edge label.

[0059] The edge labels are obtained by applying the Canny operator to the segmentation labels.

[0060] The edge detection head consists of convolution, activation layers and bilinear interpolation.

[0061] The semantic information extraction network extracts the semantic information of the original image. Specifically, the features of the original image obtained by the downsampling layer are sequentially passed through multiple residual network modules and multi-scale receptive field extraction modules to obtain features of different stages and with different receptive fields.

[0062] The semantic information extraction network adopts a step-by-step decoding structure, including multiple residual network modules, a multi-scale information extraction module, a first feature fusion module, and a second feature fusion module;

[0063] The residual network module is composed of a convolutional layer, a batch normalization layer, and an activation layer connected in sequence to achieve feature extraction;

[0064] The feature fusion module adopts the UAFM model.

[0065] The present invention constructs a real-time semantic segmentation method combining step-by-step decoding and multi-branching, which can segment street scenes in real time with high accuracy and speed and has strong stability.

[0066] A real-time semantic segmentation method combining step-by-step decoding and multi-branching. As shown in Figure 1, it includes a dataset construction module, a real-time semantic segmentation model construction module, and a real-time segmentation model training and evaluation module. The dataset construction module is used to divide the training set into a training set and a validation set to construct a dataset suitable for training and evaluation. The constructed real-time semantic segmentation model is then trained and finally evaluated on the validation set.

[0067] Real-time segmentation model construction module. As shown in Figure 2, the model consists of two main branches: the detail branch and the semantic branch. These two branches share the first convolutional layer, which is mainly composed of convolutional layers with a stride of 2. For the design of the semantic branch, the features of the first convolutional layer are first passed through the first, second, and third residual blocks to increase the number of channels and expand the receptive field. The first, second, and third residual blocks are all composed of the basic blocks of the residual network. The features of the third residual block are then input into the multi-scale information extraction module to further extract multi-scale receptive field information. In order to make more effective use of the semantic information at different stages, a step-by-step decoding structure is adopted to fuse the multi-stage semantic information. The fusion module adopts the UAFM module. For the design of the detail branch, the features of the first convolutional layer are first passed through the first convolutional layer of the detail branch to further extract features, and then the features of the first residual stage of the semantic branch are fused using the UAFM module. Finally, the features are passed through the second convolutional layer of the detail branch to further increase the representation ability of the detail branch. In order to enable the network to extract detail information more effectively, an edge head is connected after the feature fusion module of the detail branch to output edge detection results. The segmentation label is passed through the Canny operator to extract edge information as the label of the edge branch, and finally the edge loss is used to optimize the edge branch.

[0068] Building a real-time semantic segmentation method combining step-by-step decoding and multi-branching includes the following steps:

[0069] S1: Build the dataset.

[0070] S2: Build a real-time segmentation model, which mainly includes the construction of detail branches and semantic branches.

[0071] S3: Train the constructed model.

[0072] S4: Save the weights of the trained model.

[0073] S5: Evaluate the model effect.

[0074] The present invention provides a real-time semantic segmentation method combining step-by-step decoding and multiple branches. The method comprises a data set construction module, a real-time semantic segmentation model construction module, and a training and evaluation module for the real-time segmentation model. The method can segment street scene information in real time.

[0075] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A real-time semantic segmentation method combining step-by-step decoding and multi-branching, characterized in that: The following steps are involved: Obtain the original image of the autonomous driving scene and annotate it to obtain the semantically segmented label image. The original image and the label image constitute the data set; The original image is segmented in real time to obtain the scene segmentation result for autonomous driving.

2. A real-time semantic segmentation method combining step-by-step decoding and multi-branching according to claim 1, characterized in that: The real-time segmentation of the original image comprises the following steps: The detail information extraction branch extracts the features extracted from the original image through convolution, performs feature fusion with the features output by the first residual network module in the semantic information extraction branch through the first feature fusion module after the first convolution, and then obtains detail information including the boundary and texture in the original image through the second convolution; the features output by the first feature fusion module are passed through the edge detection head to obtain the edge prediction map of the original image, and the cross entropy loss calculation is performed with the real edge label map obtained by the Canny operator of the label image to optimize the features output by the feature fusion module; The semantic information extraction branch extracts the features of the original image through convolution, passes through the first residual network module, the second residual network module, the third residual network module, and the multi-scale information extraction module in sequence, and then fuses the features output by the second residual network module through the second feature fusion module. The fused features are then fused with the features output by the first residual network module through the third feature fusion module to obtain semantic information. The multi-branch information fusion module adopts the fourth feature fusion module to fuse the detail information of the detail information extraction branch with the semantic information of the semantic information extraction branch to obtain features containing detail information and semantic information as features for predicting segmentation results, and obtain the scene segmentation results of autonomous driving through the segmentation head.

3. A real-time semantic segmentation system combining step-by-step decoding and multi-branching, characterized in that: include: The dataset construction module is used to obtain the original images of the autonomous driving scene and annotate them to obtain the labeled images of semantic segmentation. The original images and labeled images constitute the dataset; The real-time semantic segmentation model building module is used to perform real-time segmentation of the original image and obtain the scene segmentation results for autonomous driving.

4. A real-time semantic segmentation system combining step-by-step decoding and multi-branching according to claim 3, characterized in that: The real-time semantic segmentation model building module includes: A detail information extraction branch is used to perform feature fusion on the features extracted from the original image by convolution, after the first convolution, and the features output by the first residual network module in the semantic information extraction branch through a first feature fusion module, and then obtain detail information including the boundary and texture in the original image through a second convolution; the features output by the first feature fusion module are passed through an edge detection head to obtain an edge prediction map of the original image, and a cross entropy loss calculation is performed on the real edge label map obtained by the Canny operator of the label image, so as to optimize the features output by the feature fusion module; The semantic information extraction branch is used to sequentially pass the features extracted from the original image by convolution through the first residual network module, the second residual network module, the third residual network module, and the multi-scale information extraction module, and then fuse the features output by the second residual network module through the second feature fusion module, and fuse the fused features with the features output by the first residual network module through the third feature fusion module to obtain semantic information; The multi-branch information fusion module is used to use the fourth feature fusion module to fuse the detail information of the detail information extraction branch with the semantic information of the semantic information extraction branch to obtain features containing detail information and semantic information as features for predicting segmentation results, and obtain the scene segmentation results of autonomous driving through the segmentation head.

5. A real-time semantic segmentation system combining step-by-step decoding and multi-branching according to claim 3, characterized in that: The first feature fusion module, the second feature fusion module, the third feature fusion module, and the fourth feature fusion module all adopt the UAFM module.

6. A real-time semantic segmentation system combining step-by-step decoding and multi-branching according to claim 3, characterized in that: The edge detection head is composed of convolution, activation function and bilinear interpolation in sequence.

7. A real-time semantic segmentation system combining step-by-step decoding and multi-branching according to claim 3, characterized in that: The segmentation head is composed of convolution, activation function and bilinear interpolation in sequence.

8. A real-time semantic segmentation device combining step-by-step decoding and multi-branching, characterized in that: It comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement a real-time semantic segmentation method combining step-by-step decoding and multi-branching as described in any one of claims 1-2 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, a real-time semantic segmentation method combining step-by-step decoding and multi-branching as described in any one of claims 1-2 is implemented.

Images