Real-time semantic segmentation method combining step-by-step decoding and multiple branches
By combining step-by-step decoding and real-time semantic segmentation methods of multi-branches, and integrating detailed information and verbal information, the problems of slow inference speed and insufficient extraction of detailed information in the existing technology are solved, and fast and high-precision segmentation in autonomous driving scenarios are achieved.
Patent Information
- Application Number
- CN202311084000.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-28
- Publication Date
- 2025-02-28
AI Technical Summary
The existing real-time semantic segmentation methods are difficult to effectively extract the detailed information of images or lack the integration of semantic information at different stages, resulting in slow inference speed in autonomous driving scenarios.
A real-time semantic segmentation method combining step-by-step decoding and multiple branches is adopted to integrate feature fusion through detailed information extraction branches and semantic information extraction branches. The multi-branch information fusion module is used to integrate detailed information and semantic information to generate high-precision segmentation results.
It realizes fast and high-precision semantic segmentation in autonomous driving scenarios, has strong robustness and stability, and can effectively reduce the computing burden.
Smart Images
Figure CN119540571A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of scene understanding of autonomous driving vehicles, and in particular to a method for constructing a real-time semantic segmentation model for autonomous driving vehicles. Background Art
[0002] Semantic segmentation is an important task in the field of computer vision. Its goal is to classify each pixel in an image to achieve a more comprehensive understanding of the image. In the application of autonomous vehicles, deep learning-based fully convolutional networks have achieved high segmentation accuracy. However, these models have slow inference speed, which is a challenge in autonomous driving. Therefore, building a semantic segmentation model with high real-time requirements and high segmentation accuracy is crucial for parsing traffic scenes, and such a model can reduce the computational burden.
[0003] With the rapid development of deep learning technology, convolutional neural networks have been widely used in image segmentation, replacing traditional manual feature extraction methods. Since the introduction of the fully convolutional network FCN, a series of novel networks have been proposed, such as DeepLabv3, SFNet, etc., aiming to improve segmentation accuracy. The continuous development and improvement of these networks have brought more accurate and robust solutions to image segmentation. Although these models have achieved impressive segmentation accuracy, their high computational cost poses a major obstacle to their application in autonomous driving scenarios. In order to meet the real-time requirements, several real-time semantic segmentation methods have been proposed recently. These methods are usually designed as progressive decoding or multi-branch networks, which can effectively improve segmentation accuracy. However, using only the progressive decoding structure may not effectively extract the detailed information of the image. On the contrary, relying solely on the multi-branch structure may lack the integration of semantic information at different stages, so the design of real-time segmentation methods needs further exploration. Summary of the invention
[0004] Based on the technical problems raised above, the present invention proposes a real-time semantic segmentation method combining step-by-step decoding and multi-branching to solve the problem that existing real-time semantic segmentation methods cannot effectively extract detailed information of images or lack the integration of semantic information at different stages.
[0005] The technical means adopted by the present invention are as follows: a real-time semantic segmentation method combining step-by-step decoding and multi-branching, comprising the following steps:
[0006] Obtain the original image of the autonomous driving scene and annotate it to obtain the semantically segmented label image. The original image and the label image constitute the data set;
[0007] The original image is segmented in real time to obtain the scene segmentation result for autonomous driving.
[0008] The real-time segmentation of the original image comprises the following steps:
[0009] The detail information extraction branch extracts the features extracted from the original image through convolution, performs feature fusion with the features output by the first residual network module in the semantic information extraction branch through the first feature fusion module after the first convolution, and then obtains detail information including the boundary and texture in the original image through the second convolution; the features output by the first feature fusion module are passed through the edge detection head to obtain the edge prediction map of the original image, and the cross entropy loss calculation is performed with the real edge label map obtained by the Canny operator of the label image to optimize the features output by the feature fusion module;
[0010] The semantic information extraction branch extracts the features of the original image through convolution, passes through the first residual network module, the second residual network module, the third residual network module, and the multi-scale information extraction module in sequence, and then fuses the features output by the second residual network module through the second feature fusion module. The fused features are then fused with the features output by the first residual network module through the third feature fusion module to obtain semantic information.
[0011] The multi-branch information fusion module adopts the fourth feature fusion module to fuse the detail information of the detail information extraction branch with the semantic information of the semantic information extraction branch to obtain features containing detail information and semantic information as features for predicting segmentation results, and obtain the scene segmentation results of autonomous driving through the segmentation head.
[0012] A real-time semantic segmentation system combining step-by-step decoding and multi-branching, comprising:
[0013] The dataset construction module is used to obtain the original images of the autonomous driving scene and annotate them to obtain the labeled images of semantic segmentation. The original images and labeled images constitute the dataset.
[0014] The real-time semantic segmentation model building module is used to perform real-time segmentation of the original image and obtain the scene segmentation results for autonomous driving.
[0015] The real-time semantic segmentation model building module includes:
[0016] A detail information extraction branch is used to perform feature fusion on the features extracted from the original image by convolution, after the first convolution, and the features output by the first residual network module in the semantic information extraction branch through a first feature fusion module, and then obtain detail information including the boundary and texture in the original image through a second convolution; the features output by the first feature fusion module are passed through an edge detection head to obtain an edge prediction map of the original image, and a cross entropy loss calculation is performed on the real edge label map obtained by the Canny operator of the label image, so as to optimize the features output by the feature fusion module;
[0017] The semantic information extraction branch is used to sequentially pass the features extracted from the original image by convolution through the first residual network module, the second residual network module, the third residual network module, and the multi-scale information extraction module, and then fuse the features output by the second residual network module through the second feature fusion module, and fuse the fused features with the features output by the first residual network module through the third feature fusion module to obtain semantic information;
[0018] The multi-branch information fusion module is used to use the fourth feature fusion module to fuse the detail information of the detail information extraction branch with the semantic information of the semantic information extraction branch to obtain features containing detail information and semantic information as features for predicting segmentation results, and obtain the scene segmentation results of autonomous driving through the segmentation head.
[0019] The first feature fusion module, the second feature fusion module, the third feature fusion module, and the fourth feature fusion module all adopt the UAFM module.
[0020] The edge detection head is composed of convolution, activation function and bilinear interpolation in sequence.
[0021] The segmentation head is composed of convolution, activation function and bilinear interpolation in sequence.
[0022] A real-time semantic segmentation device combining step-by-step decoding and multiple branches comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the real-time semantic segmentation method combining step-by-step decoding and multiple branches when executing the computer program.
[0023] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, a real-time semantic segmentation method combining step-by-step decoding and multi-branching is implemented.
[0024] The beneficial effects of the present invention are:
[0025] 1. The present invention constructs a new real-time semantic segmentation model, which combines the advantages of multi-branch structure and step-by-step decoding structure.
[0026] 2. The network structure proposed in the present invention achieves a good balance between speed and accuracy, has strong robustness, is suitable for the field of autonomous driving vehicle traffic scenarios, and can also be extended to semantic segmentation problems, with good practicality.
[0027] 3. The present invention establishes a real-time semantic segmentation method that combines step-by-step decoding and multi-branching, which can perform high-precision segmentation at a faster reasoning speed and has stronger stability. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 The overall flow chart for constructing the real-time semantic segmentation model of the present invention;
[0029] Figure 2 FIG. 4 is an overall network structure diagram of the real-time segmentation model of the present invention. DETAILED DESCRIPTION
[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.
[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0032] The present invention relates to a real-time semantic segmentation method combining step-by-step decoding and multiple branches, including a data set construction module, a real-time semantic segmentation model construction module, and a training and evaluation real-time segmentation model module. The data set construction module includes pixel-by-pixel annotation of the collected images, and division of training sets and verification sets; the real-time semantic segmentation model construction module includes a detail information extraction network and a semantic information extraction network. The detail information extraction network retains the detail information of the original image by retaining the high-resolution features of the original image and introducing boundary loss guidance, and the semantic information extraction network extracts semantic information by increasing the depth of the network and the number of channels and constructing a step-by-step decoding structure; the training and evaluation real-time segmentation model module includes training and verifying the model through a constructed data set. The present invention establishes a real-time semantic segmentation method combining step-by-step decoding and multiple branches, which can perform high-precision segmentation of different street scenes at a faster reasoning speed and has strong stability.
[0033] A real-time semantic segmentation method combining step-by-step decoding and multi-branching includes the following steps:
[0034] The dataset construction module divides the dataset into training set and validation set;
[0035] Real-time semantic segmentation model building module to achieve real-time segmentation of original images;
[0036] Training and evaluating real-time segmentation model modules, used to train and evaluate the model;
[0037] The real-time semantic segmentation model building module includes:
[0038] Detail information extraction network, which extracts detail information by retaining the high-resolution features of the original image and guiding the network through edge recognition loss;
[0039] The semantic information extraction network consists of three residual basic blocks, a step-by-step decoding structure and a feature fusion method.
[0040] The multi-branch information fusion module adopts UAFM as the information fusion method and uses the fused features as the features of the final predicted segmentation results.
[0041] The detail information extraction network extracts the detail information of the original image through the following two methods:
[0042] Preserve high-resolution features: In the design of the detail information extraction network, no downsampling method is adopted, and high-resolution features are always maintained to reduce the damage to the original image spatial structure.
[0043] Introducing edge recognition loss: By adding an edge detection head to the detail information extraction network, an additional edge detection task is constructed to guide the network to learn detail features. The label image for edge detection is obtained by applying the Canny operator to the segmentation label image.
[0044] The edge detection head is composed of basic convolution, activation function and bilinear interpolation.
[0045] The edge detection loss is a binary cross entropy loss.
[0046] The semantic information extraction network extracts the semantic information of the original image, specifically: the features of the original image obtained by the downsampling layer are sequentially passed through multiple residual network modules and multi-scale receptive field extraction modules to obtain features of different stages and with different receptive fields, and then features containing multi-scale semantic information are obtained through a step-by-step decoding method.
[0047] The semantic information extraction network adopts a U-Net structure, including three residual network modules, a multi-scale information extraction module, a first feature fusion module, and a second feature fusion module;
[0048] The residual network module is composed of a convolutional layer, a batch normalization layer and an activation layer connected in sequence to achieve feature extraction;
[0049] The first feature fusion module performs feature fusion on the features obtained by the multi-scale receptive field extraction module and the features output by the second residual network module stage;
[0050] The second feature fusion module performs feature fusion on the features output by the first feature fusion module and the features output by the first residual network module stage as the output of the semantic information extraction network.
[0051] A real-time semantic segmentation method combining step-by-step decoding and multi-branching, comprising:
[0052] The dataset construction module annotates the collected autonomous driving scene images pixel by pixel and divides them into training and validation sets;
[0053] Real-time semantic segmentation model building module, used to build a real-time segmentation model that strikes a good balance between speed and accuracy;
[0054] Train and evaluate the real-time segmentation model module, and train and verify the model through the constructed dataset.
[0055] The real-time semantic segmentation method combining step-by-step decoding and multi-branch consists of two branches, namely a detail information extraction network and a semantic information extraction network. It extracts details and semantic information from the original image and fuses them together through a feature fusion network to achieve real-time semantic segmentation of the image.
[0056] The detail information extraction network extracts detail information of the original image by retaining high-resolution features and adopting a boundary loss guided method.
[0057] The boundary loss is obtained by adding an edge detection head to the detail branch to output the edge detection result, and then calculating the binary cross entropy loss on the edge label.
[0058] The edge labels are obtained by applying the Canny operator to the segmentation labels.
[0059] The edge detection head consists of convolution, activation layers and bilinear interpolation.
[0060] The semantic information extraction network extracts the semantic information of the original image, specifically: the features of the original image obtained by the downsampling layer are sequentially passed through multiple residual network modules and multi-scale receptive field extraction modules to obtain features of different stages and with different receptive fields;
[0061] The semantic information extraction network adopts a step-by-step decoding structure, including multiple residual network modules, a multi-scale information extraction module, a first feature fusion module, and a second feature fusion module;
[0062] The residual network module is composed of a convolutional layer, a batch normalization layer and an activation layer connected in sequence to achieve feature extraction;
[0063] The feature fusion module adopts the UAFM model.
[0064] The present invention constructs a real-time semantic segmentation method combining step-by-step decoding and multi-branching, which can segment street scenes in real time with high accuracy and speed and has strong stability.
[0065] A real-time semantic segmentation method combining step-by-step decoding and multi-branching. Figure 1 As shown in the figure, it includes: a dataset construction module, a real-time semantic segmentation model construction module, and a real-time segmentation model training and evaluation module. The dataset construction module is used to divide the training set and the validation set, build a dataset that can be trained and evaluated, and then train the constructed real-time semantic segmentation model, and finally evaluate the trained model on the validation set.
[0066] Real-time segmentation model building module. Figure 2 As shown in the figure, the model is mainly composed of two branches, namely the detail branch and the semantic branch. The two branches share the first convolution layer, which is mainly composed of convolution layers with a step size of 2. For the design of the semantic branch, the features of the first convolution layer are firstly passed through the first, second and third reference blocks to increase the number of channels and expand the receptive field. The first, second and third residual blocks are all composed of basic blocks of the residual network. The features of the third residual block are then input into the multi-scale information extraction module to further extract the multi-scale receptive field information. In order to make more full use of the semantic information at different stages, a step-by-step decoding structure is used to fuse the multi-stage semantic information, and the fusion module adopts the UAFM module. For the design of detail branches, the features of the first convolutional layer are firstly further extracted through the first convolutional layer of the detail branch, and then the features of the first residual stage of the semantic branch are fused by the UAFM module. Finally, the representation ability of the detail branch is further increased through the second convolutional layer of the detail branch. In order to enable the network to extract detail information more effectively, an edge head is connected after the feature fusion module of the detail branch to output the edge detection results. The segmentation label is extracted through the Canny operator as the label of the edge branch, and finally the edge loss is used to optimize the edge branch.
[0067] Building a real-time semantic segmentation method combining step-by-step decoding and multi-branching includes the following steps:
[0068] S1: Build the dataset.
[0069] S2: Build a real-time segmentation model, which mainly includes detail branch and semantic branch construction.
[0070] S3: Train the constructed model.
[0071] S4: Save the weights of the trained model.
[0072] S5: Evaluate the model performance.
[0073] The real-time semantic segmentation method combining step-by-step decoding and multiple branches described in the present invention constructs a real-time semantic segmentation method combining step-by-step decoding and multiple branches through a data set construction module, a real-time semantic segmentation model construction module and a training and evaluation real-time segmentation model module, and can segment street scene information in real time.
[0074] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.
Claims
1. A real-time semantic segmentation method combining step-by-step decoding and multi-branching, characterized in that: The following steps are involved: Obtain the original image of the autonomous driving scene and annotate it to obtain the semantically segmented label image. The original image and the label image constitute the data set; The original image is segmented in real time to obtain the scene segmentation result for autonomous driving.
2. A real-time semantic segmentation method combining step-by-step decoding and multi-branching according to claim 1, characterized in that: The real-time segmentation of the original image comprises the following steps: The detail information extraction branch extracts the features extracted from the original image through convolution, performs feature fusion with the features output by the first residual network module in the semantic information extraction branch through the first feature fusion module after the first convolution, and then obtains detail information including the boundary and texture in the original image through the second convolution; the features output by the first feature fusion module are passed through the edge detection head to obtain the edge prediction map of the original image, and the cross entropy loss calculation is performed with the real edge label map obtained by the Canny operator of the label image to optimize the features output by the feature fusion module; The semantic information extraction branch extracts the features of the original image through convolution, passes through the first residual network module, the second residual network module, the third residual network module, and the multi-scale information extraction module in sequence, and then fuses the features output by the second residual network module through the second feature fusion module. The fused features are then fused with the features output by the first residual network module through the third feature fusion module to obtain semantic information. The multi-branch information fusion module adopts the fourth feature fusion module to fuse the detail information of the detail information extraction branch with the semantic information of the semantic information extraction branch to obtain features containing detail information and semantic information as features for predicting segmentation results, and obtain the scene segmentation results of autonomous driving through the segmentation head.
3. A real-time semantic segmentation system combining step-by-step decoding and multi-branching, characterized in that: include: The dataset construction module is used to obtain the original images of the autonomous driving scene and annotate them to obtain the labeled images of semantic segmentation. The original images and labeled images constitute the dataset. The real-time semantic segmentation model building module is used to perform real-time segmentation of the original image and obtain the scene segmentation results for autonomous driving.
4. A real-time semantic segmentation system combining step-by-step decoding and multi-branching according to claim 3, characterized in that: The real-time semantic segmentation model building module includes: A detail information extraction branch is used to perform feature fusion on the features extracted from the original image by convolution, after the first convolution, and the features output by the first residual network module in the semantic information extraction branch through a first feature fusion module, and then obtain detail information including the boundary and texture in the original image through a second convolution; the features output by the first feature fusion module are passed through an edge detection head to obtain an edge prediction map of the original image, and a cross entropy loss calculation is performed on the real edge label map obtained by the Canny operator of the label image, so as to optimize the features output by the feature fusion module; The semantic information extraction branch is used to sequentially pass the features extracted from the original image by convolution through the first residual network module, the second residual network module, the third residual network module, and the multi-scale information extraction module, and then fuse the features output by the second residual network module through the second feature fusion module, and fuse the fused features with the features output by the first residual network module through the third feature fusion module to obtain semantic information; The multi-branch information fusion module is used to use the fourth feature fusion module to fuse the detail information of the detail information extraction branch with the semantic information of the semantic information extraction branch to obtain features containing detail information and semantic information as features for predicting segmentation results, and obtain the scene segmentation results of autonomous driving through the segmentation head.
5. A real-time semantic segmentation system combining step-by-step decoding and multi-branching according to claim 3, characterized in that: The first feature fusion module, the second feature fusion module, the third feature fusion module, and the fourth feature fusion module all adopt the UAFM module.
6. A real-time semantic segmentation system combining step-by-step decoding and multi-branching according to claim 3, characterized in that: The edge detection head is composed of convolution, activation function and bilinear interpolation in sequence.
7. A real-time semantic segmentation system combining step-by-step decoding and multi-branching according to claim 3, characterized in that: The segmentation head is composed of convolution, activation function and bilinear interpolation in sequence.
8. A real-time semantic segmentation device combining step-by-step decoding and multi-branching, characterized in that: It comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement a real-time semantic segmentation method combining step-by-step decoding and multi-branching as described in any one of claims 1-2 when executing the computer program.
9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, a real-time semantic segmentation method combining step-by-step decoding and multi-branching as described in any one of claims 1-2 is implemented.