Street scene real-time semantic segmentation method and device for automatic driving vehicle

Abstract

The invention relates to an automatic driving vehicle-oriented street scene real-time semantic segmentation method and device. The method comprises the steps that an image acquisition module performs real-time original image acquisition through a monocular camera; the real-time semantic segmentation module constructs a real-time semantic segmentation model to realize real-time segmentation of the original image; and the vehicle-mounted camera real-time segmentation module obtains an environment picture in real time through a vehicle-mounted camera, and sends the environment picture to the trained real-time semantic segmentation model to obtain a final scene segmentation result. According to the method, real-time scene segmentation of the autonomous vehicle facing different street scenes is established, high-precision segmentation can be carried out on different street scenes at a high reasoning speed, and the method has high stability.

Description

Technical Field

[0001] The present invention relates to the field of scene understanding of autonomous driving vehicles, and in particular to a method for constructing a real-time semantic segmentation model for autonomous driving vehicles. Background Art

[0002] Semantic segmentation is a basic but critical task in computer vision. Its goal is to predict a specific label for each pixel in the image, so as to fully understand the information contained in the entire image scene. Many fully convolutional networks based on deep learning have achieved high segmentation accuracy, but when applied to autonomous vehicles, there is still the problem of slow model inference speed. Therefore, building a semantic segmentation model that meets real-time requirements and has high segmentation accuracy is of great research significance for traffic scene analysis.

[0003] With the rapid development of deep learning technology, convolutional neural networks have been widely used in image segmentation, replacing traditional manual feature extraction methods. Since the introduction of the fully convolutional network FCN, a series of novel networks have been proposed, such as DeepLabv3, SFNet, etc., aiming to improve segmentation accuracy. The continuous development and improvement of these networks have brought more accurate and robust solutions to image segmentation. Although these models have achieved impressive segmentation accuracy, their high computational cost poses a major obstacle to their application in autonomous driving scenarios. In order to meet the real-time requirements, in recent years, researchers have proposed several real-time semantic segmentation methods (PIDNet, BiSeNet, STDCNet), which add other branches to extract spatial information or bounding box information, and then fuse with semantic information to improve segmentation accuracy. However, for the branch design of extracting spatial information, these methods only guide the network to extract more spatial information by retaining high-resolution features or increasing segmentation loss. Unfortunately, these methods for extracting spatial information are not the most effective and need further exploration and improvement. Summary of the invention

[0004] Based on the technical problems raised above, the present invention proposes a real-time semantic segmentation method and device for street scenes for autonomous driving vehicles, which are used to solve the problems of poor effectiveness, unbalanced speed and accuracy of multi-branch structure spatial information extraction methods in existing real-time semantic segmentation methods.

[0005] The technical means adopted by the present invention are as follows: a real-time semantic segmentation method of street scenes for autonomous driving vehicles, comprising the following steps:

[0006] The image acquisition module acquires real-time raw images through a monocular camera;

[0007] The real-time semantic segmentation module builds a real-time semantic segmentation model to achieve real-time segmentation of the original image;

[0008] The vehicle camera real-time segmentation module obtains environmental images in real time through the vehicle camera and sends them to the trained real-time semantic segmentation model to obtain the final scene segmentation results.

[0009] The real-time semantic segmentation module constructs a real-time semantic segmentation model to achieve real-time segmentation of the original image, as follows:

[0010] A spatial information extraction network is used to extract the spatial information of the original image during the encoding and decoding process by using an autoencoder;

[0011] Semantic information extraction network, used to extract semantic information of the original image;

[0012] Edge information extraction network, used to extract edge information from the original image;

[0013] The UMF module is used to fuse information of multiple input features.

[0014] The semantic information extraction network extracts the semantic information of the original image, specifically: the features of the original image obtained by the downsampling layer are sequentially passed through multiple residual network modules and multi-scale receptive field extraction modules to obtain features of different stages and with different receptive fields;

[0015] The semantic information extraction network adopts a U-Net structure, including multiple residual network modules, a multi-scale receptive field extraction module, a first UMF module, and a second UMF module;

[0016] The multiple residual network modules include at least a first residual network module and a second residual network module;

[0017] The residual network module is composed of a convolutional layer, a batch normalization layer and an activation layer connected in sequence to achieve feature extraction;

[0018] The first UMF module performs feature fusion on the features obtained by the multi-scale receptive field extraction module and the features output by the second residual network module stage;

[0019] The second UMF module performs feature fusion on the features output by the first UMF module and the features output by the first residual network module stage as the output of the semantic information extraction network.

[0020] The spatial information extraction network extracts the spatial information of the original image by using an autoencoder during the encoding and decoding process, and performs the following steps:

[0021] The autoencoder includes an encoder and a decoder;

[0022] The encoder includes a first convolutional layer, a second convolutional layer, and a UMF module, which are used to encode the image; the features of the original image obtained by the downsampling layer are extracted by the first convolutional layer, and the features output by the first residual network module stage in the semantic information extraction network are extracted by the second convolutional layer; the features extracted by the first convolutional layer and the second convolutional layer are fused by the UMF module to obtain features containing spatial information;

[0023] The decoder is composed of bilinear interpolation and convolutional networks, and is used to restore images. The features containing spatial information output by the encoder are upsampled in turn through bilinear interpolation and convolutional networks to obtain an approximate representation of the original image, and the approximate representation and the original image are calculated using the minimum absolute value error loss to optimize the spatial information extraction network to extract spatial information.

[0024] The edge information extraction network includes a third convolutional layer, a fourth convolutional layer, and a UMF module, which are used to encode the image; the features of the original image obtained by the downsampling layer are extracted by the third convolutional layer, and the features output by the first residual network module stage in the semantic information extraction network are extracted by the fourth convolutional layer; the features extracted by the third convolutional layer and the fourth convolutional layer are respectively fused by the UMF module to obtain features containing edge information, and then passed through the edge head to obtain edge segmentation results;

[0025] The edge head is composed of bilinear interpolation and convolutional networks; the features containing edge information output by the edge information extraction network are upsampled in turn through bilinear interpolation and convolutional networks to obtain edge detection results, and the edge detection results and boundary labels are calculated using edge loss to optimize the edge information extraction network to extract edge information.

[0026] The boundary labels are obtained by manually marking the original images to obtain semantic segmentation labels, and then a boundary map is obtained by a canny operator as the boundary labels.

[0027] The UMF module is used to fuse at least two branch features; the feature fusion network is improved based on the spatial attention mechanism and is implemented by the following formula:

[0028] F cat =Concat(mean(P1), max(P1), mean(P2), max(P2), mean(P3), max(P3))

[0029] w1,w2,w3=Split(Sigmoid(Conu(F cat )))

[0030] out = w1×P1+w2×P2+w3×P3

[0031] Among them, P1, P2, and P3 represent three different input features, mean and max represent the operations of taking the mean and maximum values ​​of the feature map along the channel dimension, and Concat represents the concatenation operation along the channel dimension; F cat Represents the result of concatenating multiple feature information, Conv represents the convolution operation, Sigmoid represents the activation function, Split represents the splitting operation along the channel dimension, w1, w2, w3 represent the weights of the three input features, and out represents the features finally obtained by fusion of multiple features.

[0032] Real-time semantic segmentation of street scenes for autonomous vehicles, including:

[0033] An image acquisition module, used for real-time image acquisition through a monocular camera;

[0034] Real-time semantic segmentation module, used to build a real-time semantic segmentation model to achieve real-time segmentation of images;

[0035] The vehicle-mounted camera real-time segmentation module is used to obtain environmental images in real time through the vehicle-mounted camera and send them to the trained real-time semantic segmentation model to obtain the final scene segmentation results.

[0036] The real-time semantic segmentation model includes:

[0037] A spatial information extraction network is used to extract the spatial information of the original image during the encoding and decoding process by using an autoencoder;

[0038] Semantic information extraction network, used to extract semantic information of the original image;

[0039] Edge information extraction network, used to extract edge information from the original image;

[0040] The UMF module is used to fuse information of multiple input features.

[0041] The beneficial effects of the present invention are:

[0042] 1. The present invention constructs a new real-time semantic segmentation model with three branches, which respectively extract spatial, semantic and edge information. For the branch that extracts spatial information, a more effective method is proposed, which not only extracts spatial information during the encoding and decoding process by constructing an autoencoder, but also proposes a new feature fusion method UMF, which can efficiently fuse multiple features.

[0043] 2. The network structure proposed in the present invention achieves a good balance between speed and accuracy, has strong robustness, is suitable for the field of autonomous driving vehicle traffic scenarios, and can also be extended to semantic segmentation problems, with good practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is an overall schematic diagram of the real-time semantic segmentation device of the street scene of the autonomous driving vehicle of the present invention;

[0045] Figure 2 is the overall network structure diagram of the real-time segmentation model of the present invention;

[0046] Figure 3 This is a schematic diagram of the feature fusion network UMF of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0049] The present invention relates to a street scene real-time semantic segmentation method and device for autonomous driving vehicles, comprising an image acquisition module, a real-time semantic segmentation model construction module and a vehicle-mounted camera real-time segmentation module. The image acquisition module controls a monocular camera installed on a vehicle through an acquisition controller to collect images of street scenes; the real-time semantic segmentation model construction module comprises four parts, namely a spatial information extraction network, a semantic information extraction network, an edge information extraction network and a feature fusion network. The spatial information extraction network compresses the original image by constructing an autoencoder, thereby retaining the spatial information of the original image; the semantic information extraction network extracts semantic information by increasing the depth of the network and the number of channels; the edge information extraction network guides the edge branch to extract the edge features of the original image by introducing edge loss; the feature fusion module can efficiently fuse spatial information, semantic information and edge information; the vehicle-mounted camera real-time segmentation module is used to perform real-time scene segmentation on the real-time street scene pictures obtained by the vehicle-mounted camera. The present invention establishes real-time scene segmentation of autonomous driving vehicles facing different street scenes, and can perform high-precision segmentation of different street scenes at a faster reasoning speed, with strong stability.

[0050] A method and device for real-time semantic segmentation of street scenes for autonomous driving vehicles, comprising:

[0051] The monocular image acquisition module controls the camera to acquire real-time images through the acquisition controller;

[0052] Real-time semantic segmentation method building module, used to build real-time segmentation models that achieve a good balance between speed and accuracy;

[0053] The vehicle camera real-time segmentation module captures street view images in real time through the vehicle camera and inputs these images into the pre-trained real-time segmentation model to obtain the final scene segmentation results. The goal of this module is to use the vehicle camera to obtain street view images in real time and process the images through the model's segmentation algorithm to obtain segmentation results of different scenes in the image.

[0054] The real-time semantic segmentation method of street scenes for autonomous driving vehicles consists of three branches, namely a spatial branch, a semantic branch and an edge branch. It extracts spatial, semantic and edge information from street scenes and fuses them together through a feature fusion network (UMF) to achieve real-time semantic segmentation of images.

[0055] The spatial branch is composed of an autoencoder, and the spatial information of the original image is retained by constructing the autoencoder.

[0056] The autoencoder is a structure of an encoder and a decoder, which extracts the spatial structure and features of the image through the process of encoding and decoding the original image.

[0057] The semantic branch is formed by stacking residual network basic blocks. The semantic information is extracted by deepening the network depth and the number of channels. In order to obtain information of different receptive fields, the U-Net structure is used to fuse the features of different stages of the semantic branch.

[0058] The edge branch is composed of multiple convolutional networks stacked together. In order to increase the reasoning speed of the network, a smaller number of channels is used in the design of the edge branch. In order to guide the edge branch to extract more edge information, the edge segmentation loss is increased to guide the edge branch.

[0059] The present invention constructs a real-time semantic segmentation method and device for street scenes for autonomous driving vehicles, which can segment street scenes in real time with high accuracy and speed and has strong stability.

[0060] A real-time semantic segmentation device for street scenes for autonomous vehicles. Figure 1 As shown, it includes: image acquisition module, segmentation model construction module, and vehicle camera real-time segmentation module. The image acquisition module controls the monocular camera to acquire images in real time through the acquisition controller, builds a trainable data set, and then trains the constructed real-time semantic segmentation model. Finally, the trained model is deployed to the vehicle platform to perform real-time reasoning on the images collected by the vehicle camera.

[0061] Real-time segmentation model building module. Figure 2As shown in the figure, the model is mainly composed of three branches, namely edge branch, semantic branch and spatial branch. These three branches share a downsampling layer, which is mainly composed of a convolutional layer with a step size of 2. For the design of the semantic branch, the downsampled features are first passed through the F1, E2, and E3 modules to increase the number of channels and expand the receptive field. Among them, E1, E2, and E3 are all composed of basic blocks of the residual network. The features passed through E3 are then input into the PAPPM module to further extract multi-scale receptive field information. In order to make more full use of the semantic information at different stages, a U-shaped structure is used to fuse multi-stage semantic information, and the fusion module uses the UMF module. For the edge branch design, the features of the downsampling layer are first further extracted through the convolution layer, and then the features of the F1 stage of the semantic branch are convolved to reduce the optimization difficulty of the multi-branch network. The UMF module is then used to fuse the two parts of the features, and finally a convolution layer is passed to further increase the representation ability of the edge branch. In order to more effectively enable the network to extract edge information, an edge head is connected after the UMF module to output the edge detection result. The segmentation label is extracted through the Canny operator as the label of the edge branch, and finally the edge loss is used to optimize the edge branch. For the design of the spatial branch, the network structure is consistent with the edge branch. The difference is that the features extracted by the UMF module of the spatial branch are connected to the spatial head. The feature dimension output by the spatial head has the same shape as the original input image. Then the minimum absolute value error is used to construct the loss between the output of the spatial head and the input image to establish the optimization of the autoencoder. For the design of the overall network structure, the features obtained by the three branches are first input into the UMF module together to fuse multiple features. Finally, the fused features are input into the segmentation head, and the segmentation results and labels are subjected to cross entropy loss to optimize the segmentation performance of the network.

[0062] Feature Fusion Module (UMF). Figure 3 As shown in the figure, input 1, input 2 and input 3 represent three groups of input features. These three groups of features are first processed by the average and maximum values ​​to obtain features with a channel number of 2. Then, the splicing operation is performed to splice the representative features of these three groups of features together. Then, after convolution and splitting operations, three weights w1, w2, and w3 are obtained. Finally, these three weights are multiplied by the three groups of features and then added as the final fusion features. The calculation formula is as follows:

[0063] F cat =Concat(mean(P1), max(P1), mean(P2), max(P2), mean(P3), max(P3))

[0064] w1,w2,w3=Split(Sigmoid(Conv(F cat )))

[0065] out = w1×P1+w2×P2+w3×P3

[0066] Among them, P1, P2, and P3 represent the features of the multi-branch network, mean and max represent the operations of taking the mean and maximum values ​​of the feature map along the channel dimension, and Concat represents the concatenation operation along the channel dimension. cat Represents the result of concatenating multiple feature information, Conv represents the convolution operation, Sigmoid represents the activation function, Split represents the splitting operation along the channel dimension, w1, w2, w3 represent the weights of multi-branch features, and out represents the features finally fused from the multi-branch features.

[0067] Constructing a street scene real-time semantic segmentation method and device for autonomous driving vehicles includes the following steps:

[0068] S1: Establish an image acquisition controller to collect street scene pictures in real time.

[0069] S2: Build a real-time segmentation model, which mainly includes the construction of edge branch, semantic branch and spatial branch.

[0070] S3: Train the constructed model. In order to make the network have better stability, random cropping and random stretching of image size operations are introduced during training. At the same time, in order to better optimize the network structure, a deep supervised training strategy is introduced, and the loss function adopts the cross entropy loss function.

[0071] S4: Save the weights of the trained model and deploy the trained model to the vehicle platform.

[0072] S5: The on-board camera obtains street view images in real time, and the trained real-time segmentation model performs inference to obtain street view segmentation results.

[0073] The present invention discloses a method and device for real-time semantic segmentation of street scenes for autonomous driving vehicles. Through an image acquisition module, a real-time semantic segmentation model construction module and a vehicle-mounted camera real-time segmentation module, a vision-based scene segmentation method and device are constructed, which can segment street scene information in real time.

[0074] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A real-time semantic segmentation method for street scenes for autonomous vehicles, characterized in that: The following steps are involved: The image acquisition module acquires real-time raw images through a monocular camera; The real-time semantic segmentation module builds a real-time semantic segmentation model to achieve real-time segmentation of the original image; The vehicle camera real-time segmentation module obtains environmental images in real time through the vehicle camera and sends them to the trained real-time semantic segmentation model to obtain the final scene segmentation results.

2. The real-time semantic segmentation method of street scenes for autonomous driving vehicles according to claim 1, characterized in that: The real-time semantic segmentation module constructs a real-time semantic segmentation model to achieve real-time segmentation of the original image, as follows: The spatial information extraction network uses an autoencoder to extract the spatial information of the original image during the encoding and decoding process; The semantic information extraction network extracts the semantic information of the original image; The edge information extraction network extracts edge information from the original image; The UMF module performs information fusion on multiple input features.

3. The real-time semantic segmentation method of street scenes for autonomous driving vehicles according to claim 2, characterized in that: The semantic information extraction network extracts the semantic information of the original image, specifically: the features of the original image obtained by the downsampling layer are sequentially passed through multiple residual network modules and multi-scale receptive field extraction modules to obtain features of different stages and with different receptive fields; The semantic information extraction network adopts a U-Net structure, including multiple residual network modules, a multi-scale receptive field extraction module, a first UMF module, and a second UMF module; The multiple residual network modules include at least a first residual network module and a second residual network module; The residual network module is composed of a convolutional layer, a batch normalization layer and an activation layer connected in sequence to achieve feature extraction; The first UMF module performs feature fusion on the features obtained by the multi-scale receptive field extraction module and the features output by the second residual network module stage; The second UMF module performs feature fusion on the features output by the first UMF module and the features output by the first residual network module stage as the output of the semantic information extraction network.

4. The method for real-time semantic segmentation of street scenes for autonomous driving vehicles according to claim 2, characterized in that: The spatial information extraction network extracts the spatial information of the original image by using an autoencoder during the encoding and decoding process, and performs the following steps: The autoencoder includes an encoder and a decoder; The encoder includes a first convolutional layer, a second convolutional layer, and a UMF module, which are used to encode the image; The features obtained by the downsampling layer of the original image are extracted by the first convolutional layer, and the features output by the first residual network module stage in the semantic information extraction network are extracted by the second convolutional layer; the features extracted by the first convolutional layer and the second convolutional layer are fused by the UMF module to obtain features containing spatial information; The decoder is composed of bilinear interpolation and convolutional networks, and is used to restore images. The features containing spatial information output by the encoder are upsampled in turn through bilinear interpolation and convolutional networks to obtain an approximate representation of the original image, and the approximate representation and the original image are calculated using the minimum absolute value error loss to optimize the spatial information extraction network to extract spatial information.

5. The real-time semantic segmentation method of street scenes for autonomous driving vehicles according to claim 2, characterized in that: The edge information extraction network includes a third convolutional layer, a fourth convolutional layer, and a UMF module, which are used to encode the image; the features of the original image obtained by the downsampling layer are extracted by the third convolutional layer, and the features output by the first residual network module stage in the semantic information extraction network are extracted by the fourth convolutional layer; the features extracted by the third convolutional layer and the fourth convolutional layer are respectively fused by the UMF module to obtain features containing edge information, and then passed through the edge head to obtain the edge segmentation result.

6. The real-time semantic segmentation method of street scenes for autonomous driving vehicles according to claim 5, characterized in that: The edge head is composed of bilinear interpolation and convolutional networks; the features containing edge information output by the edge information extraction network are upsampled in turn through bilinear interpolation and convolutional networks to obtain edge detection results, and the edge detection results and boundary labels are calculated using edge loss to optimize the edge information extraction network to extract edge information.

7. The method for real-time semantic segmentation of street scenes for autonomous driving vehicles according to claim 6, characterized in that: The boundary labels are obtained by manually marking the original images to obtain semantic segmentation labels, and then a boundary map is obtained by a canny operator as the boundary labels.

8. The method for real-time semantic segmentation of street scenes for autonomous driving vehicles according to claim 2, characterized in that: The UMF module is used to fuse at least two branch features; The feature fusion network is improved based on the spatial attention mechanism and is implemented as follows: F cat =Concat(mean(P1),max(P1),mean(P2),max(P2),mean(P3),max(P3)) w1,w2,w3=Split(Sigmoid(Conv(F cat ))) out = w1×P1+w2×P2+w3×P3 Among them, P1, P2, and P3 represent three different input features, mean and max represent the operations of taking the mean and maximum values ​​of the feature map along the channel dimension, and Concat represents the concatenation operation along the channel dimension; F cat Represents the result of concatenating multiple feature information, Conv represents the convolution operation, Sigmoid represents the activation function, Split represents the splitting operation along the channel dimension, w1, w2, w3 represent the weights of the three input features, and out represents the features finally obtained by fusion of multiple features.

9. A real-time semantic segmentation device for street scenes for autonomous driving vehicles, characterized in that: include: An image acquisition module, used for real-time image acquisition through a monocular camera; Real-time semantic segmentation module, used to build a real-time semantic segmentation model to achieve real-time image segmentation; The vehicle-mounted camera real-time segmentation module is used to obtain environmental images in real time through the vehicle-mounted camera and send them to the trained real-time semantic segmentation model to obtain the final scene segmentation results.

10. The real-time semantic segmentation device for street scenes for autonomous driving vehicles according to claim 9, characterized in that: The real-time semantic segmentation model includes: A spatial information extraction network is used to extract the spatial information of the original image during the encoding and decoding process by using an autoencoder; Semantic information extraction network, used to extract semantic information of the original image; Edge information extraction network, used to extract edge information from the original image; The UMF module is used to fuse information of multiple input features.

Images