Fine-grained boundary extraction method for semantic segmentation of urban scenes based on laser point cloud

Through the combination of deep convolutional neural network and conditional random field, CRF learning and inference technology are used to solve the problem of low boundary extraction accuracy in urban scene semantic segmentation, and efficient fine-grained boundary extraction of LiDAR point cloud data is realized, which improves the Semantic segmentation accuracy and efficiency.

CN112819832BActive Publication Date: 2024-12-03NORTH CHINA UNIV OF WATER RESOURCES & ELECTRIC POWER
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202110145309.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-02
Publication Date
2024-12-03
Estimated Expiration
2041-02-02

AI Technical Summary

Technical Problem

The prior art has low accuracy in semantic segmentation results in urban market scene semantic segmentation, especially the fine-grained extraction effect of multi-objective segmentation boundary, especially when processing scattered LiDAR point cloud data.

Method used

After the deep convolutional neural network model is used for preliminary semantic segmentation, boundary extraction is performed in combination with post-processing or embedded conditional random fields, CRF learning and reasoning is performed through maximum likelihood estimation and average field approximation algorithm, and monomeric potential energy functions and binary The potential energy function is extracted in fine-grained boundary and maps the 2D image to the 3D point cloud through a direct transformation algorithm.

Benefits of technology

It improves the accuracy and effect of semantic segmentation in urban markets, realizes efficient extraction of fine-grained boundaries, simplifies the deep neural network structure, speeds up the training speed, and realizes independent learning of conditional random field parameters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112819832B_ABST
    Figure CN112819832B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for extracting fine-grained boundaries of semantic segmentation of urban scenes based on laser point clouds. The present invention designs a deep convolutional neural network model adapted to urban scenes, trains the model, and then uses the trained model to perform semantic segmentation on image data in the acquired data to obtain a preliminary semantic segmentation result of the urban scene based on 2D images; refinement of boundary extraction is performed through two conditional random fields, post-processing and embedded; finally, the external orientation elements of the camera are calculated according to the internal orientation elements of the camera through a direct transformation algorithm to obtain an overall mapping between the image and the corresponding laser point cloud, and on this basis, the point cloud is input to obtain a refined semantic segmentation result based on the laser point cloud. The present invention can improve the accuracy and effect of semantic segmentation of urban scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a laser point cloud-based urban scene semantic segmentation fine-grained boundary extraction method. Background Art

[0002] In recent years, with the emergence of large-scale data sets, the reduction of computer hardware costs and the improvement of GPU parallel computing capabilities, the application of deep neural networks (DCNNs) has become more extensive. Unlike traditional hand-crafted features, DCNNs can automatically learn rich feature representations from data, so they perform well in many computer vision problems such as semantic segmentation. Fully Convolutional Networks (FCNs) in DCNNs is a type of DCNN, and its performance in feature extraction is particularly outstanding. For scene semantic segmentation tasks, the global contextual information between different category labels affects their precise positioning. However, due to its full convolutional characteristics, FCNs do not have the ability to model the contextual relationship between different category labels; the semantic segmentation results obtained by convolution kernels with large receptive fields are relatively rough; at the same time, the hierarchical features generated by the pooling layer will also lose some positioning information, further reducing the possibility of outputting refined semantic segmentation results. Examples of scene segmentation effects are shown below. Figure 1 shown.

[0003] In 2017, Charles et al. from Stanford University proposed the PointNet and PointNet++ frameworks, which directly used the original point cloud as the input of the deep neural network. This is a pioneering work in the successful application of deep learning to three-dimensional point clouds, and provides a general framework for classification, part segmentation, and scene semantic segmentation. The PointNet model uses MaxPooling as a symmetric function to deal with the disorder of the point cloud model based on the characteristics of point cloud data, and uses two T-net networks to process the model for rotation invariance. The shortcoming of this model is that it only uses one MaxPooling layer to integrate single-point features, and the network is not capable of extracting local information from the model. To address this problem, in June of the same year, the research team improved PointNet and proposed its hierarchical network structure PointNet++. This model first samples (sampling) and groups (grouping) the point cloud, and uses the basic PointNet network to extract features in each small area. This process is iterated multiple times as needed, and then the global and local features of the point cloud are fused. However, since the model first selects particles for each area and then runs PointNet for each particle in its large-scale neighborhood, the computational cost is very high and the computational efficiency is far lower than that of PointNet.

[0004] Subsequently, inspired by this and based on this, some new LiDAR point cloud semantic segmentation frameworks have emerged. However, due to the massive and irregular characteristics of point cloud data, the computational complexity far exceeds the processing of 2D images, and the computational efficiency is significantly reduced. In addition, the segmentation results obtained by DCNNs model training can achieve three-dimensional point-level segmentation and maintain high accuracy, but the granularity of its segmentation edge information is not complete enough. Deep models with multiple maximum pooling layers have good classification performance, but the large receptive field brought by the dilated convolution and the inherent invariance of the model cannot achieve good target positioning, and can only produce smooth responses. The correlation between three-dimensional points is not considered in the network.

[0005] In response to this challenge, the current multi-target segmentation boundary refinement problem based on deep convolutional neural networks is mainly focused on the field of image segmentation. When dealing with multi-class image segmentation and labeling tasks, a common method is to use CRFs to perform maximum a posteriori reasoning on image pixels or image patches. The CRFs potential function incorporates a smoothing term that maximizes label consistency among similar pixels, and can integrate more complex terms that model contextual relationships between categories.

[0006] Traditionally, CRFs have been used to smooth noisy segmentation maps. Typically, these models couple adjacent nodes together, assuming that spatially adjacent nodes are similar, as a weakly supervised method to predict labels for edge-similar nodes, which can effectively remove noise and make segmentation edges smoother. Qualitatively, the main function of these short-range CRFs is to clean up spurious predictions from weak classifiers built on local hand-defined features. Compared to these weaker classifiers, modern DCNNs architectures produce score maps and semantic label predictions that are qualitatively different. Scoring Figure 1 Generally, the scores are relatively smooth and the classification results are consistent. At this time, using short-distance CRFs will bring bad results, because semantic segmentation is not to make the score map smoother, but to explore the details in the score map, such as the segmentation effect of the edge part. In 2017, DeepLab proposed a fine-grained positioning accuracy coupling method based on the recognition ability of DCNNs and fully connected conditional random fields (FC-CRF). The contrast-sensitive potential function is combined with FC-CRF to improve the positioning ability, and it is shown that it has achieved remarkable success in solving positioning challenges, producing accurate semantic segmentation results, and restoring object boundaries at a level of detail that cannot be achieved by existing methods. Subsequent researchers have continued this direction. In the field of image object segmentation, conditional random fields are used to judge pixel categories. Since the model considers the relationship between pixels and their neighboring pixels, it can efficiently distinguish the boundaries between different categories. FC-CRF further considers the relationship between each pixel and other pixels in the image, and can obtain more accurate segmentation results. FC-CRF applied to post-reasoning of semantic segmentation has the advantages of improving the fine details captured by the model, being able to capture subtle edge details, being able to adapt to long-distance dependencies, and performing fast reasoning on segmented edges.

[0007] For the problem of fine-grained extraction of multi-target segmentation boundaries in LiDAR point clouds based on deep learning, the above methods only consider regular and neat grid data, and there is little research on scattered point cloud data. At present, the combination of CRFs and LiDAR point clouds is mainly used for the segmentation of single ground targets, rather than the research on fine-grained boundary extraction in the later reasoning of urban scene segmentation, resulting in low accuracy of semantic segmentation results in existing technologies. Summary of the invention

[0008] The purpose of this application is to provide a fine-grained boundary extraction method for semantic segmentation of urban scenes based on laser point clouds, so as to solve the problem of low accuracy of semantic segmentation results in the prior art.

[0009] To achieve the above object, the present invention proposes a method for extracting fine-grained boundaries of urban scene semantic segmentation based on laser point cloud, comprising the following steps:

[0010] 1) Train a deep convolutional neural network model, and use the trained model to perform semantic segmentation on the acquired 2D images and 2D image data in the 3D point cloud to obtain preliminary semantic segmentation results of urban scenes based on 2D images;

[0011] 2) Use post-processing conditional random fields to refine the segmentation boundaries: Use the output of the deep convolutional neural network as the input of the post-processing conditional random field, and perform CRF learning and reasoning through maximum likelihood estimation and mean field approximation algorithm to refine the segmentation boundaries;

[0012] 3) Map the 2D image to the laser point cloud to obtain the semantic segmentation result of the laser point cloud urban scene.

[0013] Furthermore, the CRF learning adopts a maximum likelihood estimation algorithm to estimate the parameters of the CRFs model by maximizing the sample log-likelihood function; the CRF learning includes a univariate potential energy function and a binary potential energy function, the univariate potential energy function includes the shape, texture, position and color of the image, and the binary potential energy function uses a contrast-sensitive dual-core potential energy.

[0014] Furthermore, the CRF reasoning is to assign a label to each pixel so that the unary potential energy function and the binary potential energy function reach a minimum value as a whole.

[0015] Furthermore, the camera's exterior orientation elements are calculated according to the camera's interior orientation elements through a direct transformation algorithm, and the overall mapping between the 2D image and the laser point cloud is obtained.

[0016] Furthermore, the MS COCO, PASCAL VOC 2012 and Cityscapes datasets are used to train the deep convolutional neural network model.

[0017] The present invention also proposes another method for extracting fine-grained boundaries of urban scene semantic segmentation based on laser point cloud, which is characterized by comprising the following steps:

[0018] 1) Train a deep convolutional neural network model, and use the trained model to perform semantic segmentation on the acquired 2D images and 2D image data in the 3D point cloud to obtain preliminary semantic segmentation results of urban scenes based on 2D images;

[0019] 2) The embedded conditional random field and the deep convolutional neural network form a complete model for end-to-end training; the output of the deep convolutional neural network is used as the input of the unary potential energy of the embedded conditional random field, and the current marginal distribution estimation is used as the input of the next marginal probability estimation, and the learning is cyclically learned until the optimal parameters of the embedded conditional random field are obtained;

[0020] 3) Map the 2D image to the laser point cloud to obtain the semantic segmentation result of the laser point cloud urban scene.

[0021] Furthermore, the score map output by the deep convolutional neural network is upsampled to restore it to the original resolution, and then a network layer called a multi-stage mean field is added thereto. The original image and the preliminary segmentation results of the network output are simultaneously input into the multi-stage mean field for maximum a posteriori inference, so as to maximize the label consistency of similar pixels and pixel neighbors.

[0022] Furthermore, the camera's exterior orientation elements are calculated according to the camera's interior orientation elements through a direct transformation algorithm, and the overall mapping between the 2D image and the laser point cloud is obtained.

[0023] Furthermore, the MS COCO, PASCAL VOC 2012 and Cityscapes datasets are used to train the deep convolutional neural network model.

[0024] Traditionally, CRFs are used to smooth noisy segmentation maps as a weakly supervised method to predict labels of similar nodes on the edge, eliminate noise, and make the segmentation edge smoother; the combination of CRFs and LiDAR point clouds is also mainly used for the segmentation of single targets on the ground. The present invention improves the traditional CRFs, combines them with deep convolutional neural networks, and adopts two strategies, post-processing and embedded, to extract fine-grained boundaries for multi-target semantic segmentation in urban scenes. Among them, the post-processing conditional random field simplifies the deep neural network structure, reduces the network model learning parameters, and speeds up the network model training speed. The embedded conditional random field realizes the autonomous learning of the conditional random field parameters, avoiding the fine-grained extraction of semantic segmentation boundaries in stages. In general, the present invention improves the accuracy and effect of semantic segmentation of urban scenes. This method makes full use of the remarkable effects of deep neural networks in 2D image semantic segmentation, and uses a direct linear transformation algorithm to map the fine-grained segmentation results in 2D images to 3D laser point clouds, which reduces the difficulty of semantic segmentation and improves the efficiency of fine-grained boundary extraction in urban scene semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the result of semantic segmentation of urban scenes based on DeepLab-V2 ResNet101. (a) is the original image, (b) is the semantic label, and (c) is the segmentation result.

[0026] Figure 2 It is the technical roadmap of Example 1;

[0027] Figure 3 It is the technical roadmap of Example 2;

[0028] Figure 4 This is a statistical result chart of 6 data sets;

[0029] Figure 5 This is an example of the preliminary results of semantic segmentation based on the PASCAL VOC 2012 validation set; (a) is the original image, (b) is the semantic label, and (c) is the preliminary segmentation result;

[0030] Figure 6 The following are examples of semantic segmentation effects of urban scenes using laser point cloud. (a) is a 3D point cloud, (b) is a preliminary result of global semantic segmentation, (c) is a fine-grained boundary extraction effect of global semantic segmentation, (d) is a preliminary result of local semantic segmentation, and (e) is a fine-grained boundary extraction effect of local semantic segmentation. DETAILED DESCRIPTION

[0031] Example 1

[0032] 1. Perform statistical analysis on the 2D image dataset to select the dataset;

[0033] Through bar charts, line charts and scatter plots Figure 3 This paper compares and analyzes six benchmark datasets that are widely used in the field of urban scene semantic segmentation: SIFT-flow, PASCAL VOC2012, PASCAL-part, MS COCO, Cityscapes and PASCAL-Context. Statistics are performed on the training and validation sets, excluding the test set. First, the total number of categories and the total number of instances in the training and test sets of the six datasets are counted; then, the number of categories contained in each picture, the number of instances contained in each picture, the number of pictures containing each specific category (that is, how many pictures each specific type appears in), and the corresponding relationship between the number of categories and the number of instances are counted through program design. The statistical results of the number of categories in each picture are shown in the figure below. Figure 4As shown in the figure, it can be seen that MS COCO has the richest category information, including 80 semantic categories in total; the number of categories contained in each image in the Cityscapes dataset ranges from [4,21], and the number of categories contained in each image in the PASCAL-Context dataset ranges from [1,24]; while the maximum number of categories contained in each image in the PASCAL VOC 2012 and PASCAL-Part datasets is 6, and the maximum number of categories contained in each image in SIFT-flow is 12, indicating that the Cityscapes and PASCAL-Context datasets are more complex than other datasets. In addition, the number of categories in the PASCAL VOC2012 dataset is moderate, and the image data has different scales, which is suitable for multi-scale semantic segmentation.

[0034] Figure 4 In the figure, the horizontal axis is "Number or categories", the vertical axis is "percentage or images", and the legends from top to bottom are "SIFT-flow (4.4), MS COCO (2.9), Cityscapes (14.1), PASCAL-part (1.4), PASCAL VOC2012 (1.5), PASCAL-Context (6.5)".

[0035] Therefore, according to the statistical analysis results, the data sets used in this embodiment are MS COCO, PASCAL VOC 2012 and Cityscapes.

[0036] 2. Based on the outstanding performance of DeepLab model in the field of semantic segmentation and the influence of network model depth on segmentation performance, DeepLabV2 ResNet-101 is used as the base model, fine-tuned, and pre-trained on the selected dataset. The original base model of DeepLabV2 is VGG16, and the depth of ResNet-101 is six times that of VGG16, but it still has low complexity and is easy to optimize.

[0037] Take PASCAL VOC 2012 as an example for semantic segmentation. Taking PASCAL VOC 2012 as an example, the training set and validation set are 1,464 and 1,449 images respectively during the training process. Data gain and multi-scale random sampling (sampling factor is [0.5, 0.75, 1.0, 1, 25, 1.5]), atrous pyramid convolution (atrous_rate = [6, 12, 18, 24]) and other technologies are used. The initial results of semantic segmentation obtained by training are as follows Figure 5As shown in the figure, it can be seen that the boundary of the segmentation result based on DCNNs is relatively smooth, but the ultimate goal of semantic segmentation is not to obtain a smooth segmentation result for target recognition, but to obtain a semantic segmentation result with fine-grained boundaries. Therefore, the third step is to extract fine-grained boundaries from the semantic segmentation result based on DCNNs.

[0038] 3. Fine-grained boundary extraction

[0039] like Figure 2 As shown, this embodiment performs fine-grained boundary extraction based on post-processing random fields.

[0040] Post-processing conditional random fields are an independent process after DCNNs semantic segmentation. The output of the deep convolutional neural network is used as the input of the post-processing conditional random field, and the post-processing conditional random field does not participate in model training. The advantages are that it simplifies the structure of the deep neural network, reduces the learning parameters of the network model, and speeds up the training of the network model. Its core technologies include CRF learning and CRF derivation.

[0041] For a 2D image, it can be regarded as a graph model G = (V, E), where each vertex corresponds to a pixel, that is, V = {X1, X2, ..., X n}. Define the latent variable X i is the classification label of pixel i, and the variable range is the semantic label of the classification L = {l1, l2, l3...}; I i For each random variable X i The goal of image semantic segmentation of conditional random fields is to observe the variable I i , infer the latent variable X i The corresponding category label of (X, I) constitutes a conditional random field, whose probability distribution obeys the Gaussian (Gibbs) distribution and can be expressed as:

[0042]

[0043] Where I represents the input image and Z(I) represents the normalization factor to ensure that P is a probability distribution. E(x|I) is called the energy function of x, where x represents the label assigned to the pixel in the image. This formula converts the maximum a posteriori probability problem of CRF into a minimization problem of the energy function, which can be expressed as:

[0044]

[0045] Among them, θ i (x i ) is a single latent variable x i The univariate potential energy function term represents the hidden variable x iis the cost of a certain semantic category. The output of the last layer of the model is used as the input of the unary potential energy function, and its calculation formula is:

[0046] θ i (x i )=-logP(x i ) (3)

[0047] And θ ij (x i ,x j ) are two hidden variables connected to each other (x i ,x j ), which represents the consistency cost of two categories, can be expressed as a linear combination of Gaussian kernel functions:

[0048]

[0049] Among them, μ(x i ,x j ) is the category label compatibility function, when x i ≠x j When μ(x i ,x j )=1, otherwise 0. k(f i ,f j ) The two components represent two Gaussian kernel functions in different feature spaces, where the first bilateral kernel function is related to the pixel position (represented by P) and the RBG value (represented by I); the second kernel function is only related to the position. Hyperparameter θ α ,θ β and θ γ is a scale parameter used to control the scale of the kernel function.

[0050] The unary potential includes the shape, texture, position and color of the image. The binary potential uses a contrast-sensitive dual-core potential. The binary potential function of CRF generally describes the relationship between pixels, encouraging similar pixels to be assigned the same label, while pixels with large differences are assigned different labels. The definition of this "distance" is related to the color value and the actual relative distance. In this way, CRF can segment the image as close to the boundary as possible. The difference of the fully connected CRF model is that its binary potential function describes the relationship between each pixel and all other pixels. This model is used to establish point-to-point potential on all pixel pairs in the image, thereby achieving great refinement and segmentation.

[0051] The CRF learning problem is actually estimating the parameters of CRFs based on the training data set. CRFs is actually a log-linear model defined on time series data. The learning method used is Maximum Likelihood Estimation (MLE), which reverses the model parameters that are most likely (with the highest probability) to obtain the result based on the known sample results. The maximum likelihood function is shown in formula (5). The MLE algorithm estimates the parameters θ of the CRFs model by maximizing the sample log-likelihood function.

[0052]

[0053] Where L represents the sample log-likelihood function, is the maximum value of the parameter θ of the CRFs model, and argmax is the maximum value of the function.

[0054] CRF inference is to assign a label x to each pixel i. i , so that the unary potential energy and binary potential energy reach the minimum value as a whole. Therefore, the CRFs model inference problem is also called the CRFs energy function minimization problem. It is difficult to directly find the exact probability distribution value P(x), and the approximate distribution Q(x) = Π is usually used. i Q i (x i ) represents the product of the independent marginal probabilities of each variable, and its final derivation form is shown in formula (6).

[0055]

[0056] 4. Data Mapping

[0057] The fine-grained boundary extraction result obtained in step 3 is the segmentation result based on the 2D image. Finally, the result of step 3 needs to be mapped to 3D space. To complete the mapping from 2D image space to 3D point cloud space, it is necessary to determine the conversion parameters from the 2D image plane coordinate system to the 3D point cloud space coordinate system, including the placement parameters of the camera relative to the laser scanner and the intrinsic parameters of the camera, that is, the exterior orientation elements and interior orientation elements of the image. The placement parameters of the external frame digital camera relative to the 3D laser scanner are the exterior orientation elements of the digital camera relative to the laser scanner coordinate system. According to the collinearity condition and considering the internal parameters of the camera, the relationship equations between the image point coordinates, the object point coordinates, the camera intrinsic parameters, and the exterior orientation elements can be established. For the calibration of multi-image exterior orientation elements, the external frame camera is fixed on the laser scanner and rotated only around the Z axis of the laser scanner at an angle of ξ. The image at the initial position is calibrated once at the horizontal position, and the exterior orientation elements of the images at other positions are calculated by the rotation angle and the exterior orientation elements at the initial position.

[0058] According to DCNNs, the image is segmented to obtain the label corresponding to each pixel in the pixel coordinate system. According to the relationship between the pixel coordinate system and the image physical coordinate system, as shown in formula (6), the pixel coordinates (u, v) can be converted into image coordinates (x, y). Then, according to formula (7), the corresponding point coordinates can be obtained, and the preliminary segmentation result of the three-dimensional point cloud can be obtained.

[0059]

[0060]

[0061] Where f is the focal length, R i and T i The rotation matrix and translation vector derived for the exterior orientation elements of the ith image.

[0062] After mapping, the fine-grained boundary semantic segmentation result on the laser point cloud is obtained, and the effect is as follows Figure 6 As shown in (a), (b), (c), (d), and (e), the fully connected CRFs model can be effectively inferred at the 3D point level of the LiDAR point cloud.

[0063] Example 2

[0064] like Figure 3 As shown, the difference from Example 1 is that this embodiment uses an embedded conditional random field to perform fine-grained boundary extraction on the segmentation results of the 2D image.

[0065] The last network layer of the DeepLabV2 ResNet-101 model is an upsampling layer, which upsamples the rough score map output by the convolutional neural network to restore it to the original resolution, and then integrates the fully connected conditional random field. Specifically, a network layer called Multiple Stage Mean Field Layer (MSMFL) is added. The original image and the preliminary segmentation results of the network output are simultaneously input into MSMFL for maximum a posteriori reasoning, so that the label consistency of similar pixels and pixel neighbors is maximized. The MSMFL layer actually transforms the CRF inference algorithm, regards the transformation steps as layers of neural networks, and then reconnects and combines these layers to form a multi-stage mean field layer. The multi-stage mean field layer is embedded behind DCNNs as one of the layers, and the CRF parameters are optimized through the reverse broadcast algorithm. Specifically, it is divided into four steps: message passing, compatibility conversion, local update, and normalization. As shown in Algorithm 1, the time complexity of each step is analyzed, reflecting the breakthrough of algorithm optimization. The output of the preliminary semantic segmentation result is used as the input of the unary potential energy, and the normalized result is used as the input of the next edge distribution estimation. Compared with the traditional conditional random field algorithm, this method embeds CRF into the deep neural network, realizes the autonomous learning of conditional random field parameters, and avoids the fine-grained extraction of semantic segmentation boundaries in stages.

[0066]

[0067] The conditional random field is embedded in the redesigned deep convolutional neural network ResNet-101 in the form of a multi-stage neural network layer to achieve end-to-end training, attempting to predict all the kernel parameters of the model simultaneously and achieving autonomous learning.

Claims

1. A fine-grained boundary extraction method for semantic segmentation of urban scenes based on laser point clouds, characterized in that: The steps include: 1) Train a deep convolutional neural network model and use the trained model to perform semantic segmentation on the 2D image data in the synchronously acquired 2D image and 3D point cloud to obtain preliminary semantic segmentation results of urban scenes based on 2D images; 2) Use post-processing conditional random fields to refine the segmentation boundaries: Use the output of the deep convolutional neural network as the unary potential energy input of the post-processing conditional random field, and perform CRF learning and reasoning through maximum likelihood estimation and mean field approximation algorithm to refine the segmentation boundaries; 3) Map the 2D image to the laser point cloud to obtain the semantic segmentation result of the laser point cloud urban scene. When mapping the 2D image to the laser point cloud, determine the conversion parameters from the two-dimensional image plane coordinate system to the three-dimensional point cloud space coordinate system, including the placement parameters of the camera relative to the laser scanner and the intrinsic parameters of the camera.

2. The method according to claim 1, characterized in that: The CRF learning adopts the maximum likelihood estimation algorithm to estimate the parameters of the CRFs model by maximizing the sample log-likelihood function; the CRF learning includes a univariate potential energy function and a binary potential energy function, wherein the univariate potential energy function includes the shape, texture, position and color of the image, and the binary potential energy function uses a contrast-sensitive dual-core potential energy.

3. The method according to claim 2, characterized in that The CRF reasoning is to assign a label to each pixel so that the unary potential energy function and the binary potential energy function reach the minimum value as a whole.

4. The method according to claim 1, characterized in that: The camera's exterior orientation elements are calculated based on the camera's interior orientation elements through a direct transformation algorithm to obtain the overall mapping between the 2D image and the laser point cloud.

5. The method according to claim 1, characterized in that: The MS COCO, PASCAL VOC 2012 and Cityscapes datasets are used to train the deep convolutional neural network model.

6. A fine-grained boundary extraction method for semantic segmentation of urban scenes based on laser point clouds, characterized in that: The steps include: Train a deep convolutional neural network model, and use the trained model to perform semantic segmentation on the 2D images and 2D image data in the 3D point cloud acquired simultaneously, and obtain preliminary semantic segmentation results of urban scenes based on 2D images; The embedded conditional random field is combined with the deep convolutional neural network in the form of a recurrent neural network to form a complete model for end-to-end training, specifically: the score map output by the deep convolutional neural network is upsampled to restore it to the original resolution, and then a network layer called a multi-stage mean field is added thereto, and the original image and the preliminary segmentation result of the network output are simultaneously input into the multi-stage mean field for maximum a posteriori reasoning, so as to maximize the label consistency of similar pixels and pixel neighbors; the output of the deep convolutional neural network is used as the input of the unary potential energy of the embedded conditional random field, and the current edge distribution estimation is used as the input of the next edge probability estimation, and the learning is cyclically performed until the parameters of the optimal embedded conditional random field are obtained; 3) Map the 2D image to the 3D laser point cloud to obtain the semantic segmentation result of the laser point cloud urban scene. When mapping the 2D image to the laser point cloud, determine the conversion parameters from the 2D image plane coordinate system to the 3D point cloud space coordinate system, including the placement parameters of the camera relative to the laser scanner and the intrinsic parameters of the camera.

7. The method according to claim 6, characterized in that The camera's exterior orientation elements are calculated based on the camera's interior orientation elements through a direct transformation algorithm to obtain the overall mapping between the 2D image and the laser point cloud.

8. The method according to claim 6, characterized in that The MS COCO, PASCAL VOC 2012 and Cityscapes datasets are used to train the deep convolutional neural network model.

Citation Information

Patent Citations

  • Image semantic division method based on depth full convolution network and condition random field

    CN108062756A

  • A framework to enhance the effect of semantic segmentation model based on transfer learning

    CN109087303A

  • An image style migration and simulation method for gourd pyrography

    CN109712068A

  • Domain adaptive semantic segmentation method based on similarity space alignment

    CN110322446A

  • Semantic segmentation method, device and equipment and computer readable storage medium

    CN112085840A