A 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels

Abstract

The present invention discloses a 3D laser point cloud semantic segmentation method that takes into account the fuzzy boundary of segmentation labels, comprising: mapping the 3D laser point cloud to a spherical projection to generate a corresponding two-dimensional depth image; extracting feature quantities from the two-dimensional depth image and inputting them into a fully convolutional neural network for semantic segmentation to obtain pixel-level semantic labels; establishing an energy function model based on the semantic labels; optimizing the established energy function model using an iterative gradient descent method, minimizing the energy function model, enhancing the data items of the boundary points to update the semantic labels of the laser point cloud; and updating the semantic labels of all laser point clouds. By establishing an energy function model, the present invention calculates the average data item energy and the smoothing item energy, while taking into account local consistency and global consistency, enhancing the boundary data items, and effectively solving the problem of fuzzy boundary of segmentation labels.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and in particular relates to a 3D laser point cloud semantic segmentation method taking into account the fuzzy boundary of segmentation labels. Background Art

[0002] Due to the rapid development of intelligent driving technology in recent years, LiDAR sensors have become widely used in mass-produced vehicles. While a vehicle is driving, semantic segmentation of the 3D laser point cloud captured by LiDAR can effectively distinguish various environmental information, such as roads, vehicles, and buildings. This is particularly important for intelligent driving perception. Semantic segmentation is a deep learning-based computer vision data processing method that aims to assign each pixel or point in a camera-captured image or laser point cloud to a predefined semantic category, thereby segmenting each region and assigning a semantic label to each region.

[0003] Semantic segmentation technology was first applied in image processing to segment objects and areas in images based on color, texture information and neural network technology. With the development and application of sensor technologies such as lidar, how to semantically segment 3D laser point clouds has become a research hotspot. Chinese invention patent application number CN202011094852.3, entitled "Point cloud semantic segmentation method and system based on adversarial learning and multimodal learning", proposes to establish a correspondence between 3D point cloud data and 2D front view images, and perform semantic segmentation of laser point clouds based on adversarial learning; Chinese invention patent application number CN

[0004] 202210443785.4, entitled “Semantic Segmentation Model Training Method, Semantic Segmentation Method, Storage Medium and Terminal” proposes a training method based on the KP-FCNN network architecture that combines auxiliary tasks with semantic segmentation tasks to achieve information gain in semantic segmentation tasks and greatly improve segmentation accuracy. However, none of the above methods consider how to eliminate the fuzzy boundary problem of segmentation labels in the process of semantic segmentation of laser point clouds. 3D laser point cloud data is usually sparse, so there will be gaps between adjacent points. In the boundary area, due to insufficient point density, it is difficult to accurately capture the exact contour of the object, so there will be a problem of fuzzy boundary of segmentation labels. Summary of the Invention

[0005] In view of the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a 3D laser point cloud semantic segmentation method that takes into account the fuzzy boundary of segmentation labels, maps the 3D laser point cloud to a spherical projection, and generates a corresponding two-dimensional depth image; inputs the feature quantity of the depth image into a neural network for semantic segmentation to obtain pixel-level semantic labels; considers the fuzzy boundary problem of semantic labels after segmentation, processes the segmented labels, enhances the feature expression of boundary point cloud data, significantly improves the accuracy of laser point cloud semantic segmentation, and effectively solves the fuzzy boundary problem of semantic segmentation.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The present invention provides a 3D laser point cloud semantic segmentation method that considers the fuzzy boundary of segmentation labels, and the steps are as follows:

[0008] 1) Map the 3D laser point cloud to a spherical projection to generate the corresponding 2D depth image;

[0009] 2) extracting feature quantities from the two-dimensional depth image in step 1) and inputting them into a fully convolutional neural network for semantic segmentation to obtain pixel-level semantic labels;

[0010] 3) establishing an energy function model based on the semantic labels obtained in step 2);

[0011] 4) Optimize the energy function model established in step 3) by using the iterative gradient descent method, minimize the energy function model, enhance the data items of the boundary points to update the laser point cloud p i Semantic tags;

[0012] 5) Repeat step 4) to update the semantic labels of all laser point clouds.

[0013] Furthermore, the 3D laser point cloud in step 1) is defined as:

[0014] p i =(x,y,z)

[0015] Where p i is a single laser point cloud; i∈Σ, Σ represents the point cloud set contained in one frame of the laser radar, x, y, z are the three-dimensional coordinates of the point cloud; each laser point cloud is mapped by Π: Projected to spherical coordinates, converted to image coordinates, defined as:

[0016]

[0017] Where (u, v) is the depth image coordinate, h and w are the height and width of the required depth image representation, respectively, and f = f up +fdown is the vertical field of view of the lidar sensor, f up and f down are the highest and lowest points of the vertical field of view of the lidar sensor, r = || p i ||2 is the depth of each laser point cloud.

[0018] Furthermore, the method of extracting features from the two-dimensional depth image in step 2) is as follows: projection mapping generates a (u, v) coordinate list, which contains a pair of image coordinates for each laser point cloud; extracting features of each laser point cloud, including depth r, coordinates (x, y, z) and reflection intensity, and storing them in the depth image to create a [5×h×w] tensor.

[0019] Furthermore, the structure of the fully convolutional neural network in step 2) is an encoder-decoder hourglass architecture, which includes a downsampling encoder and an upsampling decoder; the downsampling encoder includes multiple convolution layers and pooling layers to obtain feature quantities in the depth map, and the activation function after the convolution layer and the pooling layer is ReLu to introduce nonlinear features; the decoder upsamples the features extracted by the encoder to the original image resolution and adds a convolution layer to refine the result; after each upsampling, jump connections are added between different levels of the encoder to restore high-frequency edge information lost during the downsampling process; after encoding-decoding, the last layer performs a set of kernel size [1×1] convolutions to generate an output of [n×h×w] to obtain pixel-level semantic labels, where n is the number of label categories in the data; the last layer is a softmax function on an unbounded logarithm, which gives the probability distribution of each pixel in the depth image Defined as:

[0020]

[0021] Where, logit c is the output corresponding to the label category c, e is a constant; during the training of the neural network, stochastic gradient descent and weighted cross entropy loss are used Optimize the neural network:

[0022]

[0023]

[0024] Where n is the number of semantic labels; f c is the frequency of occurrence of label category c, according to the frequency of occurrence f c The inverse of is used to penalize class c to handle imbalanced data categories.

[0025] Furthermore, the step 3) specifically includes:

[0026] 31) Calculate each laser point cloud p i The distance to its nearest neighbor is defined as the average data term energy function;

[0027] 32) Calculate the laser point cloud p i The spatial relationship between the points and their neighbors is defined as a smoothed data term energy function;

[0028] 33) An energy function model is obtained based on the smoothing term energy function in step 31) and the average data term energy function in step 32).

[0029] Furthermore, the distance calculation method of the nearest neighbor point in step 31) is: calculate the distance of each laser point cloud p i The distance to its k most similar nearest points is quantitatively described using the energy function. The energy value of the energy function represents the fit of the semantic label of the current point in the local area, that is, the semantic category relationship between the current point and its nearest neighbor points; the average data item energy function Defined as:

[0030]

[0031] Where y i Represents the laser point cloud p i The predicted semantic label, p j Represents the laser point cloud p i The jth nearest neighbor point of P(y i ∣p j ) represents the probability distribution of the predicted label.

[0032] Furthermore, the smoothing data term in step 32) considers the spatial relationship between the point and its adjacent points, and encourages adjacent points to have similar semantic labels to reduce noise in the segmentation result and enhance the overall segmentation consistency; the smoothing data term function Defined as:

[0033]

[0034] Where λ is the weight of the smoothing term, σ is the scale parameter of the smoothing term, and p j Represents the laser point cloud p i The jth nearest neighbor of .

[0035] Furthermore, the energy function model in step 33) includes an average data item and a smooth data item, wherein the average data item is the energy function defined in step 31) The data label categories of the boundary point cloud can be effectively distinguished by the energy function value; the smoothing data item is the energy function defined in step 32) It helps to adjust the semantic labels of points globally so that the segmentation results are spatially coherent. Considering local consistency and global consistency, the energy function model E(Y) is defined by calculating the sum of the average data item energy and the smoothed data item energy:

[0036]

[0037] Where, Represents the set of all neighboring points.

[0038] Furthermore, the method for minimizing the energy function model E(Y) in step 4) is specifically as follows:

[0039] The iterative conditional gradient descent method is used to decompose the optimization problem of the energy function into multiple sub-problems, and the semantic label of a point is updated in each sub-problem. The specific implementation method is: for each laser point cloud p i , fix the semantic labels of other point clouds and only optimize the laser point cloud p i Semantic labels of the laser point cloud p are retained in the energy function model. i Related items, get the sub-problem:

[0040]

[0041] Where, Represents the set of all neighboring points; use the gradient descent method to solve the subproblem, and judge whether the subproblem converges by monitoring the change of the energy function model; when the change of the energy function model is less than the threshold, the subproblem is considered to converge, the iteration ends, and the laser point cloud p is updated. i Semantic label y i ; If the subproblem does not converge, continue to iterate until the energy function model change is less than the threshold.

[0042] Furthermore, the step 5) is specifically as follows: repeatedly calculating the sub-problems decomposed in step 4) until the semantic labels of all laser point clouds are updated, so as to correct the semantic labels with blurred segmentation boundaries.

[0043] Beneficial effects of the present invention:

[0044] 1. The present invention establishes an energy function model, calculates the average data item energy and the smoothing item energy, takes into account both local consistency and global consistency, enhances the boundary data item, and effectively solves the problem of fuzzy segmentation label boundaries;

[0045] 2. The present invention uses stochastic gradient descent and weighted cross entropy loss Optimizing the neural network effectively reduces semantic segmentation errors caused by data imbalance;

[0046] 3. The present invention uses the iterative conditional gradient descent method based on the alternating direction minimization idea to decompose the optimization problem of the energy function model into multiple sub-problems, and quickly and efficiently updates the data label of each point. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flowchart of the method of the present invention.

[0048] Figure 2 This is a structural diagram of the neural network model used in the method of the present invention. DETAILED DESCRIPTION

[0049] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0050] Reference Figure 1 、 Figure 2 As shown, the present invention provides a 3D laser point cloud semantic segmentation method considering the fuzzy boundary of the segmentation label, and the steps are as follows:

[0051] 1) Map the 3D laser point cloud to a spherical projection to generate the corresponding 2D depth image;

[0052] Among them, the 3D laser point cloud is defined as:

[0053] p i =(x,y,z)

[0054] Where p i is a single laser point cloud; i∈Σ, Σ represents the point cloud set contained in one frame of the laser radar, x, y, z are the three-dimensional coordinates of the point cloud; each laser point cloud is mapped by Π: Projected to spherical coordinates, converted to image coordinates, defined as:

[0055]

[0056] Where (u, v) is the depth image coordinate, h and w are the height and width of the required depth image representation, respectively, and f = f up +f down is the vertical field of view of the lidar sensor, f up and f down are the highest and lowest points of the vertical field of view of the lidar sensor, r = || p i ||2 is the depth of each laser point cloud.

[0057] 2) extracting feature quantities from the two-dimensional depth image in step 1) and inputting them into a fully convolutional neural network for semantic segmentation to obtain pixel-level semantic labels;

[0058] The method for extracting features from the two-dimensional depth image is as follows: the projection mapping produces a list of (u, v) coordinates, which contains a pair of image coordinates for each laser point cloud; the features of each laser point cloud are extracted, including the depth r, coordinates (x, y, z) and reflection intensity, and stored in the depth image to create a [5×h×w] tensor.

[0059] The structure of the fully convolutional neural network is an encoder-decoder hourglass architecture, which includes a downsampling encoder and an upsampling decoder; the downsampling encoder contains multiple convolutional layers and pooling layers to obtain feature quantities in the depth map, and the activation function after the convolutional layer and the pooling layer is ReLu to introduce nonlinear features; the decoder upsamples the features extracted by the encoder to the original image resolution and adds a convolutional layer to refine the result; after each upsampling, jump connections are added between different levels of the encoder to restore the high-frequency edge information lost during the downsampling process; after encoding-decoding, the last layer performs a set of kernel size [1×1] convolutions to generate an output of [n×h×w] to obtain pixel-level semantic labels, where n is the number of label categories in the data (such as roads, vehicles, pedestrians, etc.); the last layer is a softmax function on unbounded logarithms, which gives the probability distribution of each pixel in the depth image Defined as:

[0060]

[0061] Where, logit c is the output corresponding to the label category c, e is a constant; during the training of the neural network, stochastic gradient descent and weighted cross entropy loss are used Optimize the neural network:

[0062]

[0063]

[0064] Where n is the number of semantic labels; f c is the frequency of occurrence of label category c, according to the frequency of occurrence f c The reciprocal of is used to penalize class C to handle unbalanced data categories; when the points of the road category are extremely redundant with the points of the pedestrian category, it can effectively reduce the semantic segmentation error caused by the imbalance of data volume.

[0065] 3) Establishing an energy function model based on the semantic labels obtained in step 2); specifically including:

[0066] 31) Calculate each laser point cloud p i The distance to its nearest neighbor is defined as the average data term energy function;

[0067] 32) Calculate the laser point cloud p i The spatial relationship between the points and their neighbors is defined as a smoothed data term energy function;

[0068] 33) An energy function model is obtained based on the smoothing term energy function in step 31) and the average data term energy function in step 32).

[0069] The method for calculating the distance of the nearest neighbor point in step 31) is as follows: calculate the distance of each laser point cloud p i The distance to its k most similar nearest points is quantitatively described using the energy function. The energy value of the energy function represents the fit of the semantic label of the current point in the local area, that is, the semantic category relationship between the current point and its nearest neighbor points; the average data item energy function Defined as:

[0070]

[0071] Where y i Represents the laser point cloud p i The predicted semantic label, p j Represents the laser point cloud p i The jth nearest neighbor point of P(y i ∣p j ) represents the probability distribution of the predicted label.

[0072] The smoothing data term in step 32) considers the spatial relationship between the point and its adjacent points, and encourages adjacent points to have similar semantic labels to reduce noise in the segmentation result and enhance the overall segmentation consistency; the smoothing data term function Defined as:

[0073]

[0074] Where λ is the weight of the smoothing term, σ is the scale parameter of the smoothing term, and p j Represents the laser point cloud p i The jth nearest neighbor of .

[0075] The energy function model in step 33) includes an average data item and a smooth data item, wherein the average data item is the energy function defined in step 31) The data label categories of the boundary point cloud can be effectively distinguished by the energy function value; the smoothing data item is the energy function defined in step 32) It helps to adjust the semantic labels of points globally so that the segmentation results are spatially coherent. Considering local consistency and global consistency, the energy function model E(Y) is defined by calculating the sum of the average data item energy and the smoothed data item energy:

[0076]

[0077] Where, Represents the set of all neighboring points.

[0078] 4) Optimize the energy function model established in step 3) by using the iterative gradient descent method, minimize the energy function model, enhance the data items of the boundary points to update the laser point cloud p i Semantic tags;

[0079] The method to minimize the energy function model E(Y) is as follows:

[0080] The iterative conditional gradient descent method is used to decompose the optimization problem of the energy function into multiple sub-problems, and the semantic label of a point is updated in each sub-problem. The specific implementation method is: for each laser point cloud p i , fix the semantic labels of other point clouds and only optimize the laser point cloud p i Semantic labels of the laser point cloud p are retained in the energy function model. i Related items, get the sub-problem:

[0081]

[0082] Where, Represents the set of all neighboring points; use the gradient descent method to solve the subproblem, and judge whether the subproblem has converged by monitoring the change of the energy function model; when the change of the energy function model is less than the threshold, the subproblem is considered to have converged, the iteration is ended, and the laser point cloud p is updated. i Semantic label y i ; If the subproblem does not converge, continue to iterate until the energy function model change is less than the threshold.

[0083] 5) Repeat step 4) to update the semantic labels of all laser point clouds; specifically, repeatedly calculate the sub-problems decomposed in step 4) until the semantic labels of all laser point clouds are updated to correct the semantic labels with blurred segmentation boundaries.

[0084] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be considered as the scope of protection of the present invention.

Claims

1. A 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels, characterized by: Here are the steps: 1) Map the 3D laser point cloud to a spherical projection to generate the corresponding 2D depth image; 2) extracting feature quantities from the two-dimensional depth image in step 1) and inputting them into a fully convolutional neural network for semantic segmentation to obtain pixel-level semantic labels; 3) establishing an energy function model based on the semantic labels obtained in step 2); 4) Optimize the energy function model established in step 3) by using the iterative gradient descent method, minimize the energy function model, enhance the data items of the boundary points to update the laser point cloud p i Semantic tags; 5) Repeat step 4) to update the semantic labels of all laser point clouds; The step 3) specifically includes: 31) Calculate each laser point cloud p i The distance to its nearest neighbor is defined as the average data term energy function; 32) Calculate the laser point cloud p i The spatial relationship between the points and their neighbors is defined as a smoothed data term energy function; 33) Calculating an energy function model based on the smoothing term energy function in step 31) and the average data term energy function in step 32); The method for minimizing the energy function model E(Y) in step 4) is specifically as follows: The iterative conditional gradient descent method is used to decompose the optimization problem of the energy function into multiple sub-problems, and the semantic label of a point is updated in each sub-problem. The specific implementation method is: for each laser point cloud p i , fix the semantic labels of other point clouds and only optimize the laser point cloud p i Semantic labels of the laser point cloud p are retained in the energy function model. i Related items, get the sub-problem: Where, Represents the set of all neighboring points; use the gradient descent method to solve the subproblem, and judge whether the subproblem has converged by monitoring the change of the energy function model; when the change of the energy function model is less than the threshold, the subproblem is considered to have converged, the iteration is ended, and the laser point cloud p is updated. i Semantic label y i ; If the subproblem does not converge, continue to iterate until the energy function model change is less than the threshold.

2. The 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels according to claim 1 is characterized in that: The 3D laser point cloud in step 1) is defined as: p i =(x,y,z) Where p i is a single laser point cloud; i∈Σ, Σ represents the point cloud set contained in one frame of the laser radar, x, y, z are the three-dimensional coordinates of the point cloud; each laser point cloud is mapped by Π: Projected to spherical coordinates, converted to image coordinates, defined as: Where (u, v) is the depth image coordinate, h and w are the height and width of the required depth image representation, respectively, and f = f up +f down is the vertical field of view of the lidar sensor, f up and f down are the highest and lowest points of the vertical field of view of the lidar sensor, r = || p i ||2 is the depth of each laser point cloud.

3. The 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels according to claim 1, characterized in that: The method for extracting features from the two-dimensional depth image in step 2) is as follows: projective mapping generates a (u, v) coordinate list containing a pair of image coordinates for each laser point cloud; extracts features of each laser point cloud, including depth, coordinates, and reflection intensity, and stores them in the depth image to create a [5×h×w] tensor.

4. The 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels according to claim 1, characterized in that: The structure of the fully convolutional neural network in step 2) is an encoder-decoder hourglass architecture, including a downsampling encoder and an upsampling decoder; wherein the downsampling encoder includes multiple convolutional layers and pooling layers to obtain feature quantities in the depth map, and the activation function after the convolutional layer and the pooling layer is ReLu to introduce nonlinear features; The decoder upsamples the features extracted by the encoder to the original image resolution and adds convolutional layers to refine the results; after each upsampling, skip connections are added between different levels of the encoder to recover the high-frequency edge information lost during the downsampling process; after encoding-decoding, the last layer performs a set of kernel size [1×1] convolutions to generate an output volume of [n×h×w] to obtain pixel-level semantic labels, where n is the number of label categories in the data; the last layer is a softmax function on the unbounded logarithm, which gives the probability distribution of each pixel in the depth image Defined as: Where, logit c is the output corresponding to the label category c, e is a constant; during the training of the neural network, stochastic gradient descent and weighted cross entropy loss are used Optimize the neural network: Where n is the number of semantic labels; f c is the frequency of occurrence of label category c, according to the frequency of occurrence f c The inverse of is used to penalize class c to handle imbalanced data categories.

5. The 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels according to claim 1, characterized in that: The method for calculating the distance of the nearest neighbor point in step 31) is: i The distance to its k most similar nearest points is quantitatively described using the energy function. The energy value of the energy function represents the fit of the semantic label of the current point in the local area and the semantic category relationship between the current point and its nearest neighbor points; the average data item energy function Defined as: Where y i Represents the laser point cloud p i The predicted semantic label, p j Represents the laser point cloud p i The jth nearest neighbor point of P(y i ∣p j ) represents the probability distribution of the predicted label.

6. The 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels according to claim 1, characterized in that: The smoothing term in step 32) considers the spatial relationship between a point and its neighboring points, and encourages neighboring points to have similar semantic labels to reduce noise in the segmentation result and enhance the overall segmentation consistency; the smoothing term function Defined as: Where λ is the weight of the smoothing term, σ is the scale parameter of the smoothing term, and p j Represents the laser point cloud p i The jth nearest neighbor of .

7. The 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels according to claim 1, characterized in that: The energy function model in step 33) includes an average data item and a smooth data item, wherein the average data item is the energy function defined in step 31) The smoothing data term is the energy function defined in step 32) Make the segmentation results coherent in space; by calculating the sum of the average data item energy and the smoothed data item energy, define the energy function model E(Y) as: Where, Represents the set of all neighboring points.

8. The 3D laser point cloud semantic segmentation method considering the fuzzy boundary of segmentation labels according to claim 1, characterized in that: The step 5) is specifically as follows: repeatedly calculating the sub-problems decomposed in step 4) until the semantic labels of all laser point clouds are updated, so as to correct the semantic labels with blurred segmentation boundaries.

Images