A point cloud semantic segmentation method based on continuous convolutional network

Abstract

The present invention relates to a point-by-point fusion point cloud semantic segmentation method based on a continuous convolutional network, comprising the following steps: obtaining an RGB image and point cloud data of a target; using a perspective projection model to establish a one-to-one correspondence between each pixel of the RGB image and the point cloud data; inputting both the RGB image and the point cloud data corresponding to the RGB image into a dual-stream feature extraction network to extract image semantic features and point cloud spatial features; inputting the point cloud spatial features and image semantic features into a fusion module to obtain fused features, which are then added to the point cloud spatial features as output features; and inputting the output features into a point cloud semantic segmentation label prediction network to obtain a semantic segmentation prediction label for each point in the point cloud data. Compared with existing technologies, the present invention can reduce the loss of appearance information, enhance the representation capability of lidar point clouds, and thus better understand semantic scenes, ensuring the safety and practicality of autonomous driving tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of fused point cloud semantic segmentation, and in particular to a point-by-point fused point cloud semantic segmentation method based on a continuous convolutional network. Background Art

[0002] Semantic scene understanding is a fundamental task in autonomous driving. It provides accurate environmental information for path planning and behavioral decision-making, ensuring the safety of autonomous driving. The main method of scene understanding is to assign category labels to input data through semantic segmentation, helping autonomous vehicles identify and understand their surroundings.

[0003] RGB images captured by cameras contain rich appearance information such as texture and color. 2D semantic segmentation methods can provide relatively detailed and accurate semantic segmentation results. However, in autonomous driving scenarios, cameras are easily affected by lighting changes, and RGB images lack spatial information, resulting in poor scene understanding.

[0004] The point cloud collected by the vehicle-mounted lidar provides reliable spatial depth information and is more robust to different lighting conditions. However, the outdoor point cloud collected in the autonomous driving scenario is relatively sparse and lacks semantic appearance information, resulting in high errors in the fine-grained segmentation task of the 3D semantic segmentation method.

[0005] There is a large field gap between RGB cameras and lidar. The fusion methods of existing technologies will cause some appearance information from RGB images to be lost, and the point cloud representation ability is weak, making it difficult to achieve the expected semantic segmentation effect. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and provide a point cloud semantic segmentation method based on a continuous convolutional network. The invention can reduce the loss of appearance information, enhance the representation ability of lidar point clouds, and improve the performance of point cloud semantic segmentation, so as to better understand the semantic scene and ensure the safety and practicality of autonomous driving tasks.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] The present invention provides a point cloud semantic segmentation method based on a continuous convolutional network, comprising the following steps:

[0009] S1: Obtain the target’s RGB image and point cloud data through the vehicle-mounted RGB camera and vehicle-mounted LiDAR respectively;

[0010] S2: Convert the point cloud data from the world coordinate system to the pixel coordinate system so that each pixel of the RGB image corresponds to the point cloud data one by one;

[0011] S3: Build and train a two-stream feature extraction network. Input the RGB image obtained by S1 and the point cloud data corresponding to the RGB image obtained by S2 into the trained two-stream feature extraction network to extract image semantic features and point cloud spatial features.

[0012] S4: Build and train a fusion module, input the point cloud spatial features and image semantic features obtained in S3 into the trained fusion module, obtain fusion features, and add the fusion features to the point cloud spatial features as output features;

[0013] S5: Build and train a point cloud semantic segmentation label prediction network, input the output features obtained in S4 into the trained point cloud semantic segmentation label prediction network, predict the point cloud semantic segmentation label, and finally obtain the semantic segmentation prediction label for each point in the point cloud data.

[0014] Preferably, in S2, the point cloud data is converted from a world coordinate system to a camera coordinate system through a perspective projection model, then projected from the camera coordinate system to an image coordinate system, and finally discretized from the image coordinate system to a pixel coordinate system.

[0015] Preferably, said S2 comprises the following steps:

[0016] S2.1: Convert the point cloud data from the world coordinate system to the camera coordinate system according to the camera's extrinsic parameter matrix;

[0017] S2.2: Based on the principle of similar triangles and the intrinsic parameters of the camera, the point cloud data converted to the camera coordinate system in S2.1 is projected into the image coordinate system.

[0018] S2.3: Discretize the point cloud data projected into the image coordinate system in S2.2 into the pixel coordinate system, and make a one-to-one correspondence between the point cloud data and the RGB image in S1.

[0019] Preferably, the dual-stream feature extraction network includes a point cloud feature extraction network branch and an RGB image feature extraction network branch; the point cloud feature extraction network branch is used to perform feature extraction on the point cloud data corresponding to the RGB image, thereby obtaining point cloud spatial features, and the RGB image feature extraction network branch is used to perform feature extraction on the RGB image, thereby obtaining image semantic features.

[0020] Preferably, the point cloud feature extraction network branch is built based on the PointNet network model.

[0021] Preferably, the point cloud feature extraction network branch includes two multi-layer perceptrons, a maximum pooling layer and two fully connected layers. The point cloud data corresponding to the RGB image passes through two multi-layer perceptrons, outputs local features in sequence, and then obtains the global features of the point cloud data through the maximum pooling layer and two fully connected layers. The global features are connected with the local features to obtain the point cloud spatial features.

[0022] Preferably, the RGB image feature extraction network branch is built based on an FCN network model with VGG-19.

[0023] Preferably, the RGB image feature extraction network branch includes 8 convolutional layers and 3 deconvolutional layers connected in sequence, and the RGB image passes through 8 convolutional layers and 3 deconvolutional layers in sequence to output image semantic features.

[0024] Preferably, said S4 comprises the following steps:

[0025] S4.1: Connect the point cloud spatial features extracted in S3 with the image semantic features to obtain the connected features, and then perform continuous convolution operations to obtain the point-by-point fused semantic features of the front view point cloud;

[0026] S4.2: The semantic features of the front view point cloud obtained in S4.1 are fused point by point, and the attention mechanism is aggregated. The features are selectively integrated through MLP to obtain the first fused feature.

[0027] S4.3: The most expressive feature is obtained from the concatenated features of S4.1 through the maximum pooling operation, and is used as the second fused feature. The first fused feature and the second fused feature are concatenated to obtain the fused feature.

[0028] S4.4: Add the fused features to the point cloud spatial features extracted in S3 to obtain the output features.

[0029] Preferably, the point cloud semantic segmentation label prediction network adopts two multi-layer perceptron networks, and after the output features are input into the two multi-layer perceptron networks, the semantic segmentation prediction labels of each point in the point cloud are output in sequence.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The present invention designs a dual-stream feature extraction network and a fusion module to simultaneously process point cloud and image information, efficiently extracting point cloud spatial features and image semantic features. The fusion module is designed to fuse point cloud data and RGB images point by point, and output the fused features as a supplementary output of the point cloud features, thereby enhancing the point cloud representation capability.

[0032] 2. Before the point cloud data and the RGB image are fused point by point, the present invention first uses a perspective projection model to make the RGB image and the point cloud data correspond one to one, thereby reducing the loss of the RGB image appearance information and facilitating the subsequent point-by-point fusion.

[0033] 3. The present invention fuses the point cloud spatial features and image semantic features by designing a fusion module, and adds the fused features to the point cloud features as the final output features, making the fusion more robust and the point cloud representation capability more prominent. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a point cloud semantic segmentation method based on a continuous convolutional network and point-by-point fusion is provided in this embodiment.

[0035] Figure 2 for Figure 1 A schematic diagram of a perspective projection model of the illustrated embodiment;

[0036] Figure 3 for Figure 1 A schematic diagram of the structure of a point cloud feature extraction network branch in the illustrated embodiment;

[0037] Figure 4 for Figure 1 A schematic diagram of the structure of the RGB image feature extraction network branch of the illustrated embodiment;

[0038] Figure 5 for Figure 1 A schematic diagram of the structure of a fusion module based on continuous convolution in the illustrated embodiment;

[0039] Figure 6 for Figure 1 A schematic diagram of the structure of a point cloud semantic segmentation label prediction network according to the embodiment shown;

[0040] Figure 7 for Figure 1 The overall network framework diagram of the illustrated embodiment. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0042] refer to Figure 1 and Figure 7 As shown, this embodiment provides a point cloud semantic segmentation method based on a continuous convolutional network, comprising the following steps:

[0043] S1: Obtain the target’s RGB image and point cloud data through the vehicle-mounted RGB camera and vehicle-mounted LiDAR respectively;

[0044] In this embodiment, the SemanticKITTI dataset and its corresponding KITTI image dataset are selected as the fusion and training datasets;

[0045] The SemanticKITTI dataset is an open source dataset focused on point cloud semantic segmentation tasks. The dataset is annotated based on the KITTI Odometry dataset. The KITTI dataset's collection vehicle is equipped with 2 grayscale cameras, 2 color cameras, 4 optical lenses, a Velodyne 64-line lidar, and a GPS navigation system, sampling at a frequency of 10Hz. The dataset contains real data collected from scenes such as urban areas, rural areas, and highways. Each frame of data contains up to 15 vehicles and 30 pedestrians, and also contains various degrees of occlusion and truncation. The KITTI Odometry dataset has a total of 22 sequences, each of which is a recorded point cloud package. SemanticKITTI provides true value labels for the point cloud packages of sequences 00-10, and sequences 11-21 are used for testing. This method combines the SemanticKITTI point cloud data and the image dataset collected by the four cameras of the KITTI dataset to verify the reliability of the point cloud semantic segmentation method based on a continuous convolutional network provided by the present invention.

[0046] S2: Convert the point cloud data from the world coordinate system to the pixel coordinate system so that each pixel of the RGB image corresponds to the point cloud data one by one.

[0047] As an optional implementation, the RGB image of the target and the lidar point cloud data are aligned through a perspective projection model, thereby avoiding the loss of appearance feature information of the RGB image and facilitating subsequent feature fusion.

[0048] refer to Figure 2 As shown, X w Axis, Y w Axis, Z w The axes are the three axes in the world coordinate system, O is the optical center of the camera, X c Axis and Y c The axis constitutes the coordinate plane of the camera, parallel to the imaging plane, Z c The axis is the optical axis of the camera and is perpendicular to the imaging plane. The imaging plane of the camera is located along the optical axis and the focal length is f, i.e. OO c =f.u p and v p Represents the pixel coordinate system established with the upper left corner O' as the origin. The perspective projection model transforms the point cloud data from the world coordinate system to the camera coordinate system, then projects it to the image coordinate system and discretizes it to the pixel coordinate system. The specific implementation steps are as follows:

[0049] S2.1: Based on the camera's extrinsic parameter matrix, transform the point cloud data from the world coordinate system to the camera coordinate system. The conversion formula is as follows:

[0050] P c =RP w +T

[0051] Where, P c =[X c , Y c , Z c , 1] T , P w =[X w , Y w , Z w , 1] T , R and T together constitute the external parameter matrix of the camera, R is the rotation matrix, and T is the translation vector;

[0052] S2.2: Based on the principle of similar triangles and the intrinsic parameters of the camera, project the point cloud data converted to the camera coordinate system in S2.1 into the image coordinate system. The conversion formula is as follows:

[0053]

[0054]

[0055] Where x and y are the horizontal and vertical coordinates of the point cloud data projected into the image coordinate system.

[0056] S2.3: Discretize the point cloud data projected into the image coordinate system in S2.2 into the pixel coordinate system, and make a one-to-one correspondence between the point cloud data and the RGB image in S1.

[0057]

[0058]

[0059] Where u and v are the discretized pixel coordinates, u0 and v0 are the horizontal and vertical pixel coordinates of the camera optical center O, respectively.

[0060] S3: Construct and train a two-stream feature extraction network based on perspective photography. Input the RGB image obtained by S1 and the point cloud data corresponding to the RGB image obtained by S2 into the trained two-stream feature extraction network to extract image semantic features and point cloud spatial features.

[0061] The dual-stream feature extraction network includes a point cloud feature extraction network branch and an RGB image feature extraction network branch; wherein the point cloud feature extraction network branch is used to extract features of point cloud data corresponding to the RGB image, and the RGB image feature extraction network branch is used to extract features of the RGB image.

[0062] As an optional implementation, refer to Figure 3 As shown in the figure, the point cloud feature extraction network branch is built based on the PointNet network model, including two multi-layer perceptrons (MLPs), a maximum pooling layer and two fully connected layers.

[0063] In this embodiment, the input of the point cloud feature extraction network branch is n five-dimensional vectors (d, X, Y, Z, r), where X, Y, Z are the three-dimensional coordinates of the point cloud in the world coordinate system, r is the reflection intensity, and d is the range value of each point.

[0064] The input information passes through two multi-layer perceptrons (MLPs), which output local features of 64, 64, 64, 128, and 1024 in sequence, and then passes through the maximum pooling layer and two fully connected layers to obtain the global features of the point cloud. Finally, the global features are connected with the local features to output the point cloud spatial features F p (n, 1, 1152);

[0065] As an optional implementation, refer to Figure 4 As shown in the figure, the RGB image feature extraction network branch is built based on the FCN network model with VGG-19, including 8 convolutional layers and 3 deconvolution layers connected in sequence.

[0066] In this example, the input image size is 1080×1080. After the first 8 convolution operations, the dimension of the image features becomes (1, 1, 4096, C), where C is the number of point cloud semantic segmentation label categories. Then, through three deconvolution layers, the image is mapped from a small resolution to a large resolution, and the image semantic features F are finally output. i (1080, 1080, C).

[0067] S4: Build and train a fusion module based on continuous convolution, input the point cloud spatial features and image semantic features obtained in S3 into the trained fusion module, obtain the fusion features, and add the fusion features to the point cloud spatial features as the output features.

[0068] Due to the difference between point cloud data and RGB images, some appearance information from the RGB image is lost after the two features are fused, making it difficult to achieve the expected semantic segmentation results. Figure 5As shown in the figure, the continuous convolution-based fusion module constructed in this embodiment realizes point-by-point feature fusion. In addition, since the camera in the autonomous driving scene is easily affected by different lighting and weather conditions, and RGB image information is easily lost, this embodiment refers to the CNN network structure and adds attention mechanism aggregation and maximum pooling operations to the fusion module. Finally, the fused features are added to the point cloud features as the final output features, making the fusion more robust and the point cloud representation capability more prominent. Specifically, the following steps are included:

[0069] S4.1: Connect the point cloud spatial features extracted in S3 with the image semantic features to obtain the connected features, and then perform continuous convolution operations to obtain the point-by-point fused semantic features of the front view point cloud;

[0070] S4.2: Perform an Attentive Aggregation operation on the semantic features of the front view point cloud obtained in S4.1, and selectively integrate the features through MLP to obtain the first fused feature.

[0071] S4.3: The most expressive feature is obtained from the connected features of S4.1 through the max-pooling operation as the second fusion feature, and the first fusion feature and the second fusion feature are connected to obtain the fusion feature.

[0072] S4.4: Add the fused features to the point cloud spatial features extracted in S3 to obtain the output features.

[0073] Specifically, this step mainly takes into account that the image semantic features of the RGB image will be affected by the illumination. In this embodiment, the output features obtained are F PI (n, 1, 1152+C).

[0074] S5: Build and train a point cloud semantic segmentation label prediction network, input the output features into the trained point cloud semantic segmentation label prediction network, predict the point cloud semantic segmentation label, and finally obtain the semantic segmentation prediction label for each point in the point cloud data.

[0075] As an optional implementation, refer to Figure 6 As shown in Figure 2, the point cloud semantic segmentation label prediction network uses two multi-layer perceptron networks (MLP).

[0076] In this embodiment, the output feature is F PI (n, 1, 1152+C) is input into two multi-layer perceptron networks (MLPs), which output channel predictions of 512, 256, 128, 128, and C in sequence. C is the category of the point cloud semantic label, and finally outputs the semantic segmentation prediction label of each point in the point cloud in sequence.

[0077] Compared with the existing technology, the present invention provides a point-by-point fusion point cloud semantic segmentation method based on a continuous convolutional network, which adopts a perspective projection model to align the lidar point cloud data and RGB image pixels to reduce the loss of appearance information; designs a dual-stream feature extraction network to process point cloud data and image information at the same time, and efficiently extracts point cloud spatial features and image semantic features; designs a fusion module based on continuous convolution to fuse the point cloud and image point by point, and uses the fused features as a supplementary output of the point cloud features to enhance the point cloud representation capability. Finally, based on the features output by the fusion module, a multi-layer perceptron network (MLP) is used to predict the point cloud semantic segmentation label. Based on a dual-stream feature extraction network and a continuous convolutional network, the present invention reduces the loss of appearance information through a point-by-point feature fusion method under camera coordinates, enhances the representation capability of the on-board lidar point cloud, and improves the point cloud semantic segmentation performance, so as to better understand the semantic scene and ensure the safety and practicality of autonomous driving.

[0078] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A point cloud semantic segmentation method based on continuous convolutional network with point-by-point fusion, characterized by: The following steps are involved: S1: Obtain the target’s RGB image and point cloud data through the vehicle-mounted RGB camera and vehicle-mounted LiDAR respectively; S2: Convert the point cloud data from the world coordinate system to the pixel coordinate system so that each pixel of the RGB image corresponds to the point cloud data one by one; S3: Build and train a two-stream feature extraction network. Input the RGB image obtained by S1 and the point cloud data corresponding to the RGB image obtained by S2 into the trained two-stream feature extraction network to extract image semantic features and point cloud spatial features. S4: Build and train a fusion module, input the point cloud spatial features and image semantic features obtained in S3 into the trained fusion module, obtain fusion features, and add the fusion features to the point cloud spatial features as output features; S5: Build and train a point cloud semantic segmentation label prediction network, input the output features obtained in S4 into the trained point cloud semantic segmentation label prediction network, predict the point cloud semantic segmentation label, and finally obtain the semantic segmentation prediction label for each point in the point cloud data.

2. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 1, characterized in that: In S2, the point cloud data is converted from the world coordinate system to the camera coordinate system through a perspective projection model, then projected from the camera coordinate system to the image coordinate system, and finally discretized from the image coordinate system to the pixel coordinate system.

3. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 2, characterized in that: The S2 comprises the following steps: S2.1: Convert the point cloud data from the world coordinate system to the camera coordinate system according to the camera's extrinsic parameter matrix; S2.2: Based on the principle of similar triangles and the intrinsic parameters of the camera, the point cloud data converted to the camera coordinate system in S2.1 is projected into the image coordinate system. S2.3: Discretize the point cloud data projected into the image coordinate system in S2.2 into the pixel coordinate system, and make a one-to-one correspondence between the point cloud data and the RGB image in S1.

4. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 1, characterized in that: The dual-stream feature extraction network includes a point cloud feature extraction network branch and an RGB image feature extraction network branch; the point cloud feature extraction network branch is used to extract features from point cloud data corresponding to the RGB image, thereby obtaining point cloud spatial features, and the RGB image feature extraction network branch is used to extract features from the RGB image, thereby obtaining image semantic features.

5. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 4, characterized in that: The point cloud feature extraction network branch is built based on the PointNet network model.

6. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 5, characterized in that: The point cloud feature extraction network branch includes two multi-layer perceptrons, a maximum pooling layer and two fully connected layers. The point cloud data corresponding to the RGB image passes through the two multi-layer perceptrons, outputs local features in sequence, and then obtains the global features of the point cloud data through the maximum pooling layer and two fully connected layers. The global features are connected with the local features to obtain the point cloud spatial features.

7. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 4, characterized in that: The RGB image feature extraction network branch is built based on the FCN network model with VGG-19.

8. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 7, characterized in that: The RGB image feature extraction network branch includes 8 convolutional layers and 3 deconvolutional layers connected in sequence. The RGB image passes through 8 convolutional layers and 3 deconvolutional layers in sequence to output image semantic features.

9. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 1, characterized in that: The S4 comprises the following steps: S4.1: Connect the point cloud spatial features extracted in S3 with the image semantic features to obtain the connected features, and then perform continuous convolution operations to obtain the point-by-point fused semantic features of the front view point cloud; S4.2: The semantic features of the front view point cloud obtained in S4.1 are fused point by point, and the attention mechanism is aggregated. The features are selectively integrated through MLP to obtain the first fused feature. S4.3: The most expressive feature is obtained from the concatenated features of S4.1 through the maximum pooling operation, and is used as the second fused feature. The first fused feature and the second fused feature are concatenated to obtain the fused feature. S4.4: Add the fused features to the point cloud spatial features extracted in S3 to obtain the output features.

10. The method for point cloud semantic segmentation based on point-by-point fusion of continuous convolutional networks according to claim 1, characterized in that: The point cloud semantic segmentation label prediction network adopts two multi-layer perceptron networks. After the output features are input into the two multi-layer perceptron networks, the semantic segmentation prediction labels of each point in the point cloud are output in sequence.

Images