A point cloud multi-modal feature fusion network method for 3D scene understanding

Abstract

The application provides a point cloud multi-modal feature fusion network method for 3D scene understanding, comprising the following steps: step 1: collecting existing point cloud data sets, wherein the data set comprises an S3DIS data set and a ModelNet40 data set, and the data in the data set comprises point cloud data containing color information; step 2: generating corresponding multi-view point cloud projection images and point frequency images from the front view, rear view, top view, bottom view, right side view and left side view according to the point cloud data; step 3: using PointNet++ to extract global features and partial local features of the point cloud; step 4: using VGG16 to extract different view image features and aggregate into a global feature; step 5: splicing and combining the features of the point cloud branch and the image branch; step 6: fusing the spliced features through two 1x1 convolution layers to fuse the point cloud features and the image features; step 7: performing bit-by-bit addition operation on the fused features obtained in step 6; step 8: splicing and combining the image features and the features obtained in step 7; step 9: repeating steps 6, 7 and 8 twice to obtain the final fused features; step 10: using a semantic segmentation network to predict point cloud semantic information for the output features of step 9, and using labeled information for supervised training; step 11: using classification to predict point cloud category information for the output features of step 9, and using labeled information for supervised training; step 12: point cloud semantic segmentation and category display, and the application is a point cloud semantic segmentation model with higher precision and more robustness, and can be used in most indoor scene understanding tasks.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, and relates to a point cloud multi-modal feature fusion network method for 3D scene understanding. BACKGROUND

[0002] In recent years, semantic segmentation and scene understanding are widely used in the field of artificial intelligence related to automatic driving, unmanned aerial vehicles, positioning and mapping (SLAM), robots and the like. Meanwhile, 3D point cloud data is rapidly growing, whether it is derived from CAD models or scanned point clouds from LiDAR sensors or RGBD cameras, and is ubiquitous. In addition, most systems directly obtain 3D point clouds rather than taking images and processing them. Therefore, scene understanding of point clouds gradually becomes crucial. However, due to the complexity of the actual environment and the inaccuracy of the depth image, there are still many challenges in the scene understanding of point cloud data. The model of point cloud can preliminarily extract global features and partial local features, while two-dimensional images can well extract features in different views of multiple scales. As a two-dimensional representation of point cloud, it is crucial to adopt a novel complementary feature fusion method. SUMMARY

[0003] Since the use of image-related information can help improve the accuracy and effect of point cloud semantic segmentation, most current semantic segmentation methods use images to extract features or supervised training.

[0004] The method proposes a point cloud multi-modal feature fusion network method for 3D scene understanding, which aims to process the multi-modal fusion of point cloud features and image features for supervised training, and finally obtain a point cloud semantic segmentation model with higher accuracy and more robustness.

[0005] The application proposes a point cloud multi-modal feature fusion network method for 3D scene understanding, comprising the following steps:

[0006] Step 1: collect existing point cloud datasets, wherein the datasets include S3DIS datasets and ModelNet40 datasets, and the data in the datasets includes point cloud data containing color information;

[0007] Step 2: generate the corresponding multi-view point cloud projection images and point frequency images from the front view, rear view, top view, bottom view, right side view and left side view according to the point cloud data;

[0008] Step 3: use PointNet++ to extract global features and partial local features of the point cloud;

[0009] Step 4: use VGG16 to extract different view image features and aggregate them into a global feature;

[0010] Step 5: The features of the point cloud branch and the image branch are spliced and combined;

[0011] Step 6: The spliced features are fused through two 1x1 convolution layers to fuse the point cloud features and the image features;

[0012] Step 7: The fused features obtained in step 6 and the spliced features are bitwise added;

[0013] Step 8: The image features and the features obtained in step 7 are spliced and combined;

[0014] Step 9: After repeating steps 6, 7, and 8 twice, the final fused features are obtained;

[0015] Step 10: The semantic segmentation network is used to predict the point cloud semantic information for the features output in step 9, and the labeled information is used for supervised training;

[0016] Step 11: The classification is used to predict the point cloud category information for the features output in step 9, and the labeled information is used for supervised training;

[0017] Step 12: The 3D scene understanding effect diagram of the point cloud is displayed.

[0018] Further, step 1 includes the following steps:

[0019] Step 1-1: Download the existing S3DIS dataset and ModelNet40 dataset;

[0020] Step 1-2: Process the obtained point cloud data.

[0021] Further, step 2 includes the following steps:

[0022] Step 2-1: Project the point cloud data from the front view, rear view, top view, bottom view, right side view, and left side view, respectively;

[0023] Step 2-2: Generate the corresponding multi-view images.

[0024] Further, step 3 includes the following steps:

[0025] Step 3-1: Construct the PointNet++ image feature extraction network and input the original point cloud data to extract the point cloud features, and load the pre-trained model parameters;

[0026] Step 3-2: The PointNet++ network head is mainly composed of sample&grouping and pointnet setabstraction for local and global feature extraction of the point cloud.

[0027] Further, step 4 includes the following steps:

[0028] Step 4-1: Construct the VGG16 image feature extraction network and load the pre-trained model parameters, input different view images to extract features;

[0029] Step 4-2: The head of the VGG16 network is composed of two 3x3 convolution layers and a 2x2 pooling layer in succession twice, plus two 3x3 convolution layers, a 1x1 convolution layer and a 2x2 pooling layer in succession three times.

[0030] Further, the method of step 5 is as follows:

[0031] The point cloud features obtained by the point cloud branch are spliced and merged with the features of the RGB image and the point frequency image of the projection view.

[0032] Further, the method of step 6 includes:

[0033] The spliced point cloud features and image features are fused through two 1x1 convolution layers, and then passed through a Relu activation layer.

[0034] Further, the method of step 7 includes the following steps:

[0035] The fused features and the spliced features are added bit by bit.

[0036] Further, the method of step 8 includes:

[0037] The image features and the features obtained in step 7 are spliced and merged.

[0038] Further, the method of step 9 is as follows:

[0039] After repeating steps 6, 7 and 8 twice, the features of two-dimensional images and three-dimensional point clouds are fused through a 1x1 convolution layer to change the channel number depth.

[0040] Further, the method of step 10 includes:

[0041] The features obtained in step 9 are used to predict the semantic information of the point cloud using a semantic segmentation network, and the labeled information is used for supervised training.

[0042] Further, the method of step 11 is as follows:

[0043] Step 11-1: The features obtained in step 9 are used to predict the category information of the point cloud using a classification network, and the labeled information is used for supervised training;

[0044] Step 11-2: The classification network is composed of several fully connected layers.

[0045] Further, the method of step 12 is as follows:

[0046] Finally, the obtained point cloud semantic segmentation and semantic category are displayed.

[0047] Advantages of the present application:

[0048] The correlation and complementarity between multi-tasks in an indoor scene are explored; a point cloud multi-modal feature fusion network method for 3D scene understanding is proposed, which can process and fuse point cloud features and image features and complementary semantic information for supervised training; finally, a point cloud semantic segmentation model with higher accuracy and more robustness can be obtained, which can be used in most indoor scene understanding tasks.

[0049] The method has achieved very high accuracy in indoor scene understanding tasks, and is widely applicable and can be adapted to various different task combinations. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above or other aspects of the present application will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings.

[0051] Figure 1 For the overall flowchart of the algorithm, steps 3 to 11 correspond to steps 3 to 11;

[0052] Figure 2 For the framework diagram of the point cloud multi-modal feature fusion network method for 3D scene understanding;

[0053] Figure 3 For the final semantic segmentation effect diagram. DETAILED DESCRIPTION

[0054] The present application will be described in detail below with reference to the accompanying drawings and examples.

[0055] The algorithm generally includes the following steps:

[0056] As shown in Figures 1-2 A point cloud multi-modal feature fusion network method for 3D scene understanding includes the following steps:

[0057] Step 1: Collect existing point cloud datasets, wherein the datasets include S3DIS datasets and ModelNet40 datasets, and the data in the datasets include point cloud data containing color information;

[0058] Step 2: Generate the corresponding multi-view point cloud projection images and point frequency images from the front view, rear view, top view, bottom view, right side view and left side view according to the point cloud data;

[0059] Step 3: Use PointNet++ to extract global features and partial local features of the point cloud;

[0060] Step 4: Extract different view image features using VGG16 and aggregate them into a global feature;

[0061] Step 5: Concatenate and merge the features of the point cloud branch and the image branch;

[0062] Step 6: Fuse the point cloud features and image features through two 1x1 convolution layers;

[0063] Step 7: Perform bitwise addition operation on the fused features obtained in step 6 and the concatenated features;

[0064] Step 8: Concatenate and merge the image features and the features obtained in step 7;

[0065] Step 9: Repeat steps 6, 7, and 8 twice to obtain the final fused features;

[0066] Step 10: Use the semantic segmentation network to predict the point cloud semantic information for the features output in step 9, and use the labeled information for supervised training;

[0067] Step 11: Use the classification to predict the point cloud category information for the features output in step 9, and use the labeled information for supervised training;

[0068] Step 12: Display the 3D scene understanding effect diagram of the point cloud.

[0069] Further, step 1 includes the following steps:

[0070] Step 1-1: Download existing S3DIS dataset and ModelNet40 dataset;

[0071] Step 1-2: Process and obtain point cloud data.

[0072] Further, step 2 includes the following steps:

[0073] Step 2-1: Project the point cloud data from the front view, rear view, top view, bottom view, right side view, and left side view;

[0074] Step 2-2: Generate the corresponding multi-view images.

[0075] Further, step 3 includes the following steps:

[0076] Step 3-1: Construct the PointNet++ image feature extraction network and input the original point cloud data to extract point cloud features, and load the pre-trained model parameters;

[0077] Step 3-2: The network head of PointNet++ is mainly composed of sample & grouping and pointnet to extract local and global features of point cloud.

[0078] Further, step 4 includes the following steps:

[0079] Step 4-1: Construct the VGG16 image feature extraction network and load the pre-trained model parameters, input different view images to extract features;

[0080] Step 4-2: The head of VGG16 network is composed of two 3x3 convolution layers and a 2x2 pooling layer for two times in succession, plus two 3x3 convolution layers, a 1x1 convolution layer and a 2x2 pooling layer for three times in succession.

[0081] Further, step 5 method is as follows:

[0082] The point cloud features obtained by the point cloud branch are spliced and merged with the features of the RGB image and the point frequency image of the projection view.

[0083] Further, step 6 method includes:

[0084] The spliced point cloud features and image features are fused through two 1x1 convolution layers, and then passed through a Relu activation layer.

[0085] Further, step 7 method includes the following steps:

[0086] The fused features and the features obtained by splicing are added bit by bit.

[0087] Further, step 8 method includes:

[0088] The image features and the features obtained by step 7 are spliced and merged.

[0089] Further, step 9 method is as follows:

[0090] After repeating steps 6, 7 and 8 twice, the features of two-dimensional image and three-dimensional point cloud are fused through 1x1 convolution layer to change the channel number depth.

[0091] Further, step 10 method contains:

[0092] The features obtained by step 9 are used to predict the semantic information of point cloud using semantic segmentation network, and the labeled information is used for supervised training.

[0093] Further, step 11 method is as follows:

[0094] Step 11-1: the feature obtained in step 9 is used to predict the category information of the point cloud by using a classification network, and the labeled information is used for supervised training;

[0095] Step 11-2: the classification network is composed of a plurality of fully connected layers.

[0096] Further, the method in step 12 is as follows:

[0097] Finally, the obtained point cloud semantic segmentation and semantic category are displayed.

[0098] Figure 3 For the final semantic segmentation effect diagram, the left column represents the original point cloud, the middle column represents the true value of semantic segmentation, and the right column is the final semantic segmentation effect of our product.

[0099] The present application has many methods and ways to achieve the technical solution, and the above description is only the preferred embodiment of the present application. It should be pointed out that for ordinary technical personnel in the technical field, without departing from the principle of the present application, a number of improvements and refinements can be made, which should be regarded as the protection scope of the present application. The components not explicitly described in the embodiment can be realized by using existing technology.

Claims

1. A point cloud multimodal feature fusion network method for 3D scene understanding, characterized in that, Includes the following steps: Step 1: Collect existing point cloud datasets, including the S3DIS dataset and the ModelNet40 dataset. The data in the datasets includes point cloud data with color information. Step 2: Generate point cloud projection images and point frequency images for the corresponding multi-view views from the front view, rear view, top view, bottom view, right view, and left view based on the point cloud data; Step 3: Use PointNet++ to extract global features and some local features of the point cloud; Step 4: Use VGG16 to extract features from different view images and aggregate them into a global feature; Step 5: Combine and merge the features of the point cloud branches and the image branches; Step 6: Fuse the stitched features with point cloud features and image features through two 1x1 convolutional layers; Step 7: Perform a bit-by-bit addition operation on the fused features obtained in Step 6 and the spliced ​​features obtained in Step 5; Step 8: Combine and merge the global image features obtained in Step 4 and the features obtained in Step 7; Step 9: Repeat steps 6, 7, and 8 twice to obtain the final fusion feature; Step 9 is as follows: After repeating steps 6, 7, and 8 twice, the features of the two-dimensional image and the three-dimensional point cloud are fused by changing the number of channels through a 1x1 convolutional layer. Step 10: Use a semantic segmentation network to predict the semantic information of the point cloud based on the features output from Step 9, and use the labeled information for supervised training; Step 11: Use the classification to predict the point cloud category information for the features output in Step 9, and use the annotation information for supervised training; Step 12: Display the 3D scene understanding effect diagram of the point cloud.

2. The method according to claim 1, characterized in that, Step 1 includes the following steps: Step 1-1: Download the existing S3DIS dataset and ModelNet40 dataset; Steps 1-2: Process and acquire point cloud data.

3. The method according to claim 2, characterized in that, Step 2 includes the following steps: Step 2-1: Project point cloud data from the front view, rear view, top view, bottom view, right view, and left view respectively; Step 2-2: Generate its corresponding multi-view image.

4. The method according to claim 3, characterized in that, Step 3 includes the following steps: Step 3-1: Construct the PointNet++ image feature extraction network and input the original point cloud data to extract point cloud features, and load them into the pre-trained model parameters; Step 3-2: The PointNet++ network head is mainly composed of sample & grouping and setabstraction, which performs local global feature extraction on the point cloud.

5. The method according to claim 4, characterized in that, Step 4 includes the following steps: Step 4-1: Construct the VGG16 image feature extraction network and load it into the pre-trained model parameters, then extract features from different view images. Step 4-2: The VGG16 network header consists of two consecutive 3x3 convolutional layers and a 2x2 pooling layer, plus three consecutive 3x3 convolutional layers, a 1x1 convolutional layer, and a 2x2 pooling layer.

6. The method according to claim 5, characterized in that, Step 5 is as follows: The point cloud features obtained from the point cloud branch are combined with the features of the RGB image and point frequency image of the projection view.

7. The method according to claim 6, characterized in that, Step 6 includes the following methods: The spliced ​​point cloud features and image features are fused through two 1x1 convolutional layers, and then activated by a ReLU layer.

8. The method according to claim 7, characterized in that, Step 8 includes the following methods: Then, the image features and the features obtained in step 7 are combined and merged.

9. The method according to claim 8, characterized in that, Step 10 includes the following methods: The features obtained in step 9 are used to predict the semantic information of the point cloud using a semantic segmentation network, and supervised training is performed using the labeled information.

10. The method according to claim 9, characterized in that, Step 11 is as follows: Step 11-1: Use the features obtained in Step 9 to predict the category information of the point cloud using a classification network, and use the annotation information for supervised training; Step 11-2: The classification network consists of several fully connected layers.

11. The method according to claim 10, characterized in that, Step 12 is as follows: Finally, the obtained point cloud semantic segmentation and semantic category are displayed.

Images