A 3D point cloud scene segmentation method and system integrating image features

Abstract

The present invention provides a method and system for segmenting a three-dimensional point cloud scene by integrating image features, which relates to the field of computer vision technology and can realize the effective integration of two-dimensional images and three-dimensional point clouds, as well as the accurate segmentation of three-dimensional scenes; the method comprises: S1, acquiring two-dimensional data including images, point cloud data, and depth data, and calculating the correlation between scene images and point clouds based on the acquired data; S2, extracting features from the two-dimensional data to obtain a high-dimensional feature map to be integrated; S3, integrating the feature map to be integrated and the point cloud data according to a fusion strategy to obtain integrated point cloud data; the fusion strategy comprises: searching for pixels adjacent to a certain point cloud data, and distorting the corresponding features of the pixels onto the point cloud data; S4, inputting the integrated point cloud data into a three-dimensional segmentation network for feature extraction, thereby obtaining the required global and local semantic information. The technical solution provided by the present invention is applicable to the process of three-dimensional point cloud processing.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a three-dimensional point cloud scene segmentation method and system integrating image features. Background Art

[0002] With the development of artificial intelligence, people are eager for machines to obtain information about the real world like humans, analyze the data intelligently, and make corresponding judgments. Computer vision, as an important technology for machines to perceive the real world, has always been a hot topic in the field of artificial intelligence. Semantic segmentation of three-dimensional scenes, as a basic task of computer vision, has important research value and application prospects, and the required accuracy is also increasing.

[0003] Scene data includes two-dimensional data represented by two-dimensional images and three-dimensional data represented by RGB-D depth maps and three-dimensional data. The imaging quality of two-dimensional images will be reduced to a certain extent due to the influence of lighting conditions, complex environments, etc. At the same time, object occlusion is also an important factor that two-dimensional images cannot better describe the real world. RGB-D depth maps are based on two-dimensional images and add pixel depth information. Three-dimensional point clouds are sparse and unstructured three-dimensional data used to describe the shape and position of objects in the real world. Compared with two-dimensional images, three-dimensional point clouds have more details and features in representing real scenes, and have richer structural information and semantic information. With the rapid development and application of laser sensors and depth sensors, the difficulty of obtaining point clouds has been reduced. The development of three-dimensional scanning technology has also promoted the growth and quality improvement of three-dimensional data. Many complete three-dimensional point cloud scene data have emerged on the Internet.

[0004] Point cloud segmentation is the basis and premise of three-dimensional scene understanding. Point cloud segmentation is to divide the points in the point set into different categories according to certain rules, corresponding to different semantic labels. After a long period of development, researchers have proposed a large number of traditional segmentation algorithms. After continuous research and improvement, traditional segmentation algorithms have continued to improve. However, since traditional segmentation algorithms require manual design of feature descriptors, they cannot achieve adaptive segmentation tasks and have poor generalization capabilities for different segmentation tasks. In recent years, deep learning has been widely used in the field of computer vision due to its self-learning ability. Among them, convolutional neural networks have shown great advantages in two-dimensional scene segmentation tasks, promoting the rapid development of the field of computer vision. Due to the characteristics of point clouds such as disorder and sparsity, scene segmentation based on three-dimensional point clouds has always been a difficult point in research.

[0005] Point clouds are unordered collections of spatial points, which do not have the orderliness and density of two-dimensional pixels. Therefore, conventional convolutional neural networks cannot be directly applied to point clouds. In recent years, researchers have proposed many methods to overcome this problem, which can be generally divided into two categories: regularization-based methods and original point cloud-based methods.

[0006] The regularization-based method converts the point cloud into a regular structure such as voxels or multi-views, and then inputs it into a convolutional neural network for processing. Representative methods of this type are VoxNet and MVCNN, where VoxNet converts the original unstructured point cloud into regular three-dimensional voxel data and uses a three-dimensional convolutional neural network to extract features on the voxel data. MVCNN is a pioneer in applying multi-view methods to point cloud recognition. It obtains the features of multi-view images through convolutional neural networks and aggregates multi-view features using pooling.

[0007] The method based on the original point cloud is to directly learn the features of the point cloud without converting it into other forms of data. PointNet is the first neural network to directly process raw point cloud data. The network obtains the global feature vector of the point cloud through a multi-layer perceptron and a maximum pooling layer, solving the problems of disorder and transformation invariance of the point cloud. Subsequently, the original author proposed a more outstanding PointNet++ model, a feature extraction structure with a multi-level receptive field, and enhanced the network's ability to extract local features of different densities. However, the position distribution information in three-dimensional space is still difficult to capture. How to convert the association information between points into semantic information is a problem that still needs to be solved.

[0008] When two-dimensional data and three-dimensional data can be registered, making full use of the two types of data is a way to improve current visual tasks. Many researchers solve visual problems by spatially mapping the two types of information. 3DMV maps 2D image features to 3D space voxel grids. Each voxel grid corresponds to multiple views, and each voxel feature is obtained by pooling the features of the corresponding view. Similarly, some researchers project three-dimensional point clouds onto two-dimensional planes, extract and fuse features from views on different two-dimensional planes, and then predict the semantic label of each point. The above methods are attempts to convert two-dimensional data into three-dimensional data and then assist in feature extraction. How to effectively combine the two types of data and improve the accuracy of the point cloud segmentation network is very challenging. First, as data in three-dimensional space, the spatial distribution characteristics and high-dimensional information between points of point clouds are difficult to capture. Secondly, three-dimensional point clouds are composed of disordered points, while two-dimensional images are ordered pixels. It is difficult to fuse two different types of data or features. Compared with the regular grid structure of image data, the sparsity and unstructured nature of point clouds make it impossible to directly extract features through CNN.

[0009] Therefore, it is necessary to study a three-dimensional point cloud scene segmentation method and system that integrates image features to address the shortcomings of the existing technology in order to solve or alleviate one or more of the above problems. Summary of the invention

[0010] In view of this, the present invention provides a three-dimensional point cloud scene segmentation method and system that integrates image features, which can achieve effective fusion of two-dimensional images and three-dimensional point clouds, as well as accurate segmentation of three-dimensional scenes.

[0011] In one aspect, the present invention provides a three-dimensional point cloud scene segmentation method integrating image features, the method comprising the following steps:

[0012] S1, acquiring two-dimensional data including an image, point cloud data, and depth data, and calculating the correlation between the scene image and the point cloud based on the acquired data;

[0013] S2, extracting features from the two-dimensional data to obtain a high-dimensional feature map to be fused;

[0014] S3, fusing the feature map to be fused and the point cloud data according to a fusion strategy to obtain fused point cloud data; the fusion strategy includes: searching for pixels adjacent to a certain point cloud data through an association relationship, and distorting the corresponding features of the pixels onto the point cloud data;

[0015] S4. Input the fused point cloud data into the 3D segmentation network for feature extraction to obtain the required global and local semantic information.

[0016] As described above and any possible implementation method, an implementation method is further provided, before fusion, the point cloud data is subjected to key point sampling processing to obtain the point cloud data to be fused, and then the point cloud data to be fused is fused with the feature map to be fused.

[0017] According to the above aspects and any possible implementation, there is further provided an implementation, wherein the content of extracting features from the two-dimensional data in step S2 includes: extracting features through a two-dimensional segmentation network;

[0018] Preferably, a U-Net-ResNet network whose encoder is ResNet is used to extract image features, and the U-Net-ResNet network is a network that has been trained with a model and has a sufficient accuracy.

[0019] According to the aspects described above and any possible implementation method, an implementation method is further provided, in which the calculation method of the association relationship between the scene image and the point cloud in step S1 is: select a certain point cloud data, project the pixel into the three-dimensional space through the depth data of the image, calculate the spatial distance between the pixel and the selected point cloud, and when the spatial distance is less than the judgment threshold, the pixel is considered to be a valid pixel; count the total number of valid pixels of all scene images in a certain scene and sort them from large to small, and select the first N scene images as the original images of the two-dimensional data in step S2; N is a positive integer.

[0020] According to the aspects described above and any possible implementation method, an implementation method is further provided, in which the point cloud data selected when counting the total number of valid pixels is all point cloud data corresponding to the scene or point cloud data of key points in the scene.

[0021] As described above and any possible implementation method, an implementation method is further provided, wherein the content of feature extraction in step S includes: capturing the feature relationship between points and neighborhoods in three-dimensional space through the RFE module and the CSA module, and performing spatial feature extraction and feature semantic correction on the fused point cloud data.

[0022] According to the above aspects and any possible implementation, an implementation is further provided, wherein the RFE module includes three RE submodules, which are used to encode the position information of the key point cloud and the neighboring points, so as to capture the spatial distribution information of the key points and the neighboring points;

[0023] The CSA module processes the neighborhood spatial features through SharedMLP, alternately generates the attention matrix and the next layer of local features and multiplies them, so as to achieve the effects of spatial feature extraction and semantic information enhancement.

[0024] As described above and any possible implementation method, an implementation method is further provided, in which the RFE module uses a shared function g containing one or more SharedMLPs to learn the features of each encoded information, and then performs nonlinear learning through an activation function.

[0025] According to the above aspects and any possible implementation, an implementation is further provided, wherein the CSA module generates an attention matrix and local features through a g function consisting of a set of SharedMLP and Softmax;

[0026] Among them, the attention matrix is ​​defined as:

[0027]

[0028] A irepresents the attention matrix of the current i layer, W is the parameter matrix learned by SharedMLP, N is the position encoding information obtained by the previous RFE module, and r i is the feature extracted from the current i-th layer. r0 represents F, which is the feature information obtained by RFE.

[0029] On the other hand, the present invention provides a three-dimensional point cloud scene segmentation system integrating image features, the system being used to implement any of the above methods; the system comprising:

[0030] A data processing module is used to obtain two-dimensional data, point cloud data and depth data, and calculate the association relationship between the scene image and the point cloud, and is also used to package the association relationship and the point cloud data together;

[0031] The two-dimensional image feature encoding module is used to extract features from two-dimensional data to obtain a high-dimensional feature map to be fused;

[0032] The fusion module is used to fuse the semantic features of pixels into point cloud data according to the positional relationship between image pixels and point cloud, effectively fusing data and enhancing semantic information;

[0033] The point cloud feature extraction module is used to extract features from the fused point cloud data and perform semantic segmentation on the fused point cloud data at multiple scales.

[0034] The terms U-Net-ResNet network, ResNet, SharedMLP, Softmax, RFE module, CSA module, etc. used in the present invention are conventional terms or English abbreviations in the field, and are commonly used tools in the field of image processing or tools improved by the present invention based on commonly used tools.

[0035] Compared with the prior art, one of the above technical solutions has the following advantages or beneficial effects: the present invention can realize the effective fusion of two-dimensional images and three-dimensional point clouds, enhance structural information and semantic information, facilitate the precise segmentation and accurate feature extraction of three-dimensional scenes, and effectively grasp the global and local semantic information;

[0036] Another technical solution in the above technical solution has the following advantages or beneficial effects: by selecting key point cloud data and selecting N most representative scene images for operation, the amount of data can be greatly reduced and the calculation efficiency can be improved while ensuring high segmentation accuracy.

[0037] Of course, any product implementing the present invention does not necessarily need to achieve all of the above-mentioned technical effects at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is a flow chart of a three-dimensional point cloud segmentation method integrating image features provided by an embodiment of the present invention;

[0040] Figure 2 is a feature fusion module diagram provided by an embodiment of the present invention;

[0041] Figure 3 is a point cloud segmentation network diagram provided by an embodiment of the present invention;

[0042] Figure 4 is a JALayer structure diagram provided by an embodiment of the present invention;

[0043] Figure 5 is a diagram of a RE{64,64} module provided by an embodiment of the present invention;

[0044] Figure 6 is a neighborhood space feature coding map provided by an embodiment of the present invention;

[0045] Figure 7 It is a CSA module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0047] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In view of the shortcomings of the prior art, the present invention designs a point cloud segmentation model that integrates image features. The network model is constructed by a two-dimensional feature detector (two-dimensional feature encoder), feature fusion and a three-dimensional segmentation network. The two-dimensional feature detector extracts the features of the two-dimensional image as accurately as possible, then fuses the image features and the point cloud, and finally inputs the three-dimensional segmentation network to achieve accurate segmentation of the three-dimensional scene. Specifically, the image and point cloud fusion strategy is first designed, that is, by searching for pixels adjacent to the point cloud and distorting the pixel features to the point cloud data. In the two-dimensional feature encoding stage, the two-dimensional segmentation network is pre-trained so that the segmentation result of the two-dimensional segmentation network reaches a certain level, and the encoder of the two-dimensional segmentation network is used as the two-dimensional feature encoder of the overall scene segmentation network. After the point cloud data is fused with the two-dimensional features extracted by the two-dimensional feature encoder, it is input into the three-dimensional segmentation network. In the three-dimensional segmentation network, the present invention designs two efficient feature extraction modules with different functions, wherein the Relative Feature Extraction (RFE) module is used to capture the feature relationship between points and neighborhoods in three-dimensional space, and better extract the association information between points, and the Cross Spatial Attention (CSA) module is used to enhance and correct the extracted semantic features of the point cloud. Feature extraction is performed on point clouds of different scales to extract features of different receptive fields, thereby effectively grasping global and local semantic information. The flowchart of the present invention for segmenting a three-dimensional point cloud scene is as follows: Figure 1 shown.

[0049] The point cloud segmentation model of the present invention mainly includes a data processing module, a two-dimensional image feature encoding module, a fusion module and a point cloud feature extraction module. The four functional modules are:

[0050] Data processing module: matches and packages two-dimensional images, depth information, and three-dimensional point clouds to facilitate network reading and training.

[0051] 2D image feature encoding module: extract features from 2D scene images to semantically supplement point cloud data.

[0052] Fusion module: Based on the positional relationship between image pixels and point cloud, the semantic features of pixels are fused to point cloud data, effectively fusing data and enhancing semantic information.

[0053] Point cloud feature extraction module: used for feature extraction of fused data, and semantic segmentation and prediction of fused data at multiple scales.

[0054] The three-dimensional point cloud scene segmentation method of the present invention mainly includes the following steps:

[0055] Step 1: Image and point cloud data preprocessing

[0056] The image and point cloud use the ScanNetV2 data set, which includes 1513 scenes and objects of 21 semantic categories. Each scene contains a video frame of scene acquisition, depth information and a scene point cloud, which can better evaluate the performance of the present invention. In the video frame of the scene, the picture with the maximum coverage of the key points of the scene point cloud is selected. In order for the feature module to play a role, the adjacent relationship between the scene point cloud and the two-dimensional image is calculated at this stage and compressed and packaged. After preprocessing and packaging operations on the picture and point cloud, the data used by the segmentation network of the present invention is obtained.

[0057] Step 2: Build a point cloud segmentation network model based on fused image features.

[0058] Image features are extracted by the trained 2D segmentation network, encoding the scene view image into a high-dimensional feature map. According to the spatial position relationship between the image and the point cloud, the high-dimensional feature map is fused with the point cloud, and the enhanced fused point cloud data is finally output. Then, the fused data is input into the 3D point cloud segmentation network to obtain the final segmentation result.

[0059] In the 3D point cloud segmentation part, in order to capture the local point cloud features, the Relative Feature Extraction (RFE) module is designed to capture the feature relationship between the point and the neighborhood in the 3D space. More specifically, the Position Encoder (PE) is used to encode the relationship between the neighboring points and the center point, and this relationship is abstracted through a neural network. In order to extract spatial features and modify feature semantics for the fused data, a Cross Spatial Attention (CSA) module is proposed. The CSA module contains three groups of cross attention modules, in which the attention module generates an attention matrix that adapts to its own feature map and performs operations with the feature map to achieve the effect of semantic modification and enhancement.

[0060] Step 3: Train the network model and optimize the network model parameters.

[0061] In order to improve the efficiency of the network, the present invention calculates the spatial position relationship between the image and the point cloud in the preprocessing stage, and compresses and packages the results, avoiding the problem of too long time in network training. The two-dimensional images used by the network are stored in different folders in units of scenes, and the spatial position relationship between the image and the point cloud and the point cloud data are packaged and stored in a single file format.

[0062] First, the 2D segmentation network is trained, and supervised learning is performed on the 2D image of the scene. After continuous iteration and optimization of parameters and models through back propagation, a 2D image feature encoder is obtained. Subsequently, the 2D image feature encoder and the 3D point cloud segmentation network are connected through the feature fusion module, and the parameter training of the 3D point cloud segmentation network is completed on the image data and packaged data. Thus, a trained 3D point cloud segmentation network model with fused image features is obtained, and the network parameters and model structure are saved in the form of a .pth file.

[0063] Step 4: Input the scene image and point cloud to be segmented into the trained neural network to obtain the segmentation result.

[0064] The test set of scene images and point clouds is input into the pre-trained network model for testing experiments to obtain a complete and high-precision scene segmentation result map.

[0065] Embodiment 1:

[0066] Step 1: Preparation and preprocessing of scene images and point cloud data

[0067] ScanNetV2 is an indoor scene dataset that is captured using the iPad's internal camera and an additional depth camera. Each scan of the dataset consists of an RGB-D sequence with associated poses, a complete scene mesh, and semantic and instance labels, with a total of 2.5M view images, 1513 point cloud meshes, and objects of 21 semantic categories. Each scene includes two-dimensional data and three-dimensional data, the two-dimensional data is the video frame of the scene acquisition, and the three-dimensional data is the point cloud information. The present invention uses 1201 scan data for training and 312 scan data for testing.

[0068] In order to improve the efficiency of network training and save training time, the relationship between the scene image and the point cloud is calculated in the preprocessing step. First, the scene point cloud is downsampled, the key points of the scene point cloud are selected, and the image pixels are projected into three-dimensional space through the depth information of the image. The spatial distance between the pixel and the point cloud is calculated, and a distance threshold is set. When the distance between the pixel and the key point is less than the threshold, it is considered that the current pixel can represent the key point. The more such pixels in the image, the stronger the ability of this image to represent the scene point cloud. For each scene, the relationship between the scene point cloud and all images is calculated, and 5 scene images with the strongest representation are selected as subsequent training data. The picture number and the scene point cloud are packaged together, and finally the training data and test data that can be used by the network are obtained.

[0069] Step 2: Image feature extraction and feature fusion module design

[0070] To solve the problem of fusion of point cloud and image, in order to fuse the two data more effectively, the present invention designs a distance-based feature fusion method, such as Figure 2 As shown in the figure, through the image depth information, the two-dimensional pixels are mapped to the three-dimensional space, and the feature fusion is performed through the global point cloud position information and the image pixel position information to achieve the effect of supplementing the semantic information of the point cloud data.

[0071] The feature information of the image can be obtained through the traditional two-dimensional segmentation network. The present invention uses the U-Net-ResNet network to extract image features, and trains the image data on the U-Net-ResNet so that the network can achieve a certain accuracy. In the U-Net-ResNet network, ResNet is used as an encoder to further improve the feature extraction capability of the image. The trained two-dimensional network is used as a two-dimensional feature encoder in the three-dimensional network training to extract the features of the image. In the next step, the image features are fused with the point cloud data in the feature fusion module.

[0072] In the feature fusion module, assume that N H x W x C two-dimensional feature maps and a set of point clouds are input. First, the pixels of the two-dimensional feature map are projected into the three-dimensional space, and the dense pixels are downsampled to obtain an M x C pixel point cloud. Each point in the pixel point cloud is calculated from the pixel coordinates and depth information, where M <N x H x W。

[0073] For a point p in the point cloud i , we use the Euclidean distance to find the point of its nearest K pixel point cloud in three-dimensional space in The spatial coordinates of the points representing the M x C pixel point cloud, Represents a splicing operation.

[0074]

[0075] By encoding the location information of pixels and point clouds Simulate and calculate each pixel P i The semantic contribution of . Finally, we get a set of neighboring pixel encodings And then fused with image features in the next step.

[0076]

[0077] like Figure 2 As shown in the figure, the feature fusion module integrates the information of each point with the features of adjacent pixels, where F i Represents the information of the fused point, which integrates point p i and the feature information of its neighboring pixels f ik and through f i k Enhance the semantic information of fused data.

[0078] Step 3: Point cloud segmentation network design

[0079] The present invention designs a point cloud segmentation network based on PointNet++, such as Figure 3 As shown in Figure 2, the network structure can better capture the point cloud features of different granularities and then predict the semantic labels of the fused point cloud data.

[0080] For the fusion data mentioned in the previous step, the present invention designs a feature extraction structure JALayer to better extract the spatial features and semantic features of the point cloud. JALayer consists of relative feature extraction (Relative feature extraction, RFE) and cross spatial attention (Cross Spatial Attention, CSA), the detailed structure is as follows Figure 4 shown.

[0081] In practical applications, a scene contains a huge number of point clouds, and even a single object point cloud is composed of a large number of points. Traditional convolutional neural networks require that the processed data is structured when performing feature extraction and prediction, and the parameters of the neural network increase with the increase in data volume. For the massive points in the point cloud data, the key points that can represent the point cloud and the neighboring points near the key points are extracted from the point cloud by sampling and grouping. In this way, the burden of the neural network is reduced and the training efficiency is improved. In JALayer, the point cloud first samples the farthest point, collects the key points that can represent the point cloud, and obtains the neighboring points of each key point through ball query. Then the coordinate information and feature information of the key points and neighboring points are sent to the RFE module for spatial feature encoding.

[0082] The RFE module consists of three Relative Encoding (RE) modules, which are designed to capture the spatial distribution information of key points and neighboring points. Figure 5 As shown in Figure 1, the RE module uses Position Encoding to encode the position information of key points and adjacent points. Given a point cloud P and the features of each point, for the center point p i The nearest k points The position information is encoded and a shared function g is designed to learn the features of each encoded information. The g function contains one or more SharedMLPs, and each SharedMLP is followed by a ReLU activation function for nonlinear learning. The Position Encoding formula is as follows:

[0083]

[0084] where p i and Respectively represent the spatial positions of the center point and its neighboring points, Represents the concatenation operator, and the “‖‖” symbol calculates the distance in Euclidean space. The relative position information of points in the local neighborhood is encoded, and this relative position information is abstracted into high-dimensional features through the g function.

[0085] like Figure 6 As shown, in RFE, for each point Two RE modules are used to encode local features and global features in the neighborhood respectively, and then concatenated into neighborhood spatial features. The formula is as follows:

[0086]

[0087] Where W0 and W1 are the weight matrices of SharedMLP respectively.

[0088] In the feature abstraction stage, the SharedMLP module, as an effective encoder, combined with the Attention mechanism, can better complete the feature extraction task. Point cloud data is information of disordered points. After being abstracted into features by SharedMLP, unlike two-dimensional convolution, the receptive field of each point is not expanded. Therefore, in our 3D network, multi-layer JALayer is used to extract features from different center points. Figure 7 As shown in the figure, we designed a Cross SpatialAttention (CSA) module. The core idea is to process the neighborhood spatial features through SharedMLP, alternately generate the attention matrix and the next layer of local features and multiply them, so as to achieve the effect of spatial feature extraction and semantic information enhancement.

[0089] We designed a g function to generate specific attention matrices and local features. Function g consists of a set of SharedMLP and Softmax. The attention matrix is ​​defined as follows:

[0090]

[0091] Among them A i represents the attention matrix of the current i layer, and W is the parameter matrix learned by SharedMLP.

[0092] r i =g(A i-1 ·r i-1 )

[0093] like Figure 7As shown, three matrix element multiplications are performed in the CSA module. For a given point cloud neighborhood spatial feature and local features The CSA module learns to aggregate local features and corrects the features through semantic enhancement. Finally, the encoded feature information vector F is obtained by maximum pooling. i .

[0094] Step 4: Build a neural network model based on multi-level feature fusion

[0095] like Figure 1 As shown, the overall architecture of the segmentation network of the present invention is divided into two branches. The upper part is the two-dimensional image feature extraction branch, and the lower part is the three-dimensional point cloud scene segmentation branch, and the two are connected by a feature fusion module. In the two-dimensional image feature extraction branch, a two-dimensional segmentation network is used to obtain the two-dimensional features of the scene view, and the two-dimensional image of the scene is encoded into a high-dimensional feature map. In the feature fusion module, the two-dimensional features and the three-dimensional point cloud data are fused in the form of position encoding, and finally the enhanced fused point cloud data is output. Then, the fused data is input into the three-dimensional point cloud segmentation network. The three-dimensional point cloud scene segmentation branch captures the feature relationship between the point and the neighborhood in the three-dimensional space through the RFE module and the CSA module, and performs spatial feature extraction and feature semantic correction on the fused data, so that the network's semantic prediction and point cloud segmentation capabilities are further enhanced.

[0096] Step 5: Train the network model and verify the network segmentation effect.

[0097] In order to improve the efficiency of the network, the present invention places images and point clouds in two folders with the same scene name, compresses and packages the image numbers and scene point clouds in the preprocessing stage, and queries images by image numbers in the training and verification stages, thereby reducing memory usage.

[0098] The experiment of the present invention uses RTX2080ti graphics processor for accelerated learning, the system platform is Ubuntu16.04, and the deep learning framework is Pytorch. The two-dimensional feature encoder is obtained by training U-Net-ResNet with scene images. The model optimizer uses Adam, the learning rate is 0.004, the batch_size of the experiment is set to 16, and the cross entropy is used as the loss function. In the three-dimensional point cloud scene segmentation network, the number of JALayer layers is set to 4, the key points of each layer are 2048, 512, 128, and 32, and the neighboring point query radius is set to 0.1m, 0.2m, 0.4m, and 0.8m, respectively. In the decoding stage, the present invention adopts the same settings as PointNet++.

[0099] In the training phase, U-Net-ResNet is first trained on the scene image dataset. The trained U-Net-ResNet is used as the two-dimensional feature encoder of the entire network. The network parameters are not updated in the subsequent stages, and only the encoder is involved in the feature extraction of the data. Secondly, the three-dimensional data network is trained. At this time, the two-dimensional feature encoder provides scene image features, and the feature fusion module fuses the scene image features with the scene point cloud. The fused data is then sent to the three-dimensional point cloud segmentation network, and the network parameters are continuously optimized and updated to complete the training of the three-dimensional network. After the training is completed, the models and parameters of the two-dimensional network and the three-dimensional network are saved in the hard disk in the form of .pth files.

[0100] The scene point cloud and image test sets are also preprocessed to obtain image data and point cloud packaged data, which are input into the trained network model for testing experiments to obtain complete prediction results.

[0101] The above is a detailed introduction to a three-dimensional point cloud scene segmentation method and system that integrates image features provided in the embodiments of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application; at the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0102] It should also be noted that the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such product or system. In the absence of further restrictions, the elements defined by the phrase "includes a ..." do not exclude the existence of other identical elements in the product or system including the elements. "Substantially" means that within an acceptable error range, a person skilled in the art can solve the technical problem within a certain error range and basically achieve the technical effect.

[0103] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms of "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. The term "and / or" used in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

Claims

1. A three-dimensional point cloud scene segmentation method integrating image features, characterized in that: The steps of the method include: S1, acquiring two-dimensional data including an image, point cloud data, and depth data, and calculating the correlation between the scene image and the point cloud based on the acquired data; S2, extracting features from the two-dimensional data to obtain a high-dimensional feature map to be fused; S3, fusing the feature map to be fused and the point cloud data according to a fusion strategy to obtain fused point cloud data; the fusion strategy includes: searching for pixels adjacent to a certain point cloud data through an association relationship, and distorting the corresponding features of the pixels onto the point cloud data; S4. Input the fused point cloud data into the 3D segmentation network for feature extraction to obtain the required global and local semantic information.

2. The three-dimensional point cloud scene segmentation method integrating image features according to claim 1, characterized in that: Before fusion, key point sampling processing is performed on the point cloud data to obtain the point cloud data to be fused, and then the point cloud data to be fused is fused with the feature map to be fused.

3. The three-dimensional point cloud scene segmentation method integrating image features according to claim 1, characterized in that: The content of extracting features from the two-dimensional data in step S2 includes: extracting features through a two-dimensional segmentation network; A U-Net-ResNet network whose encoder is ResNet is used for image feature extraction, and the U-Net-ResNet network is a network that has been trained with a model and has a sufficient accuracy rate.

4. The three-dimensional point cloud scene segmentation method integrating image features according to claim 1, characterized in that: The calculation method of the association relationship between the scene image and the point cloud in step S1 is: select a certain point cloud data, project the pixel into the three-dimensional space through the depth data of the image, calculate the spatial distance between the pixel and the selected point cloud, and when the spatial distance is less than the judgment threshold, the pixel is considered to be a valid pixel; count the total number of valid pixels of all scene images in a certain scene and sort them from large to small, and select the first N scene images as the original images of the two-dimensional data in step S2; N is a positive integer.

5. The three-dimensional point cloud scene segmentation method integrating image features according to claim 4 is characterized in that: The point cloud data selected when counting the total number of valid pixels is all point cloud data corresponding to the scene or point cloud data of key points in the scene.

6. The three-dimensional point cloud scene segmentation method integrating image features according to claim 1, characterized in that: The content of feature extraction in step S4 includes: capturing the feature relationship between the point and the neighborhood in the three-dimensional space through the RFE module and the CSA module, and performing spatial feature extraction and feature semantic correction on the fused point cloud data.

7. The three-dimensional point cloud scene segmentation method integrating image features according to claim 6, characterized in that: The RFE module includes three RE submodules, which are used to encode the location information of the key point cloud and the neighboring points, so as to capture the spatial distribution information of the key points and the neighboring points; The CSA module processes the neighborhood spatial features through SharedMLP, alternately generates the attention matrix and the next layer of local features and multiplies them, so as to achieve the effects of spatial feature extraction and semantic information enhancement.

8. The three-dimensional point cloud scene segmentation method integrating image features according to claim 7, characterized in that: The RFE module adopts a shared function g including one or more SharedMLPs to learn the features of each encoded information, and then performs nonlinear learning through an activation function.

9. The three-dimensional point cloud scene segmentation method integrating image features according to claim 7, characterized in that: The CSA module generates attention matrix and local features through a g function consisting of a set of SharedMLP and Softmax; Among them, the attention matrix is ​​defined as: A i represents the attention matrix of the current i layer, W is the parameter matrix learned by SharedMLP, N is the position encoding information obtained by the previous RFE module, and r i It is the feature extracted under the current i layer.

10. A three-dimensional point cloud scene segmentation system integrating image features, characterized in that: The system is used to implement the method according to any one of claims 1 to 9; the system comprises: A data processing module is used to obtain two-dimensional data, point cloud data and depth data, and calculate the correlation between the scene image and the point cloud; The two-dimensional image feature encoding module is used to extract features from two-dimensional data to obtain a high-dimensional feature map to be fused; The fusion module is used to fuse the semantic features of pixels into point cloud data according to the positional relationship between image pixels and point cloud, effectively fusing data and enhancing semantic information; The point cloud feature extraction module is used to extract features from the fused point cloud data and perform semantic segmentation on the fused point cloud data at multiple scales.

Images