3D point cloud instance segmentation method in indoor scene

Abstract

The invention discloses a 3D point cloud instance segmentation method in an indoor scene, and the method comprises the steps: carrying out the codec operation of multiple original point cloud data through a sparse 3D U-Net network for 3D point cloud data, obtaining initial point cloud feature information, and extracting rich multi-scale point cloud feature information through employing a multilayer feature extraction structure comprising MLP and FPS algorithms; and then data crossing and enhancement are carried out on the extracted multi-scale feature information by using a crossing perception module based on an attention mechanism, effective features are highlighted, invalid features are inhibited, and feature information of each scale of the point cloud is efficiently fused. The number of parameters of the network model is small, and the cross sensing module in the network model has portability and can be added to other instance segmentation networks for application.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a 3D point cloud instance segmentation method in an indoor scene. Background Art

[0002] With the support of various advanced technologies related to artificial intelligence, the research on two-dimensional images has reached a very good level, and has performed well in various tasks such as image classification, image segmentation and target detection. However, due to its own limitations, the current research on two-dimensional images has difficulty breaking through the bottleneck in some fields that require higher accuracy and are more complex. Therefore, more and more researchers have begun to turn their attention to three-dimensional data. Compared with images that only contain two-dimensional data, three-dimensional data contains more dimensions, which means that it contains more information. In artificial intelligence, more information means more possibilities and a higher upper limit. For example, for problems that are difficult to solve in a flat space, such as object occlusion, researchers can solve such difficult problems by switching the observation angle in the three-dimensional space constructed by three-dimensional data. As one of the most important three-dimensional data, point cloud data is composed of a large number of point data like clouds in a specific space. The attributes of each point are generally composed of Cartesian coordinates (X, Y, Z) and color information (R, G, B). Some point cloud data may contain other information such as reflection surface intensity information. Point cloud data is generally obtained by a three-dimensional scanner. Portable devices equipped with LiDAR can obtain point cloud data in a low-cost manner, making it possible for individuals to use point cloud data to obtain related services at a low cost. Through the point cloud instance segmentation task of indoor scenes, it is currently possible to identify objects in the scene only by relying on point cloud data and related algorithms, which makes it possible to realize related applications, such as instructing visually impaired people to obtain designated items through voice interaction in indoor scenes. In related applications, the accuracy of the point cloud segmentation task of indoor scenes plays an extremely important role in the quality of related services. However, the accuracy achieved by the current instance segmentation task of indoor scenes still has a lot of room for improvement. Therefore, studying the point cloud instance segmentation algorithm of indoor scenes and improving its accuracy are crucial to improving the effectiveness of related applications. The research on point cloud segmentation tasks is practical, feasible and necessary. Summary of the invention

[0003] The purpose of the present invention is to solve the above problems and to design a 3D point cloud instance segmentation method in indoor scenes.

[0004] To achieve the above object, the technical solution of the present invention is that, further, in the above-mentioned 3D point cloud instance segmentation method in an indoor scene, the 3D point cloud instance segmentation method comprises the following steps:

[0005] A sparse 3D U-Net structure is used to extract the representation of the input point cloud to obtain point cloud information. The sparse 3D U-Net structure includes at least one encoder-decoder.

[0006] The multi-scale features of the point cloud information are extracted and fused using a multi-layer semantic fusion module MSSA, so as to extract semantic information between different scales and obtain multi-scale information;

[0007] Using a cross-sensing module to cross-fuse the multi-scale information to obtain prediction information;

[0008] The prediction information is compared with the true mask through the prediction head to obtain a target prediction mask.

[0009] Furthermore, in the above-mentioned 3D point cloud instance segmentation method in an indoor scene, the multi-scale features of the point cloud information are extracted and fused using a multi-layer semantic fusion module MSSA to extract semantic information between different scales to obtain multi-scale information, including:

[0010] In the multi-layer semantic fusion module MSSA, the multi-layer farthest point sampling algorithm FPS is used to process the point cloud feature information to obtain multi-scale point cloud feature information, and then the K-nearest neighbor algorithm KNN is used to connect the point clouds into a graph before the next step of processing.

[0011] Furthermore, in the above-mentioned 3D point cloud instance segmentation method in an indoor scene, a sparse 3D U-Net structure is used to extract the representation of the point cloud from the input point cloud to obtain point cloud information. The sparse 3D U-Net structure includes at least one encoder-decoder, including:

[0012] The decoder architecture includes at least three components: a feature cross-perception module, a prediction head, and a final matching and ranking process; the decoder includes two inputs: key point features extracted by multi-scale semantic aggregation.

[0013] Furthermore, in the above-mentioned 3D point cloud instance segmentation method in an indoor scene, the key point features extracted by multi-scale semantic aggregation include:

[0014] The initial processing of the key point features is performed simultaneously in the feature cross-sensing module and learned through the query vector;

[0015] Assuming there are K learnable query vectors, the features associated with the query vector originating from each Transformer decoder layer are predefined as Z l ∈R M×D , where D represents the embedding dimension and l represents the layer index;

[0016] The feature information is enriched with the information contained in the query vector through various attention mechanisms and feed-forward neural networks to obtain a new query vector.

[0017] The query vector generated by the two parallel channels is output through a fully connected layer.

[0018] Furthermore, in the above-mentioned 3D point cloud instance segmentation method in an indoor scene, the step of comparing the prediction information with the real mask through the prediction head to obtain the target prediction mask includes:

[0019] The prediction information is directed to the prediction head and compared with the true mask S mask ∈R M×D , comparison is performed and then the final prediction mask is generated;

[0020] The Hungarian algorithm is used to implement the bipartite matching process, and the top-k sorting procedure is used to sort and obtain the target prediction mask.

[0021] Its beneficial effect lies in that a sparse 3D U-Net structure is used to extract the representation of the point cloud from the input point cloud to obtain point cloud information, and the sparse 3D U-Net structure includes at least one encoder-decoder; the multi-scale features of the point cloud information are extracted and fused using a multi-layer semantic fusion module MSSA to extract semantic information between different scales to obtain multi-scale information; the multi-scale information is cross-fused using a cross-perception module to obtain prediction information; the prediction information is compared with the real mask through the prediction head to obtain the target prediction mask. 1. The model of the network structure used in the present invention is based on the attention algorithm improved by the Transformer algorithm. The Transformer network structure is improved on the basic attention mechanism into a multi-scale feature information input, and then the cross-attention mechanism is used to enhance the fusion of features, so that point cloud feature information of different scales can be cross-fused, enriching the point cloud feature information and enhancing the robustness and prediction accuracy of the network model. 2. In addition to using the cross-attention mechanism, the network structure used in the present invention also uses a multi-layer self-attention mechanism, residual connections, and layer normalization to construct an efficient feature perception module, which strengthens the network's targeting of global and local features, highlights useful information, and suppresses useless features. 3. The network structure used in the present invention is superior to the standard method in terms of accuracy results. 4. The instance segmentation method used in the present invention is portable and can be used as a segmentation subnetwork to replace the corresponding decoder structure of any network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0023] Figure 1 It is a schematic diagram of a first embodiment of a 3D point cloud instance segmentation method in an indoor scene according to an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of point cloud data used in a training set in a 3D point cloud instance segmentation method in an indoor scene in an embodiment of the present invention;

[0025] Figure 3 A decoder model structure diagram of a 3D point cloud instance segmentation method in an indoor scene in an embodiment of the present invention;

[0026] Figure 4 This is an instance segmentation visualization result of a 3D point cloud instance segmentation method in an indoor scene according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] The present invention will be described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, a 3D point cloud instance segmentation method in an indoor scene, the 3D point cloud instance segmentation method comprises the following steps:

[0030] Step 101: extract the representation of the point cloud using a sparse 3D U-Net structure to obtain point cloud information. The sparse 3D U-Net structure includes at least one encoder-decoder.

[0031] Step 102, extract and fuse multi-scale features of the point cloud information using a multi-layer semantic fusion module MSSA, extract semantic information between different scales, and obtain multi-scale information;

[0032] Step 103: Use a cross-sensing module to cross-fuse multi-scale information to obtain prediction information;

[0033] Step 104: Compare the prediction information with the real mask through the prediction head to obtain a target prediction mask.

[0034] Specifically, in this embodiment, a multi-scale cross-sensing model is constructed, including:

[0035] Step S1: Use a sparse 3D U-Net structure to extract the representation of the point cloud from the input point cloud, where the sparse 3D U-Ne is an encoder-decoder structure.

[0036] Step S2: For the point cloud information processed by the 3D U-Net module, the multi-scale features are extracted and fused using the multi-layer semantic fusion module MSSA to fully extract the semantic information between different scales;

[0037] Step S3: Since there is no sufficient interaction between the extracted multi-scale information, the extracted multi-scale information will not be fully utilized, so the present invention uses a cross-sensing module to efficiently cross-fuse the multi-scale information and obtain prediction information;

[0038] Step S4: The result processed by the feature cross-perception module is compared with the true mask through the prediction head, and the final prediction mask is output.

[0039] It should be added that in the MSSA module of S2, the present invention uses a multi-layer farthest point sampling algorithm (FPS) to process the point cloud feature information to obtain multi-scale point cloud feature information, and then uses the K nearest neighbor algorithm (KNN) to connect the point cloud into a graph before proceeding to the next step of processing.

[0040] The decoder architecture of the present invention includes three main components: feature cross-sensing module, prediction head and final matching and ranking process. The decoder has two main inputs: key point features extracted by multi-scale semantic aggregation.

[0041] The initial processing of these features is done simultaneously in the feature cross-perception module, which is learned through the query vector. Assuming there are K learnable query vectors, the features associated with the query vector originating from each Transformer decoder layer are predefined as Z l ∈R M×D , where D represents the embedding dimension and l represents the layer index. The feature information is enriched with the information contained in the query vector through various attention mechanisms and feedforward neural networks, thereby generating a set of new query vectors. Finally, the query vector generated by the two parallel channels is output through a fully connected layer.

[0042] After being processed in the feature cross-perception module, the output is directed to the prediction head and compared with the true mask s mask ∈R M×D, and then generate the final predicted mask. In the training phase, we use the Hungarian algorithm to implement the binary matching process. In the inference process, we adopt the top-k ranking procedure. In this way, our feature decoder cleverly combines semantic features of different scales and effectively performs accurate segmentation of 3D point cloud instances.

[0043] Its beneficial effect lies in that the point cloud information is obtained by extracting the representation of the point cloud using a sparse 3D U-Net structure from the input point cloud, and the sparse 3D U-Net structure includes at least one encoder-decoder; the multi-scale features of the point cloud information are extracted and fused using a multi-layer semantic fusion module MSSA to extract semantic information between different scales to obtain multi-scale information; the multi-scale information is cross-fused using a cross-perception module to obtain prediction information; the prediction information is compared with the real mask through the prediction head to obtain the target prediction mask. 2. The model of the network structure used in the present invention is based on the attention algorithm improved by the Transformer algorithm. The Transformer network structure is improved on the basic attention mechanism into a multi-scale feature information input, and then the cross-attention mechanism is used to enhance the fusion of features, so that point cloud feature information of different scales can be cross-fused, enriching the point cloud feature information and enhancing the robustness and prediction accuracy of the network model. 2. In addition to using the cross-attention mechanism, the network structure used in the present invention also uses a multi-layer self-attention mechanism, residual connections, and layer normalization to construct an efficient feature perception module, which strengthens the network's targeting of global and local features, highlights useful information, and suppresses useless features. 3. The network structure used in the present invention is superior to the standard method in terms of accuracy results. 4. The instance segmentation method used in the present invention is portable and can be used as a segmentation subnetwork to replace the corresponding decoder structure of any network.

[0044] like Figure 3 As shown, the decoder architecture of the present invention includes three main components: feature cross-sensing module, prediction head and final matching and ranking process. The decoder has two main inputs: key point features extracted by multi-scale semantic aggregation.

[0045] The initial processing of these features is done simultaneously in the feature cross-perception module, which is learned through the query vector. Assuming there are K learnable query vectors, the features associated with the query vector originating from each Transformer decoder layer are predefined as Z l ∈R M×D , where D represents the embedding dimension and l represents the layer index. The feature information is enriched with the information contained in the query vector through various attention mechanisms and feedforward neural networks, thereby generating a set of new query vectors. Finally, the query vector generated by the two parallel channels is output through a fully connected layer.

[0046] After being processed in the feature cross-perception module, the output is directed to the prediction head and compared with the true mask s mask ∈R M×D , and then generate the final predicted mask. In the training phase, we use the Hungarian algorithm to implement the binary matching process. In the inference process, we adopt the top-k ranking procedure. In this way, our feature decoder cleverly combines semantic features of different scales and effectively performs accurate segmentation of 3D point cloud instances.

[0047] like Figure 4 As shown, it is a schematic comparison diagram of the segmentation results of the present invention. The leftmost part is the original point cloud data, the upper middle part is the segmentation result diagram of the standard method, and the lower part is the segmentation result diagram of the method of the present invention. It can be observed that the segmentation results of the present invention have obvious differences in local and global information, indicating that the multi-scale method is indeed effective for point cloud instance segmentation.

[0048] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A 3D point cloud instance segmentation method in an indoor scene, characterized in that: The 3D point cloud instance segmentation method comprises the following steps: A sparse 3D U-Net structure is used to extract the representation of the input point cloud to obtain point cloud information. The sparse 3D U-Net structure includes at least one encoder-decoder. The multi-scale features of the point cloud information are extracted and fused using a multi-layer semantic fusion module MSSA, so as to extract semantic information between different scales and obtain multi-scale information; Using a cross-sensing module to cross-fuse the multi-scale information to obtain prediction information; The prediction information is compared with the true mask through the prediction head to obtain a target prediction mask.

2. The 3D point cloud instance segmentation method in an indoor scene according to claim 1, characterized in that: The multi-scale features of the point cloud information are extracted and fused using a multi-layer semantic fusion module MSSA, the semantic information between different scales is extracted, and the multi-scale information is obtained, including: In the multi-layer semantic fusion module MSSA, the multi-layer farthest point sampling algorithm FPS is used to process the point cloud feature information to obtain multi-scale point cloud feature information, and then the K-nearest neighbor algorithm KNN is used to connect the point clouds into a graph before the next step of processing.

3. The 3D point cloud instance segmentation method in an indoor scene according to claim 1, characterized in that: The input point cloud is extracted using a sparse 3D U-Net structure to represent the point cloud and obtain point cloud information. The sparse 3D U-Net structure includes at least one encoder-decoder, including: The decoder architecture includes at least three components: a feature cross-perception module, a prediction head, and a final matching and ranking process; the decoder includes two inputs: key point features extracted by multi-scale semantic aggregation.

4. The 3D point cloud instance segmentation method in an indoor scene according to claim 3, characterized in that: The key point features extracted by multi-scale semantic aggregation include: The initial processing of the key point features is performed simultaneously in the feature cross-sensing module and learned through the query vector; Assuming there are K learnable query vectors, the features associated with the query vector originating from each Transformer decoder layer are predefined as Z l ∈R M×D , where D represents the embedding dimension and l represents the layer index; The feature information is enriched with the information contained in the query vector through various attention mechanisms and feed-forward neural networks to obtain a new query vector. The query vector generated by the two parallel channels is output through a fully connected layer.

5. The 3D point cloud instance segmentation method in an indoor scene according to claim 1, characterized in that: The step of comparing the prediction information with the real mask through the prediction head to obtain the target prediction mask includes: The prediction information is directed to the prediction head and compared with the true mask S mask ∈R M×D , comparison is performed and then the final prediction mask is generated; The Hungarian algorithm is used to implement the bipartite matching process, and the top-k sorting procedure is used to sort and obtain the target prediction mask.

Images