A point cloud segmentation method based on geometric perception attention mechanism

Abstract

The application discloses a point cloud segmentation method based on a geometry perception attention mechanism, and comprises the following steps: a point cloud segmentation network based on a geometry perception point cloud attention mechanism is established, the network comprises a geometry perception initialization module, a multi-scale geometry perception attention module, a recovery module and a classification module; in a training stage, point cloud data is input into the geometry perception initialization module to obtain encoded point cloud and features for explicitly representing geometry, then the multi-scale geometry perception attention module is used to extract global hierarchical features, a deep multi-scale geometry perception attention neural network is pre-trained by using a training set and the hierarchical features, and local and global attention information is learned from the hierarchical features by using low-level and high-level features of the network multi-scale; the point cloud data is input into the trained network to obtain point cloud semantic segmentation results of each object in a scene. The method can obtain high-quality point cloud segmentation results.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and three-dimensional perception, and in particular to a point cloud segmentation method. BACKGROUND

[0002] In the field of three-dimensional perception, point cloud, as one of the most widely used depth information expression methods, plays a huge role in automatic driving, robot control, three-dimensional reconstruction, etc. However, point cloud is limited by its own measurement method, and the data is not intuitive, often needs to be segmented to further realize data visualization, and the results of point cloud segmentation guide other tasks, such as guiding the automatic driving field to avoid obstacles during driving, guiding the robot to grasp the position and path of the target, guiding the three-dimensional reconstruction process to focus on the target area, etc. Therefore, point cloud segmentation has always been a research hotspot in the field of computer vision and three-dimensional perception.

[0003] In recent years, attention mechanism has achieved good results in the field of two-dimensional image processing, but in the field of three-dimensional perception, especially in the task of point cloud segmentation, there is still a lot of room for development. For example, the PointTransformer method based on attention mechanism has been proposed, which proves the strong potential of attention mechanism in the task of point cloud segmentation.

[0004] However, the current method mainly has three problems. First, although the PointTransformer method based on attention mechanism makes good use of global features of data, it does not fully utilize local features, which also leads to poor segmentation results in areas such as corners of walls and intersections of objects, and easy confusion problems in the process of point cloud segmentation. Second, due to the process of collecting point cloud, it is impossible to collect data through objects, so there will be a problem of point cloud data holes caused by occlusion, mirror, etc. during the collection process, which will have a great impact on the task of point cloud segmentation, because the holes will destroy the geometric structure between different objects, and thus affect the segmentation results. Finally, since the current method only calculates attention within the block when considering point cloud block attention, without calculating attention between blocks, it cannot capture the distinguishing details of different size regions in the point cloud data. At the same time, attention calculation between blocks has a larger receptive field, which allows better use of training input and more contextual information. This is very helpful for the attention model of segmenting point cloud, especially in the area of point cloud with holes.

[0005] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The main purpose of the present invention is to propose a point cloud segmentation method based on a geometrically perceived point cloud attention mechanism to solve the technical problem of low quality of point cloud segmentation results existing in the above-mentioned background technology.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A point cloud segmentation method based on a geometry-aware attention mechanism includes the following steps:

[0009] A point cloud segmentation network based on a geometry-aware point cloud attention mechanism is established. The network includes a geometry-aware initialization module, a multi-scale geometry-aware attention module, a restoration module, and a classification module. During the training phase, point cloud data is input into the geometry-aware initialization module to obtain an encoded point cloud and features that explicitly represent the geometry. The multi-scale geometry-aware attention module is then used to extract global hierarchical features. A deep multi-scale geometry-aware attention neural network is pre-trained using the training set and features at each level. The network's multi-scale low-level and high-level features are used to learn local and global attention information from the hierarchical features.

[0010] The point cloud data from the acquisition device is input into the trained network to obtain the point cloud semantic segmentation results of each object in the scene.

[0011] Further:

[0012] During the training phase, before inputting the geometric perception initialization module, the point cloud data is further enhanced, specifically including: performing data enhancement on the point cloud data by flipping, cropping, rotating, and translating.

[0013] The geometric perception initialization module has a geometric perception branch and an encoding branch. The encoding branch uses three-dimensional point cloud coordinates to encode point cloud data. The geometric perception branch represents the point cloud as a grid through gridding, and connects the encoded point cloud and explicitly represents the geometric features through splicing.

[0014] In the geometric perception branch, the input point cloud data is first converted into a grid representation that is more sensitive to geometric features; the gridded data is then input as features into the feature MLP layer and the geometric features are extracted through the BN layer, and the information of the original point cloud corresponding to the grid is passed through the position MLP layer, and the position features are extracted through the BN layer; the geometric features are added to the position features, and then the features of the geometric perception branch are obtained through the ReLU function; in the encoding branch, the input point cloud data is first input into the kpconv layer, and then into the BN layer, and activated through the LeakyReLU function. The features obtained are the features of the encoding branch, and finally the features obtained from the two branches are spliced ​​as the output of the geometric perception initialization module.

[0015] The multi-scale geometry-aware attention module first extracts global features at different scales, then performs point cloud downsampling, and extracts global features again. After multiple cycles, multi-scale attention features are obtained, and global skip connections are added to retain the original information as much as possible.

[0016] The multi-scale geometric perception attention module uses multiple stacked geometric perception attention modules and multiple point cloud downsampling modules; wherein, the entire input space is first divided into blocks of the same size, and the center of each block is obtained by sampling the data obtained by the farthest point. When the attention calculation is performed on the point cloud data in each block, not only all the data in the block are input, but also the neighborhood information obtained after the farthest point sampling of the adjacent blocks is also used as the input of the attention calculation, that is, the key points of the selected block and the adjacent blocks are used as the input of the geometric attention module each time; however, the point cloud is downsampled by the downsampling module to obtain multi-scale point cloud features; wherein, the downsampling module uses the farthest point sampling method.

[0017] The restoration module performs interpolation upsampling restoration and fuses global attention features at different scales during the restoration process, and finally performs point cloud segmentation through the classification module.

[0018] The classification module passes the multi-scale features obtained by the recovery module through the linear layer, BN layer, ReLU activation layer and linear layer in sequence to obtain logits for classification, and obtains the probability of each point in the point cloud belonging to each object category through the softmax activation function.

[0019] During the training phase, the cross entropy loss function is used to calculate the point cloud prediction probability and the cross entropy loss of the one-hot encoding of the point cloud's true category; during the testing phase, the category with the highest probability is directly selected as the predicted category to achieve point cloud classification.

[0020] A computer-readable storage medium stores a computer program, wherein the computer program implements the point cloud segmentation method when executed by a processor.

[0021] The present invention has the following beneficial effects:

[0022] The present invention proposes a point cloud segmentation method based on a geometry-aware point cloud attention mechanism, which can achieve high-quality point cloud segmentation results. The present invention proposes a method of using a geometry-aware point cloud attention mechanism in a point cloud segmentation method. The network model in the proposed point cloud segmentation method includes a geometry-aware initialization module, a multi-scale geometry-aware attention module, a recovery module, and a classification module. During the training phase, the point cloud data is input into the geometry-aware initialization module to obtain the encoded point cloud and features that explicitly represent the geometry. Then, the multi-scale geometry-aware attention module is used to extract global hierarchical features. The training set and features at each level are used to pre-train a deep multi-scale geometry-aware attention neural network. The network's multi-scale low-level and high-level features are used to learn local and global attention information from the hierarchical features. By utilizing the network's multi-scale attention features, better network representation capabilities are achieved, and ultimately high-quality point cloud segmentation results are obtained.

[0023] In the preferred embodiment, the geometric perception initialization module has two branches. The first branch encodes the point cloud data using three-dimensional point cloud coordinates, and the second branch represents the point cloud as a grid through gridding, making the data more sensitive to geometric changes, and splicing the encoded point cloud and explicitly representing the geometric features. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a simplified flowchart of a point cloud segmentation method based on a geometrically-aware attention mechanism according to an embodiment of the present invention.

[0025] Figure 2 Schematic diagram of the network structure based on the geometric-aware attention mechanism according to an embodiment of the present invention.

[0026] Figure 3 It is a structural diagram of a geometric perception initialization module in a network based on a geometric perception attention mechanism according to an embodiment of the present invention.

[0027] Figure 4 It is a structural diagram of a multi-scale geometry-aware attention module in a network based on a geometry-aware attention mechanism according to an embodiment of the present invention.

[0028] Figure 5 It is a schematic diagram of the structure of the geometric perception attention module in the multi-scale geometric perception attention module of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0030] See Figure 1, an embodiment of the present invention provides a point cloud segmentation method based on a geometry-aware attention mechanism, comprising the following steps:

[0031] A point cloud segmentation network (typically, an indoor point cloud segmentation network) based on a geometry-aware point cloud attention mechanism is established. The network includes a geometry-aware initialization module, a multi-scale geometry-aware attention module, a restoration module, and a classification module. During the training phase, point cloud data is input into the geometry-aware initialization module to obtain an encoded point cloud and features that explicitly represent the geometry. The multi-scale geometry-aware attention module is then used to extract global hierarchical features. A deep multi-scale geometry-aware attention neural network is pre-trained using the training set and features at each level. The network's multi-scale low-level and high-level features are used to learn local and global attention information from the hierarchical features.

[0032] The point cloud data from the acquisition device is input into the trained network to obtain the point cloud semantic segmentation results of each object in the scene.

[0033] Specific embodiments of the present invention are further described below.

[0034] This paper proposes a point cloud segmentation method based on geometric perception attention mechanism. The process diagram is as follows Figure 1 shown.

[0035] Step 1: Enhance the point cloud data by performing operations such as flipping, rotating, translating, and shearing on the point cloud to increase the size of the dataset.

[0036] Step 2: The processed point cloud data is input into the geometry perception initialization module of the point cloud segmentation network to obtain the encoded point cloud and the features that explicitly represent the geometry. Then, the multi-scale geometry perception attention module is used to extract the global hierarchical features. The training set and the features of each layer are used to pre-train a deep multi-scale geometry perception attention neural network. The network's multi-scale low-level and high-level features are used to learn local and global attention information from the hierarchical features.

[0037] Step 3: Input the point cloud data from the acquisition device into the trained network model to obtain the point cloud semantic segmentation results of each object in the scene.

[0038] like Figure 2 As shown, the point cloud segmentation network in this embodiment of the present invention mainly includes four modules: a geometry-aware initialization module, a multi-scale geometry-aware attention module, a restoration module, and a classification module. The network extracts global features at different scales through the multi-scale geometry-aware attention module. The restoration module then performs interpolation and upsampling restoration. During the restoration process, the global attention features at different scales are fused, and finally, the classification module performs point cloud segmentation.

[0039] The various modules of the point cloud segmentation network are described in further detail below.

[0040] Geometric perception initialization module:

[0041] like Figure 3 As shown in Figure 2, the module consists of two branches: the geometry perception branch and the encoding branch. The encoding branch encodes the point cloud data using 3D point cloud coordinates, while the geometry perception branch represents the point cloud as a grid through meshing, making the data more sensitive to geometric changes. The encoded point cloud is then connected through splicing, and the geometric features are explicitly represented.

[0042] In the geometric perception branch, the input point cloud data is first converted into a gridded representation that is more sensitive to geometric features. The gridded data is then fed into the feature MLP layer as features and passed through the BN layer to extract geometric features. The original point cloud information corresponding to the grid is passed through the position MLP layer and the BN layer to extract positional features. The geometric features are added to the positional features and then processed through the ReLU function to obtain the features of the geometric perception branch. In the encoding branch, the input point cloud data is first fed into the kpconv layer, then into the BN layer, and finally activated through the LeakyReLU function. The resulting features are the features of the encoding branch. Finally, the features obtained from the two branches are concatenated as the output of the geometric perception initialization module.

[0043] Multi-scale geometry-aware attention module:

[0044] The multi-scale geometric perception attention module first extracts global features at different scales, then performs point cloud downsampling, and extracts global features again. After four cycles in this embodiment, multi-scale attention features are obtained, and global skip connections are added to retain the original information as much as possible.

[0045] like Figure 4 As shown, the multi-scale geometric perception attention module uses multiple stacked geometric perception attention modules and multiple point cloud downsampling modules. During the calculation process, the geometric perception attention module first divides the entire input space into blocks of the same size. The center of each block is obtained by sampling the data obtained by the farthest point. When performing attention calculation on the point cloud data in each block, not only all the data in the block are input, but also the neighborhood information obtained after sampling the farthest point of the adjacent blocks is also used as the input of the attention calculation. That is, each time the key points of the selected block and the adjacent blocks are used as the input of the geometric attention module. The specific network structure of the geometric perception attention module is as follows: Figure 5As shown in the figure. By calculating geometrically aware attention on the input data, the features are made more sensitive to the geometric surface of objects, thereby reducing the impact of data holes. This improves the accuracy of point cloud segmentation. Compared with traditional attention modules, the geometric attention module is more sensitive to the surface geometric features of point cloud data, which can reduce segmentation errors caused by occlusion and holes. After this operation, the point cloud is downsampled using the point cloud downsampling module to obtain multi-scale point cloud features. Point cloud downsampling uses the farthest point sampling method.

[0046] Recovery module:

[0047] Interpolation upsampling restoration is performed in the restoration module, and the global attention features at different scales are fused during the restoration process.

[0048] Classification module:

[0049] The multi-scale features obtained by the recovery module are sequentially passed through a linear layer, a batch normalization layer, a ReLU activation layer, and a linear layer to generate logits for classification. A softmax activation function is then used to determine the probability that each point in the point cloud belongs to each object category. During the training phase, a cross-entropy loss function is used to calculate the cross-entropy loss between the predicted probability of the point cloud and the one-hot encoding of the true category of the point cloud. During the testing phase, the category with the highest probability is directly selected as the predicted category to achieve point cloud classification.

[0050] Point cloud segmentation is achieved using a point cloud segmentation network based on a geometry-aware point cloud attention mechanism according to an embodiment of the present invention. Point cloud data from an acquisition device is input into a trained network model, which sequentially passes through the four modules described above to obtain point cloud semantic segmentation results for each object in the scene. Leveraging the network's multi-scale attention features, this approach improves network representation capabilities and ultimately achieves high-quality point cloud segmentation results. Comparative test results are shown in Table 1.

[0051] Table 1 Performance comparison of this network model and other models

[0052] mIoU mAcc oAcc This network model 72.1 77.9 91.2 PointTransformer 70.4 76.5 90.8 BAAF-Net 65.4 73.1 88.9

[0053] In addition to indoor point cloud segmentation networks, the present invention is also applicable to outdoor data, and the experimental results on indoor data are better.

[0054] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0055] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0056] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0058] The background section of the present invention may contain background information about the problem or environment of the present invention, but does not necessarily describe the prior art. Therefore, the inclusion of content in the background section is not an admission by the applicant that the prior art is present.

[0059] The above description further details the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be construed as being limited to these descriptions. Persons skilled in the art will appreciate that, without departing from the spirit of the present invention, they may make various substitutions or modifications to the described embodiments, and these substitutions or modifications should be considered to fall within the scope of protection of the present invention. Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "preferred embodiments," "examples," "specific examples," or "some examples" indicates that the specific features, structures, materials, or characteristics described in conjunction with such embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Persons skilled in the art may combine and assemble the different embodiments or examples described in this specification, as well as features of different embodiments or examples, without conflicting opinions. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications may be made herein without departing from the scope of protection of the patent application.

Claims

1. A method for indoor point cloud segmentation based on geometric perception attention mechanism, characterized in that: The steps include: A point cloud segmentation network based on a geometry-aware point cloud attention mechanism is established. The network includes a geometry-aware initialization module, a multi-scale geometry-aware attention module, a restoration module, and a classification module. During the training phase, point cloud data is input into the geometry-aware initialization module to obtain an encoded point cloud and features that explicitly represent the geometry. The multi-scale geometry-aware attention module is then used to extract global hierarchical features. A deep multi-scale geometry-aware attention neural network is pre-trained using the training set and features at each level. The network's multi-scale low-level and high-level features are used to learn local and global attention information from the hierarchical features. The geometric perception initialization module has a geometric perception branch and an encoding branch. The encoding branch encodes the point cloud data using three-dimensional point cloud coordinates. The geometric perception branch represents the point cloud as a grid by gridding, and connects the encoded point cloud and explicitly represents the geometric features by splicing. In the geometric perception branch, the input point cloud data is first converted into a grid representation that is more sensitive to geometric features. The gridded data is then input as features into the feature MLP layer and the geometric features are extracted through the BN layer. The information of the original point cloud corresponding to the grid is passed through the position MLP layer, and the position features are extracted through the BN layer. The geometric features are added to the position features, and then the features of the geometric perception branch are obtained by the ReLU function. In the encoding branch, the input point cloud data is first input to the kpconv layer, then to the BN layer, and activated by the LeakyReLU function. The obtained features are the features of the encoding branch. Finally, the features obtained from the two branches are spliced ​​as the output of the geometric perception initialization module. The point cloud data from the acquisition device is input into the trained network to obtain the point cloud semantic segmentation results of each object in the scene.

2. The point cloud segmentation method based on the geometric perception attention mechanism according to claim 1, characterized in that: During the training phase, before inputting the geometric perception initialization module, the point cloud data is further enhanced, specifically including: performing data enhancement on the point cloud data by flipping, cropping, rotating, and translating.

3. The point cloud segmentation method according to any one of claims 1 to 2, characterized in that: The multi-scale geometry-aware attention module first extracts global features at different scales, then performs point cloud downsampling, and extracts global features again. After multiple cycles, multi-scale attention features are obtained, and global skip connections are added to retain the original information as much as possible.

4. The point cloud segmentation method according to claim 3, wherein: The multi-scale geometric perception attention module uses multiple stacked geometric perception attention modules and multiple point cloud downsampling modules; wherein, the entire input space is first divided into blocks of the same size, and the center of each block is obtained by sampling the data obtained by the farthest point. When the attention calculation is performed on the point cloud data in each block, not only all the data in the block are input, but also the neighborhood information obtained after the farthest point sampling of the adjacent blocks is also used as the input of the attention calculation, that is, the key points of the selected block and the adjacent blocks are used as the input of the geometric attention module each time; however, the point cloud is downsampled by the downsampling module to obtain multi-scale point cloud features; wherein, the downsampling module uses the farthest point sampling method.

5. The point cloud segmentation method according to any one of claims 1 to 2, characterized in that: The restoration module performs interpolation upsampling restoration and fuses global attention features at different scales during the restoration process, and finally performs point cloud segmentation through the classification module.

6. The point cloud segmentation method according to claim 5, wherein: The classification module passes the multi-scale features obtained by the recovery module through a linear layer, a BN layer, a ReLU activation layer, and a linear layer in sequence to obtain logits for classification, and uses a softmax activation function to obtain the probability that each point in the point cloud belongs to each object category.

7. The point cloud segmentation method according to any one of claims 1 to 2, characterized in that: During the training phase, the cross entropy loss function is used to calculate the point cloud prediction probability and the cross entropy loss of the one-hot encoding of the point cloud's true category; during the testing phase, the category with the highest probability is directly selected as the predicted category to achieve point cloud classification.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the point cloud segmentation method according to any one of claims 1 to 7 is implemented.

Images