Point cloud semantic segmentation method and system based on geometric feature fusion

Abstract

This invention provides a point cloud semantic segmentation method and system based on geometric feature fusion, belonging to the field of machine vision image processing technology. This invention introduces a polar coordinate transformation module to alleviate the problem of significant performance degradation due to object rotation. To reduce segmentation time without sacrificing model performance, random sampling with low time overhead is used in the downsampling stage for keypoint acquisition. Due to the randomness of random sampling, some feature information is inevitably lost. Therefore, a geometric feature fusion module and an improved Point Transformer module are proposed in the feature extraction and fusion stages to compensate for feature loss while extracting and fusing features. Through the polar coordinate transformation module and the geometric feature fusion module, a balance is achieved between segmentation performance and segmentation time. This invention balances segmentation performance and time overhead, solves the problems of information loss and rotation invariance, and achieves effective large-scale point cloud semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of machine vision image processing technology, specifically to a point cloud semantic segmentation method and system based on geometric feature fusion. Background Technology

[0002] Point cloud semantic segmentation aims to associate a series of regularly or irregularly distributed points in three-dimensional space with semantic labels. This is of great significance for applications such as autonomous driving, augmented reality, and robotics. For example, during road driving, it can classify obstacles, vehicles, and pedestrians, aiding the autonomous driving system in decision-making and planning, thus greatly reducing the driver's workload. Typically, point cloud semantic segmentation first requires acquiring key points and their neighboring points, and then obtaining the results through feature extraction and aggregation. Because point clouds are unstructured and disordered, recent methods for point cloud semantic segmentation can be categorized into three types: multi-view based, voxel-based, and point-based segmentation methods. Multi-view based methods effectively address the unstructured nature of point cloud data, but suffer from limitations and losses in geometric structure. Voxel-based methods address the unstructured nature of the original point cloud and compensate for the geometric limitations of multi-view based methods, but incur some loss in data representation, and the storage of voxel structures requires significant memory. Both multi-view and voxel-based methods suffer from high information loss and computational costs when processing large-scale point clouds, requiring additional processes to convert the point cloud into a regular representation and then converting the intermediate segmentation results back into a point cloud, which presents serious limitations. Unlike these methods, point-based methods can directly process 3D point clouds. To date, many point-based methods have emerged, and this approach has become the mainstream method for point cloud semantic segmentation. This paper also adopts a point-based segmentation method. Although current work has made good progress, it still has the following two shortcomings: First, when the object being segmented rotates, the segmentation model inevitably suffers from a decrease in segmentation performance (e.g., when a pedestrian is facing the scanner head-on versus from the side, the segmentation results for the pedestrian facing the scanner head-on are better). However, most models experience a significant performance degradation when dealing with rotating objects, resulting in poor rotational robustness. Second, in practical applications, most methods sacrifice runtime efficiency for better segmentation results, leading to lower real-time performance.

[0003] Therefore, in order to narrow the gap between point cloud semantic segmentation tasks and real-world applications and improve the real-time performance of methods, more and more models are trying to use the method of rotating random sampling in the key point acquisition stage, and adopting some methods in the feature extraction stage to make up for the feature loss problem caused by randomness, thereby improving the real-time performance of the model, such as RandLA-Net.

[0004] RandLA-Net uses a random sampling method to obtain key points in the downsampling stage, and then obtains the neighboring points around the key points through the K-nearest neighbor method. In the feature extraction stage, a local feature fusion method is used to supplement and extract features. Finally, the pooling layer is used to obtain the final segmentation result.

[0005] On the one hand, the performance degradation due to object rotation is unavoidable (e.g., misidentifying a pedestrian facing the scanner from the side as a tree), but most models experience a significant performance drop when the object rotates. On the other hand, real-world applications demand both high segmentation performance and real-time capabilities, but most models fail to achieve a good balance between the two. Summary of the Invention

[0006] The purpose of this invention is to provide a point cloud semantic segmentation method and system based on geometric feature fusion, so as to solve at least one of the technical problems existing in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] On the one hand, the present invention provides a point cloud semantic segmentation method based on geometric feature fusion, comprising:

[0009] The input point cloud data undergoes preliminary feature extraction through an MLP layer and a GFF layer. A PointTransformer is used as the backbone network, KNN is employed to obtain neighboring points, and four upsampling layers use linear interpolation to restore the data to its original scale. The Point Transformer consists of two MLP layers in the input and output stages, nine PointTransformer layers for feature extraction and fusion, and four downsampling layers using FPS to obtain key points.

[0010] The extracted preliminary features are input into the downsampling layer. During the downsampling stage, keypoints are obtained using random sampling, and neighboring points are obtained using the KNN method. The downsampling results are input into the GFF module. The GFF takes the point cloud's 3D coordinates and corresponding semantic features as input. For each input point, LPC processing is performed on its neighboring points to obtain the relative angles of the neighboring points and the Euclidean distance between the neighboring points and the center point. The relative angles of the neighboring points, the geometric distance between the neighboring points and the center point, and the neighboring point feature information are concatenated to obtain... Then The data is fed into the PTL layer and finally passes through the Shared MLP layer, which performs the same MLP operation on the feature dimensions of each point to adjust the dimensions and obtain the local feature information representing the neighborhood of the center point.

[0011] The local feature information representing the neighborhood of the center point is input into the upsampling layer. The upsampling method is linear interpolation. After each upsampling, the dimensions are adjusted by passing through the MLP layer. The result is then input into the last MLP layer to obtain the final segmentation result.

[0012] Furthermore, after random sampling, a center point P is obtained, and the KNN algorithm is applied to the center point to obtain P. i Neighboring points First, p i The nearest points are transformed from the Cartesian coordinate system to the polar coordinate system, that is, using... Indicate the nearest neighbor, where These are the relative coordinates of neighboring points in the Cartesian coordinate system. The transformation formula is as follows:

[0013]

[0014]

[0015]

[0016] Furthermore, calculate p i Centroid of the neighborhood The calculation uses the method of averaging the neighboring points, defining the value from p... i point to For local directions, used to represent p i and The relative angle between them; update the relative angle of neighboring points: Will and Updated to and Where δ i and θ i yes The relative angle.

[0017] Furthermore, the following formula is used for local feature stitching:

[0018]

[0019] Indicates neighboring points Euclidean distance from key points This indicates a splicing operation. and These are the nearest neighbors output by our LPC module. and p i The relative distance and the converted relative angle.

[0020] Furthermore, through the LPC output and neighboring points The coordinate information is concatenated and then processed through a point-by-point MLP to obtain... Having acquired sufficient spatial characteristics, then... and Feature information f i k By splicing For the center point p i A set of neighboring features is obtained

[0021] Furthermore, the input to the PTL module in the Point Transformer network is:

[0022]

[0023] Here, ω and ξ are two linear layers, γ consists of one MLP and two linear layers, and ρ consists of one MLP and a Softmax layer; the aim is to calculate the correlation between the center point and the neighboring points, give the features of the neighboring points a weight score, and finally fuse the neighboring features through a summation function.

[0024] Secondly, the present invention provides a point cloud semantic segmentation system based on geometric feature fusion, comprising:

[0025] The feature extraction module performs preliminary feature extraction on the input point cloud data through an MLP layer and a GFF layer. It employs a Point Transformer as the backbone network, uses KNN to obtain neighboring points, and four upsampling layers with linear interpolation to restore the data to its original scale. The Point Transformer consists of two MLP layers in the input and output stages, nine Point Transformer layers for feature extraction and fusion, and four downsampling layers using FPS to obtain key points.

[0026] The downsampling module is used to input the extracted preliminary features into the downsampling layer. During the downsampling stage, keypoints are obtained using random sampling, and neighboring points are obtained using the KNN method. The downsampling results are then input into the GFF module. The GFF takes the point cloud's 3D coordinates and corresponding semantic features as input. For each input point, LPC processing is performed on its neighboring points to obtain the relative angles of the neighboring points and the Euclidean distance between the neighboring points and the center point. The relative angles of the neighboring points, the geometric distance between the neighboring points and the center point, and the neighboring point feature information are concatenated to obtain... Then The data is fed into the PTL layer and finally passes through the Shared MLP layer, which performs the same MLP operation on the feature dimensions of each point to adjust the dimensions and obtain the local feature information representing the neighborhood of the center point.

[0027] The upsampling module is used to input local feature information representing the neighborhood of the center point into the upsampling layer. The upsampling method is linear interpolation. After each upsampling, the dimensions are adjusted by passing through the MLP layer. The result is then input into the last MLP layer to obtain the final segmentation result.

[0028] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the point cloud semantic segmentation method based on geometric feature fusion as described in the first aspect.

[0029] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the point cloud semantic segmentation method based on geometric feature fusion as described in the first aspect.

[0030] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the point cloud semantic segmentation method based on geometric feature fusion as described in the first aspect.

[0031] Terminology Explanation:

[0032] Point Cloud Semantic Segmentation (PCSS): Semantic segmentation, which associates pixels with semantic labels, is a fundamental research area in image processing. Point cloud semantic segmentation is a three-dimensional form of semantic segmentation that uses regularly or irregularly distributed points in three-dimensional space to replace regularly distributed pixels in two-dimensional images.

[0033] Random sampling (RS): Random sampling is a method of selecting samples from a population or dataset in a random manner. It is an unbiased sampling method in which every sample has an equal chance of being selected.

[0034] Encoder-decoder: An encoder is a model that transforms input data (such as images or videos) into a low-dimensional representation. It compresses the input data into a vector or feature map to make the data easier to process and reduce storage space. A decoder is a model that transforms the encoded low-dimensional representation back into the original data. It decodes the feature map or vector produced by the encoder into a form similar to the original data.

[0035] The beneficial effects of this invention are that it can balance segmentation performance and time overhead, while solving the problems of information loss and rotation invariance, thus achieving more efficient large-scale point cloud semantic segmentation.

[0036] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This section compares the network framework diagram described in the embodiments of the present invention with the Point Transformer network framework diagram. Figure 1(a) is a diagram of the overall architecture of the Point Transformer network; Figure 1 (b) is the overall architecture diagram of the GFF-Net network.

[0039] Figure 2 This is a structural framework diagram of the GFF module according to an embodiment of the present invention.

[0040] Figure 3 This is a structural framework diagram of the LPC module according to an embodiment of the present invention.

[0041] Figure 4 This is a visualization of the segmentation results of argon in the S3DIS dataset as described in an embodiment of the present invention.

[0042] Table 1 shows the segmentation performance of this invention on the S3DIS dataset;

[0043] Table 2 compares the performance and time cost of the present invention on the SemanticKITTI dataset. Detailed Implementation

[0044] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0046] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.

[0047] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0048] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0049] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0050] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0051] This invention relates to applications in autonomous driving, augmented reality, and robotics. For example, during road driving, it can classify obstacles, vehicles, and pedestrians to aid the autonomous driving system in decision-making and planning, thereby significantly reducing the driver's workload. Point cloud semantic segmentation is fundamental to 3D environment understanding. Current point cloud semantic segmentation methods typically perform poorly when objects rotate, and often sacrifice operational efficiency for better segmentation results. To address this, this paper proposes an encoder-decoder architecture, GFF-Net, for point cloud semantic segmentation. In this network, to address the issue of object rotation affecting segmentation results, a Local Polar Conversion (LPC) module is introduced to obtain a geometric space representation that does not change with Z-axis rotation. Based on this, a Geometric Feature Fusion (GFF) module is proposed, enabling the network to better supplement lost information during the downsampling stage.

[0052] Example 1

[0053] In this embodiment 1, a point cloud semantic segmentation system based on geometric feature fusion is first provided, including: a feature extraction module, used to perform preliminary feature extraction on the input point cloud data through an MLP layer and a GFF layer; wherein, a Point Transformer is used as the backbone network, KNN is used to obtain neighboring points, and four upsampling layers use linear interpolation to restore the data to the original scale; wherein, the Point Transformer includes two MLP layers in the input and output stages, nine Point Transformer layers for feature extraction and fusion, and four downsampling layers using FPS to obtain key points; a downsampling module, used to input the extracted preliminary features into the downsampling layer; in the downsampling stage, a random sampling method is used to obtain key points, and a KNN method is used to obtain neighboring points near the key points; the downsampling results are input into the GFF module; the GFF input is the three-dimensional coordinate information of the point cloud and the corresponding semantic features. For the input point, LPC processing is performed on its neighboring points to obtain the relative angle of the neighboring points and the Euclidean distance between the neighboring points and the center point; the relative angle of the neighboring points, the geometric distance between the neighboring points and the center point, and the feature information of the neighboring points are concatenated to obtain Then The data is fed into the PTL layer and then into the SharedMLP layer. The same MLP operation is performed on the feature dimensions of each point to adjust the dimensions and obtain the local feature information representing the neighborhood of the center point. The upsampling module is used to input the local feature information representing the neighborhood of the center point into the upsampling layer. The upsampling method is linear interpolation. After each upsampling, the data is passed through the MLP layer for dimension adjustment. The result is then input into the last MLP layer to obtain the final segmentation result.

[0054] In this embodiment 1, the above-described system is used to implement a point cloud semantic segmentation method based on geometric feature fusion, including: the input point cloud data undergoes preliminary feature extraction through an MLP layer and a GFF layer; wherein, PointTransformer is used as the backbone network, KNN is used to obtain neighboring points, and four upsampling layers use linear interpolation to restore the data to the original scale; wherein, PointTransformer includes two MLP layers in the input and output stages, nine PointTransformer layers for feature extraction and fusion, and four downsampling layers using FPS to obtain key points; the extracted preliminary features are input into the downsampling layer; in the downsampling stage, a random sampling method is used to obtain key points, and a KNN method is used to obtain neighboring points near the key points; the downsampling results are input into the GFF module; the GFF input is the three-dimensional coordinate information of the point cloud and the corresponding semantic features. For the input point, LPC processing is performed on its neighboring points to obtain the relative angle of the neighboring points and the Euclidean distance between the neighboring points and the center point; the relative angle of the neighboring points, the geometric distance between the neighboring points and the center point, and the feature information of the neighboring points are concatenated to obtain Then The data is fed into the PTL layer and then passes through the Shared MLP layer. This layer performs the same MLP operation on the feature dimensions of each point to adjust the dimensions and obtain local feature information representing the neighborhood of the center point. The local feature information representing the neighborhood of the center point is then input into the upsampling layer. The upsampling method is linear interpolation. After each upsampling, the data passes through the MLP layer for further dimension adjustment. The results are then input into the last MLP layer to obtain the final segmentation result.

[0055] Specifically, in this embodiment, the point cloud semantic segmentation method based on geometric feature fusion is implemented as follows:

[0056] Step S1: Use Point Transformer as the backbone network. Point Transformer consists of two MLP layers for the input and output stages, nine Point Transformer layers for feature extraction and fusion, four downsampling layers using FPS to obtain keypoints and KNN to obtain neighboring points, and four upsampling layers using linear interpolation to restore the data to its original scale. Figure 1 As shown.

[0057] Step S2: Replace the Point Transformer layer adjacent to the first MLP layer and the Point Transformer layer adjacent to the four downsampling layers with the GFF module proposed in this embodiment, and use the random sampling replacement FPS method to obtain key points in the downsampling layers. Replace the four Point Transformer layers adjacent to the upsampling stage with MLP layers as follows: Figure 1 The network architecture is shown in the diagram. This step replaces the Point Transformer layer with a GFF module to improve feature extraction capabilities; subsequently, an MLP layer is used to replace the Point Transformer layer, saving time.

[0058] Step S3: The input point cloud data first undergoes preliminary feature extraction through an MLP layer and a GFF layer. The GFF structure is as follows: Figure 2 As shown, a detailed introduction to GFF will be given in step S4. The GFF module aims to improve feature extraction and enhances the rotational robustness of the method by introducing polar coordinates.

[0059] Step S4: The output of step S3 is input into the downsampling layer. In the downsampling stage, we use a random sampling method to obtain keypoints and the KNN method to obtain neighboring points near the keypoints. The downsampling results are input into the GFF layer. The GFF input consists of the 3D coordinate information of the point cloud and the corresponding semantic features. These semantic features may be the RGB features of the original point cloud or features processed by the previous GFF layer. For the input point, LPC processing is performed on the neighboring points to obtain the relative angles of the neighboring points and the Euclidean distance between the neighboring points and the center point. Then, the relative angles of the neighboring points, the geometric distance between the neighboring points and the center point, and the neighboring point feature information are concatenated to obtain... Then The data is fed into the PTL (Point Transformer Layer) and finally passes through the Shared MLP layer. This means that the same MLP operation is performed on the feature dimensions of each point. By adjusting the dimensions, the local feature information representing the neighborhood of the center point can be obtained. The descriptions of each layer of GFF are as follows.

[0060] Step S4-1: After random sampling, obtain the center point P, and apply the KNN algorithm to the center point to obtain P. i Neighboring points First, p i The nearest points are transformed from the Cartesian coordinate system to the polar coordinate system, that is, using... Let represent neighboring points. The conversion formula is shown below. It is the relative coordinate of the nearest point in the Cartesian coordinate system.

[0061]

[0062]

[0063]

[0064] Step S4-2: First calculate p i Centroid of the neighborhood The calculation uses the method of averaging the neighboring points, as defined in this way. It can reflect p well. i The overall situation within a local neighborhood. Defined from p i point to For local directions, used to represent p i and The relative angles between them. Update the relative angles of neighboring points: and Updated to and Where δ i and θ i yes The relative angle. From formulas (4) and (5), it can be seen that when the point cloud rotates around the Z-axis, and It remains unchanged. This concludes the complete LPC process; the structure of an LPC is as follows: Figure 3 As shown.

[0065]

[0066]

[0067] Step S4-3: Use the following formula to perform local feature stitching:

[0068] Indicates neighboring points Euclidean distance from key points This indicates a splicing operation. and It is the nearest neighbor point output by the LPC module. and p i Relative distance and converted relative angle

[0069] Step S4-4: Using LPC output and neighboring points The coordinate information is concatenated and then processed through a point-by-point MLP to obtain... Having acquired sufficient spatial characteristics, then... and Feature information fi k By splicing

[0070] Through the above operations, for the center point p i A set of neighboring features can be obtained.

[0071] Step S4-5: Improvements were also made to the input portion of the PTL (Point Transformer Layer) module in the Point Transformer network. The improved formula is:

[0072] ω and ξ are two linear layers, γ consists of one MLP and two linear layers, and ρ consists of one MLP and a soft max layer. The goal is to calculate the correlation between the center point and its neighbors, assign a weight score to the features of the neighbors, and finally fuse the neighbors' features using a summation function.

[0073] Step S5: After 4 rounds (4 rounds is a limiting condition; multiple experiments have shown that too few rounds will significantly reduce the segmentation effect, while too many rounds will have a negligible improvement in the result but will bring higher time costs), the output result is input to the upsampling layer. The upsampling method is linear interpolation, and after each upsampling, the dimension is adjusted by passing through the MLP layer.

[0074] Step S6: After 4 rounds (4 rounds is a limiting condition, for the same reason as S5), the result is input into the last MLP layer to obtain the final segmentation result.

[0075] Figure 4 Table 1 shows the performance of the method described in this embodiment on the S3DIS dataset (bold indicates the best performance for the current type, and horizontal lines indicate the second-best performance). Table 2 compares the segmentation performance and segmentation time of this method with other methods on the large-scale outdoor dataset SemanticKITTI, demonstrating that the method proposed in this embodiment achieves a good balance between time and performance.

[0076] Table 1

[0077]

[0078] Table 2

[0079]

[0080] Example 2

[0081] This embodiment 2 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they implement the point cloud semantic segmentation method based on geometric feature fusion as described above.

[0082] Example 3

[0083] This embodiment 3 provides a computer device, including a memory and a processor. The processor and the memory communicate with each other. The memory stores program instructions that can be executed by the processor. The processor calls the program instructions to execute the point cloud semantic segmentation method based on geometric feature fusion as described above.

[0084] Example 4

[0085] This embodiment 4 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions to implement the point cloud semantic segmentation method based on geometric feature fusion as described above.

[0086] In summary, the point cloud semantic segmentation method based on geometric feature fusion described in this invention uses rotation-robust polar coordinates to replace the Cartesian coordinates of the point cloud data and introduces a LocalPolar Conversion (LPC) module to obtain a geometric space representation that does not change with Z-axis rotation. This measure effectively improves the rotation robustness of the model. To improve the real-time performance of the model, random sampling is used in the downsampling stage, saving time. However, this method loses some feature information. Therefore, a Geometric Feature Fusion (GFF) module is proposed. By concatenating point cloud coordinate information, feature information, and angle information, and combining it with a Point Transformer Layer improved based on the Point Transformer network, better segmentation accuracy is achieved.

[0087] This invention introduces a polar coordinate transformation module to improve the model's rotation robustness and innovatively designs a geometric feature fusion module for feature extraction, achieving a good balance between segmentation performance and segmentation time. This invention improves upon the existing model framework, further enhancing its segmentation efficiency. Inspired by polar coordinate transformation, an LPC module is introduced to improve the model's rotation robustness. In the downsampling stage, faster random sampling is chosen. To compensate for the loss of some feature information caused by random sampling, a creative GFF module is proposed, which can effectively extract features while also fully compensating for the loss of feature information. Influenced by Point Transformer, the Point Transformer Block is introduced, but improvements are also made to enable more efficient feature aggregation, which is then embedded into the GFF module. This effectively improves the efficiency of point cloud semantic segmentation.

[0088] Compared to current point-based semantic segmentation methods, most existing methods neglect the impact of object rotation on segmentation performance. Furthermore, many methods sacrifice segmentation time for higher performance, as point cloud semantic segmentation in real-world applications demands not only high-quality results but also efficient segmentation time. This necessitates that point cloud semantic segmentation networks be designed to consider both segmentation performance and time consumption. Consequently, some high-performing methods are often limited in practical applications due to time constraints. This invention introduces a polar coordinate transformation module to mitigate the significant performance drop caused by object rotation. To reduce segmentation time without sacrificing model performance, rotation is achieved through low-time random sampling during the keypoint downsampling stage. However, the randomness of sampling inevitably leads to some loss of feature information. Therefore, a geometric feature fusion module and an improved Point Transformer module are proposed in the feature extraction and fusion stages to compensate for lost feature information while effectively extracting and fusing features. Through the polar coordinate transformation and geometric feature fusion modules, a better balance between segmentation performance and time is achieved.

[0089] This invention concatenates keypoint information, distance information between keypoints and their neighbors, polar coordinate information of neighbors, and semantic information during the feature extraction stage, compensating for the loss of some feature information due to random sampling. This information is then input into an improved Point Transformer module, where feature extraction and fusion are completed after passing through an MLP layer. This effectively improves network performance. This invention provides a unique approach to compensating for feature information; other information can also be concatenated during the feature compensation stage. However, it should be noted that excessively pursuing feature information can also lead to increased segmentation time.

[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0094] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. A point cloud semantic segmentation method based on geometric feature fusion, characterized in that, include: The input point cloud data undergoes preliminary feature extraction through an MLP layer and a GFF layer. A PointTransformer is used as the backbone network, KNN is employed to obtain neighboring points, and four upsampling layers use linear interpolation to restore the data to its original scale. The Point Transformer consists of two MLP layers in the input and output stages, nine PointTransformer layers for feature extraction and fusion, and four downsampling layers using FPS to obtain key points. The extracted preliminary features are input into the downsampling layer. During the downsampling stage, keypoints are obtained using random sampling, and neighboring points are obtained using the KNN method. The downsampling results are input into the GFF module. The GFF takes the point cloud's 3D coordinates and corresponding semantic features as input. For each input point, LPC processing is performed on its neighboring points to obtain the relative angles of the neighboring points and the Euclidean distance between the neighboring points and the center point. The relative angles of the neighboring points, the geometric distance between the neighboring points and the center point, and the neighboring point feature information are concatenated to obtain... Then The data is fed into the PTL layer and finally passes through the SharedMLP layer, which performs the same MLP operation on the feature dimensions of each point to adjust the dimensions and obtain the local feature information representing the neighborhood of the center point. The local feature information representing the neighborhood of the center point is input into the upsampling layer. The upsampling method is linear interpolation. After each upsampling, the dimensions are adjusted by passing through the MLP layer. The result is then input into the last MLP layer to obtain the final segmentation result.

2. The point cloud semantic segmentation method based on geometric feature fusion according to claim 1, characterized in that, After random sampling, a center point P is obtained. The KNN algorithm is then applied to this center point to obtain P. i Neighboring points First, p i The nearest points are transformed from the Cartesian coordinate system to the polar coordinate system, that is, using... Indicate the nearest neighbor, where These are the relative coordinates of neighboring points in the Cartesian coordinate system. The transformation formula is as follows:

3. The point cloud semantic segmentation method based on geometric feature fusion according to claim 2, characterized in that, Calculate p i Centroid of the neighborhood The calculation uses the method of averaging the neighboring points, defining the value from p... i point to For local directions, used to represent p i and The relative angle between them; update the relative angle of neighboring points: Will and Updated to and Where δ i and θ i yes The relative angle.

4. The point cloud semantic segmentation method based on geometric feature fusion according to claim 3, characterized in that, The following formula is used for local feature stitching: Indicates neighboring points Euclidean distance from key points This indicates a splicing operation. and These are the nearest neighbors output by our LPC module. and p i The relative distance and the converted relative angle.

5. The point cloud semantic segmentation method based on geometric feature fusion according to claim 4, characterized in that, Through LPC output and neighboring points The coordinate information is concatenated and then processed through a point-by-point MLP to obtain r. i k ; r i k Having acquired sufficient spatial characteristics, then r i k and Feature information f i k By splicing For the center point p i A set of neighboring features is obtained 6. The point cloud semantic segmentation method based on geometric feature fusion according to claim 5, characterized in that, The input to the PTL module in the Point Transformer network is: Here, ω and ξ are two linear layers, γ consists of one MLP and two linear layers, and ρ consists of one MLP and a Softmax layer; the aim is to calculate the correlation between the center point and the neighboring points, give the features of the neighboring points a weight score, and finally fuse the neighboring features through a summation function.

7. A point cloud semantic segmentation system based on geometric feature fusion, characterized in that, include: The feature extraction module performs preliminary feature extraction on the input point cloud data through an MLP layer and a GFF layer. It employs a Point Transformer as the backbone network, uses KNN to obtain neighboring points, and four upsampling layers with linear interpolation to restore the data to its original scale. The Point Transformer consists of two MLP layers in the input and output stages, nine Point Transformer layers for feature extraction and fusion, and four downsampling layers using FPS to obtain key points. The downsampling module is used to input the extracted preliminary features into the downsampling layer. During the downsampling stage, keypoints are obtained using random sampling, and neighboring points are obtained using the KNN method. The downsampling results are then input into the GFF module. The GFF takes the point cloud's 3D coordinates and corresponding semantic features as input. For each input point, LPC processing is performed on its neighboring points to obtain the relative angles of the neighboring points and the Euclidean distance between the neighboring points and the center point. The relative angles of the neighboring points, the geometric distance between the neighboring points and the center point, and the neighboring point feature information are concatenated to obtain... Then The data is fed into the PTL layer and finally passes through the Shared MLP layer, which performs the same MLP operation on the feature dimensions of each point to adjust the dimensions and obtain the local feature information representing the neighborhood of the center point. The upsampling module is used to input local feature information representing the neighborhood of the center point into the upsampling layer. The upsampling method is linear interpolation. After each upsampling, the dimensions are adjusted by passing through the MLP layer. The result is then input into the last MLP layer to obtain the final segmentation result.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the point cloud semantic segmentation method based on geometric feature fusion as described in any one of claims 1-6.

9. A computer device, characterized in that, The method includes a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions that can be executed by the processor, and the processor calling the program instructions to execute the point cloud semantic segmentation method based on geometric feature fusion as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the point cloud semantic segmentation method based on geometric feature fusion as described in any one of claims 1-6.

Images