Three-dimensional scene point cloud segmentation enhancement method, device, storage medium and program product

Abstract

The application discloses a three-dimensional scene point cloud segmentation enhancement method and device, a storage medium and a program product, and belongs to the technical field of point cloud segmentation methods. Based on an attention mechanism, the application effectively incorporates global information by designing a good attention jump module, improves segmentation accuracy, and effectively solves the problem that the existing three-dimensional point cloud segmentation method based on the U-Net framework has simple jump connection processing and cannot comprehensively capture fine-grained texture information in point cloud data. Meanwhile, based on a model compression method of low-rank attention decomposition, a low-rank global attention jump module is introduced to reduce the computational burden caused by the calculation-intensive attention jump module when processing complex point cloud scenes and improve the model efficiency. Meanwhile, the three-dimensional scene point cloud segmentation enhancement device provided by the application can be seamlessly inserted into the existing U-Net framework to replace the traditional 'jump connection module' without any additional work.

Description

Technical Field

[0001] This invention relates to the field of point cloud segmentation methods, specifically to a three-dimensional scene point cloud segmentation enhancement method, device, storage medium, and program product. Background Technology

[0002] Large-scale point cloud data of 3D indoor spaces is commonly used in research and applications in computer vision and machine learning. As a core resource for 3D perception technology, its applications are extremely wide-ranging, covering multiple key areas from indoor navigation and positioning, object recognition and segmentation, to environmental modeling, spatial planning and management. Here are some application scenarios for 3D point cloud data: 1. Indoor Navigation and Positioning: Point cloud data provides a solid foundation for building high-precision indoor maps, helping to develop advanced navigation and positioning systems, serving cutting-edge fields such as robot navigation, smart home control, drone indoor operations, and augmented reality experiences. 2. 3D Object Recognition and Segmentation: Using point cloud data to train and optimize algorithms can significantly improve the ability to recognize and segment complex objects in indoor environments, which is crucial for applications such as autonomous driving assistance, intelligent security monitoring, and environmental monitoring. 3. Indoor Environment Analysis: Through point cloud data analysis, fine-grained division of indoor spaces, furniture layout recognition, and environmental feature extraction can be achieved, providing strong support for interior design optimization, intelligent building management, and virtual reality simulation. 4. Deep Learning Research: As a valuable resource in deep learning research, 3D point cloud data has driven the innovation and development of neural network architectures, especially in the training and testing of models such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). 5. Infrastructure Planning and Management: Urban planning, leveraging point cloud data, can more accurately plan and manage urban internal spaces, including building layout, transportation network optimization, and the allocation of public service facilities.

[0003] 3D point cloud data, with its unique three-dimensional Cartesian coordinate system, accurately captures and represents real-world objects and their attributes, including diverse information such as color and reflectivity. Compared to two-dimensional images, point cloud data exhibits more complex characteristics—sparseness and disorder; even multiple observations of the same object can result in significant differences in its point cloud representation. These characteristics necessitate that researchers develop specialized processing methods for 3D point clouds to fully unlock their potential value.

[0004] With continuous technological advancements, 3D point cloud data has demonstrated immense potential in cutting-edge fields such as autonomous driving, product quality inspection, and robotics. Particularly in these applications, the importance of point cloud segmentation technology is becoming increasingly prominent. Point cloud segmentation aims to divide complex point cloud data into a series of subsets with clear semantic meaning. This process not only helps improve the efficiency of understanding and processing point cloud information but is also a key prerequisite for realizing advanced application functions (such as obstacle recognition and path planning).

[0005] Currently, numerous methods have been proposed for point cloud segmentation, among which the U-Net-based approach stands out for its superior performance. U-Net, a classic neural network architecture, is renowned for its unique U-shaped design. This architecture cleverly combines three key components: encoder, decoder, and skip connections, achieving deep feature extraction and accurate segmentation of image or point cloud data. The encoder module captures rich contextual and spatial information from the input data through multiple feature aggregation and downsampling layers, while reducing the spatial size of the data and increasing the number of feature channels, enabling the network to capture global context and higher-level features. The decoder module utilizes feature interpolation and upsampling layers to gradually restore the spatial dimension of the data to its original size and performs precise localization based on the coarse features provided by the encoder. The skip connection module plays a crucial role throughout the process, ensuring the effective fusion of low-level spatial information and high-level contextual information by connecting the corresponding layers between the encoder and decoder, thereby preserving complex details and achieving accurate segmentation.

[0006] U-Net-based methods have achieved remarkable success in image segmentation due to their deep feature extraction at both the encoder and decoder. For example, PointNet++ innovatively learns different fine-grained features through hierarchical aggregation and sampling of point clouds, making it one of the earliest point cloud segmentation networks to adopt a U-shaped framework. KPConv employs deformable convolution operators. These operators allow the convolution kernel to adaptively adjust its shape to fit various geometries present in point cloud data.

[0007] While U-Net-based methods have achieved significant success in image segmentation, they still face challenges in processing the disordered and sparsity of point cloud data. Due to the lack of a global feature extraction module, the local and non-local features extracted by most existing U-Net-based methods for image data are unsuitable for point cloud segmentation tasks. Point clouds are known to be disordered and sparse, making global features crucial for high-precision segmentation. Specifically, disordered features refer to the absence of a predetermined arrangement or structure in a point cloud. This means that from the perspective of connections and relationships between individual points, two points that are far apart may actually be neighbors. On the other hand, during data acquisition, the limited number of samples and the sparsity of point clouds make it difficult for traditional U-Net-based methods to effectively learn good and rich representational features. Therefore, extracting global feature information is critical for processing point cloud data.

[0008] Therefore, how to integrate the global attention mechanism with the U-Net architecture to achieve segmentation enhancement of 3D point cloud data has become a technical challenge that urgently needs to be overcome by those skilled in the art. Summary of the Invention

[0009] The purpose of this invention is to provide a method, device, storage medium, and program product for enhancing point cloud segmentation in a 3D scene, in order to overcome the problem that existing 3D point cloud segmentation methods based on the U-Net framework cannot fully understand the fine-grained texture in point cloud data due to the relatively simple handling of skip connections.

[0010] The present invention solves the above-mentioned technical problems through the following technical solution:

[0011] A method for segmenting and enhancing point clouds in a 3D scene includes the following steps:

[0012] Step 1: Obtain the point cloud data to be segmented, and encode it to obtain the input features F of the current sample set. in Features F of global points glb ;

[0013] Step 2: Input features F of the current sample set in Features F of global points glb Perform layer-by-layer ordered extraction and execute linear transformations to generate the query matrix Q, the key matrix K, and the value matrix V;

[0014] Step 3: Calculate the attention weights of the query matrix Q and the key matrix K using matrix dot product to obtain the attention weight matrix A;

[0015] Step 4: By using the corresponding attention weights in the attention weight matrix A, the value vectors in the value matrix V are weighted and summed to obtain the output feature F of the attention layer. attn ;

[0016] Step 5: Based on the principle of residual networks, fuse the output features F of the attention layer. attn and the input features F of the current sample set in Randomly discard network branches to prevent overfitting and obtain the output features F of the attention jump module. out ;

[0017] Step 6: Calculate the output feature F of the attention jump module. out The output features of the deeper attention jump module are fused to enhance the point cloud segmentation of the 3D scene.

[0018] Furthermore, the query matrix Q, key matrix K, and value matrix V in step two are respectively:

[0019] Q = F in ·W q +δ in (4-1)

[0020] K = F glb ·W k +δ glb (1-5)

[0021] V = F glb ·W v +δ glb (1-6)

[0022] in, W q W k W v Both are linear transformations that can be shared and learned. N o N represents the number of points in the original data. i To calculate the number of point cloud points downsampled from the original data to the current scale, d o The feature dimension of the original data is d, and the dimension of the query vector is d. a Both the keyword vector and the value vector have a dimension of d. i δ represents the position embedding, which is obtained by performing a nonlinear transformation on the coordinates of the corresponding input point. in Embedding the position of the input point, δ glb Embed the position of the global point.

[0023] Furthermore, the attention weight matrix A in step three is specifically as follows:

[0024]

[0025] Among them, K T This is the transpose of the keyword matrix K. SoftMax() is the scaling factor and the row normalization matrix function.

[0026] Furthermore, the output feature F of the attention layer in step four attn Specifically:

[0027] F attn =A·V (2-2)

[0028] Furthermore, the output feature F of the attention jump module in step five out Specifically:

[0029] F out =LBR(F attn )+F in (2-3)

[0030] Here, LBR() represents linear transformation, batch normalization, and the ReLU function.

[0031] Furthermore, step six can be replaced by the following process: based on the query matrix Q, key matrix K, value matrix V, attention weight matrix A, and output features F of the attention layer. attn and the output feature F of the attention jump module out The output feature matrix of the attention jump module after low-rank decomposition and compression is obtained. Then, the output feature matrix of the attention jump module after low-rank decomposition and compression is fused with the output feature matrix of the deeper attention jump module after low-rank decomposition and compression to complete the enhancement of the 3D scene point cloud segmentation.

[0032] Further, in step six, the output feature matrix of the attention skipping module after low-rank decomposition and compression is obtained, specifically including the following steps:

[0033] Integrating the above expressions (1-1), (1-2), (1-3), (2-1), (2-2), and (2-3), we obtain the operator GA, which is as follows:

[0034]

[0035] Among them, F l For the queried point cloud, F r To query point clouds, The right-multiplied feature dimension transformation moment, This indicates that the operator GA needs to be of size N. l and N r Attention calculation is performed between two point clouds. To query the point cloud, multiply it by the factor matrix on the left.

[0036] By calling the GA operator, the output feature matrix of the attention skipping module is obtained, specifically:

[0037]

[0038] in, This means that the corresponding GA operator needs to complete the process from N i Attention calculation up to N0 points; F i The feature matrix is ​​a downsampled matrix. N i and C i These represent the number of point clouds and the feature dimension downsampled from the original data to the current scale; the features of global points. N o and C o These represent the number of points and feature dimensions of the original data, respectively. For a size of N i The score matrix of ×N0 represents F i and F glb Point-to-point similarity between them; To apply to F glb Linear transformation matrix on;

[0039] The score matrix is ​​decomposed into a low-rank matrix as follows:

[0040]

[0041] The linear transformation matrix is ​​decomposed into its lowest rank as follows:

[0042]

[0043] in, Representing F i and F fps Point-to-point similarity between them; Representing F fps and F glb Point-to-point similarity between them; To apply the low-rank decomposition method to F glb Linear transformation matrix on; To apply the low-rank decomposition method to F fps The linear transformation matrix on; F fps The global representative point cloud has N points. fps The rank of the low-rank decomposition in formulas (5) and (6) corresponds to N. fps < <min{N i N o};

[0044] Substituting the score matrix formula (5) and the linear transformation matrix formula (6) after low-rank decomposition into the output feature matrix (4) of the attention jump module, we obtain the output feature matrix of the attention jump module after low-rank decomposition compression, which is as follows:

[0045]

[0046] An electronic device includes: a memory and a processor;

[0047] Memory, used to store computer programs;

[0048] A processor is used to execute computer programs to implement the steps of the three-dimensional scene point cloud segmentation enhancement method described above.

[0049] A computer-readable storage medium is characterized in that it stores computer-executable instructions, which, when executed by a processor, are used to implement the steps of the above-described three-dimensional scene point cloud segmentation enhancement method.

[0050] A computer program product, characterized in that the computer program product includes computer program code, and when the computer program code is executed by a computing device, the computing device performs the steps of the above-described three-dimensional scene point cloud segmentation enhancement method.

[0051] Compared with existing technologies, the positive advancements of this invention are as follows: The 3D scene point cloud segmentation enhancement method provided by this invention, based on an attention mechanism, effectively incorporates global information and improves segmentation accuracy through a well-designed attention jump module. This effectively solves the problem that existing 3D point cloud segmentation methods based on the U-Net framework have simple jump connection processing and cannot fully capture fine-grained texture information in point cloud data. At the same time, the model compression method based on low-rank attention decomposition introduces a low-rank global attention jump module to reduce the computational burden caused by the computational density of the attention jump module when processing complex point cloud scenes, thereby improving model efficiency.

[0052] The 3D scene point cloud segmentation enhancement device provided by this invention can be inserted into the existing U-Net framework to replace the traditional "skip connection module". Through the attention skip module, global features can be easily fused with local and non-local features from the encoder and decoder, thereby obtaining a better point cloud representation. Furthermore, the replacement of the original skip connections is seamless and requires no additional work. Attached Figure Description

[0053] The accompanying drawings are provided to further understand the invention and constitute a part of this invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0054] Figure 1 This is a schematic diagram of the U-Net framework of the present invention;

[0055] Figure 2 This is the data stream of the attention-jumping module of the present invention;

[0056] Figure 3 This is the data stream of the attention layer in this invention;

[0057] Figure 4 This is a schematic diagram of the point cloud receptive field in the global attention jump module of the present invention;

[0058] Figure 5 This is a schematic diagram of the receptive field of the point cloud in the low-rank global attention jump module of the present invention;

[0059] Figure 6 This diagram illustrates the model data flow after seamlessly integrating the low-rank global attention jump module into the U-Net framework.

[0060] Figure 7 This is a schematic diagram illustrating the experimental results of seamlessly integrating the global attention jump module into three U-Net frameworks.

[0061] Figure 8 This is a schematic diagram illustrating the experimental results of seamlessly integrating the low-rank global attention skipping module into three U-Net frameworks. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0064] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0065] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0066] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0067] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0068] The present invention will be further described in detail below with reference to the accompanying drawings. These descriptions are for illustrative purposes only and are not intended to limit the scope of the invention.

[0069] Because the model needs to easily identify and distinguish each point during processing, it often needs to transform its three-dimensional coordinates into high-dimensional vectors. This task of dimensionality enhancement is called encoding, and the module that completes this task is called the encoder. This dimensionality enhancement process may be gradual, so the encoder is usually a cascaded structure. After the model completes feature recognition, it usually needs to restore the features, i.e., dimensionality reduction. This process is called decoding, and the module that completes this process is called the decoder. The decoder is also usually a hierarchical cascaded structure.

[0070] See Figure 1 The U-Net framework consists of three parts: an encoder (blue), a decoder (orange), and skip connections (red). Following the arrow direction, each module receives information, processes it internally, and then passes it to subsequent modules, ultimately outputting category labels for each point in the point cloud, thus completing the semantic segmentation of the entire point cloud. The skip connection module is a crucial component, acting as a bridge between the encoder and decoder. An overly simplistic processing procedure can lead to semantic discrepancies between different modules, directly affecting the network's output.

[0071] Represent a given point cloud dataset as a set P, where P = {F} i ,x i )|i=1,···,N∈R N×6 F i This represents the coordinate information of the i-th point in the point cloud in three-dimensional Euclidean space, x i This represents the RGB color information of the i-th point. In the encoder part, the point cloud data P undergoes multiple downsampling (using the farthest point sampling method commonly used in point cloud processing) and feature upsampling to obtain the point cloud coordinates and features of each layer. "Point cloud data" refers to the three-dimensional coordinates (three-dimensional vectors) of each point. However, in order to easily identify and distinguish each point during model processing, these three-dimensional coordinates often need to be transformed into high-dimensional vectors. "Point cloud features" refer to these high-dimensional vectors. That is, only the initial three-dimensional coordinates are called "point cloud data," and after dimensionality upscaling, they are uniformly called "point cloud features."

[0072] In each layer, the decoder receives features propagated from subsequent layers. and features from the corresponding encoder layer For example Figure 1 In The decoder of this layer (circled in orange) receives the output from the decoder of the lower layer. Output of same-level jump connection The two pieces of information were obtained through internal processing. Note the skip connection features from the same layer. Important low-level feature information is passed to the decoder to preserve detailed information. Finally, the point cloud undergoes downsampling, upsampling, and feature fusion processes to output the classification label for each point at the highest layer, completing the semantic segmentation task.

[0073] Traditional methods use multilayer perceptron (MLP) modules to directly manipulate skip connections. Due to the importance of global features, this invention provides a 3D scene point cloud segmentation enhancement method that leverages a general skip connection module to enhance the collaboration between the encoder and decoder. This enhancement method can be seamlessly integrated into the U-Net framework as a plug-and-play tool.

[0074] Example 1

[0075] A method for segmenting and enhancing point clouds in a 3D scene includes the following steps:

[0076] Step 1: Obtain the point cloud data to be segmented, and encode it to obtain the input features F of the current sample set. in Features F of global points glb ;

[0077] Step 2: Input features F of the current sample set inFeatures F of global points glb Perform layer-by-layer ordered extraction and execute linear transformations to generate the query matrix Q, the key matrix K, and the value matrix V;

[0078] Step 3: Calculate the attention weights of the query matrix Q and the key matrix K using matrix dot product to obtain the attention weight matrix A;

[0079] Step 4: By using the corresponding attention weights in the attention weight matrix A, the value vectors in the value matrix V are weighted and summed to obtain the output feature F of the attention layer. attn ;

[0080] Step 5: Based on the principle of residual networks, fuse the output features F of the attention layer. attn and the input features F of the current sample set in Randomly discard network branches to prevent overfitting and obtain the output features F of the attention jump module. out ;

[0081] Step 6: Calculate the output feature F of the attention jump module. out The output features of the deeper attention jump module are fused to enhance the point cloud segmentation of the 3D scene.

[0082] Specifically, the attention jump module is a global attention jump module, and the attention layer is a global attention layer, see [link to relevant documentation]. Figure 2 The Global Attention Skip Module (ASBlock) consists of a series of interconnected Global Attention Layers (GAlayers). ASBlock takes the features of the current sampling point (current sample set) and the initial features of the original points (global points) as combined inputs, and utilizes the series of GAlayers within the ASBlock to perform ordered feature extraction on these combined inputs. Each GAlayer aims to capture the relationships between points and the global background encompassing all points. The output features of each GAlayer are concatenated and integrated to form a comprehensive representation. Multiple outputs are combined and passed through a subsequent nonlinear fusion module, resulting in the skip connection output features of that layer. Figure 1 In Subsequently, these cascaded features ( Figure 2 The combined features at point C in the middle circle are fused using the LBR (composed of linear transformation, batch normalization, and ReLU function) module to enhance the model's effective information synthesis capability. To further enhance generalization ability, the Dropout module is also applied. The Dropout module is a general module in deep learning that prevents overfitting by randomly discarding network branches during training, thereby improving the stability and adaptability of the learned features.

[0083] Data flow in the global attention layer, such as Figure 3 As shown, GAlayer receives two inputs: the features F of the current sample set. in Features F of global points glb Both inputs correspond to a specific scale; where feature F in This indicates the encoder's output from the previous layer, while F... glb It contains all point features of the complete point cloud P.

[0084] Perform linear transformations as follows to generate the query matrix (Q), key matrix (K), and value matrix (V):

[0085]

[0086] in, W q W k W v Both are linear transformations that can be shared and learned. N o N represents the number of points in the original data. i To calculate the number of point cloud points downsampled from the original data to the current scale, d o The feature dimension of the original data is d, and the dimension of the query vector is d. a Both the keyword vector and the value vector have a dimension of d. i δ represents the position embedding, which is obtained by performing a nonlinear transformation on the coordinates of the corresponding input point. in For the position embedding of the input point, δ glb Embed the position of the global point.

[0087] Attention weights for the query and key matrices are calculated using matrix dot product. To address the vanishing gradient problem and improve training stability, the weights are divided by d. a The square root of the dot product is used to scale the result. This yields the attention score matrix A, while the output feature F of the attention layer... attn This is obtained by weighting the value vector using the corresponding attention weights in matrix A.

[0088] The specific details are as follows:

[0089]

[0090] F attn =A·V (2)

[0091] F out =LBR(F attn )+F in

[0092] Among them, K T This is the transpose of the keyword matrix K. The scaling factor is SoftMax(), which is the row normalization matrix function; LBR() represents the linear transformation, batch normalization, and ReLU function.

[0093] It's worth noting that both Softmax and weighted summation operations are commutatively invariant, ensuring that the global attention layer maintains the point order. Therefore, the entire global attention process is permutationally invariant, making it well-suited for handling the disordered and irregular nature of point cloud data. Finally, utilizing the principles of residual networks, the output features F of the attention layer are integrated. attn and the input features F of the current sample set in To generate the output features F of the entire attention jump module out .

[0094] The output feature F of the attention jump module is then processed by the decoder in the same layer as the attention jump module. out Feature fusion is performed with the output features of the attention jump module of the deeper decoder to enhance the point cloud segmentation of the 3D scene.

[0095] Due to the computational intensity of the attention jump module, it can lead to an unbearable amount of computation when dealing with complex point cloud scenarios. Therefore, this invention also proposes a method based on low-rank decomposition to compress the attention jump module, as detailed in Embodiment 2.

[0096] Example 2

[0097] Low-rank matrix decomposition is a common matrix factorization technique used to decompose a matrix M into a product of two or more low-rank matrices. This decomposition is often used for tasks such as reducing data dimensionality, denoising, and extracting features. Suppose we have a matrix A∈R... m×n The goal of low-rank matrix decomposition is to find two matrices M∈R m×k and N∈R m×k So that A≈MN T In this case, M and N have lower ranks, which is called low-rank matrix decomposition.

[0098] The difference between this implementation method and Example 1 is that step six can be replaced by the following process:

[0099] Based on the query matrix Q, key matrix K, value matrix V, attention weight matrix A, and output features F of the attention layer. attn and the output feature F of the attention jump module out The output feature matrix of the attention jump module after low-rank decomposition and compression is obtained. Then, the output feature matrix of the attention jump module after low-rank decomposition and compression is fused with the output feature matrix of the attention jump module after deeper low-rank decomposition and compression to complete the enhancement of the 3D scene point cloud segmentation.

[0100] In this implementation, the attention jump module is a low-rank global attention jump module, and the attention layer is a low-rank global attention layer.

[0101] This implementation method, based on low-rank matrix decomposition, compresses the attention skipping module and includes the following steps:

[0102] For ease of explanation, some necessary symbols are introduced, let and This represents point cloud feature matrices of different sizes, with size N. l ×N r The score matrix is ​​represented as It represents F l and F r Point-to-point similarity between them. Indicates the application of F r Linear transformation matrix on. Integrate formulas (1) and (2) into the following matrix operations:

[0103]

[0104] Among them, F l For the queried point cloud, F r To query point clouds, The right-multiplied feature dimension transformation moment, This indicates that the operator GA needs to be of size N. l and N r Attention calculation is performed between two point clouds. To query the point cloud, multiply it by the factor matrix on the left.

[0105] Operator GA can integrate F l and F r The inherent structural information between them generates a size of N. l The feature matrix of ×C. Given the downsampled feature matrix. and the unsampled complete global point feature matrix The output feature matrix of the attention skipping module can be obtained by calling the GA operator:

[0106]

[0107] Among them, formula (3) is the general form of the GA operator, and formula (4) is the specific form of the GA operator substituted into this module; This means that the corresponding GA operator needs to complete the process from N i Attention calculation up to N0 points; F i The feature matrix is ​​a downsampled matrix. N i and C iThese represent the number of point clouds and the feature dimension downsampled from the original data to the current scale, respectively; F glb Features of global points N o and C o These represent the number of points and feature dimensions of the original data, respectively. For a size of N i The score matrix of ×N0 represents F i and F glb Point-to-point similarity between them; W C0×C To apply to F glb The linear transformation matrix on.

[0108] Because the number of points N in the point cloud i Both N0 and N0 are very large, making the computational burden of the score matrix often unbearable. Inspired by the low-rank modeling concept in machine learning and data science, the score matrix... This can be approximated by multiplying two low-rank factor matrices:

[0109]

[0110] Similarly, linear transformation matrix It can also be approximated as follows:

[0111]

[0112] in, and They all have parameter forms; Representing F i and F fps Point-to-point similarity between them; Representing F fps and F glb Point-to-point similarity between them; To apply the low-rank decomposition method to F glb Linear transformation matrix on; To apply the low-rank decomposition method to F fps The linear transformation matrix on; F fps The global representative point cloud has N points. fps The rank of the low-rank decomposition in formulas (5) and (6) corresponds to N. fps <<min{N i N o}

[0113] This low-rank approximation can reduce the storage complexity from O(N) to O(N). i N o +C o C) greatly reduced to O((N) i+N o )N fps +(C o +C)C fps Substituting formulas (5) and (6) into formula (4) and using the definition of the GA operator (3), we obtain:

[0114]

[0115] Among them, formulas (4), (5), and (6) are the address decomposition and compression theory, and (7) indicates that an intermediate point cloud is needed as a medium in the actual process.

[0116] From the above equation, we can obtain the output feature matrix of the attention skipping module after low-rank decomposition and compression, specifically:

[0117]

[0118] in, For a size of N i The score matrix of ×N0 represents F i and F glb Point-to-point similarity between them; Representing F i and F fps Point-to-point similarity between them; Representing F fps and F glb Point-to-point similarity between them;

[0119] This means that the corresponding GA operator needs to complete the process from N i Attention calculation up to N0 points; To apply to F glb Linear transformation matrix on; To apply the low-rank decomposition method to F glb Linear transformation matrix on; To apply the low-rank decomposition method to F fps Linear transformation matrix on;

[0120] F i The feature matrix is ​​downsampled. F glb This is the unsampled complete global point feature matrix. F fps For the global representative point cloud, point cloud F fps It is implemented using the well-known farthest point sampling method in point cloud processing. For example... Figure 6 As shown, point cloud F fps In essence, it represents the global representative point, with N points. fpsThe rank corresponds to the low-rank decomposition in formulas (5) and (6), and N is... fps < <min{N i The global representative point cloud has a much smaller number of points than the original point cloud, N0}. fps The size of the value can directly control the degree of low-rank decomposition and compression.

[0121] Based on the above matrix low-rank decomposition, the attention jump module is compressed to obtain the output feature matrix of the attention jump module after low-rank decomposition compression. The output feature matrix after low-rank decomposition compression is fused with the output feature matrix after low-rank decomposition compression of the deeper decoder through the same layer decoder of the attention jump module to complete the enhancement of 3D scene point cloud segmentation.

[0122] I. Experimental verification of ASBlock and LrASBlock based on small data:

[0123] ShapeNet Part is a common dataset for point clouds, capable of segmenting parts. It allows users to segment smaller parts from a larger point cloud dataset. The ShapeNet Part dataset contains a large number of 3D point cloud models, each with its own 3D coordinates and corresponding semantic labels. It includes 16 object classes, 50 component labels, and a total of 16,846 samples. The dataset exhibits an imbalanced distribution of samples; for example, Table has 5,263 samples, while Earphone has only 69. Each sample contains over 2,000 points, making it a small dataset. The dataset consists of 12,137 training samples, 1,870 validation samples, and 2,839 test samples, totaling 16,846 samples. Its general applicability across different models was verified by re-experimenting with three different models and obtaining benchmark results.

[0124] After that, ASBlock (abbreviated as ASB) or LrASBlock (abbreviated as LrASB) was added to them respectively, and the training strategies of the three models remained unchanged to present the final experimental results, see Table 1.

[0125] Table 1

[0126] PointNet++ 82.5 85.4 PointNet++LrASB 82.5 85.9 PointNet++ASB 82.6 86.0 CurveNet 83.5 86.4 CurveNet+LrASB 83.7 86.6 CurveNet+ASB 83.8 86.6 PointStack 81.9 87.2 PointStack+LrASB 82.6 87.4 PointStack+ASB 82.8 87.5

[0127] In this column, each item in the Methods column represents a segmentation method. cat.mIoU (instance mIoU) and ins.mIoU (category mIoU) are evaluation metrics for point cloud semantic segmentation results, expressed as a percentage. The evaluation criteria for both are that the larger the value, the better the segmentation effect. "+LrASB" and "+ASB" indicate that skip connections are replaced by LrASBlock and ASBlock, respectively. For example, PointStack+ASB means that the skip connections of PointStack are replaced by ASBlock, that is, PointStack integrates ASBlock.

[0128] As shown in Table 1, after integrating ASBlock, PointNet++'s instance mIoU and class mIoU improved by 0.1% and 0.6%, respectively. Similarly, CurveNet and PointStack's metrics improved by 0.3% / 0.2% and 0.9% / 0.3%, respectively. These results consistently demonstrate that using ASBlock can improve network performance, thus confirming the enhancement effect of adopting ASBlock.

[0129] Furthermore, compressing ASBlock using low-order attention decomposition does not affect the module's enhancement effect on the original framework, or in other words, the loss is minimal, which also demonstrates the effectiveness of low-rank attention decomposition.

[0130] In terms of qualitative comparison of semantic segmentation results, Figure 7 The differences between the original network and the model (using PointStack as an example) are shown. It is clear from the figure that introducing ASBlock enhances the network's ability to capture comprehensive target information and mitigates complex errors in specific regions, allowing observation of details such as guitar necks, lamp bases, and lampshades.

[0131] II. Experimental Verification of LrASBlock Based on Large-Scene Data:

[0132] Due to the computational intensity of the attention jump module, it can lead to an unbearable amount of computation when dealing with complex point cloud scenes. When dealing with large scene data, LrASBlock is also needed to overcome the computational difficulty.

[0133] See Figure 6 The low-rank global attention skipping module (LrASBlock) uses the point cloud features F at the i-th scale in the U-Net network. i , and the original complete global point cloud features F glb As input, using formulas (1), (2), (3), and (8) as operators, the output of the i-th scale jump module is sent to the corresponding i-th scale decoder, thus enhancing the point cloud semantic segmentation effect of the U-Net network framework.

[0134] In indoor large-scene point cloud segmentation tasks (taking S3DIS data as an example), the enhancement effect is better than existing methods, and it can be seamlessly integrated into future new point cloud segmentation models based on the U-Net framework.

[0135] The effectiveness of the invention was tested on the S3DIS dataset. The S3DIS (Stanford 3D Indoor Scene Dataset) dataset is a point cloud dataset created by the Stanford University Computer Graphics Laboratory to facilitate research on indoor scene understanding and reconstruction. The dataset was collected from 6 areas and contains a total of 272 scenes, which can be divided into 11 different scenes.

[0136] Among them, mIoU, mAcc, and OA are evaluation metrics for point cloud semantic segmentation results. Table 1 shows the average metrics of the corresponding models in 11 scenarios, in percentage. The evaluation criteria for all three are that the larger the data, the better the segmentation effect.

[0137] The experimental results are recorded in Table 2. Each item in the "Methods" column represents a segmentation method, and the corresponding evaluation metric is the average performance of that method across 11 different scenarios. It can be seen that after combining LrASBlock with a low-rank global attention layer, KPConv's mIoU, mAcc, and OA improved by 0.8%, 0.9%, and 0.1%, respectively. The Point Transformer's metrics improved by 0.4% / 0.9% (mIoU / mAcc). The more powerful Stratified Transformer, after integrating a low-rank global attention skip connection module, improved mIoU, mAcc, and OA by 0.6%, 0.9%, and 0.2%, respectively.

[0138] Table 2

[0139]

[0140]

[0141] Figure 8 The visualization shows the segmentation results for three different regions, arranged from left to right: the ground truth, the unmodified network, and the network fused with LrASBlock. Of particular note is the targeted attention paid to the region enclosed by a black box. In this constrained region, the effect of LrASBlock fusion becomes apparent, significantly enhancing the network's overall understanding of the target entity. Furthermore, LrASBlock fusion effectively mitigates the network's tendency to segment local points within the object incorrectly.

[0142] Example 3

[0143] An electronic device includes: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the steps of the above-described three-dimensional scene point cloud segmentation enhancement method.

[0144] Example 4

[0145] A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the steps of the above-described three-dimensional scene point cloud segmentation enhancement method.

[0146] Example 5

[0147] A computer program product, comprising computer program code, wherein when the computer program code is executed by a computing device, the computing device performs the steps of the aforementioned three-dimensional scene point cloud segmentation enhancement method.

[0148] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for segmenting and enhancing point clouds in a 3D scene, characterized in that, Includes the following steps: Step 1: Obtain the point cloud data to be segmented, and encode it to obtain the input features of the current sample set. Features of global points , Includes all point features of the complete point cloud data; Step 2: Input features for the current sample set Features of global points Perform layer-by-layer ordered extraction and execute linear transformations to generate the query matrix. Key matrix Sum matrix ; Step 3: Calculate the query matrix using the matrix dot product. and key matrix The attention weights are calculated to obtain the attention weight matrix. ; Step 4: Using the attention weight matrix The corresponding attention weights in the value matrix The value vectors in the input are weighted and summed to obtain the output features of the attention layer. ; Step 5: Based on the principle of residual networks, fuse the output features of the attention layer. and the input features of the current sample set Randomly discard network branches to prevent overfitting and obtain the output features of the attention jump module. ; Step 6: Extract the output features of the attention jump module The output features of the deeper attention jump module are fused to enhance the point cloud segmentation of the 3D scene; Query matrix in step two Key matrix Sum matrix They are respectively: in, ; , Both are linear transformations that can be shared and learned. , , ; The number of points in the original data. To downsample the point cloud to the current scale from the original data, The feature dimension of the original data is [value], and the dimension of the query vector is [value]. The dimension of both the keyword vector and the value vector is 1. , The location embedding is obtained by performing a nonlinear transformation on the coordinates of the corresponding input point. Embed the position of the input point. Embed the position of the global point.

2. The method for segmenting and enhancing point clouds in a three-dimensional scene according to claim 1, characterized in that, Attention weight matrix in step three Specifically: in, Keyword matrix transpose, Scaling factor This is the row normalization matrix function.

3. The method for segmenting and enhancing point clouds in a three-dimensional scene according to claim 2, characterized in that, Output features of the attention layer in step four Specifically: 。 4. The three-dimensional scene point cloud segmentation and enhancement method according to claim 3, characterized in that, Output features of the attention jump module in step five Specifically: Where LBR() represents linear transformation, batch normalization, and the ReLU function.

5. The three-dimensional scene point cloud segmentation and enhancement method according to claim 4, characterized in that, Step six is ​​replaced by the following process: based on the query matrix Key matrix Value matrix Attention weight matrix Output features of the attention layer and the output features of the attention jump module The output feature matrix of the attention jump module after low-rank decomposition and compression is obtained. Then, the output feature matrix of the attention jump module after low-rank decomposition and compression is fused with the output feature matrix of the attention jump module after deeper low-rank decomposition and compression to complete the enhancement of the 3D scene point cloud segmentation.

6. The three-dimensional scene point cloud segmentation and enhancement method according to claim 5, characterized in that, Step six yields the output feature matrix of the attention skipping module after low-rank decomposition and compression, specifically including the following steps: Integrating the above expressions , , , , and The operator GA is obtained, specifically: in, To query point clouds, The right-multiplied feature dimension transformation moment, This indicates that the operator GA needs to be of a point size of and Attention calculation is performed between two point clouds. To query the point cloud, multiply it by the factor matrix on the left. By calling the GA operator, the output feature matrix of the attention skipping module is obtained, specifically: in, Need to complete from Attention calculation for each point; The feature matrix is ​​a downsampled matrix. , and These represent the number of point clouds and the feature dimension downsampled from the original data to the current scale; the features of global points. , and These represent the number of points and feature dimensions of the original data, respectively. For size The score matrix represents and Point-to-point similarity between them; To be applied to Linear transformation matrix on; The score matrix is ​​decomposed into a low-rank matrix as follows: The linear transformation matrix is ​​decomposed into its lowest rank as follows: in, represent and Point-to-point similarity between them; represent and Point-to-point similarity between them; To apply the low-rank decomposition method after using the low-rank decomposition method Linear transformation matrix on; To apply the low-rank decomposition method after using the low-rank decomposition method Linear transformation matrix on; The global representative point cloud has the following number of points. The rank of the low-rank decomposition in formulas (5) and (6), and ; Substituting the score matrix formula (5) and the linear transformation matrix formula (6) after low-rank decomposition into the output feature matrix (4) of the attention jump module, we obtain the output feature matrix of the attention jump module after low-rank decomposition compression, which is as follows: 。 7. An electronic device, characterized in that, include: Memory, processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code, which, when executed by a computing device, performs the steps of the method according to any one of claims 1 to 6.

Images