Three-dimensional scene point cloud segmentation enhancement method and device, storage medium and program product

Abstract

The invention discloses a three-dimensional scene point cloud segmentation enhancement method and device, a storage medium and a program product, and belongs to the technical field of point cloud segmentation methods. According to the invention, on the basis of an attention mechanism, through designing a good attention jumping module, global information is effectively incorporated, the segmentation precision is improved, and the problems that jumping connection processing is simple and fine-grained texture information in point cloud data cannot be comprehensively captured in an existing three-dimensional point cloud segmentation method based on a U-Net framework are effectively solved; meanwhile, a low-rank global attention jump module is introduced in a model compression method based on low-rank attention decomposition, so that the calculation burden caused by the calculation density of the attention jump module when a complex point cloud scene is processed is relieved and processed, and the model efficiency is improved. Meanwhile, the three-dimensional scene point cloud segmentation enhancement equipment provided by the invention can be seamlessly inserted into an existing U-Net framework to replace a traditional jumping connection module without any additional work.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud segmentation methods, and in particular to a three-dimensional scene point cloud segmentation enhancement method, device, storage medium and program product. Background Art

[0002] Large-scale point cloud data of three-dimensional indoor space is usually used for research and application in the fields of computer vision and machine learning. As the core resource of three-dimensional perception technology, its application range is extremely wide, covering multiple key areas from indoor navigation and positioning, object recognition and segmentation, to environmental modeling, space planning and management. The following are some application scenarios of three-dimensional point cloud data: 1. Indoor navigation and positioning: Point cloud data provides a solid foundation for building high-precision indoor maps, helping to develop advanced navigation and positioning systems, serving cutting-edge fields such as robot navigation, smart home control, drone indoor operations and augmented reality experience. 2. Three-dimensional object recognition and segmentation: Using point cloud data training and optimization algorithms can significantly improve the recognition and segmentation capabilities of complex objects in indoor environments, which is crucial for applications such as autonomous driving assistance, intelligent security monitoring and environmental monitoring. 3. Indoor environment analysis: Through point cloud data analysis, the fine division of indoor space, furniture layout recognition and environmental feature extraction can be achieved, providing strong support for interior design optimization, building intelligent management and virtual reality simulation. 4. Deep learning research: As a valuable resource in deep learning research, 3D point cloud data has promoted the innovation and development of neural network architecture, especially in the training and testing of models such as convolutional neural networks (CNN) and recurrent neural networks (RNN). 5. Infrastructure planning and management: With the help of point cloud data, urban planning can more accurately plan and manage urban internal space, including building layout, transportation network optimization, and public service facility configuration.

[0003] With its unique three-dimensional Cartesian coordinate system, three-dimensional point cloud data accurately captures and represents objects and their properties in the real world, including multiple information such as color and reflection intensity. Compared with two-dimensional images, point cloud data exhibits more complex characteristics - sparsity and disorder. Even multiple observations of the same object may have significant differences in its point cloud representation. These characteristics require researchers to develop processing methods specifically for three-dimensional point clouds to fully tap their potential value.

[0004] With the continuous advancement of technology, 3D point cloud data has shown great potential in cutting-edge fields such as autonomous driving, product quality inspection, and robotics. In particular, the importance of point cloud segmentation technology has become increasingly prominent in these applications. Point cloud segmentation aims to divide complex point cloud data into a series of subsets with clear semantic meanings. This process not only helps to improve the understanding and processing efficiency of point cloud information, but is also a key prerequisite for realizing advanced application functions (such as obstacle recognition, path planning, etc.).

[0005] At present, scholars have proposed many methods for point cloud segmentation tasks, among which the method based on the U-Net structure stands out for its excellent performance. As a classic neural network architecture, U-Net is famous for its unique U-shaped design. The architecture cleverly combines three key components: encoder, decoder, and jump connection to achieve deep feature extraction and accurate segmentation of image or point cloud data. The encoder module captures rich context and spatial information from the input data through multiple feature aggregation layers and downsampling layers, while reducing the spatial size of the data and increasing the number of feature channels so that the network can capture global context and higher-level features. The decoder module uses feature interpolation layers and upsampling layers to gradually restore the spatial dimension of the data to its original size, and accurately locates it based on the coarse features provided by the encoder. The jump connection module plays a vital role in the whole process. It connects the corresponding layers between the encoder and the decoder to ensure the effective fusion of low-level spatial information and high-level context information, thereby retaining complex details and achieving accurate segmentation.

[0006] U-Net-based methods have achieved remarkable results in image segmentation due to deep feature extraction on encoders and decoders. For example, PointNet++ innovatively learns different fine-grained features by layered aggregation and sampling of point clouds, making it one of the earliest point cloud segmentation networks using a U-shaped framework. KPConv uses deformable convolution operators. These operators allow the convolution kernel to adaptively adjust its shape to adapt to various geometric structures present in point cloud data.

[0007] Although U-Net-based methods have achieved remarkable achievements in the field of image segmentation, traditional U-Net methods still face challenges in processing the disorder and sparsity of point cloud data. Due to the lack of a module for extracting global feature information, most existing U-Net-based methods customize the extraction of local and non-local features for image data, which are not applicable to point cloud segmentation tasks. As we all know, point clouds are disordered and sparse, so global features are crucial for high-precision segmentation. Specifically, disordered features refer to the lack of a predetermined arrangement or structure in the point cloud, which means that from the perspective of the connection and relationship between individual points, two points far apart in the point cloud may actually become neighbors of each other; on the other hand, during the data acquisition process, due to the very limited samples, the sparsity of the point cloud makes it difficult for traditional U-Net-based methods to effectively learn good and rich representation features. Therefore, extracting global feature information is critical for processing point cloud data.

[0008] Therefore, how to integrate the global attention mechanism with the U-Net architecture to achieve segmentation enhancement of three-dimensional point cloud data has become a technical problem that technicians in this field need to overcome urgently. Summary of the invention

[0009] The purpose of the present invention is to provide a three-dimensional scene point cloud segmentation enhancement method, device, storage medium and program product to overcome the problem that the three-dimensional point cloud segmentation method based on the U-Net framework in the prior art cannot fully understand the fine-grained texture in the point cloud data due to the relatively simple processing of jump connections.

[0010] The present invention solves the above technical problems through the following technical solutions:

[0011] A three-dimensional scene point cloud segmentation enhancement method comprises the following steps:

[0012] Step 1: Obtain the point cloud data to be segmented, and obtain the input feature F of the current sample set through encoding processing in and the feature F of the global point glb ;

[0013] Step 2: Input feature F of the current sample set in and the feature F of the global point glb Perform orderly extraction layer by layer and perform linear transformation to generate query matrix Q, key matrix K and value matrix V;

[0014] Step 3: Calculate the attention weights of the query matrix Q and the key matrix K through matrix dot product to obtain the attention weight matrix A;

[0015] Step 4: By using the corresponding attention weights in the attention weight matrix A to weighted sum the value vectors in the value matrix V, we get the output feature F of the attention layer. attn ;

[0016] Step 5: Based on the principle of residual network, fuse the output features F of the attention layer attn And the input feature F of the current sample set in , randomly discard network branches to prevent overfitting, and obtain the output feature F of the attention jump module out ;

[0017] Step 6: Output feature F of the attention jump module out Fuse with the output features of the deeper attention jump module to enhance the 3D scene point cloud segmentation.

[0018] Furthermore, the query matrix Q, key matrix K and value matrix V in step 2 are respectively:

[0019] Q=F in ·W q +δ in (4-1)

[0020] K=F glb ·W k +δ glb (1-5)

[0021] V=F glb ·W v +δ glb (1-6)

[0022] in, W q , W k , W v are all linear transformations that can be shared and learned, N o is the number of points of the original data, N i is the number of point cloud points downsampled from the original data to the current scale, d o is the feature dimension of the original data, and the dimension of the query vector is d a , the dimension of keyword vector and value vector are both d i ,δ represents the position embedding, which is obtained by nonlinearly transforming the coordinates of the corresponding input points,δ in is the position embedding of the input point, δ glb is the position embedding of the global point.

[0023] Furthermore, the attention weight matrix A in step 3 is specifically:

[0024]

[0025] Among them, K T is the transpose of the keyword matrix K, is the scaling factor, and SoftMax() is the row normalization matrix function.

[0026] Furthermore, the output feature F of the attention layer in step 4 attn Specifically:

[0027] F attn =A·V (2-2)

[0028] Furthermore, the output feature F of the attention jump module in step 5 out Specifically:

[0029] F out =LBR(F attn )+F in (2-3)

[0030] Among them, LBR() represents linear transformation, batch normalization and ReLu function.

[0031] Furthermore, step six can be replaced by the following process: based on the query matrix Q, the key matrix K, the value matrix V, the attention weight matrix A, and the output feature F of the attention layer attn And the output feature F of the attention jump module out , the output feature matrix of the attention jumping module after low-rank decomposition and compression is obtained, and then the output feature matrix of the attention jumping module after low-rank decomposition and compression is fused with the output feature matrix of the attention jumping module after low-rank decomposition and compression in a deeper layer to complete the enhancement of the three-dimensional scene point cloud segmentation.

[0032] Furthermore, in step 6, the output feature matrix of the attention jump module after low-rank decomposition and compression is obtained, which specifically includes the following steps:

[0033] Integrating the above expressions (1-1), (1-2), (1-3), (2-1), (2-2) and (2-3), we get the operator GA, which is:

[0034]

[0035] Among them, F l is the queried point cloud, F r To query the point cloud, is the feature dimension transformation moment of the right multiplication, Indicates that the operator GA needs to have a point size of N l and N r Attention calculation is performed between two point clouds. Multiply the query point cloud by the factor matrix;

[0036] Call the operator GA to get the output feature matrix of the attention jump module, which is:

[0037]

[0038] in, Indicates that the corresponding GA operator needs to complete the i Attention calculation to N0 points; F i is the down-sampled feature matrix, N i and C i They are the number of point clouds and feature dimensions downsampled from the original data to the current scale; the features of the global points N o and C o are the number of points and feature dimensions of the original data respectively; For size N i ×N0 score matrix, representing F i and F glb The pointwise similarity between ; To be applied to F glb The linear transformation matrix on ;

[0039] The score matrix is ​​decomposed into the following low-rank:

[0040]

[0041] The linear transformation matrix is ​​decomposed into the following low rank:

[0042]

[0043] in, Represents F i and F fps The pointwise similarity between ; Represents F fps and F glb The pointwise similarity between ; After using the low-rank decomposition method, it is applied to F glb The linear transformation matrix on ; After using the low-rank decomposition method, it is applied to F fps The linear transformation matrix on fps is the global representative point cloud, with N points fps corresponds to the rank of the low-rank decomposition in formulas (5) and (6), and N fps < <min{N i , N o};

[0044] Substitute the score matrix formula (5) after low-rank decomposition and the linear transformation matrix formula (6) after low-rank decomposition into the output feature matrix (4) of the attention jump module to obtain the output feature matrix of the attention jump module after low-rank decomposition compression, which is:

[0045]

[0046] An electronic device, comprising: a memory, a processor;

[0047] Memory for storing computer programs;

[0048] A processor is used to execute a computer program to implement the steps of the above-mentioned three-dimensional scene point cloud segmentation enhancement method.

[0049] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions, which are used to implement the steps of the above-mentioned three-dimensional scene point cloud segmentation enhancement method when executed by a processor.

[0050] A computer program product, characterized in that the computer program product includes computer program code, and when the computer program code is executed by a computing device, the computing device executes the steps of the above-mentioned three-dimensional scene point cloud segmentation enhancement method.

[0051] Compared with the prior art, the positive progress of the present invention lies in: the three-dimensional scene point cloud segmentation enhancement method provided by the present invention is a method based on the attention mechanism, which effectively incorporates global information and improves segmentation accuracy by designing a well-designed attention jump module, and effectively solves the problem that the jump connection processing in the existing three-dimensional point cloud segmentation method based on the U-Net framework is simple and cannot fully capture the fine-grained texture information in the point cloud data; at the same time, the model compression method based on low-rank attention decomposition introduces a low-rank global attention jump module to reduce the computational burden caused by the computational intensiveness of the attention jump module when processing complex point cloud scenes, thereby improving the model efficiency.

[0052] The 3D scene point cloud segmentation enhancement device provided by the present invention can be inserted into the existing U-Net framework to replace the traditional "jump connection module". The global features can be easily fused with the local and non-local features from the encoder and decoder through the attention jump module, thereby obtaining a better point cloud representation. Moreover, the replacement of the original jump connection is seamless and does not require any additional work. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings in the specification are used to provide further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0054] Figure 1 Schematic diagram of the U-Net framework of the present invention;

[0055] Figure 2 The data stream of the attention jump module of the present invention;

[0056] Figure 3 The data flow of the attention layer of the present invention;

[0057] Figure 4 Schematic diagram of the point cloud receptive field in the global attention jump module of the present invention;

[0058] Figure 5 Schematic diagram of the point cloud receptive field in the low-rank global attention jump module of the present invention;

[0059] Figure 6 Schematic diagram of the model data flow after the low-rank global attention jump module is seamlessly inserted into the U-Net framework.

[0060] Figure 7 Schematic diagram of the visual experimental results after seamlessly inserting the global attention jump module into three U-Net frameworks.

[0061] Figure 8 Schematic diagram of the visual experimental results after seamlessly inserting the low-rank global attention jump module into three U-Net frameworks. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0065] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0066] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0067] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] The present invention is further described in detail below in conjunction with the accompanying drawings, which are intended to explain the present invention rather than to limit it.

[0069] Since the model needs to easily identify and distinguish each point during processing, it often needs to convert its three-dimensional coordinates into high-dimensional vectors. This dimensionality increase task is called encoding, and the module that completes this task is called an encoder. This dimensionality increase process may be gradual, so the encoder is usually a serial structure. When the model completes feature recognition, it usually needs to restore the features, that is, reduce the dimension. This process is called decoding, and the module that completes this process is called a decoder, and the decoder is usually also a hierarchical serial structure.

[0070] See also Figure 1 The structure of the U-Net framework consists of three parts: encoder (blue), decoder (orange) and skip connection (red). According to the direction of the arrow, each module receives information, processes it internally, and then passes it to the subsequent module, and finally outputs the category label of each point in the point cloud for the network, that is, completing the semantic segmentation of the point cloud as a whole. The skip connection module is an important part of it, serving as a bridge between the encoder and the decoder. An overly simple processing process will lead to semantic deviations between different modules, which will directly affect the output results of the network.

[0071] A given point cloud data is represented as a set P, where P = {(F i ,x i )|i=1,···,N}∈R N×6 , where F i represents the coordinate information of the i-th point in the point cloud in the three-dimensional Euclidean space, x i Indicates the RGB color information of the i-th point. In the encoder part, the point cloud data P undergoes multiple downsampling (the sampling method is the farthest point sampling method commonly used in point cloud processing) and feature dimension upscaling to obtain the point cloud coordinates and features of each layer "Point cloud data" refers to the three-dimensional coordinates (three-dimensional vector) of each point, and in order to facilitate the identification and distinction of each point during model processing, it is often necessary to convert this three-dimensional coordinate into a high-dimensional vector. "Point cloud features" refer to this high-dimensional vector. That is, only the initial three-dimensional coordinates are called "point cloud data", and after dimensionality increase processing, they are uniformly called "point cloud features".

[0072] In each layer, the decoder receives features propagated from subsequent layers and the features from the corresponding encoder layer For example Figure 1 In The decoder of this layer (orange circle) receives the output of the decoder of the lower layer And the same layer skip connection output The two parts of information are obtained through internal processing. Note that the skip connection features from the same layer Important low-level feature information is passed to the decoder to retain detailed information. Finally, the point cloud is downsampled, upsampled, and feature fused to output the classification label of each point at the highest level to complete the task of semantic segmentation.

[0073] Traditional methods use multi-layer perceptron (MLP) modules to directly operate on jump features. Due to the importance of global features, the present invention provides a 3D scene point cloud segmentation enhancement method, which aims to utilize a universal jump connection module to enhance the collaboration between the encoder and the decoder. This enhancement method can be seamlessly integrated into the U-Net framework as a plug-and-play tool.

[0074] Embodiment 1

[0075] A three-dimensional scene point cloud segmentation enhancement method comprises the following steps:

[0076] Step 1: Obtain the point cloud data to be segmented, and obtain the input feature F of the current sample set through encoding processing in and the feature F of the global point glb ;

[0077] Step 2: Input feature F of the current sample set inand the feature F of the global point glb Perform orderly extraction layer by layer and perform linear transformation to generate query matrix Q, key matrix K and value matrix V;

[0078] Step 3: Calculate the attention weights of the query matrix Q and the key matrix K through matrix dot product to obtain the attention weight matrix A;

[0079] Step 4: By using the corresponding attention weights in the attention weight matrix A to weighted sum the value vectors in the value matrix V, we get the output feature F of the attention layer. attn ;

[0080] Step 5: Based on the principle of residual network, fuse the output features F of the attention layer attn And the input feature F of the current sample set in , randomly discard network branches to prevent overfitting, and obtain the output feature F of the attention jump module out ;

[0081] Step 6: Output feature F of the attention jump module out Fuse with the output features of the deeper attention jump module to enhance the 3D scene point cloud segmentation.

[0082] Specifically, the attention jump module is a global attention jump module, and the attention layer is a global attention layer, see Figure 2 The global attention jump module (ASBlock) is composed of a series of interconnected global attention layers (GAlayers). ASBlock takes the features of the current sampling point (current sample set) and the initial features of the original point (global point) as combined inputs, and uses a series of GAlayers in the global attention jump module to perform orderly feature extraction on these combined inputs. Each GAlayer is designed to capture the relationship between points and the global background containing all points. The output features of each GAlayer are integrated in series to form a comprehensive representation. Multiple outputs are put together and after the subsequent nonlinear fusion module, the output is the jump connection output feature of the layer ( Figure 1 In ) Then, these concatenated features ( Figure 2 The features combined at the middle circle C) are fused through the LBR (consisting of linear transformation, batch normalization, and Relu function) module to enhance the model's effective information synthesis ability. In order to further enhance the generalization ability, the Dropout module is also applied. The Dropout module is a common module in deep learning. It prevents overfitting by randomly discarding network branches during training, thereby improving the stability and adaptability of the learned features.

[0083] The data flow in the global attention layer is as follows Figure 3 As shown, GAlayer receives two inputs: the feature F of the current sample set in and the feature F of the global point glb , both inputs correspond to specific scales; where the feature F in represents the output of the encoder from the previous layer, and F glb Contains all point features of the complete point cloud P.

[0084] A linear transformation is performed to generate the query matrix (Q), the key matrix (K), and the value matrix (V) in the following manner:

[0085]

[0086] in, W q , W k , W v are all linear transformations that can be shared and learned, N o is the number of points of the original data, N i is the number of point cloud points downsampled from the original data to the current scale, d o is the feature dimension of the original data, and the dimension of the query vector is d a , the dimension of keyword vector and value vector are both d i ,δ represents the position embedding, which is obtained by nonlinearly transforming the coordinates of the corresponding input points,δ in is the position embedding of the input point, δ glb is the position embedding of the global point.

[0087] The attention weights of the query matrix and the key matrix are calculated by matrix dot product. In order to solve the problem related to gradient disappearance and improve the stability of training, the attention weights of the query matrix and the key matrix are calculated by dividing by d a The square root of is used to scale the dot product result. This gives the attention score matrix A, and the output feature F of the attention layer attn It is obtained by weighted summing the value vectors using the corresponding attention weights in matrix A.

[0088] The specific description is as follows:

[0089]

[0090] F attn =A·V (2)

[0091] F out =LBR(F attn )+F in

[0092] Among them, K T is the transpose of the keyword matrix K, is the scaling factor, SoftMax() is the row normalization matrix function; LBR() represents linear transformation, batch normalization and ReLu function.

[0093] It is worth noting that both the Softmax and weighted sum operations are invariant to permutation, ensuring that the global attention layer remains invariant to the point order. Therefore, the entire global attention process is permutation invariant, which makes it very suitable for handling the disordered and irregular nature of point cloud data. Finally, using the principle of residual network, the output features F of the attention layer are integrated attn And the input feature F of the current sample set in , to generate the output feature F of the entire attention jump module out .

[0094] The output feature F of the attention jump module is then converted to out The feature fusion is performed with the output features of the attention jump module of the deeper decoder to enhance the 3D scene point cloud segmentation.

[0095] Due to the computational intensiveness of the attention jump module, it will bring unbearable computational workload when processing complex point cloud scenes. Therefore, the present invention also proposes a method based on low-rank decomposition to compress the attention jump module, see Example 2 for details.

[0096] Embodiment 2

[0097] Matrix low-rank decomposition is a common matrix decomposition technique used to decompose a matrix M into the product of two or more low-rank matrices. This decomposition is usually used for tasks such as reducing the dimension of data, denoising, and extracting features. Suppose there is a matrix A∈R m×n , the goal of matrix low-rank decomposition is to find two matrices M∈R m×k and N∈R m×k , so that A≈MN T , where the ranks of M and N are low, this is the low-rank decomposition of the matrix.

[0098] The difference between this implementation and the first embodiment is that step 6 can be replaced by the following process:

[0099] Based on the query matrix Q, key matrix K, value matrix V, attention weight matrix A, and output feature F of the attention layer attn And the output feature F of the attention jump module out , the output feature matrix of the attention jumping module after low-rank decomposition and compression is obtained, and then the output feature matrix of the attention jumping module after low-rank decomposition and compression is fused with the output feature matrix of the attention jumping module after low-rank decomposition and compression in a deeper layer to complete the enhancement of the three-dimensional scene point cloud segmentation.

[0100] When this implementation is adopted, the attention jump module is a low-rank global attention jump module, and the attention layer is a low-rank global attention layer.

[0101] This implementation method realizes compression of the attention jump module based on matrix low-rank decomposition, and specifically includes the following steps:

[0102] In order to facilitate the description, some necessary symbols are introduced. and Represents point cloud feature matrices of different sizes, with a size of N l ×N r The score matrix is ​​expressed as It stands for F l and F r The pointwise similarity between . Indicates that the F r The linear transformation matrix on . Integrate formulas (1) and (2) into the following matrix operation:

[0103]

[0104] Among them, F l is the queried point cloud, F r To query the point cloud, is the feature dimension transformation moment of the right multiplication, Indicates that the operator GA needs to have a point size of N l and N r Attention calculation is performed between two point clouds. Multiply the query point cloud by the factor matrix;

[0105] The operator GA can be integrated by F l and F r The intrinsic structure information between them is generated to a size of N l ×C feature matrix. Given the down-sampled feature matrix and the unsampled complete global point feature matrix Calling the operator GA can get the output feature matrix of the attention jump module:

[0106]

[0107] Among them, formula (3) is the general form of the GA operator, and formula (4) is the specific form of the GA operator brought into this module; Indicates that the corresponding GA operator needs to complete the i Attention calculation to N0 points; F i is the down-sampled feature matrix, N i and C iare the number of point clouds and feature dimensions downsampled from the original data to the current scale; F glb is the feature of the global point, N o and C o are the number of points and feature dimensions of the original data respectively; For size N i ×N0 score matrix, representing F i and F glb The point-wise similarity between C0×C To be applied to F glb The linear transformation matrix on .

[0108] Since the number of points in the point cloud is N i Both N0 and N1 are very large, and the computational burden of the score matrix is ​​usually unbearable. Inspired by the idea of ​​low-rank modeling in machine learning and data science, the score matrix It can be approximated by multiplying two low-rank factor matrices:

[0109]

[0110] Similarly, the linear transformation matrix It can also be approximated as follows:

[0111]

[0112] in, and All have parameter form; Represents F i and F fps The pointwise similarity between ; Represents F fps and F glb The pointwise similarity between ; After using the low-rank decomposition method, it is applied to F glb The linear transformation matrix on ; After using the low-rank decomposition method, it is applied to F fps The linear transformation matrix on fps is the global representative point cloud, with N points fps corresponds to the rank of the low-rank decomposition in formulas (5) and (6), and N fps <<min{N i ,N o}.

[0113] This low-rank approximation can reduce the storage complexity from O(N i N o +C o C) is greatly reduced to O((N i+N o )N fps +(C o +C)C fps ), substitute formula (5) and formula (6) into formula (4) and use the definition of GA operator (3), and we get:

[0114]

[0115] Among them, formulas (4), (5), and (6) are address decomposition and compression theories, and (7) indicates that in the actual process, an intermediate point cloud is required as a medium.

[0116] From the above equation, we can get the output feature matrix of the attention jump module after low-rank decomposition and compression, which is:

[0117]

[0118] in, For size N i ×N0 score matrix, representing F i and F glb The pointwise similarity between ; Represents F i and F fps The pointwise similarity between ; Represents F fps and F glb The pointwise similarity between ;

[0119] Indicates that the corresponding GA operator needs to complete the i Attention calculation to N0 points; To be applied to F glb The linear transformation matrix on ; After using the low-rank decomposition method, it is applied to F glb The linear transformation matrix on ; After using the low-rank decomposition method, it is applied to F fps The linear transformation matrix on ;

[0120] F i is the down-sampled feature matrix, F glb is the unsampled complete global point feature matrix, F fps is the global representative point cloud, point cloud F fps This is achieved through the famous farthest point sampling method in point cloud processing. Figure 6 As shown, the point cloud F fps In fact, it is the global representative point, and its number N fpsCorresponding to the rank of the low-rank decomposition in formula (5) (6), and N fps < <min{N i ,N0}, that is, the number of points in the global representative point cloud is much smaller than the number of points in the original point cloud, N fps The size of can directly control the degree of compression of low-rank decomposition.

[0121] Based on the low-rank decomposition of the above matrix, the compression of the attention jumping module is completed, and the output feature matrix of the attention jumping module after low-rank decomposition and compression is obtained. The output feature matrix compressed by low-rank decomposition and compression of the attention jumping module is fused with the output feature matrix compressed by low-rank decomposition and compression of the deeper decoder through the same-layer decoder to complete the enhancement of the three-dimensional scene point cloud segmentation.

[0122] 1. Experimental verification of ASBlock and LrASBlock based on small data:

[0123] ShapeNet Part is a common dataset in point cloud. It can complete the task of component segmentation, that is, the component knows the large segmentation of the point cloud data, and also needs to segment its small parts. The ShapeNet Part dataset contains a large number of 3D point cloud models. Each model has its 3D coordinate information and corresponding semantic part labels. There are 16 object classes and 50 component labels in total, including 16,846 samples in total. The samples in this dataset show imbalanced characteristics. For example, Table includes 5,263 samples, while Earphone only has 69 samples. Each sample contains more than 2,000 points and is a small dataset. The dataset contains 12,137 training sets, 1,870 validation sets, and 2,839 test sets, totaling 16,846. And verify its versatility on different models, re-experiment the three models separately, and obtain benchmark results.

[0124] After that, ASBlock (abbreviated as ASB) or LrASBlock (abbreviated as LrASB) are added to them respectively, and the training strategies of the three models are kept unchanged to present the final experimental results, see Table 1.

[0125] Table 1

[0126] Methods cat.mIoU ins.mIoU PointNet++ 82.5 85.4 PointNet+++LrASB 82.5 85.9 PointNet+++ASB 82.6 86.0 CurveNet 83.5 86.4 CurveNet+LrASB 83.7 86.6 CurveNet+ASB 83.8 86.6 PointStack 81.9 87.2 PointStack+LrASB 82.6 87.4 PointStack+ASB 82.8 87.5

[0127] Among them, each item in the Methods column represents a segmentation method. cat.mIoU (instance mIoU) and ins.mIoU (category mIoU) are evaluation indicators of point cloud semantic segmentation results, and the unit is %. The evaluation criteria of both are that the larger the value, the better the segmentation effect. "+LrASB" and "+ASB" respectively indicate that the jump connection is replaced by LrASBlock and ASBlock. For example, PointStack+ASB means that the jump connection of PointStack is replaced by ASBlock, that is, PointStack integrates ASBlock.

[0128] As can be seen from Table 1, after integrating ASBlock, the instance mIoU and category mIoU of PointNet++ increased by 0.1% and 0.6%, respectively. Similarly, the metrics of CurveNet and PointStack increased by 0.3% / 0.2% and 0.9% / 0.3%. These results consistently show that using ASBlock can improve network performance, which confirms the enhancement effect of adopting ASBlock.

[0129] In addition, after compressing ASBlock using the low-rank attention decomposition method, it does not affect the enhancement effect of the module on the original framework, or the loss effect is very small, which also reflects the effectiveness of the attention low-rank decomposition method.

[0130] In terms of qualitative comparison of semantic segmentation results, Figure 7 The difference between the original network and the model (taking PointStack as an example) is shown. It can be clearly seen from the figure that the introduction of ASBlock enhances the network's ability to capture comprehensive target information and alleviates complex errors in specific areas, observing details such as guitar necks, lamp bases, and lampshades.

[0131] 2. Experimental verification of LrASBlock based on large scene data:

[0132] Due to the computational intensiveness of the attention jump module, it will bring an unbearable amount of computation when processing complex point cloud scenes. When faced with large scene data, LrASBlock must be used to overcome the computational difficulty.

[0133] See also Figure 6 , low-rank global attention jump module (LrASBlock), based on the point cloud feature F of the i-th scale in the U-Net network i , and the original complete global point cloud feature F glb As input, formulas (1), (2), (3), and (8) are used as operators to obtain the output of the i-th scale jump module and transmit it to the corresponding i-th scale decoder, ultimately completing the enhancement of the point cloud semantic segmentation effect of the U-Net network framework.

[0134] In the task of indoor large-scene point cloud segmentation (taking S3DIS data as an example), the enhancement effect is better than the existing method and can be seamlessly integrated into the future new point cloud segmentation model based on the U-Net framework.

[0135] The effect of the present invention is tested on the S3DIS dataset. The S3DIS (Stanford 3D Indoor Scene Dataset) dataset is a point cloud dataset created by the Computer Graphics Laboratory of Stanford University to promote indoor scene understanding and reconstruction research. The dataset is collected from 6 areas and contains a total of 272 scenes, which can be specifically divided into 11 different scenes.

[0136] Among them, mIoU, mAcc and OA are the evaluation indicators of point cloud semantic segmentation results. Table 1 shows the average indicators of the corresponding models in 11 scenes, and the unit is %. The evaluation criteria of the three are that the larger the data, the better the segmentation effect.

[0137] The experimental results are recorded in Table 2, where each item in the Methods column represents a segmentation method. The corresponding evaluation index is the average index of the segmentation method in 11 different scenarios. It can be seen that after combining LrASBlock with the low-rank global attention layer, KPConv's mIoU, mAcc and OA increased by 0.8%, 0.9% and 0.1% respectively. The metric indicators of Point Transformer increased by 0.4% / 0.9% (mIoU / mAcc) respectively. After the more powerful Stratified Transformer integrated the low-rank global attention jump connection module, mIoU, mAcc and OA increased by 0.6%, 0.9% and 0.2% respectively.

[0138] Table 2

[0139]

[0140]

[0141] Figure 8 Visualizations of segmentation results for three different regions are shown, arranged from left to right: ground truth, unmodified network, and network with LrASBlock integrated. Particularly noteworthy is the targeted attention to the region surrounded by the black box. In this restricted region, the effect of LrASBlock integration becomes apparent, significantly enhancing the network's overall understanding of the target entity. In addition, the integration of LrASBlock effectively improves the network's tendency to segment incorrectly local points within the object.

[0142] Embodiment 3

[0143] An electronic device includes: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the steps of the above-mentioned three-dimensional scene point cloud segmentation enhancement method.

[0144] Embodiment 4

[0145] A computer-readable storage medium stores computer-executable instructions, which are used to implement the steps of the above-mentioned three-dimensional scene point cloud segmentation enhancement method when executed by a processor.

[0146] Embodiment 5

[0147] A computer program product includes computer program code. When the computer program code is executed by a computing device, the computing device executes the steps of the above-mentioned three-dimensional scene point cloud segmentation enhancement method.

[0148] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A three-dimensional scene point cloud segmentation enhancement method, characterized in that: The following steps are involved: Step 1: Obtain the point cloud data to be segmented, and obtain the input feature F of the current sample set through encoding processing in and the feature F of the global point glb ; Step 2: Input feature F of the current sample set in and the feature F of the global point glb Perform orderly extraction layer by layer and perform linear transformation to generate query matrix Q, key matrix K and value matrix V; Step 3: Calculate the attention weights of the query matrix Q and the key matrix K through matrix dot product to obtain the attention weight matrix A; Step 4: By using the corresponding attention weights in the attention weight matrix A to weighted sum the value vectors in the value matrix V, we get the output feature F of the attention layer. attn ; Step 5: Based on the principle of residual network, fuse the output features F of the attention layer attn And the input feature F of the current sample set in , randomly discard network branches to prevent overfitting, and obtain the output feature F of the attention jump module out ; Step 6: Output feature F of the attention jump module out Fuse with the output features of the deeper attention jump module to enhance the 3D scene point cloud segmentation.

2. A three-dimensional scene point cloud segmentation enhancement method according to claim 1, characterized in that: The query matrix Q, key matrix K and value matrix V in step 2 are: Q=F in ·W q +δ in (1-1) K=F glb ·W k +d glb (1-2) V=F glb ·W v +δ glb (1-3) in, W q , W k , W v are all linear transformations that can be shared and learned, N o is the number of original data points, N i is the number of point cloud points downsampled from the original data to the current scale, d o is the feature dimension of the original data, and the dimension of the query vector is d a , the dimension of keyword vector and value vector are both d i ,δ represents the position embedding, which is obtained by nonlinearly transforming the coordinates of the corresponding input points,δ in is the position embedding of the input point, δ glb is the position embedding of the global point.

3. A three-dimensional scene point cloud segmentation enhancement method according to claim 2, characterized in that: The attention weight matrix A in step 3 is specifically: Among them, K T is the transpose of the keyword matrix K, is the scaling factor, and SoftMax() is the row normalization matrix function.

4. The three-dimensional scene point cloud segmentation enhancement method according to claim 3, characterized in that: The output feature F of the attention layer in step 4 attn Specifically: F attn =A·V(2-2)。 5. A three-dimensional scene point cloud segmentation enhancement method according to claim 4, characterized in that: The output feature F of the attention jump module in step 5 out Specifically: F out =LBR(F attn )+F in (2-3) Among them, LBR() represents linear transformation, batch normalization and ReLu function.

6. A three-dimensional scene point cloud segmentation enhancement method according to claim 5, characterized in that: The step 6 can be replaced by the following process: based on the query matrix Q, the key matrix K, the value matrix V, the attention weight matrix A, the output feature F of the attention layer attn And the output feature F of the attention jump module out , the output feature matrix of the attention jumping module after low-rank decomposition and compression is obtained, and then the output feature matrix of the attention jumping module after low-rank decomposition and compression is fused with the output feature matrix of the attention jumping module after low-rank decomposition and compression in a deeper layer to complete the enhancement of the three-dimensional scene point cloud segmentation.

7. A three-dimensional scene point cloud segmentation enhancement method according to claim 6, characterized in that: In step 6, the output feature matrix of the attention jump module after low-rank decomposition and compression is obtained, which specifically includes the following steps: Integrating the above expressions (1-1), (1-2), (1-3), (2-1), (2-2) and (2-3), we get the operator GA, which is: Among them, F l is the queried point cloud, F r To query the point cloud, is the feature dimension transformation moment of the right multiplication, Indicates that the operator GA needs to have a point size of N l and N r Attention calculation is performed between two point clouds. Multiply the query point cloud by the factor matrix; Call the operator GA to get the output feature matrix of the attention jump module, which is: in, Indicates that the corresponding GA operator needs to complete the N i Attention calculation to N0 points; F i is the down-sampled feature matrix, N i and C i They are the number of point clouds and feature dimensions downsampled from the original data to the current scale; the features of the global points N o and C o are the number of points and feature dimensions of the original data respectively; For size N i ×N0 score matrix, representing F i and F glb The pointwise similarity between ; To be applied to F glb The linear transformation matrix on ; The score matrix is ​​decomposed into the following low-rank: The linear transformation matrix is ​​decomposed into the following low rank: in, Represents F i and F fps The pointwise similarity between ; Represents F fps and F glb The pointwise similarity between ; After using the low-rank decomposition method, it is applied to F glb The linear transformation matrix on ; After using the low-rank decomposition method, it is applied to F fps The linear transformation matrix on fps is the global representative point cloud, with N points fps corresponds to the rank of the low-rank decomposition in formulas (5) and (6), and N fps <<min{N i , N o }; Substitute the score matrix formula (5) after low-rank decomposition and the linear transformation matrix formula (6) after low-rank decomposition into the output feature matrix (4) of the attention jump module to obtain the output feature matrix of the attention jump module after low-rank decomposition compression, which is:

8. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises computer program codes. When the computer program codes are executed by a computing device, the computing device executes the steps of the method according to any one of claims 1 to 7.

Images