Efficient lightweight point cloud analysis method based on point-voxel attention mechanism

Abstract

The invention discloses an efficient lightweight point cloud analysis method based on a point-voxel attention mechanism, and aims to realize better precision compared with a mainstream attention method at an efficient reasoning speed and fewer calculation and storage resources. The method comprises the following five key steps of: firstly, converting an input feature into a high-dimensional space through a point embedding layer improved by local neighborhood information; 2, dividing the point cloud space into limited voxel blocks; thirdly, learning a group of smaller voxel feature sets depicting point cloud shapes by adopting a voxel encoder constructed by a self-attention layer; fourthly, calculating offset attention between a point set and a voxel feature set based on a point-voxel attention module, and capturing global shape features with approximate linear complexity relative to the input size; and finally, introducing a local compensation module to enhance the perception of each point on the local environment. According to the method, the accuracy of point cloud analysis is improved while high efficiency and light weight are ensured.

Description

Technical Field

[0001] This invention belongs to the field of 3D visual perception technology, specifically relating to an efficient and lightweight point cloud analysis method based on a point-voxel attention mechanism. Background Technology

[0002] Point clouds are a representative type of geometric data. Compared to traditional two-dimensional images, they can accurately record the three-dimensional surface structure of objects from different angles using three-dimensional coordinates and related features (such as color), providing richer geometric information. Therefore, they are widely used in fields such as urban spatial reconstruction, augmented reality, and autonomous driving. In recent years, the rapid development of 3D acquisition devices has stimulated research interest in point cloud analysis within the visual community. However, as a set of points embedded in continuous space, point clouds inherently possess irregular and disordered characteristics, which differ significantly from grid-structured images. Therefore, it is impossible to apply deep learning models based on standard convolution in computer vision to process point clouds.

[0003] PointNet, a pioneering work in point cloud deep learning, employed symmetric functions and a multilayer perceptron (MLP) to directly process point clouds, opening up new research directions for deep learning in point cloud analysis. Subsequently, PointNet++ proposed a set of abstract modules to construct a hierarchical feature learning architecture, where farthest-distance sampling and PointNet are used to define the centroid of local regions and encode the patterns of those regions, respectively. However, MLP-based models can only embed each point individually into a high-dimensional space, neglecting the interactions between points and limiting the analysis of complex scenes.

[0004] As the core of the Transformer, the attention mechanism has revolutionized the entire field of natural language processing by capturing long-range dependencies and considering interactions between elements, producing impressive results in image neighborhoods. Furthermore, the attention mechanism is inherently a set operator naturally applicable to point sets (similar to MLPs, both are invariant to the cardinality and permutation of the input, a crucial property for point cloud processing), making it a promising research direction in point cloud analysis. However, the attention mechanism has quadratic complexity in time and space relative to element cardinality, making its direct application to point clouds containing a large number of points computationally unfeasible. Therefore, existing attention-based point cloud models cleverly apply the attention mechanism to the local neighborhood of each point, achieving excellent results on various point cloud tasks. However, the large cardinality of the input point set and the lack of computational sharing between overlapping neighborhoods lead to significant resource overhead. Summary of the Invention

[0005] In terms of technical implementation, this invention studies an efficient and lightweight method by applying an attention mechanism to voxels. It transforms attention computation from focusing on the local neighborhood of all points to focusing on a finite voxel space, significantly reducing time and computational overhead while improving the accuracy of point cloud analysis. Unlike general attention-based voxelization methods that focus on a point's perception of its own voxel space, VoxelFormer considers the voxel's representation of the entire point cloud through the attention mechanism. This represents a novel application perspective of voxelization within the context of attention mechanisms.

[0006] Since the encoding effect of the voxel encoder on the local voxel space directly determines the quality of the voxel feature set in depicting the overall point cloud shape, and thus affects the acquisition of global information of points, this invention enhances it by enriching the input semantics. The point embedding layer based on MLP is improved by using the local neighborhood information of each point. This not only endows the structural semantics formed by each point and its surrounding neighbors, but also extends the receptive field of the voxel encoder to the surrounding voxel space, mitigating the information degradation in fine-grained local accuracy caused by the subsequent voxelization fragmentation of the overall shape.

[0007] Furthermore, the utilization of local structure has proven to be very important for point cloud analysis, so this invention introduces a local compensation module, in which voxel features are correspondingly connected to point features, further enhancing each point's ability to perceive its local environment.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] An efficient and lightweight point cloud analysis method based on a point-voxel attention mechanism includes the following steps:

[0010] Step 1: The core of this step lies in improving the point embedding layer based on the multilayer perceptron by leveraging the local neighborhood information of each point, thereby transforming the input features into a high-dimensional space and enriching the input semantics. This not only endows the structural semantics formed by each point and its surrounding neighbors but also extends the receptive field of the voxel encoder to the vicinity of the voxel space, mitigating the information degradation in fine-grained local accuracy caused by the subsequent voxelization that severs the overall shape.

[0011] In step 1, the point embedding layer improved by local neighborhood information embeds the point set. ( and They represent point p respectively i p in the three-dimensional coordinates and related features i Convert to

[0012]

[0013] In this context, MP is the max pooling layer, γ and β are two multilayer perceptrons, RP(p,t) represents forming a matrix by repeating vector p t times, |·| represents the cardinality of the set, and ·||· and concat represent the connection operations at the channel level and point level, respectively. It is p i The set of neighboring points in P, which is determined by the coordinates c i The sphere with radius r is obtained by querying. The effectiveness of step 1 lies not only in enriching the input semantics of the voxel encoder, but also in expanding the receptive field of the voxel encoder by points located at the edge of the voxel space. This alleviates the information degradation in fine-grained local accuracy caused by the subsequent voxelization that cuts off the overall shape.

[0014] Step 2: Divide the 3D space into non-overlapping, equally spaced voxels, and group them based on the coordinates of each point in the point set. The subset of points falling within the corresponding voxel represents the voxel, thus constructing a finite voxel block that reflects the spatial characteristics.

[0015] In step 2, voxelization first divides the 3D space into non-overlapping, equally spaced voxels. Assuming the point cloud encompasses spaces of ranges H, W, and L along the Z, Y, and X axes respectively, this step correspondingly sets the size of each voxel to h, ω, and l, thus forming H' = H / h, W' = W / ω, and L' = L / l cylinders along the Z, Y, and X axes respectively. For simplicity, H, W, and L are multiples of h, ω, and l, respectively. Then, based on the coordinates of each point in the point set, groups are formed, and a subset of points falling within the corresponding voxel represents the voxel.

[0016] Step 3: A voxel encoder constructed by stacking multiple self-attention layers encodes voxel blocks that reflect local geometry into voxel features, forming a set of voxel features with a small cardinality that characterizes the overall shape of the point cloud.

[0017] In step 3, the voxel encoder acquires data through a self-attention layer. The information is exchanged in pairs between points. Taking its first self-attention layer as an example, in The calculations are as follows:

[0018]

[0019] in for p i The point embedding in the high-dimensional space is calculated by step 1, where D is... The dimensions, η, κ, and μ, are three multilayer perceptrons.

[0020] The voxel encoder stacks multiple self-attention layers to capture voxels. Higher-order interactions in, and Encoding as voxel features s reflecting local geometry t,m,n All voxel features construct a voxel feature set S, and the elements in S (|S|=H'×W'×L') are reindexed. The specific calculation process is as follows:

[0021]

[0022] Where τ is the multilayer perceptron and t is the number of self-attention layers. In particular, when the voxels are empty, they are encoded as zero vectors.

[0023] Step 4: Calculate the offset attention between the point set and the voxel feature set using the point-voxel attention module, so that the point set can capture global information with an approximately linear complexity relative to the input size.

[0024] In step 4, the point-voxel attention layer calculates the offset attention between the point set and the voxel feature set, enabling the point set to capture global information from the voxel feature set. Taking the first point-voxel attention layer as an example, the point set... Taking the voxel feature set S as input, first calculate Attention output between S and

[0025]

[0026] in ψ and ψ are two multilayer perceptrons, and D is... Dimensions.

[0027] Then, this step uses element-wise subtraction to calculate. and The difference between the two is used to input the difference into the MLP for residual connections to help information propagate to higher layers, and finally the output of the offset attention is used. as follows:

[0028]

[0029] Where ω is MLP This is the output of the point-voxel attention layer.

[0030] We constructed a point-voxel attention module by stacking multiple point-voxel attention layers. The point set fully captures global information by interacting with the voxel feature set multiple times within the module. Compared to directly performing self-attention layers on the point set, which achieves the unaffordable global information capture, the point-voxel attention module utilizes a smaller set of voxel features that characterizes the point cloud geometry to achieve the former's... Complexity reduced to Since |S| << N, this is approximately a linear complexity. Furthermore, although constructing the voxel feature set incurs additional overhead, the overall efficiency is significantly improved due to the small value of |S|.

[0031] Step 5: Based on the local compensation module, connect the voxel features with the point features accordingly to enhance the perception of each point with respect to its local environment.

[0032] In step 5, the local compensation module connects voxel features to point features accordingly, enabling each point to perceive its local environment. Although the voxel feature set consists of voxel features reflecting local geometry, some local characteristics are blurred when the point set interacts with the voxel feature set in the point-voxel attention module to capture global information. Numerous studies have shown that local information is crucial for point cloud analysis, therefore the local compensation module enhances local modeling capabilities. Specifically, at point p... i The calculations are as follows:

[0033]

[0034] Where φ is the multilayer perceptron, and F i The output F of the representative point-voxel attention module corresponds to p i The i-th element, It is a function that maps the point index i to its corresponding voxel feature indexes t, m, n.

[0035] The beneficial effects of the point cloud analysis method based on the point-voxel attention mechanism that combines rich input semantics and enhanced local perception provided by this invention are as follows:

[0036] (1) This invention applies the attention mechanism on voxels to design an efficient and lightweight method, which achieves better accuracy with efficient reasoning and less computation and storage resources by transforming the attention computation from the local neighborhood of all points to the space of finite voxels.

[0037] (2) The present invention constructs a voxel encoder based on a self-attention layer to extract local geometric information in voxel space, and improves its effect by improving the MLP-based point embedding layer and local compensation module to enrich the input semantics and enhance local modeling, respectively.

[0038] (3) This invention innovatively proposes a point-voxel attention module, which cleverly achieves global information capture with an approximate linear complexity relative to the input size through attention interaction between the point set and the voxel feature set with a small cardinality. Attached Figure Description

[0039] To more clearly illustrate the embodiments of this application or existing technical solutions, the accompanying drawings used in the embodiments will be briefly introduced below. The accompanying drawings are only used to provide a further understanding of the present invention.

[0040] Figure 1 This is a diagram illustrating the overall framework of the efficient and lightweight point cloud analysis method based on the point-voxel attention mechanism in this invention.

[0041] Figure 2 This is a flowchart of the voxel encoder in this invention;

[0042] Figure 3 This is a flowchart of the point-voxel attention layer in this invention. Detailed Implementation

[0043] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0044] Guided by the idea of ​​point-voxel attention mechanism, this invention designs an efficient and lightweight point cloud analysis method, the overall framework of which is as follows: Figure 1 As shown. The following is a detailed introduction to the overall framework:

[0045] Step 1, the core of which lies in improving the point embedding layer based on the multilayer perceptron by leveraging the local neighborhood information of each point, thereby transforming the input features into a high-dimensional space and enriching the input semantics. This not only endows the structural semantics formed by each point and its surrounding neighbors but also extends the receptive field of the voxel encoder to the vicinity of the voxel space, mitigating the information degradation in fine-grained local accuracy caused by the subsequent voxelization that fragments the overall shape.

[0046] Step 2: This step divides the 3D space into non-overlapping, equally spaced voxels and groups them based on the coordinates of each point in the point set. A subset of points falling within the corresponding voxel represents the voxel, thus constructing a finite voxel block that reflects the spatial characteristics.

[0047] Step 3: This step uses a voxel encoder constructed from a self-attention layer to encode voxel blocks that reflect local geometry into voxel features, forming a set of voxel features with a small cardinality that characterizes the overall shape of the point cloud.

[0048] Step 4 introduces a point-voxel attention module to compute offset attention between the point set and the voxel feature set. Through efficient interaction between point-level features and voxel grid abstract features, the point set captures global information with an approximately linear complexity relative to the input size.

[0049] Step 5: This step, based on the local compensation module, connects voxel features with point features accordingly, enhancing each point's perception of its local environment. Finally, based on the corresponding task header, the corresponding point cloud analysis task is completed.

[0050] In step 1, the point embedding layer based on local neighborhood information enhancement embeds the point set. (in Point p i The three-dimensional coordinates Each point p in (representing its relevant features) i Convert to enhanced representation The conversion method is as follows:

[0051]

[0052] Where MP represents a max pooling layer, γ and β are two multilayer perceptrons, RP(p,t) represents repeating vector p t times to form a matrix, |·| represents the cardinality of the set, and ·||· and concat represent channel-level and point-level connection operations, respectively. By using coordinate c i The point p obtained by performing a sphere query with radius r i The set of neighboring points in the point set P. The design in step 1 not only enhances the semantic information input to the voxel encoder, but also expands the receptive field of the voxel encoder by introducing neighborhood information of points located at the edge of the voxel space. This mechanism effectively alleviates the problem of fine-grained local information degradation caused by the fragmentation of the overall shape in subsequent voxelization operations.

[0053] In step 2, the voxelization process first divides the three-dimensional space into non-overlapping, equally spaced voxels. Assume the point cloud covers a spatial range of height H, width W, and length L along the Z, Y, and X axes, respectively, and set the dimensions of each voxel in the three dimensions as h, ω, and l, respectively. Thus, voxel units are formed along the Z, Y, and X axes: H' = H / h, W' = W / ω, and L' = L / l, respectively. For simplicity, assume H, W, and L are integer multiples of h, ω, and l. Subsequently, based on the coordinates of each point in the point set, they are grouped into corresponding voxels. Each voxel is represented geometrically by all the points within it, i.e.

[0054]

[0055] in, Represents the voxel located at index (t,m,n), I1,…,I K This is the index of the point that falls within this voxel in the original point set.

[0056] In step 3, the voxel encoder captures voxels through a self-attention layer. Paired interaction information between midpoints. Taking its first self-attention layer as an example, for The specific output calculation is as follows:

[0057]

[0058] in, The point p calculated in step 1 i Embedding representation in high-dimensional space, D is The feature dimensions, η, κ, and μ are three multilayer perceptrons.

[0059] Figure 2 This indicates that the voxel encoder stacks multiple self-attention layers to model voxels. The higher-order interaction relationships within the voxel are encoded into voxel features s that reflect the local geometry. t,m,n The specific calculation process is as follows:

[0060]

[0061]

[0062] Here, τ represents the multilayer perceptron, and t represents the number of self-attention layers. Specifically, if a voxel is empty, it is encoded as a zero vector. All voxel features together constitute the voxel feature set S, with a size of |S| = H' × W' × L', and the elements within it are re-indexed.

[0063] In step 4, the point-voxel attention layer calculates the offset attention between the point set and the voxel feature set, enabling the point set to effectively capture global contextual information, such as... Figure 3 As shown. Taking the first point-voxel attention layer as an example, its input is a set of points. First, calculate the voxel feature set S. Attention output between S and

[0064]

[0065] in Let ψ and ψ be two multilayer perceptrons, and D represent... The feature dimensions.

[0066] This step is then performed using element-wise subtraction. and The difference between the values ​​is calculated and fused using residual connections constructed via a multilayer perceptron to facilitate information propagation to higher levels. Finally, the output of the offset attention is... Represented as:

[0067]

[0068] Where ω is a multilayer perceptron. This is the output of the point-voxel attention layer.

[0069] A point-voxel attention module was constructed by stacking multiple point-voxel attention layers. This module enables the point set to interact with the voxel feature set multiple times, thereby fully fusing global information. This is significantly more efficient than directly implementing a self-attention mechanism on the point set, which would be prohibitively expensive. The computational complexity is reduced to a minimum by introducing a voxel feature set S (|S| << N) with a size much smaller than N. The computational complexity is approximately linear. Although constructing the voxel feature set introduces some additional overhead, the overall computational efficiency is still significantly improved due to the small size of |S|.

[0070] In step 5, the local compensation module connects voxel features with corresponding point features, enabling each point to perceive its local geometric environment. For point p... i Its local compensation characteristics are calculated as follows:

[0071]

[0072] Where φ is the multilayer perceptron, and F i This indicates that the output F of the point-voxel attention module corresponds to p. i The i-th element, This is a function that maps point index i to the corresponding voxel feature index (t, m, n).

[0073] Although voxel feature sets inherently contain local geometric information, some local details may be weakened during the global interaction between point sets and voxel feature sets via point-voxel attention modules. Existing research generally indicates that local information plays a crucial role in point cloud analysis. Therefore, the introduction of a local compensation module aims to enhance the modeling ability of local structures, thereby recovering and strengthening the representation of local details based on global perception.

[0074] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A highly efficient and lightweight point cloud analysis method based on a point-voxel attention mechanism specifically includes the following steps: Step 1, the core of which lies in improving the point embedding layer based on the multilayer perceptron by leveraging the local neighborhood information of each point, thereby transforming the input features into a high-dimensional space and enriching the input semantics. This not only endows the structural semantics formed by each point and its surrounding neighbors but also extends the receptive field of the voxel encoder to the vicinity of the voxel space, mitigating the information degradation in fine-grained local accuracy caused by the subsequent voxelization that fragments the overall shape. Step 2: This step divides the 3D space into non-overlapping, equally spaced voxels and groups them based on the coordinates of each point in the point set. A subset of points falling within the corresponding voxel represents the voxel, thus constructing a finite voxel block that reflects the spatial characteristics. Step 3: This step uses a voxel encoder constructed from a self-attention layer to encode voxel blocks that reflect local geometry into voxel features, forming a set of voxel features with a small cardinality that characterizes the overall shape of the point cloud. Step 4 introduces a point-voxel attention module to compute offset attention between the point set and the voxel feature set. Through efficient interaction between point-level features and voxel grid abstract features, the point set captures global information with an approximately linear complexity relative to the input size. Step 5: This step, based on the local compensation module, connects the voxel features with the point features accordingly, enhancing the perception of each point of its local environment.

2. The efficient and lightweight point cloud analysis method based on point-voxel attention mechanism according to claim 1, characterized in that: In step 1, the point embedding layer improved by local neighborhood information embeds the point set. ( and They represent point p respectively i p in the three-dimensional coordinates and related features i Convert to In this context, MP is the max pooling layer, γ and β are two multilayer perceptrons, RP(p,t) represents forming a matrix by repeating vector p t times, |·| represents the cardinality of the set, and ·||· and concat represent the connection operations at the channel level and point level, respectively. It is p i The set of neighboring points in P, which is determined by the coordinates c i The sphere with radius r is obtained by querying. The effectiveness of step 1 lies not only in enriching the input semantics of the voxel encoder, but also in expanding the receptive field of the voxel encoder by points located at the edge of the voxel space. This alleviates the information degradation in fine-grained local accuracy caused by the subsequent voxelization that cuts off the overall shape.

3. The efficient and lightweight point cloud analysis method based on point-voxel attention mechanism according to claim 1, characterized in that: In step 2, voxelization first divides the 3D space into non-overlapping, equally spaced voxels. Assuming the point cloud encompasses spaces of ranges H, W, and L along the Z, Y, and X axes respectively, this step correspondingly sets the size of each voxel to h, ω, and ω. Then, on the Z, Y, and X axes, H' = H / h, W' = W / ω, and... A column. For simplicity, H, W, and L are h, ω, and ω, respectively. Multiples of. Then, based on the coordinates of each point in the point set, groups are formed, and a subset of points falling within the corresponding voxel is used to represent the voxel, i.e.

4. The efficient and lightweight point cloud analysis method based on point-voxel attention mechanism according to claim 1, characterized in that: In step 3, the voxel encoder acquires data through a self-attention layer. The information is exchanged in pairs between points. Taking its first self-attention layer as an example, in The calculations are as follows: in for p i The point embedding in the high-dimensional space is calculated by step 1, where D is... The dimensions, η, κ, and μ, are three multilayer perceptrons. The voxel encoder stacks multiple self-attention layers to capture voxels. Higher-order interactions in, and Encoding as voxel features s reflecting local geometry t,m,n The calculation process is as follows: Where τ is the multilayer perceptron and t is the number of self-attention layers. Specifically, when a voxel is empty, it is encoded as a zero vector. All voxel features construct the voxel feature set S, and the elements in S (|S|=H'×W'×L') are reindexed.

5. The efficient and lightweight point cloud analysis method based on point-voxel attention mechanism according to claim 1, characterized in that: In step 4, the point-voxel attention layer calculates the offset attention between the point set and the voxel feature set, enabling the point set to capture global information from the voxel feature set. Taking the first point-voxel attention layer as an example, the point set... Taking the voxel feature set S as input, first calculate Attention output between S and in ψ and ψ are two multilayer perceptrons, and D is... Dimensions. Then, this step uses element-wise subtraction to calculate. and The difference between the two is used to input the difference into a multilayer perceptron for residual connections to help information propagate to higher layers, and finally the output of the shift attention is used. as follows: Where ω is a multilayer perceptron. This is the output of the point-voxel attention layer. We stack point-voxel attention layers to construct a point-voxel attention module. The point set fully captures global information by interacting with the voxel feature set multiple times in the module. Compared with directly performing self-attention layers on the point set to achieve unaffordable global information capture, the point-voxel attention module uses a smaller set of voxel features that characterize the point cloud geometry to reduce the complexity to Since |S| << N, this is approximately a linear complexity. In addition, although constructing the voxel feature set incurs additional overhead, due to the small |S|, the overall efficiency is significantly improved.

6. The efficient and lightweight point cloud analysis method based on point-voxel attention mechanism according to claim 1, characterized in that: In step 5, the local compensation module connects the voxel features to the point features accordingly, enabling each point to perceive its local environment. At point p... i The calculations are as follows: Where φ is the multilayer perceptron, and F i The output F of the representative point-voxel attention module corresponds to p i The i-th element, This is a function that maps a point index i to its corresponding voxel feature indices t, m, n. Although the voxel feature set consists of voxel features reflecting local geometry, some local characteristics are blurred when the point set interacts with the voxel feature set in the point-voxel attention module to capture global information. Numerous studies have shown that local information is crucial for point cloud analysis, so the local compensation module enhances local modeling capabilities.

Images