Three-dimensional point cloud semantic segmentation method and system based on graph neural network and window attention mechanism, and storage medium

Abstract

The invention relates to the field of three-dimensional point cloud semantic segmentation, and particularly discloses a three-dimensional point cloud semantic segmentation method and system based on a graph neural network and a window attention mechanism, and a storage medium. The point cloud features are changed into three groups of rgb-xyz-xyz information from two groups of rgb-xyz information, so that the subsequent extraction of local geometric information is facilitated; the combination of the graph neural network and the kernel point convolution can extract richer, more real and more effective local information window attention mechanism to extract local context information, and then the local context information is combined with local geometric information extracted by the graph neural network, so that the accuracy of semantic segmentation is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional point cloud semantic segmentation, and specifically to a three-dimensional point cloud semantic segmentation method and system, and a storage medium based on a graph neural network and a window attention mechanism. Background Art

[0002] 3D data has a wide range of applications in different fields, including autonomous driving, robotics, remote sensing, medical and design industries. 3D data can usually be represented in different formats, including depth images, point clouds, meshes, and volumetric meshes. As a commonly used format, point cloud representation retains the original geometric information in 3D space without any discretization. Therefore, point cloud is the preferred representation for many scene understanding related applications (such as autonomous driving and robotics). The semantic segmentation of 3D point clouds is the basis for the semantic understanding and analysis of 3D scenes, and is the key in the fields of navigation positioning, pattern recognition, and unmanned driving.

[0003] The semantic segmentation methods of 3D point clouds are mainly divided into methods based on conventional supervised machine learning and methods based on deep learning.

[0004] Methods based on conventional supervised machine learning can be divided into two categories. One category classifies each point or each point cluster based only on individual features, such as maximum likelihood classifiers based on Gaussian mixture models, support vector machines (SVMs), random forests, and Bayesian discriminant classifiers. The other category consists of statistical context models, such as associative and non-associative Markov networks, conditional random fields (CRFs), simplified Markov random fields (MRFs), multi-level reasoning programs that focus on point cloud statistics and relational information at different scales, and spatial reasoning engines that model the medium- and long-term dependencies inherent in the data. The above algorithms are mainly designed for segmentation based on features for specific tasks, which are suitable for semantic segmentation in simple scenarios. For complex scenarios, it is difficult to find a suitable combination of features.

[0005] Deep learning based methods can be mainly divided into three categories: multi-view based, voxel based and point based.

[0006] The multi-view based method converts 3D point clouds into multi-view Figure 2 D image representation, and then processed based on 2D CNN. Subsequently, the classification result is restored to 3D. Multi-view methods solve the structuring problem of point cloud data well, but these methods have two serious shortcomings. First, the method does not make full use of the underlying geometric and structural information, because the projection step inevitably introduces information loss. Second, its performance is sensitive to viewpoint selection and occlusion, and the multi-view projection image must cover all the spaces containing points. For large and complex scenes, it is difficult to select enough suitable viewpoints for multi-view projection. Therefore, few studies have used multi-view based deep learning architectures for 3D point cloud semantic segmentation.

[0007] Voxel-based methods first divide the point cloud into a set of occupied voxels. Then, a 3D convolutional neural network is used to perform voxel segmentation. Finally, all points within the voxel are assigned the same semantic label as the voxel. However, compared with the point cloud, the resolution of the voxel structure is low, and voxelization inherently leads to information loss. In addition, high resolution will lead to higher memory and computational costs, while low resolution will lead to loss of details. Choosing a suitable resolution is not easy in practice.

[0008] Point-based methods work directly on irregular point clouds without conversion, reducing the loss of geometric information caused by dimensionality conversion and achieving end-to-end training and learning. This method is relatively universal and is currently a hot topic of research.

[0009] In the past few years, attention mechanisms have been gradually introduced into the field of 3D point cloud segmentation and have made significant progress. However, many attention mechanism-based methods ignore the role of local geometric features and make insufficient use of local geometric features, which limits the accuracy of semantic segmentation. Summary of the invention

[0010] The purpose of the present invention is to overcome the problems existing in the prior art and provide a three-dimensional point cloud semantic segmentation method, system and storage medium based on graph neural network and window attention mechanism, which can deeply integrate local geometric features into the window attention mechanism, effectively combine local geometric information and contextual information, improve the accuracy of the current three-dimensional point cloud semantic segmentation algorithm, and enrich the high-accuracy three-dimensional point cloud semantic segmentation algorithm.

[0011] In order to achieve the above object, the first aspect of the present invention provides a three-dimensional point cloud semantic segmentation method based on a graph neural network and a window attention mechanism, comprising the following steps:

[0012] S1, obtain input point cloud;

[0013] S2, reconstruct the input point cloud, swap any two sets of target information positions of the points, and add an additional set of information positions;

[0014] S3. Use the edge convolution in the graph neural network to extract the edge relationship between the point and its adjacent points on the reconstructed point cloud to obtain a point cloud containing the edge relationship. F represents the information dimension of the point, and N represents the number of points;

[0015] S4: Use kernel point convolution to process the input point cloud in step S1, extract local information of the point cloud to obtain the point cloud The point cloud containing the edge relationship And local feature point cloud Add, batch normalization and activation function processing are performed in turn to obtain the input point cloud of the attention mechanism

[0016] S5. Execute the window attention mechanism to obtain context information Will and Fuse together to obtain comprehensive local features;

[0017] S6. Downsample the comprehensive local features under the window attention mechanism, then perform multiple window attentions again, implement decoding operations by upsampling aggregated features, and finally complete semantic segmentation.

[0018] The second aspect of the present invention provides a 3D point cloud semantic segmentation system based on a graph neural network and a window attention mechanism, comprising:

[0019] The input point cloud reconstruction module is used to reconstruct the acquired input point cloud, swap any two sets of target information positions of the points, and add an additional set of information positions;

[0020] The edge convolution extraction module is used to extract the edge relationship between the reconstructed point cloud and its adjacent points using the edge convolution in the graph neural network to obtain a point cloud containing the edge relationship. F represents the information dimension of the point, and N represents the number of points;

[0021] The kernel point convolution processing module is used to process the input point cloud in step S1 using kernel point convolution, extract local information of the point cloud to obtain the point cloud The point cloud containing the edge relationship And local feature point cloud Add, batch normalization and activation function processing are performed in turn to obtain the input point cloud of the attention mechanism

[0022] Window attention mechanism module, used to execute the window attention mechanism and obtain context information Will and Fuse together to obtain comprehensive local features;

[0023] The semantic segmentation module is used to downsample the comprehensive local features under the window attention mechanism, and then perform multiple window attentions again, realize the decoding operation by upsampling the aggregated features, and finally complete the semantic segmentation.

[0024] A third aspect of the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the steps of the three-dimensional point cloud semantic segmentation method based on graph neural network and window attention mechanism described above in the present application.

[0025] A third aspect of the present invention provides a processor for running a program, wherein the program, when run, is used to execute: a three-dimensional point cloud semantic segmentation method based on a graph neural network and a window attention mechanism.

[0026] Through the above technical scheme, the present invention has the following beneficial effects: 1) an asymmetric feature is designed, that is, a coordinate feature is additionally added on the basis of the original point cloud feature, so that the point cloud feature is changed from two sets of RGB-XYZ information to three sets of RGB-XYZ-XYZ information, which is conducive to the subsequent extraction of local geometric information; 2) The combination of graph neural network and kernel point convolution can extract richer, more real and effective local information; 3) The window attention mechanism extracts local context information, and then combines it with the local geometric information extracted by the graph neural network, which significantly improves the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0028] Figure 1 A schematic diagram of the process structure of a “three-dimensional point cloud semantic segmentation network based on graph neural network and window attention” according to an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the structure of a graph neural network according to an embodiment of the present invention;

[0030] Figure 3 Schematic diagram of the structure of the window attention mechanism according to an embodiment of the present invention; wherein (a) is a schematic diagram of the window attention mechanism, and (b) is a schematic diagram of the structure of the window attention mechanism;

[0031] Figure 4 The following is a visualization comparison diagram of the segmentation effect according to the embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to further illustrate the features of the present invention, the technical solution of the present invention is described in more detail below through specific implementation methods. However, the present invention can be implemented in many other places different from the description, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited by the specific implementation disclosed below.

[0033] The existing segmentation algorithms that directly process point clouds do not fully utilize local geometric information, nor do they combine local geometric information and context information well, resulting in low segmentation accuracy. The first aspect of the embodiments of the present invention provides a 3D point cloud semantic segmentation method based on a graph neural network and a window attention mechanism, such as Figure 1 As shown, the following steps are included:

[0034] Stage I: Point Embedding Layer + Two Consecutive Window Attention Modules

[0035] 1.1 Point Embedding Layer.

[0036] First, get the input point cloud That is, there are N points in total, each point consists of color information and coordinate information, containing a total of 6-dimensional information, where the i-th point can be represented as p i =(x i ,y i ,z i ,r i ,g i ,b i ). Then, the point cloud is input into the point embedding layer consisting of edge convolution and kernel point convolution.

[0037] For the edge convolution (EdgeConv) based on graph neural network, its input is the reconstructed input point cloud, that is, the coordinate information and color information of the point cloud are swapped, and an additional set of coordinate information is added, p i =(x i ,y i ,z i ,r i ,g i ,b i ) is adjusted to p i ′=(r i ,g i ,b i ,x i ,y i ,z i ,x i ,y i ,z i ), each point contains two coordinate information, and the adjusted point cloud can be expressed as like Figure 2 As shown, the points are used as vertices of the graph The relationship between points and adjacent points is the edge of the graph Constructing a graph representing the local point cloud structure It is dynamically constructed using the nearest neighbors in the feature space, and the number of nearest neighbors is exemplarily set to 10.

[0038] Furthermore, extracting Midpoint p i ′ and adjacent point p j The edge relationship between ′ is defined as follows:

[0039] e ij =h Θ (p i ′,p′j )=h Θ (p i ′||p′ j -p i ′) (1)

[0040] where h Θ It is a neural network with a set of learnable parameters Θ, which is used to map paired 9-dimensional point cloud features to a specified F dimension. The edge function selection combines the global shape structure and the local neighborhood information asymmetric edge function, where the global shape structure is determined by the center coordinate p i ′ captures the local neighborhood information by p j ′-p′ captures, “||” indicates horizontal splicing, splicing the global shape structure and local neighborhood information together to form a point pair p i ′、p j ′’s marginal relationship.

[0041]

[0042] Furthermore, the feature of each point is updated by aggregating all edge relationships of each point through max to obtain a new point feature p i-edge , the calculation process is as shown in formula (2), and finally the point cloud containing the edge relationship is obtained F represents the information dimension of the point, and N represents the number of points.

[0043] For kernel point convolution (KPConv), its input is the original input point cloud, that is p i =(x i ,y i ,z i ,r i ,g i ,b i ). Use kernel point convolution (KPConv) to process the original input point cloud and extract the local information of the point cloud to obtain the point cloud The formula for KPConv is as follows:

[0044]

[0045] Among them, x i and x is The point in i It is point x i The corresponding features is the radius neighborhood with x as the center point, where is the chosen radius. The key part is the neighbor position y of the center point x. i =x i -x as the input kernel function g, which is defined as follows:

[0046]

[0047] The domain of g is the sphere are kernel points, y i =x i -x is the neighbor position of the center point x, It is to transform the features from dimension D in Mapping to D out The association weight matrix of . h is and i The correlation between The closer to y i When , the correlation should be higher, this paper uses linear correlation:

[0048]

[0049] The influence distance of the core point σ=1.0.

[0050] At the end of the point embedding layer, the point cloud containing the edge relationships And local feature point cloud The sum of the two is processed by batch normalization and activation function to obtain the input point cloud of the attention mechanism. at the same time, Jump to all following window attention modules.

[0051] 1.2 Window Attention Module

[0052] Use multiple window attention modules to process point clouds. Figure 3 As shown in Figure 1, the point cloud is divided into different windows and the multi-head self-attention mechanism is restricted to the local window range for calculation. Points are scattered in different windows. Each query point only needs to consider the neighbors in the same window, instead of paying attention to all points like global self-attention. Multi-head self-attention is performed independently in each window. Different windows may contain different numbers of points. t It is expressed as the number of points in the t-th window. Formally, assume that N h is the number of heads, N d is the dimension of each head, N c =N h ×N d Feature dimension, for the input point in the tth window The multi-head self-attention formula in the t-th window is

[0053] q=Linear q (x),k=Linear k (x),v=Linear v(x), (6)

[0054] attn i,j,h =q i,h ·k j,h ,

[0055] attn i,.,h =softmax(attn i,.,h ),

[0056]

[0057] in is the query, key, value obtained from x through three linear layers, representing the vector q i,h and k j,h The dot product between . is the attention map, is the aggregate feature, further projected to the output feature

[0058] Furthermore, the context-based adaptive relative position encoding cRPE is added to the attention mechanism. In cRPE, for the point in the tth window We represent the xyz coordinates as Therefore, the relative xyz coordinates between query and key The formula is:

[0059] r i,j,m =p i,m -p j,m ,1≤i,j≤k t ,m∈{1,2,3}. (7)

[0060] To map relative coordinates to corresponding positional encodings, we maintain three learnable lookup tables Corresponding to the x-axis, y-axis, and z-axis respectively. Since the relative coordinates are continuous floating point numbers, we will r i,j,m The range of (-s win ,s win ) is quantized into L discrete parts, and the relative coordinate r i,j,m Mapped to the table:

[0061]

[0062] where s win is the window size, is the quantized size, Indicates round down.

[0063] We look up these tables to retrieve the embedding corresponding to the index and add them together to get the positional encoding (9)

[0065] e i,j =t x [idx i,j,1 ]+t y [idx i,j,2 ]+t z [idx i,j,3 ]

[0066] in represents the idx-th item in table t, is a positional encoding.

[0067] In practice, the tables used for lookups, keys, and values ​​are not shared. Therefore, we distinguish them by adding superscripts, where The x-axis table represents the query. Similarly, the position codes corresponding to query, key, and value are respectively represented by e q ,e k ,e v express.

[0068] Then do a dot product of the position encoding with the query and key to get the position deviation Then it is added to the attention map. In addition, we add the value feature with the corresponding position encoding, and then weight and aggregate it. Finally, the multi-head self-attention formula (6) in the window is updated to a version with contextual relative position encoding (cRPE)

[0069]

[0070] like Figure 3 As shown, the local geometric features Local features Contextual Information Fusion together to obtain comprehensive local features, as shown below

[0071]

[0072] Then, the local features feats are input into the displacement is processed in the second window attention module.

[0073] Stage II: Downsampling + Two Consecutive Window Attention Modules

[0074] The features output from stage I are downsampled, and then two consecutive window attention modules are used to further extract features. The second window is shifted

[0075] Stage III: Downsampling + six consecutive window attention modules

[0076] The features output from stage II are downsampled, and then six consecutive window attention modules are used to further extract features. The latter window is shifted based on the previous window.

[0077] Stage IV: Downsampling + Two Consecutive Window Attention Modules

[0078] The features output from stage III are downsampled, and then two consecutive window attention modules are used to further extract features. The second window is shifted

[0079] Finally, the decoding operation is performed by upsampling and aggregating the features of each stage to achieve semantic segmentation.

[0080] The technical solution of the present invention combines the graph neural network and kernel point convolution to form an embedding layer, which can obtain more comprehensive, rich and real local information. In addition, the window attention mechanism is used to extract local context information. Each time the context information is extracted, the window is shifted by half the width. The context information of different areas is continuously obtained through the window shift operation to obtain comprehensive context information. At the same time, the local geometric information extracted by the graph neural network is combined with the context information of each window, and participates in the next window attention mechanism to achieve a deep fusion of local geometric features and the attention mechanism. It has been proven that this operation is very effective and significantly improves the segmentation accuracy.

[0081] Based on the same inventive concept, a second aspect of an embodiment of the present invention provides a three-dimensional point cloud semantic segmentation system based on a graph neural network and a window attention mechanism, comprising:

[0082] The input point cloud reconstruction module is used to reconstruct the acquired input point cloud, swap any two sets of target information positions of the points, and add an additional set of information positions;

[0083] The edge convolution extraction module is used to extract the edge relationship between the reconstructed point cloud and its adjacent points using the edge convolution in the graph neural network to obtain a point cloud containing the edge relationship. F represents the information dimension of the point, and N represents the number of points;

[0084] The kernel point convolution processing module is used to process the input point cloud in step S1 using kernel point convolution, extract local information of the point cloud to obtain the point cloud The point cloud containing the edge relationship And local feature point cloud Add, batch normalization and activation function processing are performed in turn to obtain the input point cloud of the attention mechanism

[0085] Window attention mechanism module, used to execute the window attention mechanism and obtain context information Will and Fuse together to obtain comprehensive local features;

[0086] The semantic segmentation module is used to downsample the comprehensive local features under the window attention mechanism, and then perform multiple window attentions again, realize the decoding operation by upsampling the aggregated features, and finally complete the semantic segmentation.

[0087] A third aspect of an embodiment of the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the steps of the three-dimensional point cloud semantic segmentation method based on graph neural network and window attention mechanism described above in the present application.

[0088] A fourth aspect of an embodiment of the present invention provides a processor for running a program, wherein the program, when run, is used to execute: a three-dimensional point cloud semantic segmentation method based on a graph neural network and a window attention mechanism.

[0089] Test example: The semantic segmentation performed using the embodiment of the present invention obtains the S3DIS semantic segmentation results shown in Table 1 and the ScanNetv2 semantic segmentation results shown in Table 2. It can be seen from the results in Tables 1 and 2 that on S3DIS, our performance is significantly better than other models. The average intersection over union (mIoU) of the present invention is 0.8% higher than the best model PTv2 in 2022, and the average accuracy (mAcc) is 1% higher than PTv2. The overall accuracy (OA) reaches 91.8%, which also surpasses PTv2. On ScanNetv2, the Val mIoU of our method exceeds other methods including voxel-based methods, and exceeds the voxel-based MinkowskiNet by 1.0% mIoU. The average intersection over union (Test mIoU) on the test set is also at the advanced level. As Figure 4 As shown in the figure, Input is the input point cloud, Ground Truth is the real label of the input point cloud, Baseline is the semantic segmentation result of the baseline model, Ours is the semantic segmentation result of the present invention, and one color represents an object, for example, green represents the floor, and dark blue represents the wall. The comparison between the model of the present invention and the baseline model shows the superiority of the present invention. The method of the present invention can correctly identify the object highlighted in the yellow box, which is due to the deep integration of the graph neural network and the window attention mechanism in the present invention.

[0090] Table 1 S3DIS semantic segmentation results

[0091]

[0092] Table 2 ScanNetv2 semantic segmentation results

[0093]

[0094]

[0095] In summary, the present invention has the following beneficial effects: 1) an asymmetric feature is designed, that is, coordinate features are added on the basis of the original point cloud features, so that the point cloud features are transformed from two sets of information of RGB-XYZ to three sets of information of RGB-XYZ-XYZ, which is conducive to the subsequent extraction of local geometric information; 2) the combination of graph neural network and kernel point convolution can extract more abundant, real and effective local information. 3) the window attention mechanism extracts local context information, and then combines it with the local geometric information extracted by the graph neural network, which significantly improves the accuracy of semantic segmentation.

[0096] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited thereto. Within the technical concept of the present invention, the technical solution of the present invention can be subjected to a variety of simple modifications, including the combination of various specific technical features in any suitable manner. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations. However, these simple modifications and combinations should also be regarded as the contents disclosed by the present invention and belong to the protection scope of the present invention.

Claims

1. A 3D point cloud semantic segmentation method based on graph neural network and window attention mechanism, characterized in that: The following steps are involved: S1, obtain input point cloud; S2, reconstruct the input point cloud, swap any two sets of target information positions of the points, and add an additional set of information positions; S3. Use the edge convolution in the graph neural network to extract the edge relationship between the point and its adjacent points on the reconstructed point cloud to obtain a point cloud containing the edge relationship. F represents the information dimension of the point, and N represents the number of points; S4: Use kernel point convolution to process the input point cloud in step S1, extract local information of the point cloud to obtain the point cloud The point cloud containing the edge relationship And local feature point cloud Add, batch normalization and activation function processing are performed in turn to obtain the input point cloud of the attention mechanism S5. Execute the window attention mechanism to obtain context information Will and Fuse together to obtain comprehensive local features; S6. Downsample the comprehensive local features under the window attention mechanism, then perform multiple window attentions again, implement decoding operations by upsampling aggregated features, and finally complete semantic segmentation.

2. The method according to claim 1, characterized in that In step S2, the coordinate information and color information of the point are swapped, and an additional set of coordinate information is added. In this case, the i-th point is represented as p i ′=(r i ,g i ,b i ,x i ,y i ,z i ,x i ,y i ,z i ), contains a total of 9 dimensions of information, and the entire point cloud is represented as 3. The method according to claim 1, characterized in that In step S3, the edge relationship between the reconstructed point cloud and its adjacent points is extracted using the edge convolution in the graph neural network to obtain the local geometric features: S31. Using points as vertices of a graph The relationship between points and adjacent points is the edge of the graph Constructing a graph representing the local point cloud structure It is dynamically constructed using the nearest neighbors in the feature space, and N represents the number of points; S32, Extraction graph Midpoint p i ′ and adjacent point p j The edge relationship between ′ is defined as follows: e ij =h Θ (p′ i ,p′ j )=h Θ (p′ i ||p′ j -p′ i ) where h Θ It is a neural network with a set of learnable parameters Θ, which is used to map paired 9-dimensional point cloud features to a specified F dimension. The edge function selection combines the global shape structure and the local neighborhood information asymmetric edge function, where the global shape structure is determined by the center coordinate p i ′ captures the local neighborhood information by p j ′-p′ capture, "||" represents horizontal splicing, splicing the global shape structure and local neighborhood information together to form a point pair p i ′、p j ' marginal relations; S33, update the features of each point by aggregating all edge relationships of each point by max, and obtain the new point feature p i-edge , and finally obtain a point cloud containing edge relationships New point feature p i-edge The calculation process formula is as follows:

4. The method according to claim 2, characterized in that: In step S5, the window attention mechanism is executed to obtain context information The specific process includes the following: S51, divide the point cloud into different windows and limit the multi-head self-attention mechanism to the local window range for calculation. For the input point in the tth window The multi-head self-attention formula in the t-th window is: q=Linear q (x),k=Linear k (x),v=Linear v (x), attn i,j,h =q i,h ·k j,h , attn i,.,h =softmax(attn i,.,h ), in is the query, key, value obtained from x through three linear layers, representing the vector q i,h and k j,h The dot product between is the attention map, is the aggregate feature, further projected to the output feature k t Represented as the number of points in the t-th window, N h is the number of heads, N d is the dimension of each head, N c =N h ×N d Feature dimension; S52, add context-based adaptive relative position encoding cRPE to the attention mechanism. In cRPE, for the point in the tth window Express the xyz coordinates as The relative xyz coordinates between query and key The formula is: r i,j,m =p i,m -p j,m ,1≤i,j≤k t ,m∈{1,2,3}; S53, r i,j,m The range of (-s win ,s win ) is quantized into L discrete parts, and the relative coordinate r i,j,m Mapped to the table: where s win is the window size, is the quantized size, Indicates rounding down; S54, find and maintain three learnable lookup tables Corresponding to the x-axis, y-axis, and z-axis respectively, to retrieve the embedding corresponding to the index and add them up to get the positional encoding e i,j =t x [idx i,j,1 ]+t y [idx i,j,2 ]+t z [idx i,j,3 ] in represents the idx-th item in table t, is the positional encoding; S55. Tables for query, key, and value are distinguished by adding superscripts, where The x-axis table represents the query, and the position encodings of query, key, and value are respectively expressed as e q ,e k ,e v Then do a dot product of the position encoding with the query and key to get the position deviation And add it to the attention map; add the value feature with the corresponding position encoding, then weight and aggregate it. Finally, the multi-head self-attention formula in the t-th window is updated to the context relative position encoding version formula:

5. The method according to claim 4, characterized in that The and The formula for integrating the comprehensive local features feats is as follows:

6. The method according to claim 4, characterized in that The method further includes, before performing step S6, first shifting the window The output features of the previous window are used as input and window attention is performed on the new window.

7. The method according to any one of claims 1 to 6, characterized in that: In step S4, the formula of the kernel point convolution is as follows: Among them, x i and x is The point in i It is point x i The corresponding features is the radius neighborhood with x as the center point, where is the chosen radius, with the neighbor position y of the center point x i =x i -x as the input kernel function g, which is defined as follows: The domain of g is the sphere is the core point, y i =x i -x is the neighbor position of the center point x, It is to transform the features from dimension D in Mapping to D out The association weight matrix, h is and i The correlation between them.

8. A 3D point cloud semantic segmentation system based on graph neural network and window attention mechanism, characterized in that: include The input point cloud reconstruction module is used to reconstruct the acquired input point cloud, swap any two sets of target information positions of the points, and add an additional set of information positions; The edge convolution extraction module is used to extract the edge relationship between the reconstructed point cloud and its adjacent points using the edge convolution in the graph neural network to obtain a point cloud containing the edge relationship. F represents the information dimension of the point, and N represents the number of points; The kernel point convolution processing module is used to process the input point cloud in step S1 using kernel point convolution, extract the local information of the point cloud and obtain the point cloud. The point cloud containing the edge relationship And local feature point cloud Add, batch normalization and activation function processing are performed in turn to obtain the input point cloud of the attention mechanism Window attention mechanism module, used to execute the window attention mechanism and obtain context information Will and Fuse together to obtain comprehensive local features; The semantic segmentation module is used to downsample the comprehensive local features under the window attention mechanism, and then perform multiple window attentions again, realize the decoding operation by upsampling the aggregated features, and finally complete the semantic segmentation.

9. A machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the steps of any one of the methods of claims 1 to 7 of the present application.

10. A processor, characterized in that: Used to run a program, wherein the program is used to execute when run: a three-dimensional point cloud semantic segmentation method based on a graph neural network and a window attention mechanism as described in any one of claims 1 to 7.

Images