Multi-scale image convolution three-dimensional point cloud segmentation method based on edge intelligent driving

Abstract

The invention discloses a multi-scale image convolution three-dimensional point cloud segmentation method based on edge intelligent driving, and relates to the field of three-dimensional point cloud segmentation. According to the method, a dynamic multi-hop graph convolutional network model is adopted, and the dynamic multi-hop graph convolutional network model is composed of a classification model and a segmentation model; each of the classification model and the segmentation model comprises a dynamic multi-hop graph convolution module and a multi-scale gating mechanism module; the method comprises the following steps: designing a self-adaptive multi-hop mechanism to extract high-order adjacent point features and carrying out aggregation; designing a multi-scale gating mechanism to dynamically weight and fuse the features of different levels; generating a classification score of the overall point cloud; a classification score for the semantic tag is generated. According to the method, the advantage that an edge intelligent normal form is close to a data source is utilized, edge computing power resources are reasonably scheduled, low-delay, customizable and containerized computing power support is provided for three-dimensional point cloud segmentation tasks of massive Internet of Things nodes according to needs, and the performance of a multi-scale graph convolution three-dimensional point cloud segmentation algorithm is structurally enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud segmentation, and specifically relates to an edge intelligence-driven multi-scale graph convolution three-dimensional point cloud segmentation method. Background Art

[0002] Affected by factors such as data transmission delay and communication bandwidth, the traditional cloud computing paradigm is difficult to quickly respond to the demand for computing resources of massive IoT nodes in 3D point cloud segmentation tasks. As a bridge connecting cloud platforms and IoT nodes, the edge intelligence paradigm has the advantage of being close to the data source. By rationally scheduling edge computing resources, it provides low-latency, customizable, and containerized computing support on demand for 3D point cloud segmentation tasks of massive IoT nodes, structurally enhancing the performance of the multi-scale graph convolution 3D point cloud segmentation algorithm.

[0003] Due to the characteristics of point cloud data such as sparsity and disorder, feature extraction has always faced severe challenges when performing point cloud classification and segmentation tasks. Some early methods manually converted point clouds into other data forms to extract features. As deep learning technology gradually matures, point cloud classification and segmentation methods based on deep learning have flourished. In recent years, new methods have captured the neighboring relationship between points in point clouds through dynamic graphs and extracted point cloud features using graph convolution operations, opening up the precedent for the application of graph convolution in the field of point cloud segmentation. Compared with other methods, this type of method models the local relationship between points and is better at capturing the local details and spatial relationships of point clouds, which is very beneficial to improving model performance. However, according to existing research, local features are usually extracted only from the direct neighbors of a point. This neighborhood definition is relatively narrow, which affects the model's local feature learning ability, limits the receptive field, ignores local context information, and further affects model performance. To solve this problem, some studies have proposed the concept of empty neighborhoods, but introduced additional noise points. Other studies have defined candidate point sets of different scales to achieve structural feature extraction based on MLP. However, this method of selecting neighborhood features by enumeration is inefficient.

[0004] Inspired by the above work, this paper proposes a novel dynamic multi-hop graph convolution (DMGConv) for point cloud feature extraction. The model not only aggregates direct neighbors, but also aggregates more high-order neighbors, including four steps: initializing neighbors, neighborhood adaptive multi-hop, adaptive deep convolution, and multi-scale feature aggregation. First, in order to make the model better adapt to the feature that the structure of point cloud data is constantly changing during multiple feature extractions, the neighbors are initialized at each layer input. Different from the existing neighborhood definition method, an adaptive multi-hop mechanism is adopted to define the initialization neighbors of the traditional method as first-order neighbors. In order to avoid introducing additional noise during adaptive multi-hop, the first-order neighbors only choose the direction of the neighbors that are highly correlated with them to jump. In theory, the neighborhood starting from the center point can radiate to the order point. Secondly, according to different jump degrees, an adaptive deep convolution kernel is designed to flexibly and deeply mine the local fine-grained features of the point cloud and enhance the neighborhood perception ability. Finally, cross-scale point semantic feature aggregation is achieved.

[0005] In addition, the present invention also proposes a novel gating mechanism, the closer to the bottom layer of the neural network, the more limited the receptive field of the neuron, and vice versa. This mechanism aims to enhance the interaction of global features by adjusting features of different scales in the forward-to-backward path. Summary of the invention

[0006] The purpose of the present invention is to provide an edge intelligence driven multi-scale graph convolution 3D point cloud segmentation method to solve the problems in the prior art that the graph convolution network only considers direct neighbors in the local neighborhood, is limited to extracting features with a fixed-size convolution kernel, and is difficult to fully learn the contextual information contained in the 3D point cloud data; and generally uses splicing operations to fuse multi-scale features, which limits its global information modeling capabilities.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is:

[0008] In the first aspect of the present invention, a dynamic multi-hop graph convolutional network model is proposed, which is composed of a classification model and a segmentation model; the classification model and the segmentation model both include a dynamic multi-hop graph convolution module and a multi-scale gating mechanism module;

[0009] The dynamic multi-hop graph convolution module takes n points as input and calculates the regional feature information of each point. The output of this module is used as the input of the next dynamic multi-hop graph convolution module. The output features of the last layer of dynamic multi-hop graph convolution modules are globally aggregated to form a 1D global descriptor.

[0010] The multi-scale gating mechanism module dynamically weights and fuses features among different features of adjacent layers of the input according to their importance;

[0011] The classification model extracts multi-scale features through a multi-layer dynamic multi-hop graph convolution module, and performs feature fusion through a multi-scale gating mechanism module, and a multi-layer perceptron generates a classification score of a class from a 1D global descriptor;

[0012] The segmentation model extracts multi-scale features through a multi-layer dynamic multi-hop graph convolution module, and performs feature fusion through a multi-scale gating mechanism module, combines the 1D global descriptor connecting each point and the output of all dynamic multi-hop graph convolution modules (as local descriptors), expands the classification model, and outputs the classification score of the semantic label.

[0013] Preferably, the dynamic multi-hop graph convolution module includes four parts, namely, initializing neighboring points, neighborhood adaptive multi-hop, adaptive depth convolution, and multi-scale feature aggregation; the dynamic multi-hop graph convolution module is specifically as follows:

[0014] Initialize neighbor points: Input is n Dimensional point cloud x i , In each layer of dynamic multi-hop graph convolution module, the KNN algorithm is used to calculate each x i The nearest k points are used as the initial neighboring points x ij , k neighboring points form the initial domain G = (V, E);

[0015] Neighborhood adaptive multi-hop: Starting from the initial neighbor, it adaptively jumps to k higher-order neighbors according to the graph structure; by calculating the correlation between each point and its neighbors, it adaptively selects the most relevant n-order neighbors as high-order neighbors to form a multi-hop neighborhood;

[0016] Adaptive deep convolution: After filtering out the n-order neighbor points, combine them with the initial center point x i Perform convolution; with x i is the center point, each order point is the neighboring point, and its directed edge H θ By x in the graph structure i and each step Composition, defined as:

[0017]

[0018] Here, define x j is the first-order neighbor, extending to That is x i The nth-order neighbor of ;

[0019] Multi-scale feature aggregation: Due to the disorder of point clouds, the maximum pooling is used to aggregate the final multi-scale features:

[0020]

[0021] Among them, max is the maximum pooling.

[0022] Specifically, the neighborhood adaptive multi-hop is as follows:

[0023] Define n-order neighbor points: As the initial neighbor x ij ; Through the 0th order neighbor Jump to k neighboring points and define x i The first-order neighbor of And so on x i The nth-order neighbor of :

[0024]

[0025] Among them, ·^(i) represents the kth neighboring point centered on the point,

[0026] Calculate the correlation of n-order neighboring points: After jumping k times to the kth neighbor, at the nth neighbor In the small neighborhood formed by i Defined as:

[0027]

[0028] Among them, the correlation θ i Represents the degree of correlation between the n-order neighbor point in each small neighborhood and its corresponding n-1-order center point;

[0029] Finally, filter the n-th order point: define the diffusion degree c; set the correlation θ i Arranged in descending order, to avoid introducing redundant features, the first c n-order neighbors are selected as the final n-order neighbors, and the c n-order neighbors are considered to be highly correlated with the n-1-order center point.

[0030] Preferably, the multi-scale gating mechanism module is specifically as follows:

[0031] Define U(·) as the dynamic weighting operation in the multi-scale gating mechanism module, which is used to calculate the local attention weight between two different features and improve the model's ability to extract relevant information from the input features; i and x i-1 The concatenation of is taken as input and the local attention weight U is calculated:

[0032] U=sigmoid(BN×(W2×SiLU(BN×(W1×[x i, x i-1 ]))))

[0033] Among them, [﹒ ] represents dimension connection, W1 and W2 represent output dimensions of c / r and output dimensions of c point-by-point convolution, r represents the dimensionality reduction (r=4 in the present invention), BN is batch normalization, SiLU(·) and Sigmoid(·) represent Sigmoid Linear Unit (SiLU) and Sigmoid activation function respectively;

[0034] The features of the dynamic multi-hop graph convolution modules in adjacent layers are sequentially input into the multi-scale gating mechanism module to adjust the features of different scales from bottom to top and enhance the interaction of global features.

[0035] The second aspect of the present invention proposes an edge intelligence driven multi-scale graph convolution 3D point cloud segmentation method using the above-mentioned dynamic multi-hop graph convolution network model, comprising the following steps:

[0036] S1. Design an adaptive multi-hop mechanism to extract high-order neighbor features and perform aggregation;

[0037] The dynamic multi-hop graph convolution module uses the KNN algorithm to calculate the K nearest neighbors for each point as the initial neighbors; the initial neighbors are subjected to dynamic multi-hop graph convolution operations to obtain high-order neighbors, and the most relevant high-order neighbors are adaptively selected by calculating the correlation between each point and its high-order neighbors; adaptive deep convolution operations are used for feature extraction; after multiple layers of dynamic multi-hop convolutions, multi-scale features at different levels are aggregated through maximum pooling operations to generate a 1D global descriptor;

[0038] Specifically: Input 3D point cloud data x i , through the KNN algorithm for each x i Calculate its k nearest neighbor points as the initial neighbor point x ij ; The k nearest neighbor points are combined into an initial neighborhood G = (V, E). The construction of the initial neighborhood will serve as the basic input for subsequent dynamic multi-hop graph convolution operations;

[0039] The 0th-order neighbor As the initial neighbor point, the 0th order neighbor point According to the graph structure, we adaptively jump to k high-order neighboring points, perform dynamic multi-hop graph convolution operations, and define n-order neighboring points. By calculating the correlation θ between neighboring points i , and finally select the most relevant n-order neighboring points;

[0040] After selecting the n-th order neighbor point, compare it with the initial center point x i Perform convolution; with x i As the center point, each order point is the neighboring point, and its directed edges are composed of x in the graph structure. i and each step Here, define x j is the first-order neighbor, extending to That is x i The nth-order neighbor of ;

[0041] Multi-scale features at different levels are aggregated through the maximum pooling operation to generate a 1D global descriptor;

[0042] S2. Design a multi-scale gating mechanism to dynamically weight and fuse features at different levels;

[0043] Specifically, the features generated in S1 are connected through the Gate module to achieve weighted fusion of features in adjacent layers;

[0044] S3, generate the classification score of the whole point cloud through the 1D global descriptor to complete the point cloud classification task;

[0045] S4. Combine the 1D global descriptor with local descriptors at different levels, calculate the semantic label of each point, and complete the semantic segmentation of the point cloud.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) The method of the present invention designs a dynamic multi-hop graph convolution (DMGConv) network model, which includes a dynamic multi-hop graph convolution and a multi-scale gating (Gate) mechanism; the dynamic multi-hop graph convolution breaks the direct neighbor point restriction in the local neighborhood, adaptively mines the deep context information contained in the three-dimensional point cloud data, and broadens the model receptive field; the multi-scale gating mechanism forward weights the scale characteristics of the algorithm to enhance the global feature capture capability; it realizes the advantage of the edge intelligence paradigm close to the data source, reasonably schedules the edge computing resources, and provides low-latency, customizable, containerized computing power support on demand for the three-dimensional point cloud segmentation task of massive Internet of Things nodes, structurally enhancing the performance of the multi-scale graph convolution three-dimensional point cloud segmentation algorithm.

[0048] (2) The DMGConv module proposed in the present invention not only aggregates direct neighboring points, but also effectively extracts high-order neighboring point features through an adaptive multi-hop mechanism. Compared with the limitation of the prior art that only uses direct neighboring points to extract local features, the present invention greatly expands the receptive field through a flexible jump mechanism, and can capture a wider range of contextual information, thereby improving the model's perception of local and global features of point clouds.

[0049] (3) The adaptive deep convolution kernel designed in the DMGConv module proposed in the present invention can capture deep-level neighborhood features, obtain rich multi-scale information, and flexibly represent the local features of three-dimensional point clouds.

[0050] (4) The DMGConv module proposed in the present invention aggregates the multi-scale features generated by multi-layer dynamic convolution through the maximum pooling operation to form a more expressive global descriptor. At the same time, the dynamic multi-hop mechanism effectively screens the neighboring points, avoiding the redundant feature information brought by the high-order neighboring points, so that the final extracted features have higher recognition and expression ability. Compared with the shortcomings of the existing methods that introduce noise points or low computational efficiency, the present invention greatly improves the computational efficiency while ensuring the model performance.

[0051] (5) In the Gate module proposed in the present invention, a multi-scale gating mechanism forward weighted adjustment algorithm is designed to dynamically weight and fuse features at different levels, thereby improving the interaction between global and local information. Compared with the prior art method that only relies on single-scale features, the present invention can dynamically adjust information at different scales, ensuring that the interaction between global and local features is more accurately modeled, significantly improving the accuracy of point cloud segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a block diagram of the architecture of the DMGNet neural network model in the present invention;

[0053] Figure 2 Schematic diagram of the multi-hop mechanism principle in the present invention, (a) is a schematic diagram of traditional graph convolution initialization, (b) is a schematic diagram of dynamic multi-hop graph convolution in the present invention;

[0054] Figure 3 It is a structural block diagram of the DMGConv module in the present invention;

[0055] Figure 4 It is a structural block diagram of the Gate module in the present invention;

[0056] Figure 5 It is the result diagram of the confusion matrix in the present invention;

[0057] Figure 6 t-SNE visual result diagrams in the present invention, (a) is a schematic diagram of the first stage, (b) is a schematic diagram of the second stage, (c) is a schematic diagram of the third stage, and (d) is a schematic diagram of the fourth stage;

[0058] Figure 7 This is the visual result diagram of the point cloud segmentation ShapeNet in the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Embodiment 1:

[0061] This paper proposes a multi-scale graph convolution 3D point cloud segmentation method driven by edge intelligence, which is a new 3D point cloud segmentation technology, mainly using the DMGNet neural network model. The model uses dynamic multi-hop graph convolution to extract rich features from complex point cloud data, and combines multi-scale gating mechanism to achieve efficient feature fusion and segmentation tasks.

[0062] The DMGNet neural network model proposed in this invention is as follows Figure 1 As shown in the figure. The network architecture includes a classification model (upper part) and a segmentation model (lower part). The classification model extracts multi-scale features through a multi-layer DMGConv module and performs feature fusion through a Gate module. The output of the classification model is converted into the final classification result by a multi-layer perceptron (MLP). The segmentation model combines the global descriptor with the local features to assign semantic labels and complete the point cloud segmentation task.

[0063] For the input three-dimensional point cloud data, the DMGConv module innovatively performs multi-order neighborhood feature learning through an adaptive multi-hop mechanism. This method can not only capture the fine-grained features of the local point cloud, but also expand the receptive field through jump operations, capture the long-range dependencies between points, and thus enhance the modeling ability of global context features. In addition, the present invention designs a multi-scale gating mechanism, which fuses features at different levels through a dynamic weighting mechanism, thereby improving the adaptability and robustness of the model to complex point clouds.

[0064] In this embodiment, the dynamic multi-hop graph convolutional network framework (DMGNet neural network) is as follows:

[0065] The DMGNet neural network architecture consists of DMGConv and Gate modules. Figure 1As shown. The classification model is located in the upper half of the architecture, and the segmentation model is located in the lower half. The DMGConv module takes points as input and calculates the regional feature information of each point. The output of this module is used as the input of the next DMGConv module. The output features of the last layer of DMGConv are globally aggregated to form a one-dimensional global descriptor. In the classification model, the 1D global descriptor is used to generate the classification score of the class. The segmentation model extends the classification model by connecting the 1D global descriptor of each point and all DMGConv outputs (as local descriptors) to output the classification scores of semantic labels.

[0066] In this embodiment, the dynamic multi-hop graph convolution (DMGConv) module is as follows:

[0067] In order to get rid of the problem of limited local neighborhood range of fixed convolution kernel model, each layer of the network initializes neighboring points, so that the model can be more adaptable to the changing point cloud structure in multiple feature extractions. Then, it adaptively jumps from the initialized neighboring points to the neighboring points that are highly correlated with it. In theory, the neighborhood can radiate from the center point to the n-th order point. Adaptive deep convolution kernels are used to realize long-range information transmission between multi-order neighbors and the center point, capture richer local representations, and then realize multi-scale semantic feature fusion; Figure 2 This is a comparison of the principles of dynamic multi-hop graph convolution and traditional graph convolution initialization in the present invention. Figure 3 As shown in the figure, the module consists of four parts: initializing neighbors, neighborhood adaptive multi-hop, adaptive deep convolution, and multi-scale feature aggregation.

[0068] like Figure 3 As shown in the figure, the DMGConv module completes feature extraction and aggregation through four steps: initializing neighbors, adaptive multi-hop, adaptive deep convolution, and multi-scale feature aggregation. First, the DMGConv module uses the KNN algorithm to calculate the K nearest neighbors for each point as the initial neighbors, providing basic input for subsequent multi-hop operations. Then, the model adaptively selects high-order neighbors to form a multi-hop neighborhood by calculating the correlation between each point and its neighbors, thereby expanding the receptive field of the model and capturing the contextual information between points. For the selected high-order neighbors, the model uses adaptive deep convolution operations for feature extraction, and effectively processes neighborhood features at different levels through flexible convolution kernel design to achieve fine modeling of details. Finally, the model aggregates the multi-scale features extracted by the multi-layer DMGConv module through the maximum pooling strategy to form a global descriptor, further enhancing the feature expression ability of the model.

[0069] Initialize neighboring points: Network input is n-dimensional point cloud x i , Use the KNN algorithm to calculate each x at each layer of the network i The nearest k points are used as the initial neighboring points xij , This neighboring point formation G = (V, E).

[0070] Neighborhood adaptive hopping: define the initial neighbor x ij is the 0th-order neighbor: Through 0th-order neighbors Jump to k neighboring points and define x i The first-order neighbor of And so on x i The nth-order neighbor of :

[0071]

[0072] Among them, ·^(i) represents the kth neighboring point centered on the point,

[0073] Calculate the correlation of n-order neighboring points: After jumping k times to the kth neighbor point, at these n-th order points In the small neighborhood formed by i Defined as:

[0074]

[0075] Among them, the correlation θ i Represents the degree of correlation between the n-order neighbor point in each small neighborhood and its corresponding n-1-order center point.

[0076] Finally, filter the n-th order point: define the diffusion degree c. i Arrange them in order from large to small. To avoid introducing redundant features, the first c n-order neighbors are selected as the final n-order neighbors, and these c n-order neighbors are considered to be highly correlated with the n-1-order center point.

[0077] Adaptive deep convolution: After filtering out the n-order neighbor points, combine them with the initial center point x i Perform convolution. i As the center point, each order point is the neighboring point, and its directed edges are composed of x in the graph structure. i and each step Composition, defined as:

[0078]

[0079] Here, define x j is the first-order neighbor, extending to That is x i The nth-order neighbor of .

[0080] Multi-scale feature aggregation: Due to the disorder of point clouds, the maximum pooling is used to aggregate the final multi-scale features:

[0081]

[0082] In this embodiment, the gating mechanism (Gate) module is specifically as follows:

[0083] The Gate module dynamically weights and fuses features between different features of adjacent input layers according to their importance, which not only improves the information extraction capability of the model but also enhances the interaction of global features.

[0084] Figure 4 The implementation process of the multi-scale gating mechanism is demonstrated. The model is divided into four stages, and the output of each stage is weighted and fused with the features of the subsequent stage through the Gate module. The Gate module establishes dynamic weights between the features of adjacent stages, and by gradually adjusting the feature information of different scales, it enhances the intercommunication effect of global features and further improves the feature expression ability.

[0085] U(·) is defined as the dynamic weighting operation of Gate, which is used to calculate the local attention weight between two different features and improve the model's ability to extract relevant information from the input features. Gate takes feature x i and x i-1 The concatenation of is taken as input and the local attention weight U is calculated:

[0086] U=sigmoid(BN×(W2×SiLU(BN×(W1×[x i ,x i-1 ]))))

[0087] Among them, [﹒ ] represents dimensional connection, W1 and W2 represent point-by-point convolution with output dimension c / r and output dimension c, r represents the dimensionality reduction (r=4 in this paper), BN is batch normalization, SiLU(·) and Sigmoid(·) represent Sigmoid Linear Unit (SiLU) and Sigmoid activation function, respectively. The features of adjacent layers of the network are input into the Gate module in turn, and the features of different scales are adjusted from bottom to top to enhance the interaction of global features.

[0088] The edge intelligence driven multi-scale graph convolution 3D point cloud segmentation method using the DMGNet neural network model includes the following steps:

[0089] S1. Input 3D point cloud data x i , through the KNN algorithm for each x i Calculate its k nearest neighbor points as the initial neighbor point x ijThese k neighboring points form the initial domain G = (V, E), and the construction of the neighborhood will serve as the basic input for subsequent dynamic multi-hop graph convolution operations;

[0090] S2, from the initial neighbor point (i.e., the 0th order neighbor point ), and adaptively jump to k high-order neighbors according to the graph structure, defining n-order neighbors The jump process is adaptive, by calculating the correlation θ between neighboring points i Finally, the most relevant n-order neighbor points are selected to avoid the introduction of redundant information;

[0091] S3, after selecting the n-th order neighboring point, compare it with the initial center point x i Perform convolution. i As the center point, each order point is the neighboring point, and its directed edges are composed of x in the graph structure. i and each step Here, define x j is the first-order neighbor, extending to That is x i The nth-order neighbor of ;

[0092] S4, after multiple layers of dynamic multi-hop convolution, the multi-scale features at different levels are aggregated through the maximum pooling operation to generate a 1D global description;

[0093] S5, the features generated by S2-S4 are combined through the Gate module to achieve weighted fusion of features of adjacent layers;

[0094] S6. Generate the classification score of the whole point cloud through the 1D global descriptor to complete the point cloud classification task;

[0095] S7. In the segmentation model, the 1D global descriptor is combined with the local descriptor to calculate the semantic label of each point and complete the semantic segmentation of the point cloud.

[0096] Experimental verification:

[0097] The present invention uses a 3D point cloud classification dataset, ModelNet40, and a 3D point cloud segmentation dataset, ShapeNet, to conduct a comprehensive performance evaluation of the proposed DMGNet neural network model.

[0098] ModelNet40 contains 40 object categories. The training set contains 9843 models and the test set contains 2468 models. Each time 1024 points of 3D data are sampled and input into the DMGNet classification model. ShapeNet contains 16881 3D shapes of 16 object categories. Each time 2048 points are sampled in the training sample and input into the DMGNet segmentation model.

[0099] During the evaluation process, for the ModelNet40 dataset, the present invention uses two core indicators: OA (Overall Accuracy) and mAc (Mean Accuracy). OA is used to measure the classification accuracy of the model on the entire dataset, while mAc calculates the classification accuracy of each category and then takes the average value, aiming to measure the model's classification ability for different categories. In addition, the mIoU (Mean Intersection over Union) indicator is used on the ShapeNet dataset to evaluate the performance of the model in the segmentation task.

[0100] The DMGNet neural network model in the present invention is quantitatively compared with the current advanced point cloud classification algorithm in ModelNet40, and the test results of the two core indicators mAcc and OA are shown in Table 1.

[0101] Table 1 Quantitative comparison results of point cloud classification algorithms on ModelNet40

[0102] Model mAcc(%) OA (%) DGCNN[3] 90.2 92.9 PointNet

[10] 86.0 89.2 PointNet++

[11] -- 90.7 PCT

[12] -- 93.2 APES (local-based)

[13] -- 93.5 SageMix+PointNet++

[14] -- 93.3 LCPFormers

[15] 90.7 93.6 PointMixer

[16] 91.4 93.6 PointStack

[17] 89.6 93.3 PointVector

[18] 91.0 93.5 Point-M2AE

[19] -- 92.9 DMGNet 91.5 93.9

[0103] The experimental results in Table 1 show that on the ModelNet40 classification dataset, the overall accuracy (OA) and mean category accuracy (mAcc) of DMGNet are significantly improved by 93.9% and 91.5% respectively over the existing work. The classification results show that by introducing the DMGConv module for multi-scale feature learning and combining the Gate module for global context enhancement, DMGNet is extremely competitive in classification tasks.

[0104] Among the comparison algorithms, LCPFormers

[15] improves the transformer model to learn the local structural information of point clouds and capture their long-range dependencies. The uneven density of point clouds leads to unreasonable distribution of attention mechanisms among points when learning local features, which affects the model performance. In contrast, DMGNet regards point cloud data as a graph structure, which can better adapt to irregularly sampled point cloud data and maintain robustness to sampling density and distribution. Therefore, the mAc and OA of LCPFormers are 0.8% and 0.3% lower than those of DMGNet, respectively. Similarly, Pointstack

[17] enhances global high-semantic context features by multi-layer aggregation of point features of different resolutions. Although it can reduce the loss of high-semantic features compared to directly aggregating point features, due to the inherent limitations of the downsampling method, some details and local information may be lost during sampling, which affects the model's representation and learning ability of fine-grained features. Therefore, the mAc and OA of Pointstack are 1.9% and 0.6% lower than those of DMGNet, respectively. The mAc and OA of pointVector

[18] are 0.5% and 0.4% lower than those of DMGNet, respectively. Point-M2AE

[19] improves the network structure to a pyramid and uses a staged downsampling marker encoder to obtain multi-scale local features, but its OA is only 92.9%.

[0105] The present invention is also compared with the classical network. Since PointNet

[10] and DGCNN[3] do not consider the problems mentioned in the present invention. Therefore, the mAc of DGCNN and PointNet are 1.3% and 5.5% lower than that of DMGNet, respectively, and their OA are 1% and 4.7% lower than that of DMGNet, respectively. PointNet++

[11] proposes a step-by-step sampling strategy for point clouds to increase the receptive field and obtain multi-scale features. However, this simple hierarchical sampling is difficult to handle the uneven density characteristics of point clouds. Some important local features are easily ignored by less sampled areas, thus affecting the performance of the model. Therefore, although its OA is higher than that of PointNet

[10] , it is 3.2% lower than that of DMGNet.

[0106] In order to show the relationship between the actual prediction results of the DMGNet classification model and the true value, the present invention uses the confusion matrix to evaluate the performance of DMGNet on the point cloud classification task, such as Figure 5 shown.

[0107] like Figure 6 As shown in FIG. 1 , the global features extracted by the present invention after four DMGConvs can be compared with t-SNE to verify the classification ability of the model. From the first stage to the fourth stage, similar points are increasingly clustered, and dissimilar points are gradually moving away.

[0108] To further verify the effectiveness of the DMGConv module and the Gate module, the values ​​of the DMGConv module and the Gate module (channel reduction) are simulated in detail on the ModelNet40 dataset.

[0109] Table 2 Comparison results of different k values ​​of DMGConv module on ModelNet40 dataset

[0110] k-nearest-neighbor mAcc(%) OA (%) <![CDATA[k1=15,k2=5]]> 90.3 93.3 <![CDATA[k1=15,k2=10]]> 90.3 93.4 <![CDATA[k1=15,k2=15]]> 90.5 93.6 <![CDATA[k1=15,k2=25]]> 90.0 92.5 <![CDATA[k1=20,k2=5]]> 90.7 93.9 <![CDATA[k1=20,k2=10]]> 90.5 93.8 <![CDATA[k1=20,k2=15]]> 90.3 93.6 <![CDATA[k1=20,k2=20]]> 90.1 92.8

[0111] Table 3 Comparison results of different r values ​​of Gate module on ModelNet40 dataset

[0112] r value (dimensionality reduction) mAcc(%) OA (%) 4 90.7 93.9 8 90.1 92.5 12 89.5 91.2 16 88.8 89.8

[0113] As shown in Table 2, for the sake of distinction, the neighborhoods are recorded as first-order neighborhoods, second-order neighborhoods, and so on. In the DMGNet model, only jumps to the second-order point, because in this model, jumping to a higher-order neighborhood will sacrifice computational efficiency, and it is not advisable to obtain accuracy improvement at no cost. The number of first-order neighbors is set to a multiple of 5, and the total number of neighbors + is set to [20,40]. Analysis of Table 2 shows that when k1 = 15, the overall effect is not as good as k1 = 20. Because when the total number of direct neighbors is 20, more second-order points can be extended, local structural information can be captured more comprehensively, and more contextual information can be provided, which is extremely beneficial for complex point cloud scene analysis. However, it should not be too large, because a larger direct neighborhood will introduce unnecessary noise, and the model performance will decrease instead. When k1 = 20 and k2 = 5, the model performance reaches the best state. Because too many second-order points may also introduce redundant information, thereby interfering with model performance.

[0114] As shown in Table 3, the value of r is a multiple of 4. When r = 4, the model reaches the best state. Because scaling the dimension can prompt the model to select more important and representative features, discard redundant or unimportant features, which helps to improve the generalization ability and feature expression ability of the model. However, when the value of r is too large, excessive dimensionality reduction may lead to the loss of important features during training, making it difficult to give full play to the optimization effect of the model. Therefore, when r is greater than 8, the model performance tends to decline.

[0115] The DMGNet neural network model in the present invention is quantitatively compared with the current advanced point cloud segmentation algorithm in ShapeNet, and the test results of core indicators such as mIoU are shown in Table 4.

[0116] Table 4. Quantitative comparison results of point cloud segmentation algorithms in ShapeNet

[0117]

[0118] The experimental results shown in Table 4 show that DMGNet outperforms the SOTA algorithm and has the highest accuracy. The component segmentation results show that its average instance mIoU (Mean Intersection over Union) is significantly improved by 86.6%. DMGConv has a strong ability to represent local fine structures, and the global enhancement effect of the Gate module is better.

[0119] Compared with the segmentation effects of classic PointNet

[10] and PointNet++

[11] , the architectural design of these two networks has certain limitations, making it difficult to accurately capture the local geometric information of point clouds, resulting in limited representation of local details. Therefore, their mIoU is 2.9% and 1.5% lower than that of DMGNet, respectively. Both DGCNN[3] and DMGNet implement point cloud segmentation based on graph convolution methods. However, because the classic DGCNN does not combine multi-scale information and does not introduce a global context modeling mechanism, its segmentation accuracy is 1.4% lower than that of DMGNet.

[0120] In recent years, with the continuous optimization of point cloud segmentation models, PointMLP

[20] only uses pure deep layered MLP to achieve segmentation in pursuit of a simple architecture. The pursuit of a simple architecture design makes PointMLP's artifact segmentation accuracy only 86.1%, which is 0.5% lower than DMGNet. P2p

[21] encodes the geometric information of each point, then samples the projection view and rearranges the features into an image layout point by point to obtain pixel-level features with geometric projection. This model that uses pre-trained 2D knowledge for point cloud segmentation in a 3D neighborhood inevitably relies on the feature representation ability and adaptability of the pre-trained model and cannot fully mine the features of the point cloud. Its 2D knowledge pre-training will also cause a certain amount of information loss. Therefore, the mIoU of P2p is 0.1% lower than that of DMGNet. It is worth mentioning that PointGPT

[22] applies GPT to point cloud data generation and learns point cloud features through autoregressive generation of pre-training tasks. However, it mainly solves the disorder of point clouds, the difference in information density, and the adaptability between generation tasks and downstream tasks. Despite the different tasks, it and DMGNet both show superior performance in segmentation accuracy. Figure 7 The comparison between the model performance of the DMGNet neural network model in the present invention on 9 categories in the ShapeNet dataset and the real scene is demonstrated.

[0121] The above description is only used to help understand the method of the present invention and its core essence, but the protection scope of the present invention is not limited thereto. For those skilled in the art in the art, equivalent replacement or change according to the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. Dynamic multi-hop graph convolutional network model, characterized by: The dynamic multi-hop graph convolution network model consists of a classification model and a segmentation model; the classification model and the segmentation model both include a dynamic multi-hop graph convolution module and a multi-scale gating mechanism module; The dynamic multi-hop graph convolution module takes n points as input and calculates the regional feature information of each point. The output of this module is used as the input of the next dynamic multi-hop graph convolution module. The output features of the last layer of dynamic multi-hop graph convolution modules are globally aggregated to form a 1D global descriptor. The multi-scale gating mechanism module dynamically weights and fuses features among different features of adjacent layers of the input according to their importance; The classification model extracts multi-scale features through a multi-layer dynamic multi-hop graph convolution module, and performs feature fusion through a multi-scale gating mechanism module, and a multi-layer perceptron generates a classification score of a class from a 1D global descriptor; The segmentation model extracts multi-scale features through a multi-layer dynamic multi-hop graph convolution module, and performs feature fusion through a multi-scale gating mechanism module, combines the 1D global descriptor with the output of all dynamic multi-hop graph convolution modules as local features, and outputs the classification score of the semantic label.

2. The dynamic multi-hop graph convolutional network model according to claim 1, characterized in that: The dynamic multi-hop graph convolution module includes four parts, namely, initializing neighboring points, neighborhood adaptive multi-hop, adaptive depth convolution, and multi-scale feature aggregation; the dynamic multi-hop graph convolution module is as follows: Initialize neighboring points: Input is n-dimensional point cloud x i , In each layer of dynamic multi-hop graph convolution module, the KNN algorithm is used to calculate each x i The nearest k points are used as the initial neighboring points x ij , k neighboring points form the initial domain G = (V, E); Neighborhood adaptive multi-hop: Starting from the initial neighbor, it adaptively jumps to k higher-order neighbors according to the graph structure; by calculating the correlation between each point and its neighbors, it adaptively selects the most relevant n-order neighbors as high-order neighbors to form a multi-hop neighborhood; Adaptive deep convolution: After filtering out the n-order neighbor points, combine them with the initial center point x i Perform convolution; with x i is the center point, each order point is the neighboring point, and its directed edge H θ By x in the graph structure i and each step Composition, defined as: Here, define x j is the first-order neighbor, extending to That is x i The nth-order neighbor of ; Multi-scale feature aggregation: Use maximum pooling to aggregate the final multi-scale features: Among them, max is the maximum pooling.

3. The dynamic multi-hop graph convolutional network model according to claim 2, characterized in that: The neighborhood adaptive multi-hop is specifically as follows: Define n-order neighbor points: As the initial neighbor x ij ; Through the 0th order neighbor Jump to k neighboring points and define x i The first-order neighbor of And so on x i The nth-order neighbor of : Among them, ·^(i) represents the kth neighboring point centered on the point, Calculate the correlation of n-order neighboring points: After jumping k times to the kth neighbor, at the nth neighbor In the small neighborhood formed by i Defined as: Among them, the correlation θ i Represents the degree of correlation between the n-order neighbor point and its corresponding n-1-order center point in each small neighborhood; Finally, filter the n-th order point: define the diffusion degree c; set the correlation θ i Arrange them in order from large to small, select the first c n-order neighbors as the final n-order neighbors, and the c n-order neighbors are considered to be highly correlated with the n-1-order center point.

4. The dynamic multi-hop graph convolutional network model according to claim 1, characterized in that: The multi-scale gating mechanism module is specifically as follows: Define U(·) as the dynamic weighting operation in the multi-scale gating mechanism module, which is used to calculate the local attention weight between two different features; i and x i-1 The concatenation of is taken as input and the local attention weight U is calculated: U=sigmoid(BN×(W2×SiLU(BN×(W1×[x i, x i-1 ])))) Among them, [﹒ ] represents dimension connection, W1 and W2 represent output dimensions of c / r and output dimensions of c The point-by-point convolution of , r represents the dimensionality reduction, BN is batch normalization, SiLU(·) and Sigmoid(·) represent Sigmoid Linear Unit (SiLU) and Sigmoid activation function respectively; The features of the dynamic multi-hop graph convolution modules in adjacent layers are sequentially input into the multi-scale gating mechanism module to adjust the features of different scales from bottom to top and enhance the interaction of global features.

5. A method for edge intelligence-driven multi-scale graph convolution 3D point cloud segmentation using the dynamic multi-hop graph convolution network model described in any one of claims 1 to 4, characterized in that: The following steps are involved: S1. Design an adaptive multi-hop mechanism to extract high-order neighbor features and perform aggregation; The KNN algorithm is used to calculate the K nearest neighboring points for each point as the initial neighboring points. The initial neighboring points are subjected to dynamic multi-hop graph convolution operations to obtain high-order neighboring points. The most relevant high-order neighboring points are adaptively selected by calculating the correlation between each point and its high-order neighboring points. Adaptive deep convolution operations are used for feature extraction. Multi-scale features at different levels are aggregated through maximum pooling operations to generate a 1D global descriptor. S2. Design a multi-scale gating mechanism to dynamically weight and fuse features at different levels; S3, generate the classification score of the whole point cloud through the 1D global descriptor to complete the point cloud classification task; S4. Combine the 1D global descriptor with local descriptors at different levels, calculate the semantic label of each point, and complete the semantic segmentation of the point cloud.

Images