Multi-view 3D point cloud classification method based on dynamic and static convolutional fusion neural network

Abstract

The present invention provides a multi-view three-dimensional point cloud classification method based on a dynamic and static convolution fusion neural network. The method of view selection using fixed and random viewpoints effectively avoids the overfitting phenomenon caused by traditional fixed viewpoint selection. In order to improve the adaptability of the algorithm on different data sets, a local feature extraction operator of dynamic and static convolution adaptive weight fusion is used. In particular, a lightweight adaptive dynamic convolution operator is proposed to address the problem of huge number of parameters and high computational complexity of current dynamic convolution. In addition, a global attention pooling method is used to integrate the most critical information on different views to the greatest extent. These characteristics make the method of the present invention have better overall generalization performance, better adaptability, and can extract more fine-grained detail information, further improving the classification accuracy of point cloud data, and high fusion efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud classification, and specifically relates to a multi-view point cloud classification method based on a dynamic and static convolutional fusion neural network. Figure 3 dimensional point cloud classification method. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and deep learning technology, breakthroughs have been made in the field of three-dimensional perception and understanding. As an important form of three-dimensional perception and understanding, point clouds contain richer geometric shapes and structural information. Three-dimensional point clouds are mainly collected through lidar, RGB-D cameras and other sensor devices. They are widely used in many fields such as urban environment monitoring, urban morphology analysis, road automatic driving, computer vision, robot development and reverse engineering modeling. Research areas related to three-dimensional point clouds include classification, target detection, segmentation, registration, denoising, etc. based on point clouds. Among them, the classification of three-dimensional point clouds is one of the most basic and important links. Therefore, the research on point cloud classification, especially how to improve the accuracy and efficiency of point cloud classification, has become a hot topic of widespread concern in the relevant academic circles.

[0003] At present, the classification methods of 3D point clouds mainly include voxel-based, direct point cloud-based and multi-view-based methods. These three methods have attracted a lot of attention and research from scholars. The following will briefly explain these three methods in the prior art and their problems and shortcomings.

[0004] 1. Voxel-based 3D point cloud classification

[0005] Voxels are volume pixels that represent a three-dimensional area with a constant scalar or vector. Since voxels can represent complex objects with simplifiable, discrete units such as particles, they are very powerful in simulating the behavior of complex objects in the real world, and the structure of the voxel itself is also relatively simple. Victoria Plaza et al. proposed a voxel-based natural environment three-dimensional point cloud data classification method. The model uses a multi-layer perceptron to perform statistical geometric analysis on the spatial distribution of internal points and classify voxels. The local spatial distribution characteristics around the points are defined by the principal components of the point position covariance matrix. The combination of voxels and neural networks achieves faster calculation speeds than other strategies, but does not change the inherent disadvantages of voxels themselves, which are large and time-consuming. Zhijian Liu et al. proposed a Point-Voxel convolutional neural network model, which captures the overall structure and details of an object with two modules, integrating the advantages of point cloud and grid, and has smaller data overhead and better data regularity than other voxel models, and lower memory consumption. The classification accuracy of this algorithm is better than that of various voxel-based classification models, but it has not broken through the inefficiency of voxel-based convolution and the stereotype of voxel-based and point cloud-based structures. Victoria Plaza-leiva et al. proposed a new general framework based on voxel neighborhoods to implement and compare different supervised learning classifiers. This framework uses simple features in the support region defined by the voxel itself to assign points in each non-overlapping voxel in the regular grid to the same class, effectively improving the effectiveness of 3D spatial shape feature classification, shortening processing time, and facilitating parallel processing. However, it is only a preliminary experiment, and there is a lack of in-depth analysis using different performance indicators, environments, sensors, etc. Zishu Liu et al. proposed a voxel-based generalized learning network VB-Net and used it to classify 3D objects. By converting the original point cloud into voxels, using VB-Net as a feature extractor, extracting features from voxels, and using the features for target classification of the generalized learning system (BLS), the time consumed by system training is significantly reduced, but the accuracy of model classification needs to be improved, especially as the resolution of three-dimensional objects increases, the classification accuracy drops sharply. Kazuma Hamada et al. considered the variation of voxel density along the depth direction and proposed a new three-dimensional scene classification method based on tri-projection voxel splatting. This method normalizes the 3D scene according to position and size and projects the 3D scene onto three perpendicular planes. By merging three images and applying deep learning to predict the category of each scene, the classification accuracy is greatly improved. However, since tri-projection voxel splatting requires standardization to make each 3D scene fit into one voxel, if the 3D scene is large enough, Tri-projection Voxel Splatting (TVS) may not be able to identify tiny objects.ChengWang et al. proposed a voxel-based convolutional neural network NormalNet, which uses a reflection convolution cascade (RCC) module to implement the convolution layer to extract distinguishable features for 3D vision tasks, thereby significantly reducing the number of parameters and the performance of the entire network. However, this method did not find the optimal number of reflections in the key module RCC, so the model has further room for optimization. Hui Cao et al. investigated the reasons why binary voxels are not suitable for three-dimensional convolutional representation, and improved the accuracy by about 30% by assigning distance values ​​to each voxel, and designed a fast fully connected and convolutional hybrid cascade network for the classification of three-dimensional objects. Its average inference time is faster than that of point cloud and voxel-based methods, and its accuracy is also higher, but the recognition rate of the deep network used by it for some difficult samples is not as good as that of the shallow network.

[0006] The biggest disadvantage of voxel-based 3D object classification is that the storage cost is too high. At the same time, directly sending a large and complex 3D image represented by voxels into 3D convolution for processing will inevitably require a large computational cost and consume a lot of computing time. If such problems are solved by reducing the resolution of the 3D image, the accuracy of the final trained model will be reduced.

[0007] 2. Direct processing classification based on point cloud

[0008] Point cloud is a collection of data points defined by a given coordinate system. Each point contains rich information, including three-dimensional coordinates X, Y, Z, as well as color, classification value, intensity, time, etc. It has the characteristics of disorder, sparsity, rotation and translation invariance. Compared with the voxel method, the direct processing method based on point cloud does not require additional model conversion, and directly uses the original point cloud as the processing object, which can reduce a lot of storage overhead. Charles R. Qi et al. designed a new neural network PointNet for direct training of point cloud, which uses a learned T-Net transformation matrix to ensure the invariance of the model to specific spatial transformation. The multi-layer perceptron (MLP) is used to extract features from each point cloud data, and the maximum pooling operation is performed on each dimension of the feature to obtain the final global feature, and the MLP is used to classify 3D objects. Since the point cloud is directly operated without complex preprocessing, the network model is very efficient and resistant to disturbance. However, the connection between local points is not learned by the network, and the capture of information on local features is missing. Zhongyang Zhao et al. proposed a deep neural network combining multi-scale features and PointNet, which uses the neighborhood features of multi-scale extracted points and then combines them with the global features extracted by PointNet to complete the classification of LiDAR point clouds. It achieves a relatively good classification effect, but needs to be further improved in the shortcoming of low efficiency in extracting local features. Zhuangzhuang Li et al. proposed two methods to improve the accuracy of three-dimensional classification models based on PointNet, by increasing the number of hidden layers to obtain more abstract features, and combining the softmax loss function with the center loss function to obtain discriminative features, so that the improved model has better performance than the original PointNet. However, whether a more lightweight network or a deep convolutional network can be used to further optimize the model, and whether the range of relevant parameters can be adjusted to enhance the discrimination of the target, etc., has not been carefully explored. KuangEn Zhang et al. proposed a directional PointNet to directly classify three-dimensional point clouds in order to solve the occlusion problem that may occur when classifying the environment using one-dimensional signals and two-dimensional images. This method uses the directional information of point clouds to classify various terrains, thereby helping wearable robot objects to walk in complex environments and achieve robust and efficient classification of the environment. However, the high classification accuracy of the model is limited to specific application areas. In order to solve the problem of PointNet network, that is, it cannot capture the local structure generated by the points in the metric space, thus limiting its ability to recognize fine-grained patterns and generalize to complex scenes, the authors improved this and proposed a new hierarchical network model PointNet++.The PointNet++ network adaptively combines multi-scale features through a new learning layer, and combines local point set density adaptation to effectively learn deep point set features, thereby achieving more accurate classification results. However, the time spent is also greatly increased compared to the PointNet network model, especially in the preprocessing of point cloud data. Mor Joseph-Rivlin et al. use three-dimensional coordinates as class identifiers, compare the attributes of shape moments for classification, and add polynomial functions of coordinates to enable the network to adapt to the higher-order moments of a shape. Experiments show that it has greatly improved memory usage and computational complexity, and can handle the classification problem of rigid objects well, but whether the model can still achieve very high classification accuracy when applied to other fields of design geometry analysis is still a question worth exploring. Jancheng Yang et al. used the parameter-efficient Group Shuffle Attention to replace the costly Multi-head Attention mechanism, thereby developing a Point Attention converter that can handle input data of varying sizes and has transformation equivalence. The authors also proposed for the first time an end-to-end learning and task-agnostic sampling operation (Gumble Subset Sampling, GSS). By selecting subsets in the representative hierarchy, the network can obtain a stronger representation of the input set at a lower computational cost. The experiment also showed the effectiveness and efficiency of this method in three-dimensional image classification, but GSS was not applied to general datasets to explore its effectiveness and interpretability in hierarchical multi-instance learning. Hengshuang Zhao et al. proposed a new method PointWeb to extract contextual features from local areas in point clouds. It connects each point tightly in the local neighborhood and uses a novel adaptive feature adjustment module to find interactions between points. This framework can better learn point representations for point cloud processing, but the application of the proposed module to the understanding of 3D scenes requires further verification. Jiangwen Xie et al. proposed a generative model for disordered point clouds based on the energy model. In this model, the energy function learns the coordinate encoding of each point, and then aggregates all individual point features into the energy of the entire point cloud. The model is trained based on MCMC maximum likelihood learning and its variants without any auxiliary network, and does not rely on manually drawn distance metrics to generate point clouds, so it is very efficient for point cloud classification. However, in practical applications, the model's anti-disturbance and processing of point clouds with many outliers are poor.In response to the outliers and noise in the original point cloud, Xu Yan et al. proposed a new end-to-end network (Point AdaptiveSampling and Local NonLocal module, PointASNL) for robust point cloud processing. The network model consists of two modules, the adaptive sampling module and the local non-local module. Among them, the adaptive sampling module uses the farthest point sampling algorithm FPS to sample the initial point cloud and re-weight the domain, while the local non-local module can further capture domain points and long-range dependencies. PointASNL has reached the most advanced level in robustness in point cloud classification, but the fine-tuning strategy in the adaptive sampling module needs to be continuously explored.

[0009] Due to the influence of factors such as sensors and scenes, the sparsity of the points in the point cloud actually collected for three-dimensional object classification based on point clouds varies, thus affecting the accuracy of the final classification. Therefore, it is inevitable to perform complex preprocessing operations on the point cloud. In addition, due to the disturbance of the sensor, disturbances and outliers will inevitably appear in the point cloud, that is, a point may appear within a certain radius near the sampled point or at any position in space. Rotating the point cloud will produce different point clouds, but they still represent the same three-dimensional object. Therefore, problems such as point cloud outliers and disturbances, as well as rigid transformation processing, make the classification of three-dimensional objects based on point clouds more complicated.

[0010] 3. 3D Point Cloud Classification Based on Multi-View

[0011] The multi-view based 3D point cloud classification method is to collect a series of 2D images of a 3D point cloud object from different angles, and send the collected multiple 2D images into the CNN model for classification to characterize the final 3D point cloud object. Xiaozhi Chen et al. conducted research on high-precision 3D target detection in autonomous driving scenarios and proposed a multi-view 3D network. The network consists of two sub-networks, one for generating multi-view representations of 3D objects and the other for multi-view feature fusion. Among them, the feature fusion sub-network can combine regional features from multiple views, so that the intermediate layers of different paths can interact, which greatly improves the accuracy of classification, while the calculation performance is also reduced. However, since the model uses a region-based fusion network, it is slightly insufficient in extracting feature information of objects from a global perspective. Panagiotis Papadakis et al. used a large-scale ground truth dataset and a baseline view-based recognition method to benchmark multiple multi-view hypothesis fusion schemes under different environmental assumptions and observation capabilities, highlighting the important factors that should be considered in multi-view recognition methods. However, the analysis is limited to 3D shapes, without considering the texture features of objects. Xiaohui Cheng et al. proposed a new feature selection method that embeds low-rank constraints, sparse representation, global and local structure learning into a unified framework, uses a hypergraph-based regularization term to construct the Laplacian matrix, and uses a novel optimization algorithm to solve the objective function. The method achieves good classification performance on multi-view datasets, but does not extend the method to unsupervised learning and clustering learning. Christopher Pramerdorfer Cogvis et al. proposed a 3D bounding box method for joint classification of objects and regression models in depth maps based on a novel multi-view convolutional network structure. The method is highly robust to occlusion. The network can handle views that encode object geometry and occlusion information, and output class scores and bounding box coordinates in world coordinates without any post-processing steps. The biggest advantage of the model is that it has extremely high classification accuracy and a small regression error rate, but whether the performance can be further improved through different network front-end architectures (such as ResNet) or whether the method can be integrated into a deployed detection system based on Kinect to evaluate the performance of the method is still a question worthy of further study. Aiming at the problem that the inherent hierarchical correlation and distinguishability between multiple views have not been well utilized, Yifan Feng et al. proposed a group-view convolutional neural network (GVCNN) for distinguishing 3D shape descriptions and for hierarchical correlation modeling. This method introduces a hierarchical shape description framework, including view, group, and shape level descriptors.It also considers the correlation between views of each shape and uses grouping information for shape representation. Experiments have shown that the model has achieved significant performance improvements in 3D shape classification and retrieval tasks, but it is not perfect in experiments with more and more complete views. Anan Liu et al. proposed a multi-view hierarchical fusion network (MVHFN). The method consists of two main modules, namely the visual feature learning module and the multi-view hierarchical fusion module. Module one applies a 2D convolutional neural network to extract visual features of multiple views drawn around a specific three-dimensional object, while module two fuses multiple view features into a compact descriptor. The model can discover content differentiation by learning cluster-level feature information, thereby making full use of the relevant characteristics of multiple views and greatly improving the classification accuracy. However, the number of views captured is small and the order of views is fixed, which cannot simulate object recognition in real 3D scenes well. Jinxing Li et al. proposed a probabilistic hierarchical model for multi-view classification. It first learns a latent variable to fuse multiple features obtained from the same view, sensor and form, then uses the mapping matrix of a certain view to project the latent variable from the shared space to multiple observations, and uses the expectation maximization EM algorithm to estimate parameters and latent variables, achieving multi-level and effective fusion of multi-view and multi-feature data. The calculation of related parameters is authentic and repeatable, but the complexity of the multi-view and multi-feature model itself is too high and the computational efficiency is low. Jia He et al. proposed an online Bayesian multi-view learning algorithm, which learns the prediction subspace based on the principle of maximum margin, defines the potential marginal loss, minimizes the learning problem to various Bayesian frameworks using pseudo-likelihood and data enhancement ideas, and obtains approximate posterior changes based on past samples. Compared with the most advanced methods, the model can achieve higher classification performance and can automatically infer weights and penalty parameters, but the calculation is slightly complicated when the sample data set is large. Jinxing Li et al. designed a new Gaussian process latent variable model GPVLM to represent multiple views in a common subspace. It learns another projection from observed data to shared variables through view sharing and view-specific kernel parameters under the Gaussian process structure. And through the Gaussian transformation, the latent variables are converted into label information, which well reveals the correlation between different views. The model also shows a relatively superior performance, but the specific radial basis function (RBF) cannot adapt to the actual complex distribution of data, and the authors did not take multi-kernel learning into account. Qian Yu et al. proposed a new multi-view convolutional neural network Latent-MVCNN (LMVCNN), which uses multi-view images of predefined or random views to recognize three-dimensional shapes and consists of three sub-convolutional neural networks.Among them, the first CNN outputs a probability distribution of multiple categories, the second CNN outputs a latent vector to help the first CNN select a suitable distribution, and the third CNN outputs the category probability distribution of one view into the category probability distribution of another view. LMVCNN has good recognition performance for both pre-defined views and random views, and can also show good performance when the number of views is small, but it does not solve the problem of 3D shape recognition without background interference. Therefore, how to solve the problem of detecting and recognizing objects in a real 3D environment is also a daunting challenge. Tan Yu et al. proposed a multi-view coordinated bilinear network for 3D object recognition (MHBN) from the perspective of similarity measurement between image blocks. The model applies bilinear pooling to local convolution to obtain a compact global representation, and produces a more discriminative representation by coordinating the singular values ​​of the set features. The authors experimentally verified the effectiveness of this method in 3D object recognition and achieved a high classification accuracy. However, the classification accuracy still needs to be further improved by still using fixed viewpoints to select multiple views and traditional static convolution to extract features.

[0012] Compared with voxel and direct point cloud processing methods, this method will take up less storage space because only multiple two-dimensional views need to be stored. At the same time, for the converted two-dimensional views, the two-dimensional CNN model, which is already very mature, can be fully utilized, so the time spent on training is greatly reduced, and the accuracy of model classification is also the highest among the three. However, in the process of converting three-dimensional point clouds to two-dimensional views, most multi-view-based methods directly use the traditional fixed viewpoint projection method, which obviously easily causes the similarity between view data to be too high. Therefore, for views with a large difference in similarity with the test set, the model's discriminability will deteriorate, thereby reducing the overall generalization ability of the model. Moreover, in the process of feature extraction, these methods often use a pre-trained CNN backbone model to improve the efficiency of feature extraction, but most backbone models use traditional static convolution, which cannot be adaptive between different data. In addition, in the process of feature fusion, only using methods such as maximum pooling and average pooling causes a large amount of detail information to be lost, and the fusion efficiency is very low. Summary of the invention

[0013] The present invention is made to solve the above problems, and aims to provide a three-dimensional point cloud classification method with better overall generalization, relatively lower computational complexity, better overall classification performance and high fusion efficiency. The present invention adopts the following technical solutions:

[0014] The present invention provides a multi-view convolutional neural network based on dynamic and static convolutional fusion. Figure 3 The method for classifying dimensional point clouds is characterized by comprising the following steps:

[0015] Step S1, transforming the input 3D point cloud projection into a multi-view representation through the multi-view selection part of FSDC-Net;

[0016] Step S2, extracting local features of multi-view representation through the local feature processing part of FSDC-Net;

[0017] Step S3, fusing local features into global features through the global feature fusion part of FSDC-Net;

[0018] Step S4, FSDC-Net classifies the three-dimensional point cloud based on global features.

[0019] Wherein, step S1 includes the following sub-steps:

[0020] Step S1-1, normalizing the position information of each point of the three-dimensional point cloud;

[0021] Step S1-2, selecting a fixed viewpoint, and projecting the normalized three-dimensional point cloud based on the fixed viewpoint to obtain a fixed view set;

[0022] Step S1-3, selecting a random viewpoint, and projecting the normalized three-dimensional point cloud based on the random viewpoint to obtain a random view set;

[0023] Step S1-4, selecting partial views from the fixed view set and the random view set according to a predetermined ratio, and combining them into a representation of the initial stage of multi-view, that is, the multi-view representation,

[0024] Step S2 includes the following sub-steps:

[0025] Step S2-1, performing convolution and maximum pooling on each of the views;

[0026] Step S2-2, serially passing each of the views through a plurality of FSDC layers in the FSDC-Net, wherein each of the FSDC layers applies static convolution and lightweight dynamic convolution to the input features in parallel to generate operators and extract features respectively, and then adaptively fuses the features extracted by the two branches through two learnable weights, and then obtains the output features as the local features through the activation layer,

[0027] The dynamic convolution kernel used for performing the lightweight dynamic convolution is obtained by the following sub-steps:

[0028] Step S2-2-1, the input features are fused in the Batch dimension through the BatchPool function to achieve a lightweight effect, as shown in the following formula:

[0029]

[0030] Where, X i , Z1 represents the input and output features before and after BatchPool, X i (j,c,h,w) represents an input image, b represents the size of BatchSize, c,h,w represent the set of input features in channel, height and width directions respectively;

[0031] Step S2-2-2, using convolution matrices of different sizes to perform convolution operations on the original input features from different receptive fields, so as to extract information on different receptive fields;

[0032] Step S2-2-3, integrating the information extracted from different receptive fields;

[0033] Step S2-2-4, randomly inactivating neurons in the FSDC layer according to a predetermined ratio through a Dropout layer, thereby avoiding the overfitting problem;

[0034] Step S2-2-5, for each of the FSDC layers, the original dynamic weight is changed to a predetermined number of channels by convolution, and a probability value is obtained as the dynamic weight of the FSDC layer by activation function Sigmoid, thereby obtaining a set of dynamic weights of multiple FSDC layers;

[0035] Step S2-2-6, multiplying and adding the original multiple groups of convolution kernels with the dynamic weight set to generate the final dynamic convolution kernel.

[0036] The present invention provides a multi-viewing neural network based on dynamic and static convolution fusion Figure 3 The point cloud classification method may also have such a technical feature, wherein step S3 includes the following sub-steps:

[0037] Step S3-1, performing average pooling on the n local features to obtain an n-dimensional vector representing an initial value set of each dynamic weight;

[0038] Step S3-2, applying two 1×1 convolutions with n-dimensional input and output to the n-dimensional vector in sequence, applying a nonlinear activation function Swish between the two 1×1 convolutions, and then obtaining a feature weight of each of the views through an activation function Softmax;

[0039] Step S3-3: multiply each of the views by the corresponding feature weight, and then add them in the view dimension to obtain the global feature.

[0040] The present invention provides a multi-viewing neural network based on dynamic and static convolution fusion Figure 3 The dimensional point cloud classification method may also have such a technical feature, wherein, in step S2-1, convolution is performed using a 7×7 convolution matrix, and in step S2-2, the activation layer is BN+ReLu.

[0041] The present invention provides a multi-viewing neural network based on dynamic and static convolution fusion Figure 3 The dimensional point cloud classification method may also have such a technical feature, wherein in step S2-2-2, the original input features are convolved from different receptive fields using a 1×1 convolution matrix and a 3×3 convolution matrix, respectively, as shown in the following formula:

[0042] Z2=W1(Z1)

[0043] Z3=W2(Z1)

[0044] Where W1, W2 represent 1×1 and 3×3 convolution matrices respectively, Z2, Z3 represent the corresponding output features respectively,

[0045] In step S2-2-3, the output features Z2 and Z3 are added element by element, and then shrunk to an output feature Z4 of 1×c×1×1 dimension through a spatial average pooling function, as shown in the following formula:

[0046]

[0047] Where h×w is the width and height of the local feature,

[0048] In step S2-2-4, random inactivation is performed according to the following formula:

[0049] Z5=f dropout (Z4,p)

[0050] In the formula, p represents the inactivation ratio, and 0 <p<1。

[0051] The present invention provides a multi-viewing neural network based on dynamic and static convolution fusion Figure 3 The dimensional point cloud classification method may also have such a technical feature, wherein in step S1-1, the normalization is performed based on the following formula:

[0052]

[0053] In the formula, p i,j,k Represents any point in the 3D point cloud, min(P X,Y,Z ) and max(PX,Y,Z ) represent the points obtained by taking the minimum and maximum values ​​of the three coordinate axes X, Y, and Z of the three-dimensional Cartesian coordinate system in the three-dimensional point cloud.

[0054] The present invention provides a multi-viewing neural network based on dynamic and static convolution fusion Figure 3 The three-dimensional point cloud classification method can also have such a technical feature, wherein, in step S1-2, a virtual camera is placed at a fixed position, two coordinate axes in the three-dimensional point cloud are kept unchanged, and the third coordinate axis is rotated at equally spaced angles, and then the virtual camera is used for two-dimensional projection to obtain the fixed view set.

[0055] The present invention provides a multi-viewing neural network based on dynamic and static convolution fusion Figure 3 The point cloud classification method may also have such a technical feature, wherein in step S1-4, selection and combination are performed according to the following formula:

[0056]

[0057] Where fixed(Θ) is the fixed view set, rand(Θ) is the random view set, n is the total number of views, t is the number of random views, and F i is the i-th view, and P is the multi-view representation.

[0058] Function and Effect of the Invention

[0059] According to the present invention, the multi-view based on dynamic and static convolutional fusion neural network Figure 3 The three-dimensional point cloud classification method adopts a fixed-random viewpoint multi-view preprocessing algorithm to improve the generalization performance of the model on different views in order to solve the problem of overfitting in the preprocessing stage of the traditional fixed viewpoint selection multi-view method. At the same time, in order to extract richer fine-grained feature information, an adaptive weight fusion operator of static and dynamic convolution is used for local feature extraction. In addition, an adaptive global attention pooling method is adopted to more effectively fuse local features on different views and obtain the most critical details in the global representation of the point cloud. In summary, compared with some of the most advanced three-dimensional point cloud classification algorithm models and methods in the prior art, the method of the present invention has better overall generalization performance, better overall classification performance and high fusion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a multi-view based on dynamic and static convolutional fusion neural network in the embodiment of the present invention. Figure 3 Flowchart of the dimensional point cloud classification method;

[0061] Figure 2 It is a working principle diagram of the FSDC-Net network model in this embodiment;

[0062] Figure 3 is a schematic diagram of a flow chart of a multi-view preprocessing algorithm of fixed-random viewpoints in an embodiment of the present invention;

[0063] Figure 4 is a schematic diagram of a local feature extraction process in an embodiment of the present invention;

[0064] Figure 5 is a schematic diagram of a process of lightweight dynamic convolution in an embodiment of the present invention;

[0065] Figure 6 is a schematic diagram of the process of adaptive attention pooling in an embodiment of the present invention;

[0066] Figure 7 This is an example diagram of the ModelNet40 data set in an embodiment of the present invention;

[0067] Figure 8 It is an example diagram of the Sydney city dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, the following is a multi-view image processing method based on a dynamic and static convolutional fusion neural network of the present invention in combination with the embodiments and drawings. Figure 3 The classification method of 3D point cloud is described in detail.

[0069] <Example>

[0070] Figure 1 This is a multi-view convolutional neural network based on dynamic and static convolution. Figure 3 Flowchart of the dimensional point cloud classification method.

[0071] Figure 2 It is a working principle diagram of the FSDC-Net network model in this embodiment.

[0072] like Figure 1 and Figure 2 As shown in the figure, in this embodiment, the FSDC-Net network model includes three parts: (a) a multi-view selection part of fixed-random viewpoints: used to preprocess the original point cloud, convert it into a two-dimensional view through fixed and random viewpoints, and then combine them together in a certain proportion; (b) a local feature processing part: by combining lightweight dynamic convolution with traditional static convolution in parallel, a local operator for point cloud feature extraction is formed to improve the extraction efficiency of local features; (c) a global feature fusion part: using the attention pooling method, in the process of fusing the local features of multiple views into global features, different local features are given different influence weights, so as to achieve a higher classification accuracy.

[0073] Multi-view based on the FSDC-Net network model Figure 3 The point cloud classification method specifically includes the following steps:

[0074] Step S1, transforming the input 3D point cloud projection into a multi-view representation through the multi-view selection part of the FSDC-Net network model.

[0075] This embodiment involves a multi-view point cloud classification method. First, the three-dimensional point cloud projection must be transformed into a view. Common three-dimensional point cloud projection methods include orthogonal projection and perspective projection. Since the original point cloud is a three-dimensional object composed of three-dimensional coordinate points, it cannot be directly processed by a two-dimensional CNN. Therefore, it is necessary to first orthogonally or perspectively project the three-dimensional point cloud in the form of a two-dimensional view. Here we choose to use orthogonal projection because it ensures that the size of each point is the same and will not be different due to the distance from the camera.

[0076] Since MVCNN and other point cloud classification methods were proposed, the conversion method from point cloud to multiple views is to fix an angle and then rotate the point cloud 360 degrees at that angle. If you want to get the projection of multiple views, do a 2D perspective projection every 360 / n degrees, where n represents the number of views required.

[0077] Obviously, this fixed viewpoint is not conducive to the generalization of the model and may cause overfitting problems. This is because CNN learns the features of objects from the pre-processed two-dimensional view of the fixed viewpoint, and for those views that are very different from the fixed viewpoint, the discrimination power of the CNN model will decrease. In other words, the preprocessing method of the fixed viewpoint makes it possible for the CNN model to perform well in multiple views at certain angles, but poorly at other angles. To this end, this embodiment proposes a multi-view preprocessing algorithm for generating point clouds from fixed-random viewpoints, and the multi-view selection part of the FSDC-Net network model adopts this algorithm.

[0078] Figure 3 4 is a flow chart of a multi-view preprocessing algorithm of fixed-random viewpoints in this embodiment.

[0079] like Figure 3 As shown, taking the 3D point cloud of a car as an example, the algorithm is divided into four steps:

[0080] Step S1-1, normalizing the position information of all points in the point cloud.

[0081] That is, the standardization of the spatial position of the point cloud. Due to the differences in the acquisition equipment and related scanning parameter settings, the position distribution of the original point cloud may be inconsistent, which in turn causes the virtual camera to frequently move its position for different objects. Therefore, the position information of all points in the point cloud is normalized so that its position distribution revolves around the origin of the three-dimensional coordinate system. The specific normalization method is shown in the following formula (1):

[0082]

[0083] In the formula, p i,j,k Represents any point in the point cloud, min(P X,Y,Z ) and max(P X,Y,Z ) represent the points obtained by taking the minimum and maximum values ​​of the three coordinate axes X, Y, and Z of the three-dimensional Cartesian coordinate system in the point cloud.

[0084] Step S1-2: Select a fixed viewpoint, and project the normalized three-dimensional point cloud based on the fixed viewpoint to obtain a fixed view set.

[0085] After the point cloud is standardized, it is necessary to select the viewpoint. In this embodiment, for the convenience of operation, it is equivalently converted into another method, that is, fixing the virtual camera and then rotating the object in space. Specifically, by artificially defining a rotation angle set θ = {(θ x ,θ y ,θ z )|θ x ,θ y ,θ z ∈[0,360°)}, where θ x ,θ y ,θ z Respectively represent the angles of rotation of the original point cloud on X, Y, and Z. Place the virtual camera in a fixed position, and then keep two of the coordinate axes of the point cloud X, Y, and Z unchanged, and only rotate the third coordinate axis at equal intervals, and then use the virtual camera for two-dimensional projection, and select the required view at the fixed viewpoint.

[0086] Step S1-3: Select a random viewpoint, and project the normalized three-dimensional point cloud based on the random viewpoint to obtain a random view set.

[0087] For the original point cloud, a random rotation angle (θ x ,θ y ,θ z ), rotate the center point of the object to this angle. At this time, use the virtual camera for projection, that is, obtain the required view at this random viewpoint.

[0088] Step S1-4: select partial views from the fixed view set and the random view set according to a predetermined ratio, and combine them into a representation of the initial stage of multi-view (hereinafter referred to as multi-view representation).

[0089] If a fixed view set fixed(Θ) and a random view set rand(Θ) have been obtained by fixed and random projection methods, the total number of views is n, the number of random views is t, and the i-th view is denoted as F i , then the final generated multi-view combination (representation) is P, which can be represented by the following formula (2):

[0090]

[0091] In this embodiment, when the number of combined views is less than 6, that is, n<6, the view obtained from the fixed viewpoint is directly taken, that is, t=0; for other cases, a random view is added for every 5 fixed views, that is, The process is shown in the following formula (3):

[0092]

[0093] As described above, through step S1, multiple views are obtained by projecting the three-dimensional point cloud.

[0094] Step S2, extracting local features of the multi-view representation through the local feature processing part of the FSDC-Net network model.

[0095] Figure 4 Schematic diagram of the local feature extraction process in this embodiment.

[0096] like Figure 4 As shown, the local feature extraction process is to convert the i-th view in the multi-view representation P of the three-dimensional point cloud obtained in step S1 into a local view representation For each view, the conversion process includes the following steps:

[0097] In step S2-1, the view is subjected to 7×7 convolution and maximum pooling.

[0098] Step S2-2, the convolution and maximum pooling views are serially passed through n FSDC layers, where the i-th FSDC layer processes the input feature X i Static convolution and lightweight dynamic convolution are applied in parallel to generate operators and extract features respectively, and then the features of the two branches are adaptively fused through two learnable weights β and γ, and then the output feature X is obtained through BN and ReLu layers. i+1 , as the local feature of the view.

[0099] Among them, in step S2-2, the traditional static convolution and the lightweight dynamic convolution proposed in this embodiment are combined.

[0100] Convolutional neural networks extract features by performing a series of operations such as convolution and pooling on images. Its advantage is that the convolution kernel is shared, which is suitable for processing high-dimensional data. However, its disadvantage is that the parameters of the entire network no longer change after training, and its adaptability is poor and not flexible enough. Dynamic convolution can well avoid the shortcomings of traditional static convolution, but its computational complexity and memory overhead are too large. Therefore, in view of the shortcomings of dynamic convolution, a more lightweight dynamic convolution is proposed in this embodiment.

[0101] Figure 5 Schematic diagram of the structure of lightweight dynamic convolution in this embodiment.

[0102] like Figure 5 As shown, the process of lightweight dynamic convolution generation operator can be divided into the following 6 steps:

[0103] Step S2-2-1, pooling of BatchSize dimension: define the BatchPool function, which achieves a lightweight effect by fusing the feature maps of the original input in the Batch dimension, and integrates the most critical feature information of the Batch dimension, as shown in the following formula (4):

[0104]

[0105] In the above formula, X i , Z1 represents the input and output features before and after BatchPool, X i (j,c,h,w) represents an input image, b represents the size of BatchSize, c, h, w represent the set of input features in the channel, height, and width directions respectively.

[0106] Original X i The dimension of X is b×c×h×w. After being processed by the BatchPool function, i , the dimension of Z1 is 1×c×h×w.

[0107] Step S2-2-2, convolution operations on different receptive fields: convolution matrices of different sizes are used to perform convolution operations on the original input features from different receptive fields.

[0108] In this embodiment, 1×1 and 3×3 convolution matrices are used to convolve the original input features from different receptive fields, that is, the following formulas (5) and (6) are obtained:

[0109] Z2=W1(Z1) (5)

[0110] Z3=W2(Z1) (6)

[0111] In the above formula, W1 and W2 represent 1×1 and 3×3 convolution matrices respectively, and Z2 and Z3 represent the corresponding output features respectively. Obviously, the receptive field of the 1×1 convolution matrix is ​​small, which is used to integrate each pixel in the spatial dimension, while the receptive field of the 3×3 convolution matrix is ​​larger, which can obtain more field information around the pixel.

[0112] Step S2-2-3, fuse the information extracted from different receptive fields: add the output features Z2 and Z3 element by element, and then shrink them to Z4 of 1×c×1×1 dimension through a spatial average pooling function, as shown in the following formula (7):

[0113]

[0114] Step S2-2-4, Dropout to avoid overfitting: Use the Dropout layer to randomly inactivate the neurons in the FSDC layer in a certain proportion, thereby avoiding the overfitting problem of the network in the process of dynamically generating convolution, as shown in the following formula (8):

[0115] Z5=f dropout (Z4,p) (8)

[0116] In the above formula, p represents the inactivation ratio, and 0 <p<1。

[0117] Step S2-2-5, generation of dynamic weights: The original dynamic weights are changed to the appropriate number of channels by using a 1×1 convolution matrix W3, and then the activation function Sigmoid is used to make it nonlinear and obtain a probability value between (0, 1) as the dynamic weight for generating the final convolution layer, as shown in the following formula (9):

[0118] Ω=(ω1,...,ω i ,...,ω n ,)=σ sigmoid W3(Z5) (9)

[0119] In the above formula, Ω represents the final generated dynamic convolution weight set, Ω = (ω1, ω2, ..., ω n ), whose dimension is 1×n.

[0120] Step S2-2-6, generation of dynamic convolution kernel: the original n groups of convolution kernels are represented as K, that is, K = (Conv1, ..., Conv i ,...,Conv n ), and multiply and add it with the dynamically generated weight set Ω to generate the final dynamic convolution kernel, as shown in the following formula (10):

[0121]

[0122] Through the above step S2-1, a dynamic convolution kernel for performing lightweight dynamic convolution is obtained. In this embodiment, the performance of the lightweight dynamic convolution is also analyzed, as shown in Table 1.

[0123] Table 1 Performance comparison of static convolution, dynamic convolution and FSDC

[0124]

[0125] Performance analysis includes the following parts:

[0126] (1) Parameter analysis

[0127] For ease of explanation, assume that the number of input and output channels of the convolution operation is c in and c out And the size is c, the size of the output feature map is n, n = h out × out ,h out and w out That is, the height and width of the output feature map, the convolution kernel size is k×k, and the BatchSize size is b. Since FSDC dynamic convolution uses the pooling operation of the BatchSize dimension, it converts the original dynamic convolution bc 2 k 2 The number of parameters is reduced to c 2 k 2 , and the number of parameters of the traditional static convolution is also c 2 k 2 , so the final parameter amount is 2c 2 k 2 , as shown in Table 1.

[0128] (2) Calculation Amount Analysis

[0129] Since the computational complexity of traditional static convolution is approximately, as shown in the following formula (11):

[0130]

[0131] The FSDC dynamic convolution part of this embodiment mainly involves the calculation of the dynamic convolution kernel generated in the end. Therefore, the calculation amount can refer to the traditional static convolution, which is also approximately 2nc 2 k 2 , as shown in formula (11). So the total amount of calculation is 4nc 2 k 2 , while the general dynamic convolution calculation amount is bnc 2 k2 , as shown in Table 1. The FSDC convolution of this embodiment has only doubled the number of parameters and the amount of calculation compared to the traditional static convolution, but in general it is much smaller than the previous dynamic convolution method, because the amount of calculation of the previous dynamic convolution method will increase with the increase of b, and the relevant literature also shows that in order to achieve better results for the model, the value of b is often very large (b>>4). Therefore, the method of this embodiment can well combine the advantages of traditional static convolution and dynamic convolution while greatly reducing the number of parameters and the amount of calculation.

[0132] In addition, step S2-2 also involves an adaptive process of fusing static convolution and dynamic convolution.

[0133] In view of the advantages of traditional convolution and dynamic convolution, static convolution and lightweight dynamic convolution are combined in this embodiment to make full use of their advantages. i Apply two different convolution branches to obtain the output features X s and X d .Here X s represents the output features obtained by static convolution, X d Represents the output feature obtained by dynamic convolution. Two learnable parameters β and γ are introduced as the output feature X s and X d Therefore, the final output feature X i+1 It can be expressed as the following formula (12):

[0134] X i+1 =β·X s +γ·X d (12)

[0135] Obviously, when β=1, γ=0, the output feature X i+1 Will degenerate into the feature X extracted by the traditional static convolution s , and when β=0,γ=1, the output feature X i+1 It will become the feature X extracted by the lightweight dynamic convolution d Therefore, by continuously optimizing the two parameters β and γ, different weights can be automatically assigned according to different input data, which means that the features learned by static convolution or dynamic convolution are more emphasized, thereby achieving better feature extraction effects.

[0136] In step S3, the multiple local features obtained in step S2 are fused into global features through the global feature fusion part of the FSDC-Net network model.

[0137] What is extracted by the FSDC local feature processing operator are local view features, which need to be fused into a global feature in some way for the final classification. Traditional methods use maximum pooling, sum pooling, and their combination to fuse and generate global features. Although these methods are relatively simple to operate, they do not take into account the importance of different views, but use the same pooling method for all views, which is inefficient. Obviously, for local view features with higher discrimination, they need to have a higher weight, while those local view features with high similarity to other objects should be given a lower weight. To this end, this embodiment proposes an adaptive global attention pooling, which is used to fuse multiple local view features according to their own importance into a final global view representation for classification tasks.

[0138] Figure 6 It is a schematic diagram of the process of adaptive global attention pooling in this embodiment.

[0139] like Figure 6 As shown, the adaptive global attention pooling of this embodiment can well take into account the influence of different views, making the final global view representation more typical and representative, which specifically includes the following three steps:

[0140] Step S3-1, global average pooling (GAP) of local features: for n local features Average pooling is performed to obtain an n-dimensional vector, which represents the initial value set A1 of each dynamic weight. The process can be represented by formula (13):

[0141]

[0142] In the above formula, A1∈R n , and h×w is the width and height of the local feature. Usually, h and w have a value of 1, and f i,j Indicates the feature value of a specific feature at a certain subscript position.

[0143] Step S3-2, generation of local feature dynamic weights: Apply two 1×1 convolutions with n-dimensional input and output to the n-dimensional vector after global average pooling, and add a nonlinear activation function Swish between the two 1×1 convolutions. This is similar to the SENet attention mechanism, but channel compression and expansion are not used in this embodiment. In addition, Swish is used in this embodiment to increase the nonlinear representation, and then the feature weights of each view are obtained through the Softmax activation function. The specific generation process is as follows Figure 6 (a) and the following formula (14):

[0144] A2=σ softmax (Conv 1×1 (σswish (Conv 1×1 (A1)))) (14)

[0145] In the above formula, A2 represents the dynamically generated weight set, σ softmax ,σ swish Represents Softmax and Swish activation functions, Conv 1×1 It is a 1×1 convolution with n input and output channels.

[0146] Step S3-3, global feature fusion: by multiplying each view with its learned feature weight, and then adding them in the view dimension, a global view feature F is finally generated, which contains the feature information of all local views. The global feature fusion process is as follows: Figure 6 (b) and is represented by the following formula (15):

[0147]

[0148] In step S4, the FSDC-Net network model classifies the input three-dimensional point cloud based on the global features obtained in step S3.

[0149] Based on the above steps, in this embodiment, the training and testing data sets used are the ModelNet40 data set and the Sydney city object data set, respectively. ModelNet40 is a representative of dense point cloud data sets, which is completely synthesized by computer and simulates point cloud data under idealized conditions. At the same time, since it does not have any noise data, it can be used to test the accuracy of the method of this embodiment in point cloud classification. The Sydney city data set is a representative of sparse point cloud data sets, which is data obtained by scanning objects in urban outdoor scenes through LiDAR. It contains a lot of noise, and also has problems such as local incompleteness and uneven point density. It can be used to test the robustness and high accuracy of the method of this embodiment in real sparse point cloud classification tasks.

[0150] As a multi-view Figure 3 The ModelNet dataset is one of the commonly used datasets for point cloud classification. It is a dataset released by Princeton University, which contains 662 object categories and a total of 127,915 CAD models. ModelNet40 selects 40 categories of objects including airplanes, bathtubs, beds, benches, and a total of 12,314 CAD models. The Sydney City Object Dataset is a real sparse point cloud dataset obtained by scanning outdoor objects in Sydney, Australia using lidar. It contains 26 object categories including common urban road objects such as vehicles, pedestrians, high-rise buildings, and trees, with a total of 631 point clouds. The example data of the two datasets are as follows: Figure 7 and Figure 8shown.

[0151] In this embodiment, the basic hardware environment is Nvidia Titan Xp dual graphics card and 64GB of RAM, the operating system is Ubuntu20.04, and the software used includes blender2.92.0, which is mainly used for preprocessing of point clouds. The Pytorch version is 1.7.1, the cuda version is 10.1, the learning rate is set to 0.0001, the BatchSize of a single view is 128, the BatchSize of 6 views is 16, and the BatchSize of 12 views is set to 8. For the original ModelNet40 grid dataset, we construct the point cloud dataset corresponding to the original ModelNet40 by sampling points for each grid; for the Sydney city point cloud dataset, since it is already in the point cloud format obtained by scanning, there is no need to perform point sampling. For the training cycle, the ModelNet40 dataset is 30 epochs and the Sydney city dataset is 50 epochs.

[0152] In this embodiment, the following measurement criteria are adopted:

[0153] For the ModelNet40 dataset, the overall accuracy (OA) and average accuracy (AA) are mainly used to compare the FSDC-Net of this embodiment with the network model in the prior art to evaluate the quality of the model. For ease of explanation, assume that Δ represents the total number of samples in the test set, Δ true Indicates the number of samples correctly classified by the model on the test set, δ i represents the number of samples of a certain category in the test set, δ i(true) Indicates the number of samples in the test set that are correctly classified by the model. i is the accuracy of the i-th category, n is the number of categories, and the specific calculation processes of OA and AA are as follows:

[0154]

[0155]

[0156] For the Sydney City Object Dataset, the amount of data for each category of objects varies greatly. For example, the number of car samples is 88, while the number of scooter samples is only 1. Obviously, this is unfair, because correctly judging small samples requires less reasoning consumption than large samples, but the weight of the impact on the final AA is indeed the same. AA will be affected by small sample categories and produce drastic fluctuations. Instead, the F1 score is used instead of AA, which can effectively reduce the impact of such problems on the final evaluation. Therefore, the measurement criteria for the Sydney City Object Dataset are OA and F1 score. The calculation method of the F1 score is shown in formula (18):

[0157]

[0158] In the above formula, P is the average precision of all categories, and R is the average recall of all categories.

[0159] The following Tables 2-7 show the comparison results between the method of this embodiment and other classification methods (models) in the prior art.

[0160] Table 2 Classification performance comparison on ModelNet40 dataset

[0161]

[0162] Table 3 Classification performance comparison on Sydney city dataset

[0163]

[0164]

[0165] As shown in Table 2-3, the FSDC-Net of this embodiment is compared with various state-of-the-art point cloud classification models in the prior art. The state-of-the-art point cloud classification models can be further divided into three categories: voxel-based VoxelNet, VRN, ORION, LightNet, direct point cloud-based PointNet, PointNet++, PointGrid, PointASNL, ECC, and view-based MVCNN, GVCNN, and MHBN.

[0166] As can be seen from Table 2, on the ModelNet40 dataset, the OA and AA of FSDCNet in this embodiment on 6 views reached 94.6% and 93.3% respectively, while the best MHBN among other methods had OA and AA on 6 views of 94.1% and 92.2% respectively, which were 0.5% and 1.1% higher than them. The OA and AA of FSDCNet under 12 views reached 95.3% and 93.6%, which were the highest among all algorithm models.

[0167] As can be seen from Table 3, on the Sydney City Object Dataset, our FSDCNet has an OA of 85.3% on 6 views and an F1 score of 83.6%. Among the above-mentioned point cloud classification algorithm models, the model with the highest classification accuracy performance is LightNet, with an F1 score of 79.8%. Our model is 3.8% higher than LightNet in F1 score. FSDCNet has an OA of 85.5% on 12 views and an F1 score of 83.7%. Compared with several other point cloud classification algorithm models, it also has the best evaluation indicators on the Sydney City Dataset.

[0168] Therefore, the results of Table 2-3 show that the FSDC-Net model of this embodiment has wide applicability and can achieve the most advanced classification effect on both dense point clouds and sparse point clouds. On the one hand, the traditional fixed viewpoint selects multiple views, which is not conducive to the generalization of the model, that is, the accuracy of the model will drop significantly on unseen viewpoints or viewpoints with large differences. In comparison, we use a fixed plus random viewpoint selection method, which can increase the generalization performance of the model, that is, improve the accuracy of the model classification on unknown viewpoints, and is more flexible. On the other hand, the dynamic and static convolution fusion method, as our local feature extraction operator, can combine the advantages of the two in an adaptive way to achieve better feature extraction effects, while most of the algorithm models compared with it use traditional static convolutions and cannot extract richer fine-grained information. In addition, the model we proposed is to give different weights to each view according to its importance through adaptive global attention pooling, so that the final fused global view is more representative. This is also an effect that other point cloud classification pooling methods such as average pooling and maximum pooling cannot achieve.

[0169] Table 4 Classification performance comparison of different viewpoint selection methods on ModelNet40 dataset

[0170]

[0171] Table 5 Comparison of classification performance of different viewpoint selection methods on Sydney city dataset

[0172]

[0173]

[0174] As shown in Table 4-5, two different viewpoint selection methods are used, the first is fixed viewpoint selection of multiple views, and the second is fixed plus random viewpoint selection of multiple views, and the AA metrics of the two methods are compared on 6 views and 12 views. According to the above formula (3), in the fixed plus random viewpoint selection method, 6 views are 5 fixed views plus 1 random view, and 12 views are 10 fixed views and 2 random views.

[0175] It can be seen from Table 4-5 that both the ModelNet40 dataset and the Sydney city object dataset have a certain percentage improvement in their AA indicators on 6 views and 12 views after using the fixed plus random viewpoint selection multi-view method.

[0176] As can be seen from Table 4, on the ModelNet dataset, for the 6-view and 12-view data, if fixed plus random viewpoints are used, the OA is 94.6% and 95.3% respectively, while if only fixed viewpoints are used, the OA is 94.1% and 94.7% respectively, which is an improvement of 0.5% and 0.6%.

[0177] As can be seen from Table 5, on the Sydney City Objects dataset, FSDCNet achieves OA of 85.3% and 85.5% on 6 views and 12 views respectively if fixed plus random viewpoints are used, while it achieves OA of 84.1% and 84.9% using only fixed viewpoints, which are improved by 0.7% and 0.8% respectively.

[0178] Therefore, it can be concluded that combining some views obtained from random viewpoints with the views selected from the traditional fixed viewpoint can increase the accuracy of the model classification accuracy, and even the robustness, so that the model can learn some features that cannot be learned under the fixed viewpoint, thereby improving the final performance. However, the number of views selected from random viewpoints cannot be too large, otherwise the accuracy of the model will drop sharply. This is because the differences between the different views generated by these random viewpoints are large, which will reduce the correlation between the data on the training set and the test set, or even completely different, and ultimately affect the accuracy of the classification. Therefore, in this embodiment, a method of inserting a random viewpoint selected view for every 6 views is adopted.

[0179] Table 6 Comparison of classification performance with different numbers of views on the ModelNet40 dataset

[0180]

[0181] Table 7 Comparison of classification performance with different number of views on Sydney city dataset

[0182]

[0183]

[0184] As shown in Tables 6-7, by evaluating the classification performance of FSDCNet on different numbers of views, a balance can be achieved between classification performance and computational storage cost. In the experiment, the number of views is set to include single view, 6 views, and 12 views to compare the evaluation indicators of OA, AA, and F1 of FSDCNet with other state-of-the-art multi-view classification models on ModelNet40 and Sydney city object datasets.

[0185] As can be seen from Table 6, the OA and AA indicators of FSDC-Net in this embodiment on the ModelNet40 dataset are 93.8% and 91.2% respectively on a single view, which are significantly higher than the OA indicators of MVCNN on 6 views and 12 views by 1.8% and 2.3%, and higher than the OA indicator of GVCNN on 12 views by 1.2%. The OA and AA indicators of FSDCNet on 6 views are 94.6% and 93.2%, and the corresponding indicators of the same type of MHBN reach 94.1% and 92.2%, compared with which our FSDCNet is 0.5% and 1.0% higher respectively. FSDCNet reaches 95.3% and 93.6% on 12 views, and the OA and AA indicators are far higher than the current most advanced multi-view classification models.

[0186] As can be seen from Table 7, the OA index and F1 score of FSDC-Net in this embodiment on the Sydney city object dataset are 81.2% and 80.1%, 85.3% and 83.6% for 6 views, and 85.5% and 83.7% for 12 views. Among the models compared with it, LightNet, which has a good performance index, has an F1 score of only 79.8%, while the F1 index of FSDC-Net in this embodiment is only 0.3% higher than that in a single view, and 3.8% and 3.9% higher in 6 views and 12 views, which also proves that its effect is the best. It is worth noting that, whether on ModelNet40 or the Sydney city object dataset, the classification performance of FSDC-Net in a single view is better than that of most of the compared point cloud classification models. This also proves that the model of this embodiment has reached the most advanced level.

[0187] In general, as the number of views increases, the OA, AA and F1 values ​​of the FSDC-Net in this embodiment, which are used to measure the quality of the model, are also increasing. In contrast, the classification performance of MHBN on 12 views is worse than that on 6 views, which shows that FSDC-Net has more outstanding robustness than MHBN, and the model is not prone to jitter. However, the effect of increasing the corresponding index value on the ModelNet40 dataset is significantly better than that on the Sydney City Objects dataset. This is mainly because the Sydney City dataset is a sparse point cloud dataset in real scenes, and its overall number of samples is small, so the model does not have enough learning data. In addition, since the Sydney City Objects dataset is data obtained by laser radar scanning in real scenes, it contains a lot of noise, and as the number of views increases, the noise also increases, so the model inevitably learns more noise data, which will eventually affect the classification performance.

[0188] Table 8 Comparison of classification performance on different local feature extraction architectures

[0189]

[0190]

[0191] As shown in Table 8, the FSDC-Net local feature extraction operator can be divided into two parts: static convolution and dynamic convolution. In fact, we can use the convolution of any existing two-dimensional CNN model for the static convolution part. Here, we use the three most commonly used models, namely ResNet50, ResNext50, and SENet50, and compare the OA, AA, and F1 indicators of the models with and without the lightweight dynamic convolution proposed by us. It should be noted that in order to balance the storage overhead and accuracy improvement, all comparisons are performed under 6 views. It can be seen from Table 5 that after adding the lightweight dynamic convolution proposed by us and the adaptive weight parameter fusion, the classification accuracy on both datasets has been improved to a certain extent. For the ResNet50 network, its OA and AA on the ModelNet40 dataset are 93.0% and 92.0% respectively. After adding the lightweight dynamic convolution branch, its OA and AA are 93.8% and 92.6% respectively, which are 0.8% and 0.6% higher overall. On the Sydney City Objects dataset, the original ResNet50 has OA and F1 of 83.1% and 81.7% respectively. After adding the lightweight dynamic convolution, OA and F1 become 84.5% and 82.3%, respectively, an increase of 1.4% and 0.6%. Similarly, in Table 8, it can be seen that for the ResNext50 network, its OA and AA on the ModelNet40 dataset are improved by 1.2% and 0.5% respectively, while its OA and F1 on the Sydney City dataset are improved by 1.0% and 0.8% respectively. For the SENet50 network, its OA and AA on the ModelNet40 dataset are improved by 1.1% and 1.0% respectively, while on the Sydney City Objects dataset, they are improved by 1.3% and 1.4%. Through the above comparison, it is not difficult to find that after using the traditional static convolution and the lightweight dynamic convolution branch we proposed in parallel, the metric indicators have been improved to varying degrees.

[0192] The reason is that the lightweight dynamic convolution of this embodiment can extract some fine-grained feature information that traditional static convolution cannot extract, and these detailed features will be more helpful for the final model classification, making the classification effect generally better than the model without the dynamic convolution branch. At the same time, the lightweight dynamic convolution and the traditional static convolution are combined in an adaptive way, which can better integrate features of different dimensions, thus achieving higher classification performance compared to the traditional static convolution model.

[0193] Table 9 Comparison of classification performance of different global pooling methods

[0194]

[0195] As shown in Table 9, in order to make a trade-off between storage overhead and accuracy improvement, 9 views can be used as the corresponding evaluation benchmark. By comparing the most commonly used pooling methods, including maximum pooling, average pooling, maximum pooling plus average pooling, and soft pooling methods with the adaptive attention pooling method of FSDC-Net, the optimal effectiveness of the FSDC-Net pooling method of this embodiment is proved. As can be seen from Table 9, the OA and AA of FSDC-Net on ModelNet40 are 94.6% and 93.3% respectively, and the corresponding maximum pooling is 94.2% and 92.5%. The pooling method of FSDC-Net is 0.4% and 0.8% higher than the maximum pooling, respectively. On the Sydney City Object Dataset, the OA and F1 of FSDC-Net are 85.3% and 83.6% respectively, and the corresponding maximum pooling is 84.9% and 83.1%. The pooling method of this embodiment is 0.4% and 0.5% higher than the maximum pooling, respectively. This also proves that the adaptive global attention pooling of this embodiment has achieved the highest performance among the existing methods. This is because adaptive global attention pooling can learn a weight for each different local view feature according to its importance, and then fuse the different view features into the final global view representation with the learned weights, that is, strengthen important features, weaken less obvious features, and integrate the most critical information in multiple local views, which helps the final classification and judgment of the 3D point cloud.

[0196] In this embodiment, parts not described in detail are well-known technologies in the art.

[0197] Example Function and Effect

[0198] According to the multi-view based on dynamic and static convolutional fusion neural network provided by this embodiment Figure 3 The three-dimensional point cloud classification method adopts a fixed-random viewpoint multi-view preprocessing algorithm to improve the generalization performance of the model on different views in order to solve the problem of overfitting in the preprocessing stage of the traditional fixed viewpoint selection multi-view method. At the same time, in order to extract richer fine-grained feature information, an adaptive weight fusion operator of static and dynamic convolution is used for local feature extraction. In addition, an adaptive global attention pooling method is adopted to more effectively fuse local features on different views and obtain the most critical details in the global representation of the point cloud. In summary, compared with some of the most advanced three-dimensional point cloud classification algorithm models and methods in the prior art, the method of the present invention has better overall generalization performance, better overall classification performance and high fusion efficiency.

[0199] In the embodiment, it can be seen that the method of this embodiment has achieved the highest classification level in terms of OA, AA and F1 metrics. Experiments on the ModelNet40 dataset and the Sydney city object dataset show that the method of this embodiment is not only suitable for the classification of dense point cloud data, but also can achieve the best results for sparse point clouds with noise and local block incompleteness, which proves that the method of this embodiment and the neural network FSDCNet have wide applicability.

[0200] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A multi-view 3D point cloud classification method based on a dynamic and static convolutional fusion neural network, characterized in that: The steps include: Step S1, transforming the input 3D point cloud projection into a multi-view representation through the multi-view selection part of FSDC-Net; Step S2, extracting local features of the multi-view representation through the local feature processing part of the FSDC-Net; Step S3, fusing the local features into global features through the global feature fusion part of the FSDC-Net; Step S4, the FSDC-Net classifies the three-dimensional point cloud based on the global features, Wherein, step S1 includes the following sub-steps: Step S1-1, normalizing the position information of each point of the three-dimensional point cloud; Step S1-2, selecting a fixed viewpoint, and projecting the normalized three-dimensional point cloud based on the fixed viewpoint to obtain a fixed view set; Step S1-3, selecting a random viewpoint, and projecting the normalized three-dimensional point cloud based on the random viewpoint to obtain a random view set; Step S1-4, selecting partial views from the fixed view set and the random view set according to a predetermined ratio, and combining them into a representation of the initial stage of multi-view, that is, the multi-view representation, Step S2 includes the following sub-steps: Step S2-1, performing convolution and maximum pooling on each of the views; Step S2-2, serially passing each of the views through a plurality of FSDC layers in the FSDC-Net, wherein each of the FSDC layers applies static convolution and lightweight dynamic convolution to the input features in parallel to generate operators and extract features respectively, and then adaptively fuses the features extracted by the two branches through two learnable weights, and then obtains the output features as the local features through the activation layer, The dynamic convolution kernel used for performing the lightweight dynamic convolution is obtained by the following sub-steps: Step S2-2-1, the input features are fused in the Batch dimension through the BatchPool function to achieve a lightweight effect, as shown in the following formula: Where, X i , Z1 represents the input and output features before and after BatchPool, X i (j,c,h,w) represents an input image, b represents the size of BatchSize, c,h,w represent the set of input features in channel, height and width directions respectively; Step S2-2-2, using convolution matrices of different sizes to perform convolution operations on the original input features from different receptive fields, so as to extract information on different receptive fields; Step S2-2-3, integrating the information extracted from different receptive fields; Step S2-2-4, randomly inactivating neurons in the FSDC layer according to a predetermined ratio through a Dropout layer, thereby avoiding the overfitting problem; Step S2-2-5, for each of the FSDC layers, the original dynamic weight is changed to a predetermined number of channels by convolution, and a probability value is obtained as the dynamic weight of the FSDC layer by activation function Sigmoid, thereby obtaining a set of dynamic weights of multiple FSDC layers; Step S2-2-6, multiplying and adding the original multiple groups of convolution kernels with the dynamic weight set to generate the final dynamic convolution kernel.

2. The multi-view three-dimensional point cloud classification method based on dynamic and static convolutional fusion neural network according to claim 1, Features: Wherein, step S3 includes the following sub-steps: Step S3-1, performing average pooling on the n local features to obtain an n-dimensional vector representing an initial value set of each dynamic weight; Step S3-2, applying two 1×1 convolutions with n-dimensional input and output to the n-dimensional vector in sequence, applying a nonlinear activation function Swish between the two 1×1 convolutions, and then obtaining a feature weight of each of the views through an activation function Softmax; Step S3-3: multiply each of the views by the corresponding feature weight, and then add them in the view dimension to obtain the global feature.

3. The multi-view 3D point cloud classification method based on dynamic and static convolutional fusion neural network according to claim 1, characterized in that: in, In step S2-1, convolution is performed using a 7×7 convolution matrix. In step S2-2, the activation layer is BN+ReLu.

4. The multi-view 3D point cloud classification method based on dynamic and static convolutional fusion neural network according to claim 1, characterized in that: in, In step S2-2-2, the original input features are convolved using a 1×1 convolution matrix and a 3×3 convolution matrix from different receptive fields, as shown in the following formula: Z2=W1(Z1) Z3=W2(Z1) Where W1, W2 represent 1×1 and 3×3 convolution matrices respectively, Z2, Z3 represent the corresponding output features respectively, In step S2-2-3, the output features Z2 and Z3 are added element by element, and then shrunk to an output feature Z4 of 1×c×1×1 dimension through a spatial average pooling function, as shown in the following formula: Where h×w is the width and height of the local feature, In step S2-2-4, random inactivation is performed according to the following formula: Z5=f dropout (Z4,p) In the formula, p represents the inactivation ratio, and 0 <p<1。 5. The multi-view 3D point cloud classification method based on dynamic and static convolutional fusion neural network according to claim 1, characterized in that: in, In step S1-1, the normalization is performed based on the following formula: In the formula, p i,j,k Represents any point in the 3D point cloud, min(P X,Y,Z ) and max(P X,Y,Z ) represent the points obtained by taking the minimum and maximum values ​​of the three coordinate axes X, Y, and Z of the three-dimensional Cartesian coordinate system in the three-dimensional point cloud.

6. The multi-view 3D point cloud classification method based on dynamic and static convolutional fusion neural network according to claim 1, characterized in that: in, In step S1-2, the virtual camera is placed at a fixed position, two coordinate axes of X, Y, and Z of the three-dimensional point cloud are kept unchanged, and the third coordinate axis is rotated at equal intervals, and then the virtual camera is used for two-dimensional projection to obtain the fixed view set.

7. The multi-view 3D point cloud classification method based on dynamic and static convolutional fusion neural network according to claim 1, characterized in that: in, In step S1-4, selection and combination are performed according to the following formula: Where fixed(Θ) is the fixed view set, rand(Θ) is the random view set, n is the total number of views, t is the number of random views, and F i is the i-th view, and P is the multi-view representation.

Images