A point cloud super-resolution method based on FPFH features and deep neural network

By combining FPFH features and deep neural networks with dense connection of convolutional neural networks and k-nearest neighbor graphs, the problem of poor effect of point cloud super-resolution algorithm in complex noise scenarios is solved, and the local and global information extraction capabilities of point clouds are improved. The generated high-resolution point clouds are closer to the true distribution, surpassing the performance of mainstream methods.

CN115018710BActive Publication Date: 2024-12-03SOUTHEAST UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210741028.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-27
Publication Date
2024-12-03
Estimated Expiration
2042-06-27

AI Technical Summary

Technical Problem

The existing point cloud super-resolution algorithms are not effective in scenarios with complex noise and high noise intensity, and they lack the ability to extract local feature details information. Traditional methods rely on high calculation accuracy of normal vectors, making it difficult to adapt to diverse noise environments.

Method used

Using a method based on FPFH features and deep neural networks, combined with densely connected convolutional neural networks, k-nearest neighbor graphs and feature expansion modules, high-resolution point clouds are generated through supervised training, normal vectors and FPFH features are estimated, and local and global information are fused. , dynamically adjust network parameters to improve super resolution effect.

Benefits of technology

The super-resolution effect of point clouds is improved, and the generated high-resolution point clouds are closer to the real point cloud distribution, surpassing the performance of mainstream deep learning methods, making them more adaptable, and can better handle complex noise scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115018710B_ABST
    Figure CN115018710B_ABST
Patent Text Reader

Abstract

The present invention discloses a point cloud super-resolution method based on FPFH features and deep neural networks, including: obtaining a public point cloud super-resolution data set, constructing a training set and a validation set, and performing normalization processing; estimating the normal vector of a low-resolution point cloud, and using the estimated normal vector to calculate the FPFH features of the point cloud; constructing a deep neural network based on a densely connected 1×1 convolution kernel convolutional neural network, a k-nearest neighbor graph, and feature expansion; inputting the coordinates and FPFH features of the low-resolution point cloud into the neural network, combining the high-resolution point cloud of the corresponding sample, training the neural network in a supervised learning manner, and outputting a high-resolution point cloud; and evaluating the super-resolution performance of the network model on a test set. The neural network designed by the present invention can dynamically set the network parameter amount, the algorithm can focus on more surface geometry information, the generated high-resolution point cloud is closer to the real point cloud distribution, and the super-resolution effect is better than the mainstream method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of point cloud super-resolution technology in computer vision, and in particular to a point cloud super-resolution method based on FPFH features and a deep neural network. Background Art

[0002] Point cloud is an image format in three-dimensional space, which is generally obtained by three-dimensional scanning of real-world objects through three-dimensional image acquisition equipment such as three-dimensional laser scanners. The point cloud data structure is simplified and flexible, with strong expression ability. The point cloud information is processed by scientific methods, which plays a positive role in auxiliary work in engineering applications such as resource exploration, urban planning, agricultural development, and environmental monitoring. Due to the accuracy and acquisition distance of the acquisition equipment, the point cloud directly collected is prone to problems such as uneven point distribution and small number of local points, resulting in insufficient information. Generally, it is necessary to perform post-processing and repair manually according to the specific surface distribution of the actual three-dimensional object, so that the point cloud data can be better applied to industry. Point cloud super-resolution is an algorithm that adaptively upsamples the point cloud according to the spatial geometric structure of the existing point cloud. Using this algorithm to process low-resolution point clouds helps to reduce the difficulty of manual repair and increase the detail information of the point cloud.

[0003] Compared with images, point cloud data has an irregular structure and cannot generally be directly input into convolutional neural network training. Traditional point cloud super-resolution algorithms require sufficient knowledge of computer graphics to calculate specific geometric features of three-dimensional space, estimate the point cloud surface, calculate the intermediate points for interpolation, and achieve point cloud super-resolution. Most of these methods are based on the prior assumption that the point cloud surface is smooth, and the super-resolution effect on the edges and sharp parts of objects is average. In the case of only point cloud coordinates, these algorithms need to estimate the normal vector of each point, and the upsampling results are too dependent on the calculation accuracy of the point cloud normal vector. In addition, the noise of real point cloud data is diverse, and traditional algorithms are greatly affected by noise, which is not suitable for scenes with complex noise sources and high noise intensity.

[0004] Since the birth of deep neural networks such as PointNet for point cloud processing, researchers have focused on point cloud processing based on deep learning methods, using neural network models to extract spatial geometric information and deep semantic information of point clouds, and using deep learning classification, regression and other methods to obtain target results. Deep neural networks such as PointNet provide effective neural network templates for point cloud information extraction. These networks have achieved remarkable results in the application of classification and regression tasks such as point cloud segmentation and point cloud target detection. Point cloud super-resolution tasks require neural networks to have certain generation capabilities and understand the spatial geometric structure of input point clouds as much as possible, and have higher requirements for the extraction of local information and the fusion of global and local information. The early deep neural networks used for point cloud super-resolution basically referred to the information extraction modes of PointNet and PointNet++. These information extraction methods have better solved the problem of irregular point cloud data and difficulty in inputting neural network processing, and can effectively extract the global information of point clouds. The method based on PointNet++ can generate multi-scale feature maps, so that the deep neural network can focus on local information of different scales. These methods have a positive effect on improving the final point cloud super-resolution effect by extracting global and local information. However, these information extraction methods do not have an explicit definition of local features, their ability to extract detailed information is not strong enough, and the training process is highly dependent on the adaptive adjustment of the neural network. When dealing with samples with large differences in surface distribution, the super-resolution effects of the algorithms are mixed.

[0005] In recent years, point cloud super-resolution methods based on deep neural networks have shown a trend of flourishing, and the structures of these networks vary greatly. Deep learning methods generally achieve upsampling of point cloud points by designing specific feature expansion networks. The current feature expansion network design schemes vary, and each has its own advantages and disadvantages in different scenarios. Although it cannot replace traditional methods, deep neural networks have powerful fitting capabilities. The trained network has higher adaptability to point cloud data from various environments and scales, is less affected by noise, and has super-resolution effects that surpass traditional methods in many scenarios. Point cloud super-resolution based on deep neural networks needs further development and has certain research value. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a point cloud super-resolution method based on FPFH features and deep neural networks. This method builds a deep neural network for supervised training, combines densely connected convolutional neural networks, k-nearest neighbor graphs, feature expansion and other methods, and uses neural networks to achieve super-resolution of point clouds. At the same time, the normal vector and FPFH features of the input point cloud are estimated and input into the neural network to assist the neural network in information extraction. The point cloud super-resolution effect of the proposed method is better than that of mainstream deep learning methods.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A point cloud super-resolution method based on FPFH features and deep neural network, comprising:

[0009] Step S1, obtaining a point cloud data set, and performing normalization processing on the point cloud data set, wherein the point cloud data set includes high-resolution and low-resolution point cloud pairs;

[0010] Step S2, estimating the normal vector of the low-resolution point cloud at different selection radii, using the estimated normal vector to calculate the SPFH feature of the point cloud, and fusing the SPFH features at different selection radii to obtain the FPFH feature;

[0011] Step S3, constructing a deep neural network, which includes three network modules based on densely connected convolutional neural network, k nearest neighbor graph and feature expansion, wherein, first, the input feature is mapped to the high-dimensional space through the densely connected convolutional neural network, and the basic feature extraction is completed; then, the output feature of the densely connected convolutional neural network is input into the k nearest neighbor graph module, the k nearest neighbor graph is established, the k nearest neighbor graph feature is generated, and the k nearest neighbor graph feature and the output feature of the densely connected convolutional neural network establish a residual relationship to accelerate the network convergence; finally, the output features of the two network modules based on the densely connected convolutional neural network and the k nearest neighbor graph are spliced ​​and input into the feature expansion module to increase the number of feature channels and perform scale transformation on the features;

[0012] Step S4, using the low-resolution point cloud pairs in the point cloud data set in step S1 and the FPFH features corresponding to the low-resolution point cloud as the input of the deep neural network constructed in step S3, and using the high-resolution point cloud pairs corresponding to the low-resolution point cloud pairs as labels for supervised training, obtaining a training model after the training is completed, and then obtaining a super-resolution point cloud through the output of the training model.

[0013] Furthermore, the step S1 includes:

[0014] Step S101, obtain the public point cloud dataset PU1K, the training validation set of the dataset contains 69,000 point cloud pairs, the low-resolution point cloud contains 256 points, and the high-resolution point cloud contains 2048 points, which is a 4x super-resolution dataset; the test set contains point cloud data of 127 objects, each object has 4x super-resolution point cloud pairs of 4 scales, namely 256 points-1024 points, 512 points-2048 points, 1024 points-4096 points and 2048 points-8192 points. The point cloud data of the test set has not been processed by block processing, and each point cloud image represents a complete three-dimensional object;

[0015] Step S102, randomly dividing the training validation set into a training set and a validation set, wherein the training set contains 65,000 point cloud pairs and the validation set contains 4,000 point cloud pairs;

[0016] Step S103: normalize the input point cloud.

[0017] Furthermore, the step S103 specifically includes:

[0018] First, the center point of the entire point cloud is obtained by averaging the three coordinates of all points in the low-resolution point cloud;

[0019] Then, the Euclidean distances from all points to the center point are calculated for the low-resolution point cloud, and the maximum distance is taken as the normalized distance;

[0020] Finally, the center point coordinates are subtracted from the overall coordinates of the low and high resolution point clouds and the normalized distance is applied to normalize the values ​​of the three coordinates of the point cloud of the dataset to between -1 and 1 to generate a normalized point cloud.

[0021] Furthermore, the step S2 comprises:

[0022] Step S201, by statistically normalizing the distance distribution between points of the low-resolution point cloud, calling the open3d library normal vector estimation function, setting the neighbor point selection radius to 0.2 and 0.4, and estimating the normal vector information of each point in the point cloud;

[0023] Step S202, combining the normal vector information and the normalized coordinates of the point cloud, calling the FPFH feature calculation function of the open3d library, and using the neighboring point information within different selection radii to calculate the FPFH features of each point in the point cloud under different selection radii;

[0024] Step S203: design a convolutional neural network to perform feature fusion on the FPFH features under multiple selected radii, and the dimension of the fused feature is the same as that of a single FPFH feature.

[0025] Furthermore, in step S202, the FPFH feature is calculated as follows:

[0026] Step S2021: for the target point p s With one of its neighboring points p t Establish a local coordinate system, which includes:

[0027] Define p s With p t The corresponding normal vectors are n s 、n t , and then use represents the square of the L2 norm, then the three coordinate axes of the local coordinate system are represented by u, v, and w as follows:

[0028] u=n s (1)

[0029]

[0030] w=u×v (3)

[0031] Step S2022: For the local coordinate system constructed in step S2021, the SPFH features of the two points are characterized by the following three angle values:

[0032] α=v·n t (4)

[0033]

[0034] θ=arctan(w·n t ,u·n t ) (6)

[0035] SPFH(p t )=(α,φ,θ) (7)

[0036] Step S2023: for any query point p q and its k neighbors p k , the FPFH features of the query points are calculated from their respective SPFH features:

[0037]

[0038] Furthermore, in step S3, the densely connected convolutional neural network includes:

[0039] Each convolutional layer uses a 1×1 convolution kernel; the network contains multiple layers of intermediate features, and the scale of each layer of intermediate features is the same. All the previous intermediate features are concatenated in the feature dimension and then sent to the convolution layer for convolution calculation;

[0040] The input and output feature dimensions of the densely connected convolutional neural network are set to the same, and multiple densely connected convolutional neural networks with the same structure are cascaded to enhance the ability to extract point cloud information;

[0041] Among them, a 4-level densely connected convolutional neural network with the same structure was used to fuse the FPFH features of the point cloud input from the 1st and 3rd level networks, and the coordinate features of the 2nd and 4th level network input.

[0042] Furthermore, in step S3, the k nearest neighbor graph specifically includes:

[0043] Each convolution layer uses a 1×1 convolution kernel. For the input features of each convolution layer, each point feature first uses the k nearest neighbor algorithm, and the k nearest neighbor point features constitute the neighborhood feature of the point. Then, the point feature is subtracted from the corresponding neighborhood feature to obtain the relative feature. The relative feature and the input point feature constitute the k nearest neighbor graph feature, which is sent to the convolution layer for feature transformation.

[0044] By setting the output feature dimension of the convolutional layer to be the same as the input feature, multiple k-nearest neighbor graph convolutional neural networks with the same structure are cascaded, and each network uses the k-nearest neighbor algorithm once to dynamically generate the k-nearest neighbor graph features of the network at that level and send them to the convolutional layer of the network at that level;

[0045] Residual connections are established between the features output by each network, and the final output features obtained have the same dimension as the output features of the densely connected convolutional neural network and are summed with the output features of the k-nearest neighbor graph.

[0046] Furthermore, in step S3, the feature expansion specifically includes:

[0047] Each convolution layer uses a 1×1 convolution kernel; the feature expansion module contains multiple convolution layers, and the number of convolution channels gradually increases from the number of input channels to 4 times; the output features of the densely connected convolutional neural network and the k-nearest neighbor graph are concatenated and sent to the module to obtain output features that are 4 times the number of input channels. After rearranging the feature dimensions, the number of feature points is expanded to 4 times the number of input points.

[0048] Furthermore, the step S4 specifically includes:

[0049] Step S401: Using the point cloud data set preprocessed in step S1, extract the FPFH features of the low-resolution point cloud through step S2, fuse the FPFH under different selected radii, input the point cloud coordinates and the fused FPFH features into the deep neural network designed in step S3, and output the predicted high-resolution point cloud;

[0050] Step S402: Use the chamfer distance as a loss function to constrain the distance between the point cloud predicted in step S401 and the corresponding real high-resolution point cloud in the data set, perform supervised training on the deep neural network, and obtain a training model after the loss function converges;

[0051] Step S403: Perform a performance test on the training model on the test set of the public point cloud dataset PU1K.

[0052] Furthermore, in step S402, the chamfer distance is expressed by the following formula:

[0053]

[0054] In formula (8), S1 and S2 represent two different point clouds, x and y represent the coordinates of any point in the two point clouds, respectively. Represents the square of the L2 norm.

[0055] The beneficial effects of the present invention are:

[0056] The present invention uses a densely connected convolutional neural network to fuse point cloud coordinates and FPFH features, uses a k-nearest neighbor graph to establish an explicit neighbor point relationship, further extracts local information from the fused features, increases the number of point clouds through feature expansion, and generates high-resolution point clouds. FPFH features are generated based on the original point cloud coordinates, and the angle relationship between each point and the neighbor point and the neighbor point normal vector is statistically calculated. It has the ability to characterize surface geometric information, and its fusion with the point coordinate feature helps the neural network focus on the surface distribution of the point cloud. The k-nearest neighbor graph sets the input and output dimensions to be the same, and multiple networks with the same structure can be cascaded. Each network dynamically establishes the k-nearest neighbor graph of this layer, and the cascaded network extracts multi-layer neighborhood information from shallow to deep. The modules of the model of this method are clearly hierarchical, and the same structure network inside the module can be cascaded in any number. The model parameter quantity is dynamically determined according to demand, and the degree of freedom is high. When only the point cloud coordinates are used as input features, the neural network must establish a positional relationship based on the relative coordinates of the points, and fit the surface distribution of the point cloud from the relative position information. The introduction of FPFH features helps the network focus on more geometric information, reduces the difficulty of network fitting, and ultimately improves the super-resolution effect. This algorithm can dynamically set the network parameters and introduce explicit geometric information to assist the neural network fitting information. The generated high-resolution point cloud is closer to the real point cloud distribution, and the super-resolution effect is better than the mainstream method. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of the point cloud super-resolution method based on FPFH features and deep neural network provided in Example 1;

[0058] Figure 2 It is the FPFH feature fusion flow chart provided in Example 1;

[0059] Figure 3 This is the overall structure diagram of the deep neural network model provided in Example 1;

[0060] Figure 4 A densely connected convolutional neural network structure diagram provided in Example 1;

[0061] Figure 5 The k-nearest-neighbor graph forward propagation structure diagram provided in Example 1;

[0062] Figure 6 This is a feature expansion structure diagram provided in Example 1;

[0063] Figure 7 The performance comparison between the method of Example 1 and the mainstream point cloud super-resolution algorithm is shown;

[0064] Figure 8 This is an example of the point cloud super-resolution effect of the method in Example 1. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Example 1

[0067] See also Figure 1-Figure 8 This embodiment provides a point cloud super-resolution method based on FPFH features and deep neural network. The specific process of the method is as follows: Figure 1 As shown, the method comprises the following steps:

[0068] Step S1, obtaining a point cloud data set, and performing normalization processing on the point cloud data set, wherein the point cloud data set includes high-resolution and low-resolution point cloud pairs;

[0069] Specifically, step S1 includes:

[0070] Step S101, obtain the public point cloud dataset PU1K, the training and validation set of which contains 69,000 point cloud pairs, each point cloud is a point cloud block intercepted from the complete and clean point cloud data, representing a local area of ​​the object, the low-resolution point cloud contains 256 points, and the high-resolution point cloud contains 2048 points, which is a 4x super-resolution dataset;

[0071] The test set contains point cloud data of 127 objects. Each object has 4 super-resolution point cloud pairs of 4 scales, namely 256 points to 1024 points, 512 points to 2048 points, 1024 points to 4096 points, and 2048 points to 8192 points. The point cloud data of the test set has not been processed in blocks, and each point cloud image represents a complete 3D object.

[0072] Step S102, randomly dividing the training validation set into a training set and a validation set, wherein the training set contains 65,000 point cloud pairs and the validation set contains 4,000 point cloud pairs;

[0073] Step S103: normalizing the input point cloud, which includes:

[0074] First, the center point of the entire point cloud is obtained by averaging the three coordinates of all points in the low-resolution point cloud;

[0075] Then, the Euclidean distances from all points to the center point are calculated for the low-resolution point cloud, and the maximum distance is taken as the normalized distance;

[0076] Finally, the center point coordinates are subtracted from the overall coordinates of the low and high resolution point clouds and the normalized distance is applied to normalize the values ​​of the three coordinates of the point cloud of the dataset to between -1 and 1 to generate a normalized point cloud.

[0077] Step S2, estimating the normal vector of the low-resolution point cloud at different selection radii, using the estimated normal vector to calculate the SPFH feature of the point cloud, and fusing the SPFH features at different selection radii to obtain the FPFH feature;

[0078] Specifically, step S2 includes:

[0079] Step S201, by statistically normalizing the distance distribution between points of the low-resolution point cloud, calling the open3d library normal vector estimation function, setting the neighbor point selection radius to 0.2 and 0.4, and estimating the normal vector information of each point in the point cloud;

[0080] Step S202: Combine the normal vector information and normalized coordinates of the point cloud, call the FPFH feature calculation function of the open3d library, and use the neighboring point information within different selection radii to calculate the FPFH features of each point in the point cloud under different selection radii. The calculation method of the FPFH feature is:

[0081] Step S2021: for the target point p s With one of its neighboring points p t Establish a local coordinate system, which includes:

[0082] Define p s With p t The corresponding normal vectors are n s 、n t , and then use ||·||22 to represent the square of the L2 norm, then the three coordinate axes of the local coordinate system are represented by u, v, and w as follows:

[0083] u=n s (1)

[0084]

[0085] w=u×v (3)

[0086] Step S2022: For the local coordinate system constructed in step S2021, the SPFH features of the two points are characterized by the following three angle values:

[0087] α=v·n t (4)

[0088]

[0089] θ=arctan(w·n t ,u·n t ) (6)

[0090] SPFH(p t )=(α,φ,θ) (7)

[0091] Step S2023: for any query point p q and its k neighbors p k , the FPFH features of the query points are calculated from their respective SPFH features:

[0092]

[0093] Step S203: Figure 3 As shown, a convolutional neural network is designed to perform feature fusion on FPFH features under multiple selected radii, and the dimension of the fused feature is the same as that of a single FPFH feature.

[0094] Specifically, in this embodiment, if Figure 2 As shown in the figure, any number of selection radii are selected to extract FPFH features of various scales, which are then concatenated and input into the convolutional neural network. The feature channel dimension of the concatenated fused features can be freely defined, and the fused features have the same feature dimension when the number of selected radii is different. In this example, the channel dimension of the fused feature is the same as that of a single FPFH feature.

[0095] Step S3, constructing a deep neural network based on densely connected convolutional neural network, k-nearest neighbor graph and feature expansion;

[0096] Specifically, in this embodiment, step S3 includes:

[0097] The above-mentioned deep neural network is composed of three cascaded network modules, namely, the network structure corresponding to the densely connected convolutional neural network, the k-nearest neighbor graph, and the feature expansion. The output feature level of each network module is progressive.

[0098] First, the input features are passed through a densely connected convolutional neural network, whose internal network uses a densely connected structure to map lower-dimensional coordinate information to a high-dimensional space to complete basic feature extraction.

[0099] Then, in order to further explore the relationship between features and realize the feature fusion of point features and neighboring point features, a k-nearest neighbor graph is established for the output features of the densely connected convolutional neural network to generate k-nearest neighbor graph features, and a residual relationship is established between the k-nearest neighbor graph features and the output features of the densely connected convolutional neural network to accelerate network convergence.

[0100] Finally, considering that the output features of the densely connected convolutional neural network better preserve the original information of the point, and the k-nearest neighbor graph features better integrate the neighboring point information relationship, by splicing these two features in the channel dimension, the same attention can be paid to single point features and neighboring point features. The spliced ​​feature input feature is expanded into the corresponding neural network structure, the number of feature channels is increased, and the feature is scaled, and the increase in channel dimension is converted into an increase in the number of point features. Therefore, the number of output expanded feature point features becomes 4 times the number of original input point features. After the expanded feature input is regressed in the fully connected layer, 4 times the number of high-resolution point clouds can be generated.

[0101] More specifically, in this embodiment, the above three network modules are described in more detail, including:

[0102] Based on densely connected convolutional neural networks, such as Figure 4 As shown in the figure, a densely connected neural network structure is designed, and each convolution layer uses a 1×1 convolution kernel; the network contains multiple layers of intermediate features, and the scale of each layer of intermediate features is the same. All the previous intermediate features are concatenated in the feature dimension and then sent to the convolution layer for convolution calculation;

[0103] The input and output feature dimensions of the densely connected convolutional neural network are set to the same, and multiple densely connected convolutional neural networks with the same structure can be cascaded to enhance the ability to extract point cloud information.

[0104] This embodiment uses a 4-level densely connected convolutional neural network with the same structure to fuse the FPFH features of the point cloud input from the 1st and 3rd level networks, and the coordinate features of the 2nd and 4th level networks;

[0105] k nearest neighbor graph, such as Figure 5As shown in the figure, a convolutional neural network structure based on the k nearest neighbor graph is designed, and each convolution layer uses a 1×1 convolution kernel. For the input features of each convolution layer, each point feature first uses the k nearest neighbor algorithm, and the k nearest neighbor point features constitute the neighborhood feature of the point. Then, the point feature is subtracted from the corresponding neighborhood feature to obtain the relative feature. The relative feature and the input point feature constitute the k nearest neighbor graph feature, which is sent to the convolution layer for feature transformation. By setting the output feature dimension of the convolution layer to be the same as the input feature, multiple k nearest neighbor graph convolutional neural networks with the same structure can be cascaded, and each network uses the k nearest neighbor algorithm once to dynamically generate the k nearest neighbor graph features of the network at this level and send them to the convolution layer of the network at this level. Residual connections are established between the features output by each network, and the final output features obtained are the same as the output feature dimension of the densely connected convolutional neural network, and are summed with the output features of the k nearest neighbor graph.

[0106] In this embodiment, two layers of k-nearest neighbor graph calculation modules with the same structure are used, and each module performs dynamic calculation of the k-nearest neighbor graph three times;

[0107] Feature expansion, such as Figure 6 As shown in the figure, a convolutional neural network structure with expanded feature number is designed, and each convolution layer uses a 1×1 convolution kernel. The feature expansion module contains multiple convolutional layers, and the number of convolution channels gradually increases from the number of input channels to 4 times. The output features of the densely connected convolutional neural network and the k-nearest neighbor graph are concatenated and sent to the module to obtain output features that are 4 times the number of input channels. After rearranging the feature dimensions, the number of feature points is expanded to 4 times the number of input points.

[0108] Step S4, using the low-resolution point cloud pairs in the point cloud data set in step S1 and the FPFH features corresponding to the low-resolution point cloud as the input of the deep neural network constructed in step S3, and using the high-resolution point cloud pairs corresponding to the low-resolution point cloud pairs as labels for supervised training, obtaining a training model after the training is completed, and then obtaining a super-resolution point cloud through the output of the training model.

[0109] Specifically, in this embodiment, step S4 includes:

[0110] Step S401: Use the point cloud data set preprocessed in step S1 to extract the FPFH features of the low-resolution point cloud through step S2, fuse the FPFH under different selected radii, input the point cloud coordinates and the fused FPFH features into the deep neural network designed in step S3, and output the predicted 4 times high-resolution point cloud;

[0111] Step S402: Use the chamfer distance as a loss function to constrain the distance between the point cloud predicted in step S401 and the corresponding real high-resolution point cloud in the data set, perform supervised training on the deep neural network, and obtain a training model after the loss function converges;

[0112] More specifically, in this embodiment, the chamfer distance can be expressed by the following formula:

[0113]

[0114] In formula (8), S1 and S2 represent two different point clouds, x and y represent the coordinates of any point in the two point clouds, respectively. represents the square of L2 norm;

[0115] Step S403: Perform a performance test on the trained model on the test set of the public point cloud dataset PU1K. The PU1K test set contains low-resolution point cloud pairs with different number of points. The number of points of the selected test set low-resolution point cloud is 2048, and the number of points of the high-resolution point cloud is 8192. The performance of the trained model is evaluated using the chamfer distance as an indicator.

[0116] like Figure 7 The figure shows the performance comparison between the method of this embodiment and the mainstream deep learning method of point cloud super-resolution. The method of this embodiment is tested on the test set of the PU1K public data set, and the average chamfer distance with the real sample is the lowest, indicating that the super-resolution performance of the method of this embodiment is better than other mainstream methods.

[0117] like Figure 8 The figure shows an example of the point cloud super-resolution effect of the method of this embodiment. The generated high-resolution point cloud has basically the same three-dimensional surface structure as the original low-resolution point cloud, and the distance between points is relatively uniform, so the generated point cloud contains more surface structure information than the original point cloud.

[0118] The matters not described in detail in the present invention are all known technologies to those skilled in the art.

[0119] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A point cloud super-resolution method based on FPFH features and deep neural network, characterized in that: The method comprises: Step S1, obtaining a point cloud data set, and performing normalization processing on the point cloud data set, wherein the point cloud data set includes high-resolution and low-resolution point cloud pairs; Step S2, estimating the normal vector of the low-resolution point cloud at different selection radii, using the estimated normal vector to calculate the SPFH feature of the point cloud, and fusing the SPFH features at different selection radii to obtain the FPFH feature; Step S3, constructing a deep neural network, which includes three network modules based on densely connected convolutional neural network, k nearest neighbor graph and feature expansion, wherein, first, the input feature is mapped to the high-dimensional space through the densely connected convolutional neural network, and the basic feature extraction is completed; then, the output feature of the densely connected convolutional neural network is input into the k nearest neighbor graph module, the k nearest neighbor graph is established, the k nearest neighbor graph feature is generated, and the k nearest neighbor graph feature and the output feature of the densely connected convolutional neural network establish a residual relationship to accelerate the network convergence; finally, the output features of the two network modules based on the densely connected convolutional neural network and the k nearest neighbor graph are spliced ​​and input into the feature expansion module to increase the number of feature channels and perform scale transformation on the features; Step S4, using the low-resolution point cloud pairs in the point cloud data set in step S1 and the FPFH features corresponding to the low-resolution point cloud as inputs to the deep neural network constructed in step S3, using the high-resolution point cloud pairs corresponding to the low-resolution point cloud pairs as labels for supervised training, obtaining a training model after the training is completed, and then obtaining a super-resolution point cloud through the output of the training model; The step S2 comprises: Step S201, by statistically normalizing the distance distribution between points of the low-resolution point cloud, calling the open3d library normal vector estimation function, setting the neighbor point selection radius to 0.2 and 0.4, and estimating the normal vector information of each point in the point cloud; Step S202, combining the normal vector information and the normalized coordinates of the point cloud, calling the FPFH feature calculation function of the open3d library, and using the neighboring point information within different selection radii to calculate the FPFH features of each point in the point cloud under different selection radii; Step S203, designing a convolutional neural network to perform feature fusion on the FPFH features under multiple selected radii, where the dimension of the fused feature is the same as that of a single FPFH feature; In step S202, the FPFH feature is calculated as follows: Step S2021: for the target point p s With one of its neighboring points p t Establish a local coordinate system, which includes: Define p s With p t The corresponding normal vectors are n s 、n t , and then use represents the square of the L2 norm, then the three coordinate axes of the local coordinate system are represented by u, v, and w as follows: u=n s (1) w=u×v(3) Step S2022: For the local coordinate system constructed in step S2021, the SPFH features of the two points are characterized by the following three angle values: α=v·n t (4) θ=arctan(w·n t ,u·n t )(6) SPFH(pt)=(α,φ,θ)(7) Step S2023: for any query point p q and its k neighbors p k , the FPFH features of the query points are calculated from their respective SPFH features: In step S3, the feature expansion specifically includes: Each convolution layer uses a 1×1 convolution kernel; the feature expansion module contains multiple convolution layers, and the number of convolution channels gradually increases from the number of input channels to 4 times; the output features of the densely connected convolutional neural network and the k-nearest neighbor graph are concatenated and sent to the module to obtain output features that are 4 times the number of input channels. After rearranging the feature dimensions, the number of feature points is expanded to 4 times the number of input points.

2. A point cloud super-resolution method based on FPFH features and deep neural network according to claim 1, characterized in that: The step S1 comprises: Step S101, obtain the public point cloud dataset PU1K, the training validation set of the dataset contains 69,000 point cloud pairs, the low-resolution point cloud contains 256 points, and the high-resolution point cloud contains 2048 points, which is a 4x super-resolution dataset; the test set contains point cloud data of 127 objects, each object has 4x super-resolution point cloud pairs of 4 scales, namely 256 points-1024 points, 512 points-2048 points, 1024 points-4096 points and 2048 points-8192 points. The point cloud data of the test set has not been processed by block processing, and each point cloud image represents a complete three-dimensional object; Step S102, randomly dividing the training validation set into a training set and a validation set, wherein the training set contains 65,000 point cloud pairs and the validation set contains 4,000 point cloud pairs; Step S103: normalize the input point cloud.

3. The point cloud super-resolution method based on FPFH features and deep neural network according to claim 2 is characterized in that: The step S103 specifically includes: First, the center point of the entire point cloud is obtained by averaging the three coordinates of all points in the low-resolution point cloud; Then, the Euclidean distances from all points to the center point are calculated for the low-resolution point cloud, and the maximum distance is taken as the normalized distance; Finally, the center point coordinates are subtracted from the overall coordinates of the low and high resolution point clouds and the normalized distance is applied to normalize the values ​​of the three coordinates of the point cloud of the dataset to between -1 and 1 to generate a normalized point cloud.

4. The point cloud super-resolution method based on FPFH features and deep neural network according to claim 1, characterized in that: In the step S3, the densely connected convolutional neural network includes: Each convolutional layer uses a 1×1 convolution kernel; the network contains multiple layers of intermediate features, and the scale of each layer of intermediate features is the same. All the previous intermediate features are concatenated in feature dimensions and then sent to the convolutional layer for convolution calculation. The input and output feature dimensions of the densely connected convolutional neural network are set to the same, and multiple densely connected convolutional neural networks with the same structure are cascaded to enhance the ability to extract point cloud information; Among them, a 4-level densely connected convolutional neural network with the same structure was used to fuse the FPFH features of the point cloud input from the 1st and 3rd level networks, and the coordinate features of the 2nd and 4th level network input.

5. The point cloud super-resolution method based on FPFH features and deep neural network according to claim 1, characterized in that: In step S3, the k nearest neighbor graph specifically includes: Each convolution layer uses a 1×1 convolution kernel. For the input features of each convolution layer, each point feature first uses the k nearest neighbor algorithm, and the k nearest neighbor point features constitute the neighborhood feature of the point. Then, the point feature is subtracted from the corresponding neighborhood feature to obtain the relative feature. The relative feature and the input point feature constitute the k nearest neighbor graph feature, which is sent to the convolution layer for feature transformation. By setting the output feature dimension of the convolutional layer to be the same as the input feature, multiple k-nearest neighbor graph convolutional neural networks with the same structure are cascaded, and each network uses the k-nearest neighbor algorithm once to dynamically generate the k-nearest neighbor graph features of the network at that level and send them to the convolutional layer of the network at that level; Residual connections are established between the features output by each network, and the final output features obtained have the same dimension as the output features of the densely connected convolutional neural network and are summed with the output features of the k-nearest neighbor graph.

6. The point cloud super-resolution method based on FPFH features and deep neural network according to claim 1, characterized in that: The step S4 specifically includes: Step S401: Using the point cloud data set preprocessed in step S1, extract the FPFH features of the low-resolution point cloud through step S2, fuse the FPFH under different selected radii, input the point cloud coordinates and the fused FPFH features into the deep neural network designed in step S3, and output the predicted high-resolution point cloud; Step S402: Use the chamfer distance as a loss function to constrain the distance between the point cloud predicted in step S401 and the corresponding real high-resolution point cloud in the data set, perform supervised training on the deep neural network, and obtain a training model after the loss function converges; Step S403: Perform a performance test on the training model on the test set of the public point cloud dataset PU1K.

7. The point cloud super-resolution method based on FPFH features and deep neural network according to claim 6, characterized in that: In step S402, the chamfer distance is expressed by the following formula: In formula (8), S1 and S2 represent two different point clouds, x and y represent the coordinates of any point in the two point clouds, respectively. Represents the square of the L2 norm.

Citation Information

Patent Citations

  • Reference image guided super-resolution method based on dense matching and adaptive fusion

    CN111915484A

  • Depth image matching method based on point cloud registration

    CN112017225A

  • Image super-resolution reconstruction method and apparatus for sharpening of label data

    WO2021052261A1