Vector neural network-based point cloud scene reconstruction method

Abstract

The invention discloses a point cloud scene reconstruction method based on a vector neural network, and belongs to the technical field of computer vision and three-dimensional reconstruction. Aiming at the defects of the existing point cloud scene reconstruction method in processing rotation change and complex scene reconstruction, the method comprises the following steps of: performing instance segmentation on point cloud scene data by utilizing PointNet, and identifying and segmenting each object in a scene; establishing a CAD model library covering common indoor scene objects, training a VN network, and performing feature extraction on each point and a neighborhood thereof by using a vector convolution layer; obtaining feature representations of different scales by stacking a plurality of vector convolution layers and combining residual connection and jump connection; then calculating the similarity between the point cloud features and the CAD model features, performing retrieval by adopting a Chamfer distance, and selecting the most similar CAD model as a candidate; and finally, aligning CAD model candidates with scene point clouds one by one, and introducing a physical violation check module to ensure the accuracy and physical compliance of the reconstructed scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and three-dimensional reconstruction, and in particular relates to a point cloud scene reconstruction method based on a vector neural network. Background Art

[0002] With the rapid development of artificial intelligence, household robots have gradually entered people's lives, assisting people in completing various daily tasks. Understanding and reconstructing scenes is one of the key challenges for robots to achieve autonomous navigation, environmental perception, and augmented reality experience. The environmental data obtained using 2D images is very limited and cannot meet the requirements of robots for scene reconstruction in unstructured environments. Therefore, using point cloud data to obtain scene information has become an effective means. However, single-view point cloud data itself has problems of incompleteness and noise, especially in complex scenes, and it is often impossible to fully capture the shape and position of all objects. In order to solve this problem, the CAD model of each object in the scene is required. The objects in the point cloud data are replaced with complete CAD models through model matching methods, thereby achieving complete modeling of the entire scene.

[0003] With the development of virtual reality, augmented reality and human-computer interaction technology, it is crucial to build realistic and accurate three-dimensional virtual scenes. At present, there are still some challenges in the process of 3D model replacement, including low accuracy of CAD model matching, the need to improve the precision of model alignment, and the inability to provide interactive scenes for intelligent applications such as autonomous decision-making and planning of robots. These technical bottlenecks have severely restricted the development of the robotics industry. Summary of the invention

[0004] In view of the shortcomings of current point cloud scene reconstruction methods in dealing with rotation changes and complex scene reconstruction, the present invention provides a point cloud scene reconstruction method based on a vector neural network, which improves the accuracy and efficiency of three-dimensional scene reconstruction and overcomes the limitations of existing technologies in dealing with rotation changes and complex scene reconstruction.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A point cloud scene reconstruction method based on a vector neural network comprises the following steps:

[0007] Step 1, point cloud scene data acquisition: obtain the single-view point cloud scanning data of the scene, filter, denoise and downsample it, eliminate noise and redundant points, and obtain optimized point cloud data;

[0008] Step 2: Instance segmentation: Use PointNet to perform instance segmentation on the optimized point cloud data, identify the objects in the scene, and finally output the instance labels of different objects in the point cloud;

[0009] Step 3, feature extraction: Calculate the normal vector of the point cloud to obtain local geometric information. Then, use the principal component analysis (PCA) method to extract the category, geometric features (such as normal vector, curvature) and color features (such as color histogram, color space components) of each object, obtain the point-by-point features and global features of the partial point cloud, and the point-by-point features and global features of the corresponding complete CAD model, and form the feature descriptor of the point cloud as the input of CAD model retrieval;

[0010] Step 4, build a CAD model library: build a database containing multiple CAD models, covering various objects appearing in the scene; discretize each CAD model, generate a point cloud representation of the model, extract features from the model point cloud, obtain its global and local feature descriptors; and store them in the database for subsequent matching and alignment tasks;

[0011] Step 5, CAD model rough matching: build a model training environment, input the point cloud data into the vector neural network model for training, optimize the model weight parameters, output the model candidate set, and obtain the best model weight parameters;

[0012] Step 6, CAD model fine alignment: Select the CAD model with the highest similarity (minimum matching loss) with the target point cloud from the candidate set; resize and align the selected CAD model with 9DoF posture based on the local feature descriptor to achieve precise alignment of the point cloud and the CAD model; fuse the aligned CAD model with the point cloud to generate a complete scene reconstruction result;

[0013] Step 7, result verification: calculate the alignment error between the aligned model and the target point cloud. If the error is within the preset range, the match is considered successful, and the successfully matched CAD model and alignment parameters are output; if the alignment error exceeds the preset threshold, return to step 6 and select the next candidate CAD model for alignment;

[0014] Step 8: Output the reconstructed scene: Replace the identified objects in the scene with the successfully matched and precisely aligned CAD models to generate a complete reconstructed scene.

[0015] Furthermore, the point cloud data of the scene collected in step 1 is collected using a high-precision three-dimensional laser scanning device called LiDAR to collect point cloud data of the target scene. Under a single viewing angle, the scene is completely scanned once to ensure that the required area is captured as completely as possible. According to the complexity of the scene, parameters such as scanning resolution and scanning speed are adjusted to balance data accuracy and collection efficiency. The collected point cloud data is saved in the PLY standard format.

[0016] Eliminate points with infinite distance, zero coordinates or outliers in the point cloud data. For each point p i , calculate the average distance of points in its neighborhood

[0017]

[0018] Among them, knn is the size of the neighborhood in the point cloud, indicating that each point p i The number of k-nearest neighbors, j k Represents point p i A point in the neighborhood of , that is, the jth point in the k-nearest neighbor k points;

[0019] By setting a threshold, outliers beyond a certain standard deviation range are removed to eliminate isolated noise points. A fixed radius r is set, and the number of neighboring points within the radius of each point is counted. If the number of neighboring points is less than the preset threshold, the point is considered a noise point and removed.

[0020] Perform surface fitting and smoothing on the point cloud to smooth the point cloud data while retaining edge features. Divide the point cloud space into a voxel grid of size l×l×l, and in each voxel, only one representative point (centroid) is retained for each voxel:

[0021]

[0022] Among them, p voxel Represents the coordinates of the representative point (center of mass) in the voxel; N voxel Indicates the number of points contained in the current voxel;

[0023] Reduce the number of points and evenly distribute the point cloud. Normalize the point cloud coordinates to the unit cube, calculate the bounding box of the point cloud, translate the center of the point cloud to the origin, and scale it according to the maximum side length. Convert the preprocessed point cloud data into the HDF5 format required for VN network input.

[0024] Furthermore, the instance segmentation using PointNet in step 2 specifically includes the following steps:

[0025] Step 2.1, for each point p in the point cloud i A series of fully connected layers are applied independently to extract local features of each point;

[0026] h i =MLP(p i ) (3)

[0027] Step 2.2: Use Max Pooling to aggregate the local features of all points to obtain the global feature representation.

[0028]

[0029] Step 2.3, concatenate the local features of each point with the global features to form enhanced features;

[0030] g i =[h i ,g] (5)

[0031] Step 2.4, use the enhanced features to perform instance segmentation on each point and output the instance label of each object in the scene after segmentation;

[0032]

[0033] Furthermore, in step 4, all models are converted to the same OBJ file format, and the right-handed coordinate system is uniformly used to calculate the bounding box size of the model (l x ,l y ,i z ), choose the maximum edge length:

[0034] l max =max(l x ,l y ,l z ) (7)

[0035] Scale all vertex coordinates of the model proportionally:

[0036]

[0037] Through principal component analysis (PCA), the direction of the main axis of the model is calculated, the model is rotated so that its main axis is aligned with the global coordinate system axis, and the bottom of the model is ensured to be located on the z = 0 plane. Each model is assigned an object category (such as "chair", "table", "lamp", etc.), and the attributes of the model (such as material, color, etc.) are recorded.

[0038] Furthermore, in step 4, each CAD model is discretized to generate a point cloud representation of the model, which specifically includes the following steps:

[0039] Step 4.1, sample points at uniform intervals on the model surface. For each triangular patch, determine the number of sampling points on the patch based on its area and the total number of sampling points required:

[0040]

[0041] Among them, A j is the area of ​​the jth patch, A total is the total surface area of ​​the model, N total is the expected total number of sampling points;

[0042] Step 4.2, on each patch, use the random barycentric coordinate method to generate sampling points:

[0043] P = αv 1 +βv 2 +γv 3 (10)

[0044] Among them, α, β, and γ are not 0, and α+β+γ=1;

[0045] Step 4.3, for each vertex, calculate the weighted average of its adjacent face normal vectors:

[0046]

[0047] Among them, F i is the set of faces adjacent to the vertex, n j is the patch normal vector;

[0048] Step 4.4, normalize each normal vector to a unit vector:

[0049]

[0050] Furthermore, the vector neural network in step 5 expands the feature from a scalar to a vector, that is, each feature dimension is a vector: α =[v α1 ,v α2 ,..,v αt ],v αt ∈R 3 ;

[0051] Among them, f α represents the αth eigenvector; v αt Denotes the eigenvector f α The vector value in the tth dimension;

[0052] For an input vector v∈R 3 , calculate its length:

[0053]

[0054] Where ||v|| represents the Euclidean norm of any input vector v, that is, the geometric distance of the vector from the origin to its end point; Respectively represent the square values ​​of the components of vector v in the x, y, and z directions;

[0055] Then, the nonlinear activation function is applied to the length of the vector and multiplied by the unit vector of the original vector, expressed as:

[0056]

[0057] Among them, u represents the activated vector; represents the scalar nonlinear activation function ReLU; v represents any input;

[0058] The vector neural network applies a nonlinear activation function to each component of the input vector to make the vector obtain element-wise nonlinearity (Element-wiseActivation). The calculation formula is as follows:

[0059]

[0060] Where T represents the transpose of a matrix or vector;

[0061] The vector neural network is orthogonal and has a determinant of 1 through the rotation matrix R∈SO(3), so that for any input v, it satisfies:

[0062] f(Rv)=Rf(v) (16)

[0063] Ensure rotation equivariance, that is, when the input of the network is rotated, the output will rotate in the same way, maintaining the geometric relationship between the output and the input.

[0064] Furthermore, the vector neural network model in step 5 includes an input layer, a vector neuron feature extraction layer, a feature aggregation layer, and a matching layer. The specific training steps are as follows:

[0065] Step 5.1, data preprocessing: receive point cloud data, the coordinates and attributes (such as color, normal vector, etc.) of each point, and then organize the data (including format conversion and data cleaning); then perform data enhancement operations (such as random rotation, scaling, translation and adding noise) on the point cloud data to enhance the generalization ability of the model, and finally normalize the point cloud coordinates to a unified coordinate system or scale range;

[0066] Step 5.2, feature extraction: input the preprocessed point cloud data into the input layer of the vector neural network model for initial feature mapping; extract features for each point and its neighborhood points through the vector convolution layer in the network to capture fine-grained geometric shape and texture information; obtain feature representations of different scales by stacking multiple vector convolution layers and using residual connections and skip connections; gradually increase the dimension of local features through multi-level feature extraction to extract global shape features;

[0067] Step 5.3, feature fusion: The features extracted at different levels are fused through the feature aggregation layer, and the feature fusion module is used to integrate local and global information; the attention mechanism is introduced to weight the fusion features, increase the focus on important features, suppress the influence of noise, and enhance the discrimination ability of the model;

[0068] Step 5.4, CAD model matching: Calculate the similarity between the fused point cloud features and the model features in the CAD model library, use the matching error to measure the matching accuracy, and select the best matching CAD model; based on the loss function, use the optimizer to update the model weight parameters to improve the matching and alignment capabilities of the model;

[0069] Step 5.5, matching output: output the top k CAD models with the smallest loss as the model candidate set of the target point cloud.

[0070] Furthermore, the matching error in step 5.4 includes Chamfer distance loss, symmetry loss, residual retrieval loss and residual regularization loss; the matching output in step 5.5 is to assign different weights to the Chamfer distance loss, symmetry loss, residual retrieval loss and residual regularization loss in step 5.4 to jointly measure the similarity between the partial scan and the complete CAD model; finally, the top k CAD models with the smallest matching errors are output as the candidate set.

[0071] Chamfer distance loss, used to measure the similarity between two point sets;

[0072] Given a source point set P and a target point set Q:

[0073]

[0074] Symmetry loss:

[0075]

[0076] Wherein, S(p) represents the symmetric point of point p with respect to the symmetry axis or symmetry plane.

[0077] Residual retrieval loss, which measures the residual loss between the model prediction and the true alignment:

[0078] L Residual =∑||T pred (p)-T gt (q)|| 2 (19)

[0079] Among them, T pred is the predicted transformation, T gt It is a real transformation.

[0080] Residual regularization loss, used to prevent overfitting:

[0081]

[0082] Where W represents the weight matrix in the network:

[0083]

[0084] ||.|| F represents the Frobenius norm, which is used to calculate the square root of the sum of the squares of the matrix elements; different loss functions are weighted and integrated to form a total loss function L (matching error), which is assigned different weights to jointly measure the similarity between the target point cloud (partial scan) and the source point cloud (complete CAD model);

[0085] L=αL Chamfer +βL Symmetry +γL Residual +δL Regularization (twenty two)

[0086] Among them, α, β, γ, δ are weight coefficients; finally, the first km CAD model candidates with the smallest matching error are output.

[0087] Furthermore, in step 6, based on the local feature descriptor, the selected CAD model is resized and 9DoF posture aligned to achieve accurate alignment of the point cloud and the CAD model. Specifically, the following steps are performed:

[0088] According to the features of the point cloud (such as center point, normal vector) and the corresponding features of the CAD model, the random sampling consistency RANSAC algorithm is used to find matching feature point pairs. The iterative closest point (ICP) algorithm is used to calculate the optimal rigid transformation matrix (including translation, rotation and scaling parameters) between the point cloud to be matched and the CAD model. The CAD model is transformed so that it is accurately aligned with the original point cloud in space.

[0089] Further, the specific operation of step 7 includes the following steps:

[0090] Step 7.1, calculate the root mean square error (RMSE) and alignment residual between the aligned model and the target point cloud. If the error is within the threshold range, the match is successful;

[0091] Step 7.2, introduce the physical violation check module to check the physical violation of the scene after fine alignment; the relationship between objects in the scene is divided into close proximity relationship and support relationship; for close proximity relationship, check whether the 3D bounding boxes of adjacent objects intersect or overlap, if there is intersection or overlap, there is a physical violation; at the same time, use heuristic methods to detect the support relationship between objects, the distance between the lower surface of the upper object and the upper surface of the lower object is less than a certain threshold, the contact surface area is greater than a certain threshold, and the angle of the contact surface is within a preset range. The heuristic conditions are set by experience;

[0092] Step 7.3: If the alignment error exceeds the threshold or the aligned model has physical violations, the current model is discarded, the next model candidate is selected, and the alignment and checking process is repeated until all objects in the scene are physically compliant or the candidate set is exhausted.

[0093] Compared with the prior art, the present invention has the following advantages:

[0094] 1. Improve the accuracy and completeness of scene reconstruction: By introducing a vector neural network, the geometric and texture information of point cloud data is fully utilized, the processing capabilities of rotational changes and complex scenes are enhanced, and the problem of insufficient reconstruction accuracy in existing technologies is effectively solved.

[0095] 2. Realize efficient extraction and fusion of local and global features: Use vector convolutional layers to gradually extract local and global features, and fuse the features through residual connections, skip connections, and attention mechanisms, which enhances the model's focus on important features and suppresses noise interference, thereby improving the expressiveness and discrimination capabilities of the features.

[0096] 3. Support precise alignment of multiple objects in complex scenes: By building a CAD model library and combining coarse matching and fine alignment steps, the RANSAC and ICP algorithms are used to achieve 9DoF posture alignment of multiple objects, ensuring precise spatial matching between the point cloud and the CAD model, thereby improving the precision of scene reconstruction.

[0097] 4. Strong robustness and adaptability to a variety of complex scenarios: Through the introduction of data enhancement (such as random rotation, scaling, translation, etc.) and nonlinear activation functions, the model's robustness to rotation, scale changes and noise is enhanced, and it can adapt to a variety of complex unstructured scenarios.

[0098] 5. Joint optimization of multiple loss functions for more accurate matching: Combining multiple loss functions such as Chamfer distance loss, symmetry loss, and residual retrieval loss, a weighted comprehensive measure of the similarity between the point cloud and the CAD model is performed, which significantly improves the accuracy of model matching.

[0099] 6. Physical violation check function to ensure scene rationality: Introduce a physical violation check module to verify the physical rationality of the reconstructed scene (such as proximity and support relationship checks) to ensure that the arrangement of objects in the scene complies with physical laws, thereby improving the actual usability of the scene.

[0100] 7. Efficient point cloud preprocessing and feature normalization: Through preprocessing methods such as filtering, denoising, downsampling and normalization, the redundancy and noise in the point cloud data are reduced, the distribution and representation of the point cloud are optimized, and high-quality input is provided for subsequent feature extraction and matching.

[0101] 8. Unified CAD model standardization processing: Discretization, principal axis alignment, normalization and other processing are performed on CAD models to ensure data consistency in the model library and lay the foundation for efficient matching and alignment.

[0102] 9. Efficient training and reasoning: Through the design optimization of vector neural network, combined with feature aggregation and attention mechanism, the training efficiency and reasoning speed of the model are improved, which is suitable for large-scale scene reconstruction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 This is a sample scene of the standard dataset Scannet.

[0104] Figure 2 It is a constructed CAD model library.

[0105] Figure 3 This is an example scene after instance segmentation.

[0106] Figure 4 It is a vector neural network structure diagram.

[0107] Figure 5 is a schematic diagram of the CAD model matching and alignment process.

[0108] Figure 6 This is an example of the output reconstructed scene. DETAILED DESCRIPTION

[0109] In order to gain a deeper understanding of the present invention, a comprehensive and detailed description will be given herein. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen a comprehensive understanding of the disclosure of the present invention.

[0110] Example 1

[0111] A point cloud scene reconstruction method based on a vector neural network comprises the following steps:

[0112] Step 1, point cloud scene data acquisition: Use a laser scanner to scan the indoor scene, obtain the single-view point cloud scan data of the scene, filter, denoise and downsample it, eliminate noise and redundant points, and obtain optimized point cloud data; ensure computational efficiency while retaining key features. Normalize the bounding box of the point cloud, translate its center to the origin, and scale it according to the maximum side length. Convert the preprocessed point cloud data into the HDF5 format required for VN network input;

[0113] Step 2: Instance segmentation: Use PointNet to perform instance segmentation on the optimized point cloud data, identify the objects in the scene, and finally output the instance labels of different objects in the point cloud;

[0114] Step 2.1, apply a series of fully connected layers to each point in the point cloud independently to extract the local features of each point;

[0115] h i =MLP(p i )

[0116] Step 2.2, use maximum pooling to aggregate the local features of all points to obtain the global feature representation;

[0117]

[0118] Step 2.3, concatenate the local features of each point with the global features to form enhanced features;

[0119] g i =[h i ,g]

[0120] Step 2.4, use the enhanced features to perform instance segmentation on each point and output the instance label of each object in the scene after segmentation;

[0121]

[0122] Step 3, feature extraction: Calculate the normal vector of the point cloud to obtain local geometric information. Then, use the principal component analysis (PCA) method to extract the category, geometric features (such as normal vector, curvature) and color features (such as color histogram, color space components) of each object, obtain the point-by-point features and global features of the partial point cloud, and the point-by-point features and global features of the corresponding complete CAD model, and form the feature descriptor of the point cloud as the input of CAD model retrieval;

[0123] Step 4, build a CAD model library: collect a large number of free and diverse CAD models of target objects from public 3D model databases such as ShapeNet and ModelNet. The categories include common furniture (chairs, tables, sofas), home appliances (TVs, refrigerators), daily necessities (books, cups), etc. in indoor scenes, and build a database containing multiple CAD models, covering various objects that appear in the scene; discretize each CAD model, generate a point cloud representation of the model, extract features from the model point cloud, and obtain its global and local feature descriptors; and store them in the database for subsequent matching and alignment tasks;

[0124] Step 4.1, sample points at uniform intervals on the model surface. For each triangular patch, determine the number of sampling points on the patch based on its area and the total number of sampling points required:

[0125]

[0126] Among them, A j is the area of ​​the jth patch, A total is the total surface area of ​​the model, F total is the expected total number of sampling points;

[0127] Step 4.2, on each patch, use the random barycentric coordinate method to generate sampling points:

[0128] P = αv 1 +βv 2 +γv 3

[0129] Among them, α, β, and γ are not 0, and α+β+γ=1;

[0130] Step 4.3, for each vertex, calculate the weighted average of its adjacent face normal vectors:

[0131]

[0132] Among them, F i is the set of faces adjacent to the vertex, n j is the patch normal vector;

[0133] Step 4.4, normalize each normal vector to a unit vector:

[0134]

[0135] Step 5, CAD model rough matching: build a model training environment, download and configure the Vector Neuron Network model code, input the point cloud data into the Vector Neuron Network model for training, optimize the model weight parameters, output the model candidate set, and obtain the best model weight parameters;

[0136] The vector neural network model comprises an input layer, a vector neural network feature extraction layer, a feature aggregation layer and a matching layer.

[0137] The specific training steps are as follows:

[0138] Step 5.1, data preprocessing: receiving point cloud data, the coordinates and attributes of each point, and then sorting the data; performing data enhancement operations on the point cloud data and finally normalizing the point cloud coordinates to a unified coordinate system or scale range;

[0139] Step 5.2, feature extraction: input the preprocessed point cloud data into the input layer of the vector neural network model for initial feature mapping; extract features for each point and its neighborhood points through the vector convolution layer in the network to capture fine-grained geometric shape and texture information; obtain feature representations of different scales by stacking multiple vector convolution layers and using residual connections and skip connections; gradually increase the dimension of local features through multi-level feature extraction to extract global shape features;

[0140] Step 5.3, feature fusion: The features extracted at different levels are fused through the feature aggregation layer, and the feature fusion module is used to integrate local and global information; the attention mechanism is introduced to weight the fusion features, increase the focus on important features, suppress the influence of noise, and enhance the discrimination ability of the model;

[0141] Step 5.4, CAD model matching: Calculate the similarity between the fused point cloud features and the model features in the CAD model library, use the matching error to measure the matching accuracy, and select the best matching CAD model; based on the loss function, use the optimizer to update the model weight parameters to improve the matching and alignment capabilities of the model; the matching error includes Chamfer distance loss, symmetry loss, residual retrieval loss, and residual regularization loss;

[0142] Step 5.5, matching output: output the first km of CAD models with the smallest loss as the model candidate set of the target point cloud.

[0143] Different weights are assigned to Chamfer distance loss, symmetry loss, residual retrieval loss and residual regularization loss to form a matching error, which is used to measure the similarity between the partial scan (target point cloud) and the complete CAD model (source point cloud); finally, the first km of CAD model candidates with the smallest matching error are output.

[0144] Step 6, precise alignment of CAD models: select the CAD model with the highest similarity (minimum matching loss) with the target point cloud from the candidate set; based on the local feature descriptor, resize and align the selected CAD model with 9DoF posture to achieve precise alignment of the point cloud and the CAD model; use the random sampling consistency RANSAC algorithm to find matching feature point pairs based on the features of the point cloud (such as center point, normal vector) and the corresponding features of the CAD model, and use the iterative closest point (ICP) algorithm to calculate the optimal rigid transformation matrix (including translation, rotation and scaling parameters) between the point cloud to be matched and the CAD model, and transform the CAD model to align it precisely with the original point cloud in space; fuse the aligned CAD model with the point cloud to generate a complete scene reconstruction result;

[0145] Step 7, result verification: calculate the alignment error between the aligned model and the target point cloud. If the error is within the preset range, the match is considered successful, and the successfully matched CAD model and alignment parameters are output; if the alignment error exceeds the preset threshold, return to step 6 and select the next candidate CAD model for alignment;

[0146] Step 7.1, calculate the root mean square error (RMSE) and alignment residual between the aligned model and the target point cloud. If the error is within the threshold range, the match is successful;

[0147] Step 7.2, introduce the physical violation check module to check the physical violation of the scene after fine alignment; the relationship between objects in the scene is divided into close proximity relationship and support relationship; for close proximity relationship, check whether the 3D bounding boxes of adjacent objects intersect or overlap, if there is intersection or overlap, there is a physical violation; at the same time, use heuristic methods to detect the support relationship between objects, the distance between the lower surface of the upper object and the upper surface of the lower object is less than a certain threshold, the contact surface area is greater than a certain threshold, and the angle of the contact surface is within a preset range. The heuristic conditions are set by experience;

[0148] Step 7.3: If the alignment error exceeds the threshold or the aligned model has physical violations, the current model is discarded, the next model candidate is selected, and the alignment and checking process is repeated until all objects in the scene are physically compliant or the candidate set is exhausted;

[0149] Step 8: Output the reconstructed scene: Replace the identified objects in the scene with the successfully matched and precisely aligned CAD models to generate a complete reconstructed scene, including the CAD models of each object and its position and posture in the scene. This scene can be used in applications such as virtual reality and robot navigation.

[0150] Example 2

[0151] Experimental evaluation criteria:

[0152] In the point cloud scene reconstruction algorithm based on vector neural network, in order to comprehensively evaluate the reconstruction accuracy and efficiency of the algorithm in different scenarios, experimental tests were conducted on three types of object point cloud data sets. Combining common 3D point cloud evaluation indicators with general indicators used in some reconstruction fields, reconstruction accuracy (ReconstructionAccuracy, Ra), recall (recall, R), mean average precision (mAP), parameters (Parameters, Pa) and mean reconstruction error (MeanReconstructionError, MRE) were used as evaluation indicators for this experiment. The following are the main meanings and calculation methods of each indicator for reference in the subsequent experimental results analysis:

[0153]

[0154] Among them, TP represents the number of correctly reconstructed objects (real objects are correctly reconstructed), FP represents the number of incorrectly reconstructed objects (non-existent objects are incorrectly reconstructed), and FN represents the situation where real objects are missed or misclassified as other categories. For point cloud scene reconstruction, TP, FP, and FN correspond to the number of objects that are correctly fitted to the CAD model in the reconstructed scene, the number of objects that are incorrectly fitted or misidentified, and the number of objects that are missed. n is the total number of categories, AP i is the average precision of the category, calculated as the area under the precision-recall (PR) curve. i is the number of parameters of the i-th layer, and la is the total number of layers in the model. i To reconstruct the i-th point in the point cloud, Y i is the i-th point in the real point cloud, and N is the number of points in the point cloud.

[0155] Data preparation:

[0156] This paper selects a large-scale indoor scene dataset (ScanNet) and a self-made indoor scene point cloud dataset to test the proposed point cloud reconstruction method based on vector neural network. In the data preprocessing stage, the collected or public point cloud data is filtered, denoised, downsampled and normalized to ensure that the point cloud is evenly distributed and retains the main geometric features.

[0157] Dataset experimental environment:

[0158] The experiments were conducted in the following hardware and software environment: 12th Gen Intel(R) Core(TM) i7-12700CPU, RTX 3090GPU, Ubuntu 20.04 operating system, using Pytorch 1.13.1 deep learning framework to build algorithms, Python 3.9.10 compiled language, and CUDA 11.6 to accelerate training.

[0159] Dataset experiments:

[0160] The present invention conducts point cloud scene reconstruction experiments based on vector neural network (VN network), the training batch size is set to 64, the initial learning rate is 0.001, the optimizer uses adam, the number of training rounds is 300, the momentum parameter is set to 0.9, the weight decay is set to 0.0005, and the point cloud resolution is set to 1024 points. The experimental results are as follows.

[0161] Table 1 Experimental results of different methods on ScanNet dataset

[0162]

[0163]

[0164] Table 2 Experimental results of different methods on self-made indoor scene dataset

[0165]

[0166] Ablation experiment:

[0167] In order to verify the effectiveness of the various improved modules proposed in this paper, the improved modules are gradually introduced into the original VN network for ablation experiments. This ablation experiment is divided into four groups, which are trained on the ScanNet dataset and the self-made indoor scene dataset respectively, and the experimental parameters are kept consistent. Experiment A introduces the attention mechanism; Experiment B improves the vectorization of the feature extraction layer; Experiment C adds a neighborhood-adaptive aggregation strategy to the feature aggregation layer; Experiment D introduces physical violation checks and lightweight processing in the alignment module. The ablation experiment results of training 300 epochs are shown in Tables 3 and 4.

[0168] Table 3 Ablation experiment results on ScanNet dataset

[0169]

[0170] Table 4 Ablation experiment results on the self-made indoor scene dataset

[0171]

[0172] In this example, on ScanNet and self-made indoor scene datasets, the point cloud scene reconstruction method based on vector neural network proposed in the present invention has significantly improved reconstruction accuracy, recall rate and average precision compared with existing methods (such as PointNet++, DGCNN and VN), while the number of parameters and reconstruction error have been significantly reduced. Ablation experiments show that each improved module proposed in this paper has a positive effect on model performance. The present invention shows strong robustness and efficiency in point cloud scene reconstruction tasks, and is suitable for three-dimensional reconstruction tasks of complex scenes.

[0173] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A point cloud scene reconstruction method based on vector neural network, characterized in that: The following steps are involved: Step 1, point cloud scene data acquisition: obtain the single-view point cloud scanning data of the scene, filter, denoise and downsample it, eliminate noise and redundant points, and obtain optimized point cloud data; Step 2: Instance segmentation: Use PointNet to perform instance segmentation on the optimized point cloud data, identify the objects in the scene, and finally output the instance labels of different objects in the point cloud; Step 3, feature extraction: Calculate the normal vector of the point cloud to obtain local geometric information. Then, use the principal component analysis method to extract the category, geometric features and color feature information of each object, obtain the point-by-point features and global features of the partial point cloud, and the point-by-point features and global features of the corresponding complete CAD model, and form the feature descriptor of the point cloud as the input of CAD model retrieval; Step 4, build a CAD model library: build a database containing multiple CAD models, covering various objects appearing in the scene; discretize each CAD model, generate a point cloud representation of the model, extract features from the model point cloud, and obtain its global and local feature descriptors; Step 5, CAD model rough matching: build a model training environment, input the point cloud data into the vector neural network model for training, optimize the model weight parameters, output the model candidate set, and obtain the best model weight parameters; Step 6, CAD model alignment: Select the CAD model with the highest similarity to the target point cloud from the candidate set; resize and align the selected CAD model with 9DoF posture based on the local feature descriptor to achieve precise alignment of the point cloud and the CAD model; fuse the aligned CAD model with the point cloud to generate a complete scene reconstruction result; Step 7, result verification: calculate the alignment error between the aligned model and the target point cloud. If the error is within the preset range, the match is considered successful, and the successfully matched CAD model and alignment parameters are output; If the alignment error exceeds the preset threshold, return to step 6 and select the next candidate CAD model for alignment; Step 8: Output the reconstructed scene: Replace the identified objects in the scene with the successfully matched and precisely aligned CAD models to generate a complete reconstructed scene.

2. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: The step 1, point cloud scene data acquisition: obtain the single-view point cloud scanning data of the scene, filter, denoise and downsample it, eliminate noise and redundant points, and obtain optimized point cloud data. Specifically: Eliminate points with infinite distance, zero coordinates or outliers in the point cloud data. For each point p i , calculate the average distance of points in its neighborhood Among them, knn is the size of the neighborhood in the point cloud, indicating that each point p i The number of k-nearest neighbors, j k Represents point p i A point in the neighborhood of , that is, the jth point in the k-nearest neighbor k points; By setting a threshold, outliers beyond a certain standard deviation range are removed to eliminate isolated noise points; a fixed radius r is set, and the number of neighboring points within the radius of each point is counted. If the number of neighboring points is less than the preset threshold, the point is considered a noise point and is removed; Perform surface fitting and smoothing on the point cloud, smooth the point cloud data while retaining edge features; divide the point cloud space into a voxel grid of size l×l×l, and in each voxel, only retain one representative point centroid for each voxel: Among them, p voxel Represents the coordinates of the representative point in the voxel; N voxel Indicates the number of points contained in the current voxel; Reduce the number of points and evenly distribute the point cloud; normalize the point cloud coordinates to the unit cube, calculate the bounding box of the point cloud, translate the center of the point cloud to the origin, and scale it according to the maximum side length.

3. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: In step 2, using PointNet to perform instance segmentation specifically includes the following steps: Step 2.1, apply a series of fully connected layers to each point in the point cloud independently to extract the local features of each point; h i =MLP(p i ) (3) Step 2.2, use maximum pooling to aggregate the local features of all points to obtain the global feature representation; Step 2.3, concatenate the local features of each point with the global features to form enhanced features; g i =[h i ,g] (5) Step 2.4, use the enhanced features to perform instance segmentation on each point and output the instance label of each object in the scene after segmentation; 4. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: In step 4, each CAD model is discretized to generate a point cloud representation of the model, which specifically includes the following steps: Step 4.1, sample points at uniform intervals on the model surface. For each triangular patch, determine the number of sampling points on the patch based on its area and the total number of sampling points required: Among them, A j is the area of ​​the jth patch, A total is the total surface area of ​​the model, N total is the expected total number of sampling points; Step 4.2, on each patch, use the random barycentric coordinate method to generate sampling points: P=αv1+βv2+γv3 (8) Among them, α, β, and γ are not 0, and α+β+γ=1; Step 4.3, for each vertex, calculate the weighted average of its adjacent face normal vectors: Among them, F i is the set of faces adjacent to the vertex, n j is the patch normal vector; Step 4.4, normalize each normal vector to a unit vector:

5. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: The vector neural network model comprises an input layer, a vector neural network feature extraction layer, a feature aggregation layer and a matching layer.

6. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: The step 5 inputs the point cloud data into the vector neural network model for training. The specific training steps are as follows: Step 5.1, data preprocessing: receiving point cloud data, the coordinates and attributes of each point, and then sorting the data; performing data enhancement operations on the point cloud data and finally normalizing the point cloud coordinates to a unified coordinate system or scale range; Step 5.2, feature extraction: input the preprocessed point cloud data into the input layer of the vector neural network model for initial feature mapping; extract features for each point and its neighborhood points through the vector convolution layer in the network to capture fine-grained geometric shape and texture information; obtain feature representations of different scales by stacking multiple vector convolution layers and using residual connections and skip connections; gradually increase the dimension of local features through multi-level feature extraction to extract global shape features; Step 5.3, feature fusion: The features extracted at different levels are fused through the feature aggregation layer, and the feature fusion module is used to integrate local and global information; the attention mechanism is introduced to weight the fusion features, increase the focus on important features, suppress the influence of noise, and enhance the discrimination ability of the model; Step 5.4, CAD model matching: Calculate the similarity between the fused point cloud features and the model features in the CAD model library, use the matching error to measure the matching accuracy, and select the best matching CAD model; based on the loss function, use the optimizer to update the model weight parameters to improve the matching and alignment capabilities of the model; Step 5.5, matching output: output the first km of CAD models with the smallest loss as the model candidate set of the target point cloud.

7. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: The matching error in step 5.4 includes Chamfer distance loss, symmetry loss, residual retrieval loss and residual regularization loss; the matching output in step 5.5 is to give different weights to the Chamfer distance loss, symmetry loss, residual retrieval loss and residual regularization loss in step 5.4 to jointly measure the similarity between the partial scan and the complete CAD model; finally, the first km CAD models with the smallest matching errors are output as candidate sets.

8. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: The vector neural network in step 5 expands the features from scalars to vectors, that is, each feature dimension is a vector: α =[v α1 ,v α2 ,..,v αt ],v αt ∈R 3 ; Among them, f α represents the αth eigenvector; v αt Denotes the eigenvector f α The vector value in the tth dimension; For an input vector v∈R 3 , calculate its length: Where ||v|| represents the Euclidean norm of any input vector v, that is, the geometric distance of the vector from the origin to its end point; Respectively represent the square values ​​of the components of vector v in the x, y, and z directions; Then, the nonlinear activation function is applied to the length of the vector and multiplied by the unit vector of the original vector, expressed as: Among them, u represents the activated vector; represents the scalar nonlinear activation function ReLU; v represents any input; The vector neural network applies a nonlinear activation function to each component of the input vector to make the vector element-by-element nonlinear. The calculation formula is as follows: Where T represents the transpose of a matrix or vector; The vector neural network is orthogonal and has a determinant of 1 through the rotation matrix R∈SO(3), so that for any input v, it satisfies: f(Rv)=Rf(v) (14) Ensure rotational equivalence.

9. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: In step 6, based on the local feature descriptor, the selected CAD model is resized and 9DoF posture aligned to achieve accurate alignment of the point cloud and the CAD model. Specifically, the following steps are performed: According to the features of the point cloud and the corresponding features of the CAD model, the random sampling consistency RANSAC algorithm is used to find matching feature point pairs. The iterative closest point algorithm is used to calculate the optimal rigid transformation matrix between the point cloud to be matched and the CAD model. The CAD model is transformed so that it is accurately aligned with the original point cloud in space.

10. The point cloud scene reconstruction method based on vector neural network according to claim 1, characterized in that: The specific operation of step 7 includes the following steps: Step 7.1, calculate the root mean square error and alignment residual between the aligned model and the target point cloud. If the error is within the threshold range, the match is successful; Step 7.2, introduce the physical violation check module to check the physical violation of the scene after fine alignment; the relationship between objects in the scene is divided into close proximity relationship and support relationship; for close proximity relationship, check whether the 3D bounding boxes of adjacent objects intersect or overlap, if there is intersection or overlap, there is a physical violation; at the same time, use heuristic methods to detect the support relationship between objects, the distance between the lower surface of the upper object and the upper surface of the lower object is less than a certain threshold, the contact surface area is greater than a certain threshold, and the angle of the contact surface is within a preset range. The heuristic conditions are set by experience; Step 7.3: If the alignment error exceeds the threshold or the aligned model has physical violations, the current model is discarded, the next model candidate is selected, and the alignment and checking process is repeated until all objects in the scene are physically compliant or the candidate set is exhausted.

Images