Point cloud data completion method and device based on CP decomposition and neural network

Abstract

The embodiment of the invention provides a point cloud data completion method and device based on CP decomposition and a neural network. The method is applied to image visual processing. The method comprises the steps of mapping points in to-be-complemented point cloud data into a voxel grid, and obtaining tensor data of the voxel grid; preprocessing the tensor data, inputting the preprocessed tensor data into a trained neural network model, so that the neural network model performs cp decomposition on the preprocessed tensor data, generates a factor matrix, performs cp inverse decomposition on missing tensor data according to the generated factor matrix, and outputs reconstructed tensor data; and mapping the reconstructed tensor data to a three-dimensional space to obtain complemented point cloud data. In this way, the data integrity and the completion precision are improved, computing resources are reduced, and the limitation of an existing point cloud completion method is effectively solved.

Description

Technical Field

[0001] The present disclosure relates to the field of image vision processing, and in particular to a point cloud data completion method and device based on CP decomposition and neural network. Background Art

[0002] As a discrete representation of spatial information, three-dimensional point cloud data is widely used in many fields such as autonomous driving, computer vision, medical imaging, digital manufacturing, and geographic information systems. In autonomous driving, point cloud data is used for environmental perception, obstacle detection, and high-precision map construction; in medical image analysis, three-dimensional point clouds can assist in organ modeling, lesion detection, and surgical planning; in manufacturing and cultural heritage protection, point cloud data is widely used for high-precision three-dimensional reconstruction and reverse engineering. However, since the collection of point cloud data usually relies on sensing devices such as lidar, structured light cameras, or stereo vision, these devices may be affected by factors such as occlusion, illumination changes, sensor noise, and perspective deviation in complex environments, resulting in missing, sparse, or incomplete local structures in the collected point cloud data. The lack of point cloud data not only affects the integrity of the target object, but also reduces the accuracy of subsequent three-dimensional reconstruction, target detection, semantic segmentation, and other tasks. For example, in autonomous driving scenarios, incomplete point cloud data may lead to errors in obstacle recognition, which in turn affects the safety decision of the vehicle; in medical imaging, missing point cloud information may affect the precise detection of lesion areas and reduce the accuracy of diagnosis. In this context, how to efficiently and accurately complete the missing point cloud data and restore its geometric structure and detail information has become a key issue in current research.

[0003] At present, point cloud completion methods mainly use technologies such as geometric modeling, deep learning, and multi-view fusion. Among them, geometric modeling methods usually rely on prior shapes and have limited adaptability to complex structures. Deep learning such as 3D convolutional networks and graph neural networks can learn completion patterns from data, but they have high computational complexity and are sensitive to data scale, making them difficult to apply to large-scale point cloud completion tasks. Multi-view fusion methods require additional registration steps, which increases computational overhead and may introduce error propagation. Although these methods have made some progress in solving the point cloud completion problem, they still have the following key defects: (1) High dependence on data quality. Most existing methods rely on image depth information, multi-view registration, or external shape prior information, which makes them have high requirements on the quality of input data. For example, image-based completion methods perform poorly on low-quality or noisy image data and cannot effectively process high-noise or low-resolution point cloud data; (2) Insufficient recovery of local details. When processing point cloud data, many methods restore point clouds through coarse-grained completion or multi-scale feature processing. However, these methods often ignore the local geometric details of the point cloud, resulting in a lack of precision in the completion results. (3) High computational complexity. Some deep learning-based methods, such as neural networks and ResUNet, have good performance in completion accuracy, but their computational overhead is large. Especially when processing large-scale data sets, they may face significant computational bottlenecks, which limits their application in real-time or large-scale point cloud completion tasks. (4) Difficulty in model training and optimization. Many existing methods rely on complex network architectures and multi-stage training. The training process is complex and prone to overfitting. Especially when data is sparse or missing, the model may not be able to converge stably, resulting in unsatisfactory completion results. Summary of the invention

[0004] The present invention provides a point cloud data completion method based on CP decomposition and neural network, an apparatus, a device and a storage medium.

[0005] According to a first aspect of the present disclosure, a point cloud data completion method based on CP decomposition and neural network is provided. The method comprises:

[0006] Map the points in the point cloud data to be completed into a voxel grid, and obtain the tensor data of the voxel grid;

[0007] Preprocessing the tensor data, inputting the preprocessed tensor data into a trained neural network model, so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix, and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs reconstructed tensor data;

[0008] Map the reconstructed tensor data to three-dimensional space to obtain the completed point cloud data.

[0009] In some implementations of the first aspect, mapping points in the point cloud data to be completed into a voxel grid includes:

[0010] According to the following mapping formula, the mapping position of the points in the point cloud data to be completed in the voxel grid is calculated:

[0011]

[0012] Among them, x i ,y i 、z i Respectively represent the position coordinates of the ith point in the point cloud data to be completed in the x-axis, y-axis, and z-axis directions, m, j, and k represent the position coordinates of the voxel corresponding to the ith point in the point cloud data to be completed in the voxel grid in the x-axis, y-axis, and z-axis directions, Δx, Δy, and Δz represent the sizes of the voxel corresponding to the ith point in the point cloud data to be completed in the voxel grid in the x-axis, y-axis, and z-axis directions, and [·] represents the rounding down operation.

[0013] In some implementations of the first aspect, cp decomposition is performed using the following formula:

[0014]

[0015] Among them, T represents the tensor data to be decomposed, r represents the rank of decomposition, and Q represents the maximum rank of decomposition. represents the Nth factor vector of rank 1, represents the outer product operation, A (N) represents the Nth factor matrix.

[0016] In some implementations of the first aspect, the neural network model is trained by the following method:

[0017] Map the points in the existing point cloud data to the voxel grid, obtain the corresponding tensor data, perform mask processing and preprocessing on the obtained tensor data, and use the preprocessed tensor data as training samples;

[0018] Input the training samples into the pre-trained neural network model to train the neural network model, so that the neural network model performs CP decomposition on the training samples, generates a factor matrix, performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs the reconstructed tensor data;

[0019] Calculating the difference between the output reconstructed tensor data and the tensor data in the training sample, adjusting the model parameters according to the difference, and re-inputting the adjusted model parameters into the neural network model;

[0020] Repeat model training, calculate difference values, adjust model parameters, and re-enter the model until the comprehensive loss function converges.

[0021] In some implementations of the first aspect, the neural network model includes an input layer, a hidden layer, and an output layer; wherein,

[0022] The input layer is a multi-channel input, each channel receives tensor data coordinate information on a coordinate dimension;

[0023] The hidden layer includes a structure composed of multiple neurons, each neuron performs a linear transformation on the input, and then performs a nonlinear transformation through an activation function. The data after the nonlinear transformation undergoes a linear transformation again to extract features at different levels from the input and understand the CP decomposition task;

[0024] The output layer outputs a factor matrix.

[0025] In some implementations of the first aspect, the comprehensive loss function includes a reconstruction loss function, a smoothing loss function and a regularization loss function; wherein;

[0026] The reconstruction loss function L1 is expressed as L1=||TT c |‖ F , T c represents the reconstructed tensor data, T represents the tensor data to be decomposed (i.e., the observed tensor data), ||·|| F represents the F-norm;

[0027] The smooth loss function L2 is expressed as L2=∑ u，v ||D(u)-D(v)||2, D(u) and D(v) represent the tensor of the u-th point and the v-th point in the reconstructed tensor data, respectively, and the difference between u and v is 1;

[0028] The regularized loss function L3 is expressed as A (n) represents the nth factor matrix;

[0029] The comprehensive loss function L is expressed as L=L1+λ1L2+λ2L3, where λ1 and λ2 represent weight hyperparameters corresponding to the smoothing loss function and the regularization loss function, respectively.

[0030] According to a second aspect of the present disclosure, a point cloud data completion device based on CP decomposition and neural network is provided. The device comprises:

[0031] A tensor acquisition module is used to map points in the point cloud data to be completed into a voxel grid and obtain tensor data of the voxel grid;

[0032] A tensor reconstruction module is used to preprocess the tensor data, input the preprocessed tensor data into a trained neural network model, so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix, and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs the reconstructed tensor data;

[0033] The point cloud completion module is used to map the reconstructed tensor data into three-dimensional space and obtain the completed point cloud data.

[0034] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method described above.

[0035] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described above.

[0036] In the present disclosure, a method and device for completing point cloud data based on CP decomposition and neural network are provided, the method comprising mapping points in the point cloud data to be completed to a voxel grid, obtaining tensor data of the voxel grid; preprocessing the tensor data, inputting the preprocessed tensor data into a trained neural network model, so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix, and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs reconstructed tensor data; maps the reconstructed tensor data to a three-dimensional space, and obtains the completed point cloud data. In this way, data integrity and completion accuracy are improved, computing resources are reduced, and the limitations of existing point cloud completion methods are effectively solved. For sparse point cloud and high-dimensional point cloud data, the point cloud data is efficiently completed while retaining the structural characteristics of the point cloud data, more local details can be restored in complex scenes, and missing point cloud data can be efficiently restored based on single-view point cloud data, thereby enhancing the robustness of completion, improving completion accuracy, and reducing computing resources.

[0037] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0039] Figure 1 A flowchart of a point cloud data completion method based on CP decomposition and neural network provided by an embodiment of the present disclosure is shown;

[0040] Figure 2 A network architecture diagram for CP decomposition of a neural network model provided by an embodiment of the present disclosure is shown;

[0041] Figure 3 A structural diagram of a point cloud data completion device based on CP decomposition and neural network provided by an embodiment of the present disclosure is shown;

[0042] Figure 4 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0044] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0045] In view of the problems in the background technology, the embodiment of the present disclosure provides a method and device for completing point cloud data based on CP decomposition and neural network, the method comprising mapping points in the point cloud data to be completed to a voxel grid, obtaining tensor data of the voxel grid; preprocessing the tensor data, inputting the preprocessed tensor data into a trained neural network model, so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix, and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs reconstructed tensor data; maps the reconstructed tensor data to a three-dimensional space, and obtains the completed point cloud data. In this way, the data integrity and completion accuracy are improved, the computing resources are reduced, and the limitations of the existing point cloud completion method are effectively solved. For sparse point cloud and high-dimensional point cloud data, the point cloud data is efficiently completed while retaining the structural characteristics of the point cloud data, more local details can be restored in complex scenes, and the missing point cloud data can be efficiently restored based on the single-view point cloud data, thereby enhancing the robustness of the completion, improving the completion accuracy and reducing the computing resources.

[0046] In the following, in conjunction with the accompanying drawings, the point cloud data completion method and device based on CP decomposition and neural network provided by the embodiment of the present disclosure are described in detail through specific embodiments.

[0047] Figure 1 A flowchart of a point cloud data completion method based on CP decomposition and neural network provided by an embodiment of the present disclosure is shown. The method 100 includes the following steps:

[0048] S110, mapping points in the point cloud data to be completed to a voxel grid, and obtaining tensor data of the voxel grid.

[0049] In some embodiments, each point in the point cloud data to be completed needs to be spatially discretized so that the points in the point cloud data to be completed can be mapped to a voxel grid.

[0050] In some embodiments, mapping points in the point cloud data to be completed into a voxel grid includes:

[0051] According to the following mapping formula, the mapping position of the points in the point cloud data to be completed in the voxel grid is calculated:

[0052]

[0053] Among them, x i ,y i 、z iRespectively represent the position coordinates of the i-th point in the point cloud data to be completed in the x-axis, y-axis, and z-axis directions, m, j, and k represent the position coordinates of the voxel corresponding to the i-th point in the point cloud data to be completed in the voxel grid in the x-axis, y-axis, and z-axis directions, Δx, Δy, and Δz represent the size of the voxel corresponding to the i-th point in the point cloud data to be completed in the voxel grid in the x-axis, y-axis, and z-axis directions, and [·] represents the rounding down operation;

[0054] Through the above mapping formula, all points in the point cloud data to be completed can be mapped to the corresponding positions of the voxel grid.

[0055] In some embodiments, the point cloud data to be completed can be converted into corresponding tensor data T, T∈R by mapping points in the point cloud data to be completed to a voxel grid and obtaining tensor data of the voxel grid. W×H×G , R is the real number field, W, H, and G are the resolutions of the voxel grid in the x-axis, y-axis, and z-axis directions, respectively. In the tensor data T, the voxel value is 1 or 0. A voxel value of 1 indicates that the voxel contains at least one point in the point cloud data, that is, the voxel is occupied by a point in the point cloud data.

[0056] In some embodiments, a dense voxel grid may be constructed by minimizing the following function:

[0057]

[0058] Where Ω represents the set of three-dimensional position coordinates corresponding to non-empty voxels in the voxel grid X, X mjk , They represent the voxel value corresponding to the voxel with coordinates (m, j, k) and the voxel reference value corresponding to the voxel with coordinates (m, j, k), λ is the regularization parameter, and ||X|| * is the tensor nuclear norm.

[0059] S120, preprocessing the tensor data, inputting the preprocessed tensor data into the trained neural network model, so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs the reconstructed tensor data.

[0060] In some embodiments, preprocessing includes standardizing and normalizing the tensor data corresponding to the point cloud data to be completed to ensure the stability and effectiveness of the data when it is input into the neural network model. Standardization and normalization can eliminate dimensional differences and avoid excessive influence of certain features.

[0061] In some embodiments, cp decomposition is performed by the following formula:

[0062]

[0063] Among them, T represents the tensor data to be decomposed, r represents the rank of decomposition, and Q represents the maximum rank of decomposition. represents the Nth factor vector of rank 1, represents the outer product operation, A (N) represents the Nth factor matrix.

[0064] In some embodiments, the three-dimensional tensor data to be decomposed is subjected to CP decomposition to obtain three factor matrices, A (1) , A (2) , A (3) , the dimensions of these three factor matrices are I1×Q, I2×Q, and I3×Q respectively.

[0065] In some embodiments, performing CP decomposition on the missing tensor data according to the generated factor matrix and outputting reconstructed tensor data includes:

[0066] According to the formula for CP decomposition, the missing tensor data is CP-decomposed by the generated factor matrix to obtain reconstructed tensor data.

[0067] In some embodiments, the neural network model is trained by:

[0068] Map the points in the existing point cloud data to the voxel grid, obtain the corresponding tensor data, perform mask processing and preprocessing on the obtained tensor data, and use the preprocessed tensor data as training samples;

[0069] Input the training samples into the pre-trained neural network model to train the neural network model, so that the neural network model performs CP decomposition on the training samples, generates a factor matrix, performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs the reconstructed tensor data;

[0070] Calculating the difference between the output reconstructed tensor data and the tensor data in the training sample, adjusting the model parameters according to the difference, and re-inputting the adjusted model parameters into the neural network model;

[0071] Repeat model training, calculate difference values, adjust model parameters, and re-enter the model until the comprehensive loss function converges.

[0072] In some embodiments, the acquired tensor data is masked and some values ​​of the tensor data are set to 0 to simulate missing conditions.

[0073] In some embodiments, the input layer, hidden layer and output layer constitute the network architecture of the neural network model for cp decomposition, such as Figure 2 As shown, the neural network model includes an input layer, a hidden layer and an output layer; wherein,

[0074] The input layer is a multi-channel input, each channel receives the coordinate information of the tensor data on a coordinate dimension; for example, the first input channel receives the coordinate information M of the tensor data on the x-axis. 1×E , the second input channel receives the coordinate information J of the tensor data on the y-axis 1×E , the third input channel receives the coordinate information k of the tensor data on the z-axis 1×E , E represents the number of tensor data;

[0075] The hidden layer includes a structure composed of multiple neurons, each neuron performs a linear transformation on the input through a linear function, and then performs a nonlinear transformation through an activation function (ReLU). The nonlinearly transformed data is then linearly transformed through a linear function to extract features at different levels from the input and understand the cp decomposition task;

[0076] The output layer outputs a factor matrix through multiple channels. For example, the output dimension of the first output channel is I 1×Q The factor matrix The output dimension of the second output channel is I 2×Q The factor matrix The output dimension of the third output channel is I 3×Q The factor matrix

[0077] In some embodiments, the design of the network architecture of the neural network model for cp decomposition can adapt to the characteristics of high-dimensional tensor data and extract deep features in the data representation;

[0078] For example, the input layer provides the neural network model with the position information of the tensor data in each dimension, so that the neural network model can learn the corresponding factor matrix based on each position information in the high-dimensional space;

[0079] The hidden layer introduces nonlinear processing by using an activation function, which enhances the model's fitting ability and enables it to handle complex patterns and high-dimensional relationships in tensor data. The activation function formula is ReLU(q)=max(0,q), where q represents the sum of the inputs of a neuron. After processing by the hidden layer, the model can effectively learn the relationship between the coordinate information received by the input layer and the column vector of the factor matrix, where the column vector of the factor matrix represents a low-rank representation of the tensor.

[0080] The output layer maps the high-dimensional features learned by the hidden layer to the column vectors of each factor matrix to achieve CP decomposition.

[0081] In some embodiments, when the neural network model performs cp decomposition for the first time, a random initialization method is used to initialize the factor matrix obtained by the decomposition.

[0082] In some embodiments, the Adam optimizer is used to execute a gradient descent strategy to optimize the comprehensive loss function and converge the comprehensive loss function. During the process of optimizing the comprehensive loss function, the model automatically adjusts the model parameters, the weights of the factor matrix, etc. through the back propagation algorithm until the optimal cp decomposition parameters are obtained.

[0083] In some embodiments, the comprehensive loss function includes a reconstruction loss function, a smoothing loss function and a regularization loss function; wherein;

[0084] The reconstruction loss function L1 is expressed as L1=||TT c || F , T c represents the reconstructed tensor data, T represents the tensor data to be decomposed (i.e., the observed tensor data), ||·|| F represents the F-norm;

[0085] The smooth loss function L2 is expressed as L2=∑ u，v ||D(u)-D(v)||2, D(u) and D(v) represent the tensor of the u-th point and the v-th point in the reconstructed tensor data, respectively, and the difference between u and v is 1 (u and v are adjacent points);

[0086] The regularized loss function L3 is expressed as A (n) represents the nth factor matrix;

[0087] The comprehensive loss function L is expressed as L=L1+λ1L2+λ2L3, where λ1 and λ2 represent weight hyperparameters corresponding to the smoothing loss function and the regularization loss function, respectively.

[0088] In some embodiments, a smooth loss function is used to ensure the continuity and smoothness of the reconstructed tensor data. In particular, for dense or irregular data, adding a smooth loss function can avoid generating jumpy point clouds or discrete structures, thereby making the completed point cloud data more continuous.

[0089] The regularization loss function is used to prevent the model from overfitting. By constraining the model to learn the CP decomposition process through the regularization loss function, it helps to control the size of the factor matrix, prevent the model from overfitting the training samples, and control the complexity of the model to ensure the generalization ability of the model.

[0090] S130, mapping the reconstructed tensor data to three-dimensional space to obtain completed point cloud data.

[0091] In some embodiments, the reconstructed tensor data includes all missing values ​​completed based on the low-rank structure. The reconstruction process can maintain the overall pattern and structure of the original tensor (tensor data corresponding to the point cloud data to be completed), ensure that the completed tensor data conforms to the distribution of the original tensor data, thereby effectively recovering the missing parts in the sparse tensor, mapping the reconstructed tensor data to three-dimensional space, and obtaining the completed point cloud data. The completed point cloud data contains the complete spatial information of the original point cloud data to be completed.

[0092] According to an embodiment of the present disclosure, a point cloud data completion method and device based on CP decomposition and neural network are provided, the method comprising mapping points in the point cloud data to be completed into a voxel grid, and obtaining tensor data of the voxel grid; preprocessing the tensor data, and inputting the preprocessed tensor data into a trained neural network model, so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs reconstructed tensor data; mapping the reconstructed tensor data into three-dimensional space to obtain the completed point cloud data. This method converts the point cloud completion problem into a tensor completion problem, uses CP decomposition to perform low-rank decomposition on the tensor, and represents the high-dimensional point cloud data in the form of the outer product of the factor matrix, thereby extracting the potential low-rank structural features of the high-dimensional point cloud data (i.e., the factor matrix), and then uses the neural network model to optimize the factor matrix to improve the stability of the tensor decomposition and enhance the adaptability to different data distributions. Based on the optimized factor matrix, the CP inverse decomposition method is used to reconstruct the tensor data to achieve high-precision completion of the missing point cloud, ensuring the consistency of the completion result in the global structure and the accuracy of the local geometric details. In this way, this method not only improves data integrity and completion accuracy, reduces computing resources, but also effectively solves the limitations of existing point cloud completion methods. For sparse point clouds and high-dimensional point cloud data, the point cloud data can be efficiently completed while retaining the structural characteristics of the point cloud data. More local details can be restored in complex scenes, and the missing point cloud data can be efficiently restored based on the single-view point cloud data, which enhances the robustness of the completion, improves the completion accuracy, and reduces computing resources.

[0093] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0094] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0095] Figure 3 The structure diagram of a point cloud data completion device based on CP decomposition and neural network provided by an embodiment of the present disclosure is shown. The device 300 includes:

[0096] The tensor acquisition module 310 is used to map points in the point cloud data to be completed into a voxel grid and acquire tensor data of the voxel grid.

[0097] In some embodiments, the tensor acquisition module 310 is specifically used to:

[0098] The mapping of points in the point cloud data to be completed into a voxel grid includes:

[0099] According to the following mapping formula, the mapping position of the points in the point cloud data to be completed in the voxel grid is calculated:

[0100]

[0101] Among them, x i ,y i 、z i Respectively represent the position coordinates of the ith point in the point cloud data to be completed in the x-axis, y-axis, and z-axis directions, m, j, and k represent the position coordinates of the voxel corresponding to the ith point in the point cloud data to be completed in the voxel grid in the x-axis, y-axis, and z-axis directions, Δx, Δy, and Δz represent the sizes of the voxel corresponding to the ith point in the point cloud data to be completed in the voxel grid in the x-axis, y-axis, and z-axis directions, and [·] represents the rounding down operation.

[0102] The tensor reconstruction module 320 is used to preprocess the tensor data and input the preprocessed tensor data into the trained neural network model so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs the reconstructed tensor data.

[0103] In some embodiments, the tensor reconstruction module 320 is specifically used to:

[0104] The CP decomposition is performed using the following formula:

[0105]

[0106] Among them, T represents the tensor data to be decomposed, r represents the rank of decomposition, and Q represents the maximum rank of decomposition. represents the Nth factor vector of rank 1, represents the outer product operation, A (N)represents the Nth factor matrix.

[0107] In some embodiments, the tensor reconstruction module 320 is further configured to:

[0108] The neural network model is trained by the following method:

[0109] Map the points in the existing point cloud data to the voxel grid, obtain the corresponding tensor data, perform mask processing and preprocessing on the obtained tensor data, and use the preprocessed tensor data as training samples;

[0110] Input the training samples into the pre-trained neural network model to train the neural network model, so that the neural network model performs CP decomposition on the training samples, generates a factor matrix, performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs the reconstructed tensor data;

[0111] Calculating the difference between the output reconstructed tensor data and the tensor data in the training sample, adjusting the model parameters according to the difference, and re-inputting the adjusted model parameters into the neural network model;

[0112] Repeat model training, calculate difference values, adjust model parameters, and re-enter the model until the comprehensive loss function converges.

[0113] In some embodiments, the tensor reconstruction module 320 is further configured to:

[0114] The neural network model includes an input layer, a hidden layer and an output layer; wherein,

[0115] The input layer is a multi-channel input, each channel receives tensor data coordinate information on a coordinate dimension;

[0116] The hidden layer includes a structure composed of multiple neurons, each neuron performs a linear transformation on the input, and then performs a nonlinear transformation through an activation function. The data after the nonlinear transformation undergoes a linear transformation again to extract features at different levels from the input and understand the CP decomposition task;

[0117] The output layer outputs a factor matrix.

[0118] In some embodiments, the tensor reconstruction module 320 is further configured to:

[0119] The comprehensive loss function includes a reconstruction loss function, a smoothing loss function and a regularization loss function; wherein;

[0120] The reconstruction loss function L1 is expressed as L1=||TT c || F , T crepresents the reconstructed tensor data, T represents the tensor data to be decomposed (i.e., the observed tensor data), ||·|| F represents the F-norm;

[0121] The smooth loss function L2 is expressed as L2=∑ u，v ||D(u)-D(v)||2, D(u) and D(v) represent the tensor of the u-th point and the v-th point in the reconstructed tensor data, respectively, and the difference between u and v is 1;

[0122] The regularized loss function L3 is expressed as A (n) represents the nth factor matrix;

[0123] The comprehensive loss function L is expressed as L=L1+λ1L2+λ2L3, where λ1 and λ2 represent weight hyperparameters corresponding to the smoothing loss function and the regularization loss function, respectively.

[0124] The point cloud completion module 330 is used to map the reconstructed tensor data into three-dimensional space to obtain completed point cloud data.

[0125] Understandably, Figure 3 Each module / unit in the illustrated device 300 has the function of implementing each step in the method 100 provided in the embodiment of the present disclosure and can achieve its corresponding technical effect. For the sake of brevity, they will not be described in detail here.

[0126] Figure 4 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. Electronic device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 400 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0127] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An I / O interface 405 is also connected to the bus 404.

[0128] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0132] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] It should be noted that the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in the embodiment of the present disclosure. For the sake of concise description, they are not repeated here.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A point cloud data completion method based on CP decomposition and neural network, characterized in that: include: Map the points in the point cloud data to be completed into a voxel grid, and obtain the tensor data of the voxel grid; Preprocessing the tensor data, inputting the preprocessed tensor data into a trained neural network model, so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix, and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs reconstructed tensor data; Map the reconstructed tensor data to three-dimensional space to obtain the completed point cloud data.

2. The method according to claim 1, characterized in that The mapping of points in the point cloud data to be completed into a voxel grid includes: According to the following mapping formula, the mapping position of the points in the point cloud data to be completed in the voxel grid is calculated: Among them, x i ,y i 、z i Respectively represent the position coordinates of the ith point in the point cloud data to be completed in the x-axis, y-axis, and z-axis directions, m, j, and k represent the position coordinates of the voxel corresponding to the ith point in the point cloud data to be completed in the voxel grid in the x-axis, y-axis, and z-axis directions, Δx, Δy, and Δz represent the sizes of the voxel corresponding to the ith point in the point cloud data to be completed in the voxel grid in the x-axis, y-axis, and z-axis directions, and [·] represents the rounding down operation.

3. The method according to claim 1, characterized in that The CP decomposition is performed using the following formula: Among them, T represents the tensor data to be decomposed, r represents the rank of decomposition, and Q represents the maximum rank of decomposition. represents the Nth rank 1 factor vector, ° represents the outer product operation, A (N) represents the Nth factor matrix.

4. The method according to claim 3, characterized in that The neural network model is trained by the following method: Map the points in the existing point cloud data to the voxel grid, obtain the corresponding tensor data, perform mask processing and preprocessing on the obtained tensor data, and use the preprocessed tensor data as training samples; Input the training samples into the pre-trained neural network model to train the neural network model, so that the neural network model performs CP decomposition on the training samples, generates a factor matrix, performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs the reconstructed tensor data; Calculating the difference between the output reconstructed tensor data and the tensor data in the training sample, adjusting the model parameters according to the difference, and re-inputting the adjusted model parameters into the neural network model; Repeat model training, calculate difference values, adjust model parameters, and re-enter the model until the comprehensive loss function converges.

5. The method according to claim 4, characterized in that The neural network model includes an input layer, a hidden layer and an output layer; wherein, The input layer is a multi-channel input, each channel receives tensor data coordinate information on a coordinate dimension; The hidden layer includes a structure composed of multiple neurons, each neuron performs a linear transformation on the input, and then performs a nonlinear transformation through an activation function. The data after the nonlinear transformation undergoes a linear transformation again to extract features at different levels from the input and understand the CP decomposition task; The output layer outputs a factor matrix.

6. The method according to claim 4, characterized in that The comprehensive loss function includes a reconstruction loss function, a smoothing loss function and a regularization loss function; wherein; The reconstruction loss function L1 is expressed as L1=||TT c || F , T c represents the reconstruction of tensor data, ||·|| F represents the F-norm; The smooth loss function L2 is expressed as L2=∑ u，v ||D(u)-D(v)||2, D(u) and D(v) represent the tensor of the u-th point and the v-th point in the reconstructed tensor data, respectively, and the difference between u and v is 1; The regularized loss function L3 is expressed as A (n) represents the nth factor matrix; The comprehensive loss function L is expressed as L=L1+λ1L2+λ2L3, where λ1 and λ2 represent weight hyperparameters corresponding to the smoothing loss function and the regularization loss function, respectively.

7. A point cloud data completion device based on CP decomposition and neural network, characterized in that: include: A tensor acquisition module is used to map points in the point cloud data to be completed into a voxel grid and obtain tensor data of the voxel grid; A tensor reconstruction module is used to preprocess the tensor data, input the preprocessed tensor data into a trained neural network model, so that the neural network model performs CP decomposition on the preprocessed tensor data, generates a factor matrix, and performs CP inverse decomposition on the missing tensor data according to the generated factor matrix, and outputs the reconstructed tensor data; The point cloud completion module is used to map the reconstructed tensor data into three-dimensional space and obtain the completed point cloud data.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

Images