A lightweight 3D point cloud completion method, apparatus, and processing equipment

Abstract

This application provides a lightweight 3D point cloud completion method, apparatus, and processing device. It designs a novel lightweight 3D point cloud completion network structure, which is based on the SVDFormer completion network structure and is fully simplified using various methods with extremely low complexity. It retains the global attention mechanism only in a very few core links, thereby achieving the goal of high completion accuracy and high stability with as few computational resources as possible, and meeting the requirements of high-quality 3D point cloud completion.

Description

Technical Field

[0001] This application relates to the field of 3D point cloud completion, specifically to a lightweight 3D point cloud completion method, apparatus, and processing equipment. Background Technology

[0002] Point clouds are typically acquired by sensors such as LiDAR and depth cameras. However, due to factors such as object occlusion and shooting distance, the actual acquired point clouds often have partial and detailed missing parts. In order not to affect the performance of subsequent point cloud application services such as digital twins, target recognition, detection and tracking, it is necessary to perform relevant point cloud completion processing to infer the shape and structure of the missing parts and supplement local detailed information, thereby providing a complete point cloud.

[0003] Currently, commonly used point cloud completion methods can be mainly divided into two categories. The first category is based on multilayer perceptrons, with Point Completion Network (PCN) being one of the representative methods. This method uses a shared multilayer perceptron to transform the 3D coordinates of each point in the incomplete point cloud of the target into the feature space, then obtains the target features through max pooling, and finally obtains a complete point cloud with coarse-to-fine granularity through layer-by-layer decoding by the multilayer perceptron. This method is computationally simple and has strong anti-interference ability, but the completion accuracy is very low. The second category is based on global attention mechanisms, with Self-View Augmentation and Dual-generator (SVDFormer) being one of the representative methods. This method introduces a large number of global attention mechanisms to calculate the features of points based on the first category of methods, achieving leading completion accuracy on multiple benchmark sets. However, the computational overhead of the global attention mechanism is very large, which greatly increases the training and inference time and the required GPU memory space of this method.

[0004] Therefore, how to minimize the consumption of computing resources by point cloud completion methods while maintaining high completion accuracy and stability is an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a lightweight 3D point cloud completion method, apparatus, and processing device. It designs a novel lightweight 3D point cloud completion network structure, which is based on the SVDFormer completion network structure and is fully simplified using various methods with extremely low complexity. It retains the global attention mechanism only in a very few core links, thereby achieving the goal of high completion accuracy and high stability with as few computational resources as possible, and meeting the requirements of high-quality 3D point cloud completion.

[0006] Firstly, this application provides a lightweight 3D point cloud completion method, the method comprising:

[0007] Obtain the defect cloud to be completed, which is specifically a 3D point cloud;

[0008] The incomplete point cloud to be completed is input into a pre-configured lightweight point cloud completion network. The lightweight point cloud completion network includes an incomplete point cloud encoder based on a fine-tuning point cloud completion network, a lightweight coarse-grained point cloud decoder, a medium-grained point cloud decoder based on incomplete point cloud feature reuse, and a fine-grained point cloud decoder based on incomplete point cloud feature reuse. The incomplete point cloud encoder based on the fine-tuning point cloud completion network is used to extract the incomplete point cloud features of the incomplete point cloud input to the network. The lightweight coarse-grained point cloud decoder is used to generate a coarse-grained complete point cloud. The medium-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a medium-grained complete point cloud. The fine-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a fine-grained complete point cloud.

[0009] Extract the fine-grained complete point cloud from the output of the lightweight point cloud completion network as the point cloud completion result.

[0010] Secondly, this application provides a lightweight 3D point cloud completion device, the device comprising:

[0011] The acquisition unit is used to acquire the defect cloud to be completed, which is specifically a three-dimensional point cloud;

[0012] The completion unit is used to input the incomplete point cloud to be completed into a pre-configured lightweight point cloud completion network. The lightweight point cloud completion network includes an incomplete point cloud encoder based on a fine-tuning point completion network, a lightweight coarse-grained point cloud decoder, a medium-grained point cloud decoder based on incomplete point cloud feature reuse, and a fine-grained point cloud decoder based on incomplete point cloud feature reuse. The incomplete point cloud encoder based on the fine-tuning point completion network is used to extract the incomplete point cloud features of the incomplete point cloud input to the network. The lightweight coarse-grained point cloud decoder is used to generate a coarse-grained complete point cloud. The medium-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a medium-grained complete point cloud. The fine-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a fine-grained complete point cloud.

[0013] The extraction unit is used to extract the fine-grained complete point cloud output by the lightweight point cloud completion network as the point cloud completion result.

[0014] Thirdly, this application provides a processing device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the method provided in the first aspect of this application when it invokes the computer program in the memory.

[0015] Fourthly, this application provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the method provided in the first aspect of this application.

[0016] From the above, it can be concluded that this application has the following beneficial effects:

[0017] For the goal of 3D point cloud completion, this application designs a novel lightweight 3D point cloud completion network structure. Based on the SVDFormer completion network structure, this structure is greatly simplified using various methods with extremely low complexity. The global attention mechanism is retained only in a very few core components, so that the goal of high completion accuracy and high stability can be achieved with as few computational resources as possible, thus meeting the requirements of high-quality 3D point cloud completion. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a lightweight 3D point cloud completion method according to this application.

[0020] Figure 2 This is a logical diagram illustrating the 3D point cloud completion processing in this application.

[0021] Figure 3 This is another logical diagram illustrating the 3D point cloud completion processing in this application.

[0022] Figure 4 This is a schematic diagram of a defective cloud encoder based on a fine-tuning point completion network, as described in this application.

[0023] Figure 5 This is a schematic diagram of a lightweight coarse-grained point cloud decoder according to this application.

[0024] Figure 6 This is a schematic diagram of a medium / fine-grained point cloud decoder based on residual point cloud feature reuse in this application.

[0025] Figure 7 This is a schematic diagram of one structure of the lightweight residual module of this application;

[0026] Figure 8 This is a schematic diagram of a lightweight 3D point cloud completion device according to this application.

[0027] Figure 9This is a schematic diagram of one type of processing equipment used in this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such process, method, product, or apparatus. The naming or numbering of steps in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved.

[0030] The module division described in this application is a logical division. In practical applications, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between modules shown or discussed may be through some interfaces, and the indirect coupling or communication connection between modules may be electrical or other similar forms, none of which are limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules may be selected to achieve the purpose of the solution in this application according to actual needs.

[0031] Before introducing the lightweight 3D point cloud completion method provided in this application, we will first introduce the background content involved in this application.

[0032] The lightweight 3D point cloud completion method, apparatus, and computer-readable storage medium provided in this application can be applied to processing devices. A novel lightweight 3D point cloud completion network structure is designed. Based on the SVDFormer completion network structure, this structure is fully simplified using various methods with extremely low complexity. The global attention mechanism is retained only in a very few core links, so that the goal of high completion accuracy and high stability can be achieved with as few computational resources as possible, thus meeting the requirements of high-quality 3D point cloud completion.

[0033] The lightweight 3D point cloud completion method mentioned in this application can be implemented by a lightweight 3D point cloud completion device, or by different types of processing devices such as servers, physical hosts, or user equipment (UE) that integrate the lightweight 3D point cloud completion device. The lightweight 3D point cloud completion device can be implemented in hardware or software. The UE can be a terminal device such as a smartphone, tablet, laptop, desktop computer, or personal digital assistant (PDA). The processing devices can be configured in a device cluster.

[0034] It is understandable that, considering that the solution in this application is mainly based on a pre-configured lightweight point cloud completion network, the processing device that executes the lightweight 3D point cloud completion method of this application or is equipped with the application service corresponding to the lightweight 3D point cloud completion method of this application generally only needs to meet the required data processing capabilities. The specific device type and device deployment form are quite flexible and can be flexibly configured according to the actual situation.

[0035] If it involves the acquisition of 3D point clouds or the display of 3D point cloud completion results, and other specific data applications that are easy to understand, then further configuration of the processing equipment in terms of both hardware and software is required.

[0036] As an example, a processing device can display the processing progress / results of 3D point clouds through its own configured display screen (including touch screen), an external display device, or other devices with a display screen.

[0037] As another example, the processing device may include a first device part that performs training of a lightweight point cloud completion network, a second device part that performs application of the lightweight point cloud completion network, and a third device part that performs 3D point cloud acquisition.

[0038] The lightweight 3D point cloud completion method provided in this application will now be introduced.

[0039] First, refer to Figure 1 , Figure 1 This paper illustrates a flowchart of the lightweight 3D point cloud completion method of this application. The lightweight 3D point cloud completion method provided by this application may specifically include the following steps S101 to S103:

[0040] Step S101: Obtain the defect cloud to be filled, wherein the defect cloud to be filled is specifically a three-dimensional point cloud;

[0041] Understandably, this application will begin by explaining a pre-configured lightweight point cloud completion network.

[0042] In the specific application of lightweight point cloud completion networks, the corresponding 3D point cloud completion processing is usually initiated in the form of work tasks. Correspondingly, it may also involve the processing of acquiring 3D point cloud completion tasks. These tasks can be initiated manually, received from other devices, or initiated autonomously according to a corresponding autonomous initiation strategy. All of these are possible.

[0043] As for the 3D point cloud that needs to be completed, which is the incomplete point cloud, when a 3D point cloud completion task is involved, it can be carried in the task information, or it can be extracted from the corresponding storage location according to the instructions of the task information, or it can be collected and processed in real time according to the instructions of the task information, or it can be entered manually.

[0044] Furthermore, the acquired 3D point cloud is usually assumed to be incomplete and needs to be completed. Of course, in some cases, it may be determined after detection that it is incomplete and needs to be completed, and then the specific 3D point cloud completion process can be carried out through the solution of this application.

[0045] Three-dimensional point clouds are typically acquired using methods such as lidar and depth cameras.

[0046] Step S102: Input the incomplete point cloud to be completed into a pre-configured lightweight point cloud completion network. The lightweight point cloud completion network includes an incomplete point cloud encoder based on a fine-tuning point completion network, a lightweight coarse-grained point cloud decoder, a medium-grained point cloud decoder based on incomplete point cloud feature reuse, and a fine-grained point cloud decoder based on incomplete point cloud feature reuse. The incomplete point cloud encoder based on the fine-tuning point completion network is used to extract the incomplete point cloud features of the incomplete point cloud input to the network. The lightweight coarse-grained point cloud decoder is used to generate a coarse-grained complete point cloud. The medium-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a medium-grained complete point cloud. The fine-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a fine-grained complete point cloud.

[0047] As can be seen, this application designs a novel lightweight 3D point cloud completion network structure, which consists of four main parts: a residual point cloud encoder based on a fine-tuned point cloud completion network, a lightweight coarse-grained point cloud decoder, a medium-grained point cloud decoder based on residual point cloud feature reuse, and a fine-grained point cloud decoder based on residual point cloud feature reuse. These parts are connected sequentially, with the residual point cloud encoder corresponding to the network input side and the fine-grained point cloud decoder based on residual point cloud feature reuse corresponding to the network output side. Following a processing approach from residual point cloud to coarse-grained point cloud to medium-grained point cloud to fine-grained point cloud, the network gradually completes the completion / reconstruction of the residual point cloud to be completed in the input network. (See reference [link to reference here]). Figure 2 The diagram shown illustrates a logic diagram of the three-dimensional point cloud completion process of this application.

[0048] It is worth noting that the lightweight point cloud completion network of this application is obtained by further improving and matching other structures on the basis of the SVDFormer completion network. Specifically, the SVDFormer completion network structure has been fully simplified and optimized, thus forming the aforementioned lightweight point cloud completion network structure, thereby achieving a 3D point cloud completion effect that balances low resource consumption, high completion accuracy, and high stability.

[0049] In the application of 3D point cloud networks, after configuring the network structure in the early stage, this application can also involve related network training. This corresponds to the situation that the network training effect with low resource consumption, high completion accuracy and high stability can be promoted under the network structure design of this application.

[0050] In this regard, the method of this application may also include the following in its specific operation:

[0051] Obtain incomplete point cloud samples and their corresponding fine-grained complete point cloud ground truth values;

[0052] A lightweight point cloud completion network is trained based on incomplete point cloud samples and their corresponding fine-grained complete point cloud ground truth values.

[0053] In addition to referring to the above content on obtaining the incomplete ...

[0054] The fine-grained complete point cloud ground truth corresponding to the incomplete point cloud sample is usually completed manually by cropping and splicing point cloud images from multiple perspectives at close range. Alternatively, it can be a fine-grained complete point cloud generated during the construction of a virtual incomplete point cloud, or it can be obtained by retrieving the incomplete point cloud sample from a large-scale fine-grained complete dataset.

[0055] In the specific network training process, it is understandable that the specific network training scheme that can be adopted (e.g., 80% training set and 20% validation set), as well as the loss function used in the training process (e.g., chamfer distance loss function and earth movement distance loss function), are not the focus of this application, and relevant existing schemes can be adopted. Therefore, they are not elaborated on here.

[0056] Of course, it is understandable that, in practice, both can make further optimizations and improvements based on the existing solutions, or even adopt entirely new self-developed solutions, which is possible in some cases.

[0057] Step S103: Extract the fine-grained complete point cloud output by the lightweight point cloud completion network as the point cloud completion result.

[0058] It is easy to understand that after performing the corresponding 3D point cloud completion processing on the current incomplete point cloud to be completed through the lightweight point cloud completion network, the point cloud completion result output by the lightweight point cloud completion network can be extracted. That is, the fine-grained complete point cloud generated by the fine-grained point cloud decoder based on the feature reuse of incomplete point cloud in the last layer of the lightweight point cloud completion network, thus completing a 3D point cloud completion task.

[0059] Furthermore, it is understandable that 3D point clouds can be collected and configured for related point cloud application services. In this regard, point cloud completion can also be used as a data preprocessing method to improve the quality of specific point cloud application services such as digital twins, target recognition, detection and tracking.

[0060] Correspondingly, after extracting the fine-grained complete point cloud output from the lightweight point cloud completion network, it can be stored locally, stored off-site, displayed, pushed to the system, prompted when the output completion is complete, or used for further point cloud application services.

[0061] from Figure 1 As can be seen from the embodiments shown, this application designs a novel lightweight 3D point cloud completion network structure for the 3D point cloud completion target. Based on the SVDFormer completion network structure, this structure is fully simplified using various methods with extremely low complexity. It retains the global attention mechanism only in a very few core links, so as to achieve the goal of high completion accuracy and high stability with as few computational resources as possible, thus meeting the requirements of high-quality 3D point cloud completion.

[0062] Next, we will further elaborate on the lightweight point cloud completion network specifically designed in this application.

[0063] At the overall level, refer to Figure 3 The diagram shown represents another logical representation of the 3D point cloud completion processing in this application. The input of the lightweight point cloud completion network is specifically the 3D coordinate matrix of the incomplete point cloud, and the output is the 3D coordinate matrix of the fine-grained complete point cloud (the 3D coordinate matrix is ​​the specific representation of the 3D point cloud). The goal of the lightweight point cloud completion network is to establish a mapping relationship between the input and output through the design and training of the lightweight completion network, so as to achieve accurate, robust and efficient completion of the incomplete point cloud in the test dataset.

[0064] like Figure 3 As shown, in the point cloud completion process, the first stage will... The three-dimensional coordinate matrix of the defect cloud (i.e.) Inputting the 3D coordinate matrix of the defect cloud of each point into the defect cloud encoder based on the fine-tuning point completion network, we obtain... Feature vector of the defective cloud;

[0065] The second phase will The feature vector of the dimensional defect cloud is input into the lightweight coarse-grained point cloud decoder to generate... 3D coordinate matrix of coarse-grained point cloud;

[0066] The third phase will 3D coordinate matrix of the defect cloud Dimensional residual cloud feature vector and The three-dimensional coordinate matrix of a coarse-grained point cloud is input into a medium-grained point cloud decoder based on residual point cloud feature reuse, resulting in... 3D coordinate matrix of point cloud with medium granularity;

[0067] The fourth phase will 3D coordinate matrix of the defect cloud Dimensional residual cloud feature vector and The 3D coordinate matrix of the mid-grained point cloud is input into a fine-grained point cloud decoder based on residual point cloud feature reuse to obtain... The output is a fine-grained point cloud 3D coordinate matrix.

[0068] from Figure 3 As can be seen from this, compared to SVDFormer, the lightweight point cloud completion network designed in this application has the following three differences:

[0069] (1) The encoder removed the time-consuming multi-view feature extraction branch and used fine-tuned PCN instead of PointNet++ to directly extract residual cloud features;

[0070] (2) The coarse-grained decoder eliminates the time-consuming self-attention module;

[0071] (3) The medium and fine-grained point cloud decoders directly reuse the residual point cloud features output by the encoder instead of consuming a lot of computing resources to recalculate the residual point cloud features.

[0072] The above measures have greatly improved the computational efficiency of the network. Experiments on multiple benchmark sets have confirmed that the network maintains the accuracy of SVDFormer while reducing training and inference time by half and memory usage by half, thereby significantly reducing the training and inference costs for users.

[0073] Next, refer to Figure 4 The diagram shown is a logical representation of a defect cloud encoder based on a fine-tuning point completion network, as described in this application. The first layer of the lightweight point cloud completion network, which is the defect cloud encoder based on the fine-tuning point completion network, can specifically include a one-dimensional convolutional module. Maximum pooling module Copy module splicing module One-dimensional convolution module and max pooling module Let the dimension of the three-dimensional coordinate matrix of the defective cloud input to the network be... , It is the number of points in the defective cloud;

[0074] In this case, the residual cloud encoder based on the fine-tuning point completion network can specifically include the following processing:

[0075] 1.1) will The 3D coordinate matrix of the defect cloud is input into the 1D convolution module. ,get 3D initial feature matrix (i.e., residual feature cloud) (256-dimensional initial features of each point);

[0076] 1.2) will The initial feature matrix is ​​input to the max pooling module. ,get Initial eigenvectors;

[0077] 1.3) By the copy module Will The initial feature vector is copied in the row direction. Next, the splicing module Copy module The replication results, and The initial feature matrix (obtained here from 1.1) is concatenated along the column direction to obtain... 3D concatenation of feature matrices;

[0078] 1.4) 3D concatenation feature matrix input to 1D convolution module ,get 3D fusion feature matrix;

[0079] 1.5) 3D fusion feature matrix input max pooling module ,get The feature vector of the defect cloud is used as the feature of the defect cloud.

[0080] Given the specific network structure provided above, it is understandable that, compared to SVDFormer's defect cloud encoder, the defect cloud encoder based on a fine-tuning point completion network designed in this application differs in the following ways:

[0081] (1) The time-consuming multi-view feature extraction branch in the encoder of SVDFormer, as well as the subsequent time-consuming multi-view feature and residual cloud feature fusion network based on self-attention mechanism, were removed, and only the residual cloud feature extraction branch was retained. Experiments on multiple standard test sets have shown that this only leads to a slight decrease in overall completion accuracy (which is negligible in practice), but the overall computational efficiency is greatly improved;

[0082] (2) Replace the PointNet++-based defect cloud feature extraction branch in the encoder of SVDFormer with the encoder of PCN to directly extract the defect cloud features. This avoids the time-consuming neighborhood calculation of points, which not only improves the computational efficiency, but also has strong robustness to various changes in the input defect cloud points, such as density distribution, noise, outliers and other factors that cause neighborhood changes.

[0083] (3) Experiments on multiple standard test sets have shown that only simple adjustments to the encoder structure of PCN are needed, such as adding a one-dimensional convolution module. The number of convolutional layers can compensate for the loss of completion accuracy caused by removing the time-consuming multi-view feature extraction branch and feature fusion network in the SVDFormer encoder with minimal computational cost.

[0084] Next, refer to Figure 5 The diagram shown is a logical schematic of a lightweight coarse-grained point cloud decoder according to this application. The second layer network structure in the lightweight point cloud completion network, namely the lightweight coarse-grained point cloud decoder, may specifically include a transposed convolution module and a copying module. splicing module One-dimensional convolution module Shape reshaping module and one-dimensional convolutional modules ;

[0085] In this case, the lightweight coarse-grained point cloud decoder includes the following processing:

[0086] 2.1) will The feature vector of the defective cloud is input into the transposed convolution module (to perform transposed convolution), and the result is... 3D transposed convolution feature matrix;

[0087] 2.2) By the copy module Will The feature vector of the defective cloud is copied 128 times in the row direction, and then the concatenation module performs the stitching. Copy module The replication results, and The transposed convolutional feature matrix (obtained from 2.1) is concatenated along the column direction to obtain... 3D concatenation of feature matrices;

[0088] 2.3) will 3D concatenation feature matrix input to 1D convolution module (Perform feature fusion) to obtain 3D fusion feature matrix;

[0089] 2.4) Shape reshaping module Shape reshaping (or feature reshaping) is performed on the dimensional fusion feature matrix to obtain Reshape the feature matrix in 3D;

[0090] 2.5) 3D reshaping feature matrix input one-dimensional convolution module ,get The three-dimensional coordinate matrix of the coarse-grained point cloud is used as the coarse-grained complete point cloud.

[0091] Given the specific network architecture described above, it's understandable that the lightweight coarse-grained point cloud decoder designed differs from SVDFormer's coarse-grained point cloud decoder in the following ways:

[0092] (1) The self-attention module of the SVDFormer coarse-grained point cloud decoder was removed. Experiments on multiple standard test sets have shown that this only leads to a slight decrease in the overall completion accuracy (which can be ignored in actual cases), but the overall computational efficiency is greatly improved.

[0093] (2) The splicing and fusion operations of the coarse-grained point cloud decoder of SVDFormer to reshape the feature matrix and the copied residual point cloud feature matrix have been removed, which further reduces the consumption of computing resources without affecting the completion performance.

[0094] (3) Experiments on multiple standard test sets have shown that only simple fine-tuning of the coarse-grained point cloud encoder structure is needed, such as adding a one-dimensional convolution module. The number of convolutional layers can compensate for the loss of completion accuracy caused by replacing the time-consuming global attention module in the SVDFormer coarse-grained point cloud decoder with minimal computational cost.

[0095] Next, refer to Figure 6 The diagram shown is a logical schematic of a medium / fine-grained point cloud decoder based on residual point cloud feature reuse in this application. The subscripted letters after the forward slash and the number of dimensions in parentheses correspond to the fine-grained point cloud decoder based on residual point cloud feature reuse. When focusing on the medium-grained point cloud decoder based on residual point cloud feature reuse, the content after the forward slash and in parentheses is ignored.

[0096] As shown in the figure, the third layer of the lightweight point cloud completion network, which is a medium-granularity point cloud decoder based on the reuse of residual point cloud features, can specifically include a merging module. Farthest point sampling module One-dimensional convolution module Lightweight residual module One-dimensional convolution module Copy module splicing module One-dimensional convolution module Self-attention module One-dimensional convolution module Lightweight residual module Copy module splicing module One-dimensional convolution module Shape reshaping module One-dimensional convolution module ;

[0097] In this case, a medium-granularity point cloud decoder based on residual point cloud feature reuse can specifically include the following processing:

[0098] 3.1) By the merging module Will The three-dimensional coordinate matrix of the defect cloud (the initial network input) and The three-dimensional coordinate matrices of coarse-grained point clouds are merged to obtain... 3D coordinate matrix of merged point cloud;

[0099] 3.2) will 3D point cloud 3D coordinate matrix input farthest point sampling module ,get The 3D coordinate matrix of the sampled point cloud, where, Choose 512 or 1024;

[0100] 3.3) will 3D sampling point cloud 3D coordinate matrix input to 1D convolution module ,get Low-level feature matrix;

[0101] 3.4) will Low-dimensional feature matrix input lightweight residual module ,get Low-dimensional residual feature matrix;

[0102] 3.5) will The feature vector of the residual cloud (obtained from 1.5, corresponding to the concept of reuse) is input into the one-dimensional convolutional module. ,get Compressed feature vectors of dimensional residual cloud;

[0103] 3.6) By the copy module Will The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module Copy module The replication results and The low-level residual feature matrix (obtained from 3.4) is concatenated along the column direction to obtain... Mid-level concatenation of feature matrices;

[0104] 3.7) will The feature matrix is ​​concatenated in the middle layer and input into the one-dimensional convolution module. ,get Mid-level fusion feature matrix;

[0105] 3.8) will The mid-level fusion feature matrix is ​​input into the attention module. ,get Mid-level self-attention feature matrix;

[0106] 3.9) will The self-attention feature matrix in the middle layer is input into the one-dimensional convolutional module. ,get Mid-level convolutional feature matrix;

[0107] 3.10) will The mid-level convolutional feature matrix is ​​input into a lightweight residual module. ,get Mid-level residual feature matrix;

[0108] 3.11) By the copy module Will The feature vector of the residual cloud (obtained here from 1.5) is copied in the row direction. Next, the splicing module Copy module The replication results and The residual feature matrix of the middle layer (obtained from 3.10) is concatenated in the column direction to obtain... High-level splicing feature matrix;

[0109] 3.12) will High-level concatenated feature matrix input to one-dimensional convolution module ,get High-dimensional compressed feature matrix;

[0110] 3.13) will High-dimensional compressed feature matrix input shape reshaping module ,get Reshape the feature matrix in 3D;

[0111] 3.14) will 3D reshaping feature matrix input one-dimensional convolution module This yields the three-dimensional incremental coordinates of 2048 points;

[0112] 3.15) will The 3D coordinate matrix of the sampled point cloud (obtained from 3.2) is copied in the row direction. This is then added to the 3D incremental coordinates of 2048 points (obtained here from 3.14) to obtain... The three-dimensional coordinate matrix of the medium-grained point cloud is used as the medium-grained complete point cloud.

[0113] Under the very specific network structure settings provided above, it can be understood that, compared to the medium / fine-grained point cloud decoder of SVDFormer, the designed medium / fine-grained point cloud decoder based on residual point cloud feature reuse has the following differences:

[0114] (1) The residual cloud feature vector obtained by the residual cloud encoder based on fine-tuned PCN is directly reused. This avoids the SVDFormer from recalculating the residual cloud feature matrix by using time-consuming edge convolution and downsampling. This not only greatly improves the overall computational efficiency, but also avoids the neighborhood calculation of the midpoint of the edge convolution, thereby enhancing the robustness to various changes of the midpoint of the input residual cloud.

[0115] (2) In the process of calculating high-level splicing features from mid-level fusion features, only one four-layer one-dimensional convolutional module is used. and a lightweight residual module SVDFormer uses a cross-attention module and two self-attention modules. Experiments on multiple standard test sets have shown that this only leads to a slight decrease in overall completion accuracy (which is negligible in practice), but a significant improvement in overall computational efficiency.

[0116] (3) Experiments on multiple standard test sets have confirmed that adding a lightweight residual module can effectively address the issue. Enhance low-level features and fine-tune the structure of the medium / fine-grained point cloud decoder, such as by adding a one-dimensional convolutional module. and The number of convolutional layers can compensate for the loss of completion accuracy caused by the replacement of the SVDFormer attention module with very little computational cost;

[0117] (4) The parameters of the self-attention module used for mid-level fusion feature enhancement are restricted. That is, a four-head self-attention mechanism is used instead of the eight-head attention mechanism in SVDFormer. Experiments on multiple standard test sets have shown that although the self-attention module is irreplaceable, otherwise the completion accuracy will be greatly reduced, the computational load can be further reduced while maintaining the completion accuracy by restricting its parameter size.

[0118] Next, we focus on fine-grained point cloud decoders based on the reuse of incomplete point cloud features, and continue reading... Figure 6 In terms of details, the number of dimensions inside the parentheses should be used as the standard; that is, the number of dimensions inside the parentheses should replace the number of dimensions outside the parentheses. For example, initially it should be... The 3D coordinate matrix of a point cloud with medium granularity, rather than Three-dimensional coordinate matrix of coarse-grained point cloud.

[0119] As shown in the figure, the fourth layer of the lightweight point cloud completion network, which is a fine-grained point cloud decoder based on the reuse of residual point cloud features, can specifically include a merging module. Farthest point sampling module One-dimensional convolution module Lightweight residual module One-dimensional convolution module Copy module splicing module One-dimensional convolution module Self-attention module One-dimensional convolution module Lightweight residual module Copy module splicing module One-dimensional convolution module Shape reshaping module One-dimensional convolution module ;

[0120] In this case, a fine-grained point cloud decoder based on residual point cloud feature reuse can specifically include the following processing:

[0121] 4.1) By the merging module Will The three-dimensional coordinate matrix of the defect cloud (the initial network input) and The 3D coordinate matrices of the mid-grained point clouds are merged to obtain... 3D coordinate matrix of merged point cloud;

[0122] 4.2) 3D point cloud 3D coordinate matrix input farthest point sampling module ,get 3D coordinate matrix of sampled point cloud;

[0123] 4.3) will 3D sampling point cloud 3D coordinate matrix input to 1D convolution module ,get Low-level feature matrix;

[0124] 4.4) will Low-dimensional feature matrix input lightweight residual module ,get Low-dimensional residual feature matrix;

[0125] 4.5) The feature vector of the residual cloud (obtained from 1.5, corresponding to the concept of reuse) is input into the one-dimensional convolutional module. ,get Compressed feature vectors of dimensional residual cloud;

[0126] 4.6) By the copy module Will The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module Copy module The replication results and The low-level residual feature matrix (obtained from 4.4) is concatenated along the column direction to obtain... Mid-level concatenation of feature matrices;

[0127] 4.7) will The feature matrix is ​​concatenated in the middle layer and input into the one-dimensional convolution module. ,get Mid-level fusion feature matrix;

[0128] 4.8) will The mid-level fusion feature matrix is ​​input into the attention module. ,get Mid-level self-attention feature matrix;

[0129] 4.9) will The self-attention feature matrix in the middle layer is input into the one-dimensional convolutional module. ,get Mid-level convolutional feature matrix;

[0130] 4.10) will The mid-level convolutional feature matrix is ​​input into a lightweight residual module. ,get Mid-level residual feature matrix;

[0131] 4.11) By the copy module Will The feature vector of the residual cloud (obtained here from 1.5) is copied in the row direction. Next, the splicing module Copy module The replication results and The residual feature matrices of the middle layer are concatenated in the column direction to obtain... High-level splicing feature matrix;

[0132] 4.12) will High-level concatenated feature matrix input to one-dimensional convolution module ,get High-dimensional compressed feature matrix;

[0133] 4.13) will High-dimensional compressed feature matrix input shape reshaping module ,get Reshape the feature matrix in 3D;

[0134] 4.14) will 3D reshaping feature matrix input one-dimensional convolution module This yields 16,384 points in three-dimensional incremental coordinates.

[0135] 4.15) will The 3D coordinate matrix of the sampled point cloud (obtained from 4.2) is copied in the row direction. This is then added to the 3D incremental coordinates of 16384 points (obtained from 4.14) to obtain... The three-dimensional coordinate matrix of the fine-grained point cloud is used as the fine-grained complete point cloud.

[0136] Under the specific network structure described above, it can be understood that the fine-grained point cloud decoder based on residual point cloud feature reuse and the medium-grained point cloud decoder based on residual point cloud feature reuse have the same decoder structure. The difference lies in the input and the parameters with different details, thus generating point clouds with richer details. The advantages of the 3D coordinate matrix of fine-grained point cloud (fine-grained complete point cloud) can be found in the previous description of the structure of the medium-grained point cloud decoder based on the reuse of residual point cloud features, and will not be repeated here.

[0137] Furthermore, both of the decoders mentioned above involve the application of residual modules for efficient feature matrix enhancement. For more information, please refer to... Figure 7 The diagram shown illustrates a structural schematic of the lightweight residual module of this application. It represents the module structure of the lightweight residual module between a medium-granularity point cloud decoder based on residual point cloud feature reuse and a fine-granularity point cloud decoder based on residual point cloud feature reuse. Specifically, it may include a linear layer. Linear layer Layer normalization layer and activation layer ;

[0138] In this case, the corresponding residual feature matrix extraction process may specifically include the following processing steps:

[0139] The original feature matrix of dimension 1 passes through a linear layer in sequence. and linear layer The number of feature points is halved. Then revert to the original points. Then, it goes through the normalization layers in sequence. and activation layer Finally, the output will be Dimensional results and Adding the original feature matrices in dimension 1 gives the corresponding The residual feature matrix is ​​used as the output.

[0140] Under the very specific network architecture configuration provided above, it can be understood that:

[0141] (1) Compared with the attention module, the residual module can also utilize the correlation between different feature points through the linear layer to achieve a similar feature enhancement effect. At the same time, the residual module uses the layer normalization function instead of the normalization exponential function, which greatly reduces the amount of computation. This is the reason for using the residual module to replace the attention module.

[0142] (2) Compared with the classic residual module, the lightweight residual module removes an activation layer between two linear layers and moves the second activation layer before the residual connection. Experiments on multiple standard test sets have confirmed that this can improve performance completion accuracy while reducing computational cost, which is the reason for using the lightweight residual module instead of the classic residual module.

[0143] Finally, regarding the specific configuration of each layer of the lightweight point cloud completion network, it can be summarized that this application systematically adopts various methods with extremely low complexity to fully simplify the SVDFormer completion network structure. These methods mainly include removing the time-consuming multi-view feature extraction branch in the SVDFormer encoder, reusing the residual point cloud features extracted by the encoder based on fine-tuned PCN, and extensively using lightweight residual modules to replace the time-consuming attention modules in the SVDFormer method. Experiments on multiple benchmark sets have confirmed that, while maintaining the accuracy of the SVDFormer method, the training time and inference time are reduced by half, and the memory usage is reduced by half, thereby significantly reducing the user's training and inference costs.

[0144] Meanwhile, this application fully adopts the feature extraction method of PCN encoder to extract residual point cloud features. Since the feature extraction method of PCN encoder does not involve the division of local point cloud and the calculation of the neighborhood of points in each stage of feature extraction, compared with the SVDFormer method, the network of this application has stronger robustness to various changes in the points in the input residual point cloud, such as neighborhood changes caused by factors such as density distribution, noise, and outliers.

[0145] The above is an introduction to the lightweight 3D point cloud completion method provided in this application. In order to facilitate better implementation of the lightweight 3D point cloud completion method provided in this application, this application also provides a lightweight 3D point cloud completion device from the perspective of functional modules.

[0146] See Figure 8 , Figure 8 This is a schematic diagram of a lightweight 3D point cloud completion device according to this application. In this application, the lightweight 3D point cloud completion device 800 may specifically include the following structure:

[0147] The acquisition unit 801 is used to acquire the defect cloud to be completed, wherein the defect cloud to be completed is specifically a three-dimensional point cloud;

[0148] The completion unit 802 is used to input the incomplete point cloud to be completed into a pre-configured lightweight point cloud completion network. The lightweight point cloud completion network includes an incomplete point cloud encoder based on a fine-tuning point completion network, a lightweight coarse-grained point cloud decoder, a medium-grained point cloud decoder based on incomplete point cloud feature reuse, and a fine-grained point cloud decoder based on incomplete point cloud feature reuse. The incomplete point cloud encoder of the fine-tuning point completion network is used to extract the incomplete point cloud features of the incomplete point cloud input to the network. The lightweight coarse-grained point cloud decoder is used to generate a coarse-grained complete point cloud. The medium-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a medium-grained complete point cloud. The fine-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a fine-grained complete point cloud.

[0149] Extraction unit 803 is used to extract fine-grained complete point cloud output from the lightweight point cloud completion network as the point cloud completion result.

[0150] In one exemplary embodiment, the defect cloud encoder based on a fine-tuning point completion network includes a one-dimensional convolutional module. Maximum pooling module Copy module splicing module One-dimensional convolution module and max pooling module Let the dimension of the three-dimensional coordinate matrix of the defective cloud input to the network be... , This refers to the number of points in the defect cloud. The defect cloud encoder based on the fine-tuning point completion network includes the following processing:

[0151] 1.1) will The 3D coordinate matrix of the defect cloud is input into the 1D convolution module. ,get dimensional initial feature matrix;

[0152] 1.2) will The initial feature matrix is ​​input to the max pooling module. ,get Initial eigenvectors;

[0153] 1.3) By the copy module Will The initial feature vector is copied in the row direction. Next, the splicing module Copy module The replication results, and The initial feature matrix is ​​concatenated along the column direction to obtain 3D concatenation of feature matrices;

[0154] 1.4) 3D concatenation feature matrix input to 1D convolution module ,get 3D fusion feature matrix;

[0155] 1.5) 3D fusion feature matrix input max pooling module ,get The feature vector of the defect cloud is used as the feature of the defect cloud.

[0156] In yet another exemplary embodiment, the lightweight coarse-grained point cloud decoder includes a transposed convolution module and a copy module. splicing module One-dimensional convolution module Shape reshaping module and one-dimensional convolutional modules The lightweight coarse-grained point cloud decoder includes the following processing:

[0157] 2.1) will The feature vector of the defective cloud is input into the transposed convolution module to obtain... 3D transposed convolution feature matrix;

[0158] 2.2) By the copy module Will The feature vector of the defective cloud is copied 128 times in the row direction, and then the concatenation module performs the stitching. Copy module The replication results, and The transposed convolutional feature matrices are concatenated along the column direction to obtain... 3D concatenation of feature matrices;

[0159] 2.3) will 3D concatenation feature matrix input to 1D convolution module ,get 3D fusion feature matrix;

[0160] 2.4) Shape Reshaping Module right Shape reshaping is performed on the 3D fusion feature matrix to obtain Reshape the feature matrix in 3D;

[0161] 2.5) 3D reshaping feature matrix input one-dimensional convolution module ,get The three-dimensional coordinate matrix of the coarse-grained point cloud is used as the coarse-grained complete point cloud.

[0162] In yet another exemplary embodiment, the medium-granularity point cloud decoder based on residual point cloud feature reuse includes a merging module. Farthest point sampling module One-dimensional convolution module Lightweight residual module One-dimensional convolution module Copy module splicing module One-dimensional convolution module Self-attention module One-dimensional convolution module Lightweight residual module Copy module splicing module One-dimensional convolution module Shape reshaping module One-dimensional convolution module The medium-granularity point cloud decoder based on residual point cloud feature reuse includes the following processing:

[0163] 3.1) By the merging module Will 3D coordinate matrix of defect cloud and The three-dimensional coordinate matrices of coarse-grained point clouds are merged to obtain... 3D coordinate matrix of merged point cloud;

[0164] 3.2) will 3D point cloud 3D coordinate matrix input farthest point sampling module ,get The 3D coordinate matrix of the sampled point cloud, where, Choose 512 or 1024;

[0165] 3.3) will 3D sampling point cloud 3D coordinate matrix input to 1D convolution module ,get Low-level feature matrix;

[0166] 3.4) will Low-dimensional feature matrix input lightweight residual module ,get Low-dimensional residual feature matrix;

[0167] 3.5) will 3D residual feature vector input to 1D convolution module ,get Compressed feature vectors of dimensional residual cloud;

[0168] 3.6) By the copy module Will The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module Copy module The replication results and The low-level residual feature matrices are concatenated along the column direction to obtain... Mid-level concatenation of feature matrices;

[0169] 3.7) will The feature matrix is ​​concatenated in the middle layer and input into the one-dimensional convolution module. ,get Mid-level fusion feature matrix;

[0170] 3.8) will The mid-level fusion feature matrix is ​​input into the attention module. ,get Mid-level self-attention feature matrix;

[0171] 3.9) will The self-attention feature matrix in the middle layer is input into the one-dimensional convolutional module. ,get Mid-level convolutional feature matrix;

[0172] 3.10) will The mid-level convolutional feature matrix is ​​input into a lightweight residual module. ,get Mid-level residual feature matrix;

[0173] 3.11) By the copy module Will The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module Copy module The replication results and The residual feature matrices of the middle layer are concatenated in the column direction to obtain... High-level splicing feature matrix;

[0174] 3.12) will High-level concatenated feature matrix input to one-dimensional convolution module ,get High-dimensional compressed feature matrix;

[0175] 3.13) will High-dimensional compressed feature matrix input shape reshaping module ,get Reshape the feature matrix in 3D;

[0176] 3.14) will 3D reshaping feature matrix input one-dimensional convolution module This yields the three-dimensional incremental coordinates of 2048 points;

[0177] 3.15) will Copying the 3D coordinate matrix of the 3D sampled point cloud in the row direction This is repeated and added to the three-dimensional incremental coordinates of 2048 points to obtain... The three-dimensional coordinate matrix of the medium-grained point cloud is used as the medium-grained complete point cloud.

[0178] In yet another exemplary embodiment, the fine-grained point cloud decoder based on residual point cloud feature reuse includes a merging module. Farthest point sampling module One-dimensional convolution module Lightweight residual module One-dimensional convolution module Copy module splicing module One-dimensional convolution module Self-attention module One-dimensional convolution module Lightweight residual module Copy module splicing module One-dimensional convolution module Shape reshaping module One-dimensional convolution module The fine-grained point cloud decoder based on residual point cloud feature reuse includes the following processing:

[0179] 4.1) By the merging module Will 3D coordinate matrix of defect cloud and The 3D coordinate matrices of the mid-grained point clouds are merged to obtain... 3D coordinate matrix of merged point cloud;

[0180] 4.2) 3D point cloud 3D coordinate matrix input farthest point sampling module ,get 3D coordinate matrix of sampled point cloud;

[0181] 4.3) will 3D sampling point cloud 3D coordinate matrix input to 1D convolution module ,get Low-level feature matrix;

[0182] 4.4) will Low-dimensional feature matrix input lightweight residual module ,get Low-dimensional residual feature matrix;

[0183] 4.5) 3D residual feature vector input to 1D convolution module ,get Compressed feature vectors of dimensional residual cloud;

[0184] 4.6) By the copy module Will The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module Copy module The replication results and The low-level residual feature matrices are concatenated along the column direction to obtain... Mid-level concatenation of feature matrices;

[0185] 4.7) will The feature matrix is ​​concatenated in the middle layer and input into the one-dimensional convolution module. ,get Mid-level fusion feature matrix;

[0186] 4.8) will The mid-level fusion feature matrix is ​​input into the attention module. ,get Mid-level self-attention feature matrix;

[0187] 4.9) will The self-attention feature matrix in the middle layer is input into the one-dimensional convolutional module. ,get Mid-level convolutional feature matrix;

[0188] 4.10) will The mid-level convolutional feature matrix is ​​input into a lightweight residual module. ,get Mid-level residual feature matrix;

[0189] 4.11) By the copy module Will The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module Copy module The replication results and The residual feature matrices of the middle layer are concatenated in the column direction to obtain... High-level splicing feature matrix;

[0190] 4.12) will High-level concatenated feature matrix input to one-dimensional convolution module ,get High-dimensional compressed feature matrix;

[0191] 4.13) will High-dimensional compressed feature matrix input shape reshaping module ,get Reshape the feature matrix in 3D;

[0192] 4.14) will 3D reshaping feature matrix input one-dimensional convolution module This yields 16,384 points in three-dimensional incremental coordinates.

[0193] 4.15) will Copying the 3D coordinate matrix of the 3D sampled point cloud in the row direction This is repeated and added to the three-dimensional incremental coordinates of 16384 points to obtain... The three-dimensional coordinate matrix of the fine-grained point cloud is used as the fine-grained complete point cloud.

[0194] In yet another exemplary embodiment, the module structure of the lightweight residual module in both the medium-granularity point cloud decoder based on residual point cloud feature reuse and the fine-granularity point cloud decoder based on residual point cloud feature reuse includes a linear layer. Linear layer Layer normalization layer and activation layer The corresponding residual feature matrix extraction process includes the following steps:

[0195] The original feature matrix of dimension 1 passes through a linear layer in sequence. and linear layer The number of feature points is halved. After 2, the points are restored to their original value. Then, it goes through the normalization layers in sequence. and activation layer Finally, the output will be Dimensional results and Adding the original feature matrices in dimension 1 gives the corresponding The residual feature matrix is ​​used as the output.

[0196] In yet another exemplary embodiment, the apparatus further includes a training unit 804 for:

[0197] Obtain incomplete point cloud samples and their corresponding fine-grained complete point cloud ground truth values;

[0198] A lightweight point cloud completion network is trained based on incomplete point cloud samples and their corresponding fine-grained complete point cloud ground truth values.

[0199] This application also provides a processing device from a hardware architecture perspective, see [link / reference]. Figure 9 , Figure 9This diagram illustrates a structural schematic of the processing device of this application. Specifically, the processing device may include a processor 901, a memory 902, and an input / output device 903. The processor 901 executes the computer program stored in the memory 902 to implement, for example... Figure 1 The corresponding steps of the lightweight 3D point cloud completion method in the embodiment; or, when the processor 901 executes the computer program stored in the memory 902, it implements as follows: Figure 8 Corresponding to the functions of each unit in the embodiment, the memory 902 is used to store the functions executed by the processor 901 as described above. Figure 1 The computer program required for the lightweight 3D point cloud completion method in the corresponding embodiment.

[0200] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 902 and executed by processor 901 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0201] The processing device may include, but is not limited to, processor 901, memory 902, and input / output device 903. Those skilled in the art will understand that the illustrations are merely examples of the processing device and do not constitute a limitation on the processing device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the processing device may also include network access devices, buses, etc., and processor 901, memory 902, input / output device 903, etc., are connected via a bus.

[0202] The processor 901 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the processing device, connecting various parts of the device through various interfaces and lines.

[0203] The memory 902 can be used to store computer programs and / or modules. The processor 901 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 902 and by calling data stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required to complete at least one function, etc.; the data storage area may store data created according to the use of the processing device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, etc., or other volatile solid-state storage devices.

[0204] When processor 901 executes a computer program stored in memory 902, it can specifically perform the following functions:

[0205] Obtain the defect cloud to be completed, which is specifically a 3D point cloud;

[0206] The incomplete point cloud to be completed is input into a pre-configured lightweight point cloud completion network. The lightweight point cloud completion network includes an incomplete point cloud encoder based on a fine-tuning point cloud completion network, a lightweight coarse-grained point cloud decoder, a medium-grained point cloud decoder based on incomplete point cloud feature reuse, and a fine-grained point cloud decoder based on incomplete point cloud feature reuse. The incomplete point cloud encoder based on the fine-tuning point cloud completion network is used to extract the incomplete point cloud features of the incomplete point cloud input to the network. The lightweight coarse-grained point cloud decoder is used to generate a coarse-grained complete point cloud. The medium-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a medium-grained complete point cloud. The fine-grained point cloud decoder based on incomplete point cloud feature reuse is used to generate a fine-grained complete point cloud.

[0207] Extract the fine-grained complete point cloud from the output of the lightweight point cloud completion network as the point cloud completion result.

[0208] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the lightweight 3D point cloud completion device, processing equipment, and its corresponding units described above can be found in, for example... Figure 1 The description of the lightweight 3D point cloud completion method in the corresponding embodiment will not be repeated here.

[0209] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0210] Therefore, this application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the present application. Figure 1 For details on the steps of the lightweight 3D point cloud completion method in the corresponding embodiment, please refer to the following: Figure 1 The description of the lightweight 3D point cloud completion method in the corresponding embodiment will not be repeated here.

[0211] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0212] Because of the instructions stored in the computer-readable storage medium, the present application can be executed as described above. Figure 1 The steps of the lightweight 3D point cloud completion method in the corresponding embodiment can therefore achieve the results of this application. Figure 1 The beneficial effects that the lightweight 3D point cloud completion method can achieve in the corresponding embodiments are detailed in the preceding description and will not be repeated here.

[0213] The lightweight 3D point cloud completion method, apparatus, processing device, and computer-readable storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A lightweight 3D point cloud completion method, characterized in that, The method includes: Obtain the defect cloud to be completed, wherein the defect cloud to be completed is specifically a three-dimensional point cloud; The incomplete point cloud to be completed is input into a pre-configured lightweight point cloud completion network. The lightweight point cloud completion network includes an incomplete point cloud encoder based on a fine-tuning point cloud completion network, a lightweight coarse-grained point cloud decoder, a medium-grained point cloud decoder based on incomplete point cloud feature reuse, and a fine-grained point cloud decoder based on incomplete point cloud feature reuse. The incomplete point cloud encoder based on the fine-tuning point cloud completion network is used to extract the incomplete point cloud features of the incomplete point cloud input to the network. The lightweight coarse-grained point cloud decoder is used to generate a coarse-grained complete point cloud. The medium-grained point cloud decoder based on the incomplete point cloud feature reuse is used to generate a medium-grained complete point cloud. The fine-grained point cloud decoder based on the incomplete point cloud feature reuse is used to generate a fine-grained complete point cloud. Extract the fine-grained complete point cloud output by the lightweight point cloud completion network as the point cloud completion result; The residual cloud encoder based on the fine-tuning point completion network includes a one-dimensional convolutional module. Maximum pooling module Copy module splicing module One-dimensional convolution module and max pooling module Let the dimension of the three-dimensional coordinate matrix of the residual cloud input to the network be... , This refers to the number of points in the defect cloud. The defect cloud encoder based on the fine-tuning point completion network includes the following processing: Will The three-dimensional coordinate matrix of the defect cloud is input into the one-dimensional convolution module. ,get dimensional initial feature matrix; The The initial feature matrix is ​​input into the max pooling module. ,get Initial eigenvectors; By the copy module The The initial feature vector is copied in the row direction. Next, the splicing module The copy module The replication result, and the... The initial feature matrix is ​​concatenated along the column direction to obtain 3D concatenation of feature matrices; The The 3D concatenated feature matrix is ​​input into the 1D convolution module. ,get 3D fusion feature matrix; The The 3D fusion feature matrix is ​​input into the max pooling module. ,get The feature vector of the residual defect cloud is used as the feature of the residual defect cloud.

2. The method according to claim 1, characterized in that, The lightweight coarse-grained point cloud decoder includes a transposed convolution module and a copy module. splicing module One-dimensional convolution module Shape reshaping module and one-dimensional convolutional modules The lightweight coarse-grained point cloud decoder includes the following processing functions: The The feature vector of the defective cloud is input into the transposed convolution module to obtain... 3D transposed convolution feature matrix; By the copy module The The feature vector of the defective cloud is copied 128 times in the row direction, and then the splicing module is used. The copy module The replication result, and the... The transposed convolutional feature matrices are concatenated along the column direction to obtain... 3D concatenation of feature matrices; The The 3D concatenated feature matrix is ​​input into the 1D convolution module. ,get 3D fusion feature matrix; By the shape reshaping module Regarding the Shape reshaping is performed on the 3D fusion feature matrix to obtain Reshape the feature matrix in 3D; The The 3D reshaped feature matrix is ​​input into the 1D convolution module. ,get The three-dimensional coordinate matrix of the coarse-grained point cloud is used as the coarse-grained complete point cloud.

3. The method according to claim 2, characterized in that, The medium-granularity point cloud decoder based on the reuse of residual point cloud features includes a merging module. Farthest point sampling module One-dimensional convolution module Lightweight residual module One-dimensional convolution module Copy module splicing module One-dimensional convolution module Self-attention module One-dimensional convolution module Lightweight residual module Copy module splicing module One-dimensional convolution module Shape reshaping module One-dimensional convolution module The medium-granularity point cloud decoder based on the reuse of the residual point cloud features includes the following processing: By the merging module The The three-dimensional coordinate matrix of the defect cloud and the above The three-dimensional coordinate matrices of coarse-grained point clouds are merged to obtain... 3D coordinate matrix of merged point cloud; The The three-dimensional coordinate matrix of the merged point cloud is input to the farthest point sampling module. ,get The 3D coordinate matrix of the sampled point cloud, where, Choose 512 or 1024; The The 3D coordinate matrix of the 3D sampled point cloud is input into the 1D convolution module. ,get Low-level feature matrix; The The low-dimensional feature matrix is ​​input into the lightweight residual module. ,get Low-dimensional residual feature matrix; The The feature vector of the defect cloud is input into the one-dimensional convolution module. ,get Compressed feature vectors of dimensional residual cloud; By the copy module The The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module The copy module The replication result is the same as the stated The low-level residual feature matrices are concatenated along the column direction to obtain... Mid-level concatenation of feature matrices; The The feature matrix concatenated in the middle layer is input into the one-dimensional convolution module. ,get Mid-level fusion feature matrix; The The self-attention module is input with the fusion feature matrix of the middle layer. ,get Mid-level self-attention feature matrix; The The self-attention feature matrix of the middle layer is input into the one-dimensional convolutional module. ,get Mid-level convolutional feature matrix; The The lightweight residual module is input to the mid-level convolutional feature matrix. ,get Mid-level residual feature matrix; By the copy module The The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module The copy module The replication result is the same as the stated The residual feature matrices of the middle layer are concatenated in the column direction to obtain... High-level splicing feature matrix; The The high-level concatenated feature matrix is ​​input into the one-dimensional convolution module. ,get High-dimensional compressed feature matrix; The The high-dimensional compressed feature matrix is ​​input into the shape reshaping module. ,get Reshape the feature matrix in 3D; The The 3D reshaped feature matrix is ​​input into the 1D convolution module. This yields the three-dimensional incremental coordinates of 2048 points; The Copying the 3D coordinate matrix of the 3D sampled point cloud in the row direction This is repeated and then added to the three-dimensional incremental coordinates of the 2048 points to obtain... The three-dimensional coordinate matrix of the medium-grained point cloud is used as the medium-grained complete point cloud.

4. The method according to claim 3, characterized in that, The fine-grained point cloud decoder based on the reuse of the residual point cloud features includes a merging module. Farthest point sampling module One-dimensional convolution module Lightweight residual module One-dimensional convolution module Copy module splicing module One-dimensional convolution module Self-attention module One-dimensional convolution module Lightweight residual module Copy module splicing module One-dimensional convolution module Shape reshaping module One-dimensional convolution module The fine-grained point cloud decoder based on the reuse of the residual point cloud features includes the following processing: By the merging module The The three-dimensional coordinate matrix of the defect cloud and the above The 3D coordinate matrices of the mid-grained point clouds are merged to obtain... 3D coordinate matrix of merged point cloud; The The three-dimensional coordinate matrix of the merged point cloud is input to the farthest point sampling module. ,get 3D coordinate matrix of sampled point cloud; The The 3D coordinate matrix of the 3D sampled point cloud is input into the 1D convolution module. ,get Low-level feature matrix; The The low-dimensional feature matrix is ​​input into the lightweight residual module. ,get Low-dimensional residual feature matrix; The The feature vector of the defect cloud is input into the one-dimensional convolution module. , to obtain the Compressed feature vectors of dimensional residual cloud; By the copy module The The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module The copy module The replication result is the same as the stated The low-level residual feature matrices are concatenated along the column direction to obtain... Mid-level concatenation of feature matrices; The The feature matrix concatenated in the middle layer is input into the one-dimensional convolution module. ,get Mid-level fusion feature matrix; The The self-attention module is input with the fusion feature matrix of the middle layer. ,get Mid-level self-attention feature matrix; The The self-attention feature matrix of the middle layer is input into the one-dimensional convolutional module. ,get Mid-level convolutional feature matrix; The The lightweight residual module is input to the mid-level convolutional feature matrix. ,get Mid-level residual feature matrix; By the copy module The The feature vector of the dimensional residual cloud is copied in the row direction. Next, the splicing module The copy module The replication result is the same as the stated The residual feature matrices of the middle layer are concatenated in the column direction to obtain... High-level splicing feature matrix; The The high-level concatenated feature matrix is ​​input into the one-dimensional convolution module. ,get High-dimensional compressed feature matrix; The The high-dimensional compressed feature matrix is ​​input into the shape reshaping module. ,get Reshape the feature matrix in 3D; The The 3D reshaped feature matrix is ​​input into the 1D convolution module. This yields 16,384 points in three-dimensional incremental coordinates. The Copying the 3D coordinate matrix of the 3D sampled point cloud in the row direction This is repeated and then added to the three-dimensional incremental coordinates of the 16384 points to obtain... The three-dimensional coordinate matrix of the fine-grained point cloud is used as the fine-grained complete point cloud.

5. The method according to claim 4, characterized in that, The module structure of the lightweight residual module in both the medium-granularity point cloud decoder and the fine-granularity point cloud decoder based on the residual point cloud feature reuse includes a linear layer. Linear layer Layer normalization layer and activation layer The corresponding residual feature matrix extraction process includes the following steps: The original feature matrix of dimension 1 passes through the linear layer in sequence. and the linear layer The number of feature points is halved. Then revert to the original points. Then, it passes through the layer normalization layer in sequence. and the activation layer Finally, the output will be Dimensional results and the above Adding the original feature matrices in one dimension yields the corresponding The residual feature matrix is ​​used as the output.

6. The method according to claim 1, characterized in that, The method further includes: Obtain incomplete point cloud samples and their corresponding fine-grained complete point cloud ground truth values; The lightweight point cloud completion network is trained based on the incomplete point cloud samples and their corresponding fine-grained complete point cloud ground truth values.

7. A lightweight 3D point cloud completion device, characterized in that, The device includes: An acquisition unit is used to acquire a defect cloud to be completed, wherein the defect cloud to be completed is specifically a three-dimensional point cloud; The completion unit is used to input the incomplete point cloud to be completed into a pre-configured lightweight point cloud completion network. The lightweight point cloud completion network includes an incomplete point cloud encoder based on a fine-tuning point completion network, a lightweight coarse-grained point cloud decoder, a medium-grained point cloud decoder based on incomplete point cloud feature reuse, and a fine-grained point cloud decoder based on incomplete point cloud feature reuse. The incomplete point cloud encoder based on the fine-tuning point completion network is used to extract the incomplete point cloud features of the incomplete point cloud input to the network. The lightweight coarse-grained point cloud decoder is used to generate a coarse-grained complete point cloud. The medium-grained point cloud decoder based on the incomplete point cloud feature reuse is used to generate a medium-grained complete point cloud. The fine-grained point cloud decoder based on the incomplete point cloud feature reuse is used to generate a fine-grained complete point cloud. The extraction unit is used to extract the fine-grained complete point cloud output by the lightweight point cloud completion network as the point cloud completion result; The residual cloud encoder based on the fine-tuning point completion network includes a one-dimensional convolutional module. Maximum pooling module Copy module splicing module One-dimensional convolution module and max pooling module Let the dimension of the three-dimensional coordinate matrix of the residual cloud input to the network be... , This refers to the number of points in the defect cloud. The defect cloud encoder based on the fine-tuning point completion network includes the following processing: Will The three-dimensional coordinate matrix of the defect cloud is input into the one-dimensional convolution module. ,get dimensional initial feature matrix; The The initial feature matrix is ​​input into the max pooling module. ,get Initial eigenvectors; By the copy module The The initial feature vector is copied in the row direction. Next, the splicing module The copy module The replication result, and the... The initial feature matrix is ​​concatenated along the column direction to obtain 3D concatenation of feature matrices; The The 3D concatenated feature matrix is ​​input into the 1D convolution module. ,get 3D fusion feature matrix; The The 3D fusion feature matrix is ​​input into the max pooling module. ,get The feature vector of the residual defect cloud is used as the feature of the residual defect cloud.

8. A processing apparatus, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the method as described in any one of claims 1 to 6 when it invokes the computer program in the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the method of any one of claims 1 to 6.

Images