Improved three-dimensional point cloud completion method and device and processing equipment

Abstract

The invention provides an improved three-dimensional point cloud completion method, an improved three-dimensional point cloud completion device and processing equipment, a novel three-dimensional point cloud completion network structure is specially designed, a mapping relation is established between input and output through design and training of a high-precision completion network, and on the basis of deep analysis of various correlations in point cloud, a new three-dimensional point cloud completion network is established. The correlation is utilized to the maximum extent, each level feature extracted in the whole point cloud complementing process and generated point clouds with different granularities are corrected in time, and finally a high-precision, high-stability and efficient three-dimensional point cloud complementing effect is formed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional point cloud completion, in particular to an improved three-dimensional point cloud completion method, device and processing equipment. BACKGROUND

[0002] With the popularization of various three-dimensional scanning devices such as laser radar and depth camera, the acquisition of point cloud data has become more convenient, and has been widely applied in the fields of robots, autonomous driving and three-dimensional modeling.

[0003] However, the original point cloud data directly collected by these three-dimensional scanning devices often presents sparsity and incompleteness due to occlusion, device resolution or acquisition distance limitations. Therefore, how to generate complete point cloud from partial observation data is of great significance to promote the development of downstream applications.

[0004] Point Completion Network (PCN) is one of the pioneering works in the field of point cloud completion. This method adopts an encoder-decoder structure, in which the encoder extracts global features from the incomplete point cloud using a shared perception machine, and the decoder adopts a top-down approach, i.e., first generating a complete coarse-grained point cloud using a multi-layer perception machine, and then using a shared perception machine to complete the local details. The advantage of this method is that the calculation is simple, and the completion accuracy is very stable under various changes of the target point cloud. The disadvantage is that the completion process is very rough, and the various correlations in the point cloud are not fully utilized to timely correct the extracted features at different levels and the generated point clouds at different granularities, resulting in very low completion accuracy.

[0005] In order to improve the completion accuracy, various attention mechanisms are introduced on the basis of the above-mentioned encoder-decoder structure. Representative works include Point Cloud Completion with Geometry-Aware Transformers (PointTr), Patch Seeds based Point Cloud Completion with Upsample Transformer (SeedFormer) and Complementing Point Cloud via Self-view Augmentation and Self-structure Dual-generator (SVDFormer).

[0006] PointTr introduces a self-attention mechanism in a missing point cloud encoder based on a dynamic graph convolutional neural network (DGCNN), uses the local neighborhood feature correlation and global feature correlation of the points in the missing point cloud to correct the features of the points layer by layer; SeedFormer uses a self-attention mechanism between the abstraction layers of a missing point cloud encoder based on PointNet++, uses the global feature correlation of the points in the missing point cloud at different scales to correct the features of the points layer by layer; meanwhile, a cross-attention mechanism is introduced in the medium-granularity and fine-granularity point cloud decoders, and the mutual correlation of the coarse-granularity point cloud features and other granularity point cloud features is used to correct the features of other granularity point cloud layer by layer; SVDFormer introduces a self-attention mechanism in a missing point cloud encoder based on three-view depth maps, uses the global feature correlation of the depth maps to correct the global features of the depth maps, and meanwhile introduces a cross-attention mechanism in the medium-granularity and fine-granularity point cloud decoders, and uses the mutual correlation of the missing point cloud features and the coarse-granularity point cloud features to correct the coarse-granularity point cloud features. These methods effectively improve the completion accuracy.

[0007] However, the inventors of the present application found that whether the various correlations on which the above methods are based exist in actual applications, and if so, how strong or weak they are, have not been thoroughly analyzed, which leads to the fact that after using certain correlations, the above methods not only waste a large amount of computing resources, but also do not improve the overall completion accuracy.

[0008] Therefore, how to make the most of the various correlations in the point cloud based on in-depth analysis of the correlations, and timely correct the features extracted at each level and the different granularity point clouds generated in the point cloud completion process, so as to further improve the overall completion accuracy, is still a problem to be solved. SUMMARY

[0009] The present application provides an improved three-dimensional point cloud completion method, device and processing equipment, and a novel three-dimensional point cloud completion network structure is specially designed. Through the design and training of the high-precision completion network, a mapping relationship is established between the input and the output, the various correlations in the point cloud are maximized based on in-depth analysis of the correlations, and the features extracted at each level and the different granularity point clouds generated in the point cloud completion process are timely corrected, so as to finally form a high-precision, high-stability and efficient three-dimensional point cloud completion effect.

[0010] In a first aspect, the present application provides an improved three-dimensional point cloud completion method, which comprises: obtaining a missing point cloud to be completed, wherein the missing point cloud to be completed is specifically a three-dimensional point cloud; input the incomplete point cloud to be completed into a pre-configured three-dimensional point cloud completion network, wherein the three-dimensional point cloud completion network sequentially comprises an incomplete point cloud encoder based on a global feature fusion point completion network, a coarse-grained point cloud decoder based on residual fine-tuning, a medium-grained point cloud decoder based on feature layer-by-layer fusion correction, and a fine-grained point cloud decoder based on feature layer-by-layer fusion correction, the incomplete point cloud encoder extracts incomplete point cloud features according to the input point cloud of the network, the coarse-grained point cloud decoder generates a coarse-grained complete point cloud according to the incomplete point cloud features, the medium-grained point cloud decoder generates a medium-grained complete point cloud according to the input point cloud of the network, the incomplete point cloud features, and the coarse-grained complete point cloud, and the fine-grained point cloud decoder generates a fine-grained complete point cloud according to the input point cloud of the network, the incomplete point cloud features, and the medium-grained complete point cloud; extract the fine-grained complete point cloud output by the three-dimensional point cloud completion network as a point cloud completion result.

[0011] In a second aspect, the present application provides an improved three-dimensional point cloud completion device, which comprises: an acquisition unit configured to acquire an incomplete point cloud to be completed, wherein the incomplete point cloud to be completed is a three-dimensional point cloud; a completion unit configured to input the incomplete point cloud to be completed into a pre-configured three-dimensional point cloud completion network, wherein the three-dimensional point cloud completion network sequentially comprises an incomplete point cloud encoder based on a global feature fusion point completion network, a coarse-grained point cloud decoder based on residual fine-tuning, a medium-grained point cloud decoder based on feature layer-by-layer fusion correction, and a fine-grained point cloud decoder based on feature layer-by-layer fusion correction, the incomplete point cloud encoder extracts incomplete point cloud features according to the input point cloud of the network, the coarse-grained point cloud decoder generates a coarse-grained complete point cloud according to the incomplete point cloud features, the medium-grained point cloud decoder generates a medium-grained complete point cloud according to the input point cloud of the network, the incomplete point cloud features, and the coarse-grained complete point cloud, and the fine-grained point cloud decoder generates a fine-grained complete point cloud according to the input point cloud of the network, the incomplete point cloud features, and the medium-grained complete point cloud; an extraction unit configured to extract the fine-grained complete point cloud output by the three-dimensional point cloud completion network as a point cloud completion result.

[0012] In a third aspect, the present application provides a processing device comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program in the memory to perform the method provided in the first aspect of the present application.

[0013] In a fourth aspect, the present application provides a computer readable storage medium, which stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to execute the method provided in the first aspect of the present application.

[0014] From the above, the present application has the following beneficial effects: For the three-dimensional point cloud completion target, a novel three-dimensional point cloud completion network structure is specially designed, a mapping relationship is established between the input and the output through the design and training of the high-precision completion network, the correlation in the point cloud is deeply analyzed, the correlation is maximally utilized, and each level feature extracted in the point cloud completion process and different granularity point clouds generated are timely corrected, so that a high-precision, high-stability and high-efficiency three-dimensional point cloud completion effect is finally formed. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A flowchart of the improved three-dimensional point cloud completion method of the present application; Figure 2 A working diagram of the three-dimensional point cloud completion network of the present application; Figure 3 Another working diagram of the three-dimensional point cloud completion network of the present application; Figure 4 A working diagram of the incomplete point cloud encoder based on the global feature fusion point completion network of the present application; Figure 5 A working diagram of the coarse-grained point cloud decoder based on residual fine-tuning of the present application; Figure 6 A working diagram of the medium / fine-grained point cloud decoder based on feature layer-by-layer fusion correction of the present application; Figure 7 A structural diagram of the feature correction module of the present application; Figure 8 A structural diagram of the improved three-dimensional point cloud completion device of the present application; Figure 9 A structural diagram of the processing device of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0018] The terms "first", "second", and the like in the description and in the claims of the present application and above-described drawings are used to distinguish similar objects and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments described herein are capable of functioning in other sequences than the one described or illustrated herein. Furthermore, the terms "comprise", "comprising", "include", "including", and "has", "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a list of steps or modules is not necessarily limited to those steps or modules that are expressly listed, but can include additional steps or modules not expressly listed or inherent to such process, method, product, or apparatus. The naming or numbering of the steps appearing in the present application does not mean that the steps in the method flow must be executed in the order / sequential order indicated by the naming or numbering, and the steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0019] The division of modules appearing in the present application is a logical division, and in actual application, there can be another division manner, for example, multiple modules can be combined or integrated in another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, the indirect coupling or communication connection between the modules can be electrical or other similar forms, which are not limited in the present application. In addition, the modules or sub-modules described as separate components can or can not be physically separated, can or can not be physical modules, or can be distributed to multiple circuit modules, and part or all of the modules can be selected according to actual needs to achieve the purpose of the present application.

[0020] Before introducing the improved three-dimensional point cloud completion method provided by the present application, first introduce the background content involved in the present application.

[0021] The present application provides an improved three-dimensional point cloud completion method, device, processing equipment and computer readable storage medium. The present application specially designs a novel three-dimensional point cloud completion network structure, establishes a mapping relationship between the input and the output through the design and training of the high-precision completion network, realizes the maximum utilization of various correlations in the point cloud on the basis of in-depth analysis of the correlations, and timely corrects the features extracted at each level and the different granularity point clouds generated in the point cloud completion process, and finally forms a high-precision, high-stability and efficient three-dimensional point cloud completion effect.

[0022] The improved 3D point cloud completion method mentioned in this application can be executed by an improved 3D point cloud completion device, or by various types of processing devices, such as a server, physical host, or user equipment (UE) that integrates the improved 3D point cloud completion device. The improved 3D point cloud completion device can be implemented using hardware or software. The UE can specifically be a terminal device such as a smartphone, tablet computer, laptop computer, desktop computer, or personal digital assistant (PDA). The processing device can be configured in a device cluster.

[0023] It can be understood that, considering that the solution of this application is mainly a point cloud completion processing carried out on the basis of a pre-configured three-dimensional point cloud completion network, the processing equipment that executes the improved three-dimensional point cloud completion method of this application or is equipped with the corresponding application service of the improved three-dimensional point cloud completion method of this application, generally speaking, only needs to meet the required data processing capabilities, and the specific equipment type and equipment deployment form are relatively flexible and can be flexibly configured according to actual conditions.

[0024] If it involves specific data applications such as the collection of three-dimensional point clouds or the display of three-dimensional point cloud completion results, it is easy to understand that further configuration of the processing equipment in terms of software and hardware is required.

[0025] As an example, the processing device can display the processing process / results such as a three-dimensional point cloud through its own configured display screen (including a touch screen), an external display device, or other device with a display screen.

[0026] As another example, the processing device may include a first device part that performs training work for a 3D point cloud completion network, a second device part that performs application work for the 3D point cloud completion network, and a third device part that performs 3D point cloud acquisition work.

[0027] Next, we will introduce the improved three-dimensional point cloud completion method provided by this application.

[0028] First, see Figure 1 , Figure 1 A schematic flow chart of the improved three-dimensional point cloud completion method of the present application is shown. The improved three-dimensional point cloud completion method provided by the present application may specifically include the following steps S101 to S103: Step S101, obtaining a defect cloud to be completed, wherein the defect cloud to be completed is specifically a three-dimensional point cloud; It can be understood that the scheme of the application is described starting from a pre-configured three-dimensional point cloud completion network, and in the specific application of the three-dimensional point cloud completion network, the corresponding three-dimensional point cloud completion processing is usually initiated in the form of a work task, and the three-dimensional point cloud completion task can also be obtained, which can be initiated manually, received from other devices, or initiated autonomously according to the corresponding autonomous initiation strategy.

[0029] For the three-dimensional point cloud to be completed, that is, the incomplete point cloud to be completed, in the case of a three-dimensional point cloud completion task, it can be carried in the task information, extracted from the corresponding storage location according to the task information, or collected in real time according to the task information, or entered manually.

[0030] In addition, for the obtained three-dimensional point cloud, it is usually incomplete by default and needs to be completed. Of course, in some cases, it can be determined to be incomplete and needs to be completed after detection, and then the specific three-dimensional point cloud completion processing is promoted by the scheme of the application.

[0031] The three-dimensional point cloud is usually collected by various three-dimensional scanning devices such as laser radar and depth camera.

[0032] In step S102, the incomplete point cloud to be completed is input into a pre-configured three-dimensional point cloud completion network, wherein the three-dimensional point cloud completion network sequentially includes an incomplete point cloud encoder based on a global feature fusion point completion network, a coarse-grained point cloud decoder based on residual fine-tuning, a medium-grained point cloud decoder based on feature layer-by-layer fusion correction, and a fine-grained point cloud decoder based on feature layer-by-layer fusion correction. The incomplete point cloud encoder extracts the incomplete point cloud feature according to the input point cloud of the network, the coarse-grained point cloud decoder generates a coarse-grained complete point cloud according to the incomplete point cloud feature, the medium-grained point cloud decoder generates a medium-grained complete point cloud according to the input point cloud of the network, the incomplete point cloud feature and the coarse-grained complete point cloud, and the fine-grained point cloud decoder generates a fine-grained complete point cloud according to the input point cloud of the network, the incomplete point cloud feature and the medium-grained complete point cloud; It can be seen that the application designs a novel three-dimensional point cloud completion network structure, which is composed of four parts, namely an incomplete point cloud encoder based on a global feature fusion point completion network, a coarse-grained point cloud decoder based on residual fine-tuning, a medium-grained point cloud decoder based on feature layer-by-layer fusion correction, and a fine-grained point cloud decoder based on feature layer-by-layer fusion correction, which are sequentially connected. In addition, there is an additional connection relationship between the interval modules.

[0033] Specifically, in combination with Figure 2A working diagram of the three-dimensional point cloud completion network of the present application is shown. In the three-dimensional point cloud completion network, the incomplete point cloud encoder based on global feature fusion point completion network corresponds to the network input side, the fine-grained point cloud decoder based on feature layer-by-layer fusion correction corresponds to the network output side, and the processing idea of incomplete point cloud-coarse-grained point cloud-medium-grained point cloud-fine-grained point cloud is followed to gradually complete the completion / reconstruction of the input network of the incomplete point cloud to be completed. Here, please refer to Figure 2 A logical diagram of the three-dimensional point cloud completion process of the present application is shown.

[0034] Corresponding to the application of the three-dimensional point cloud completion network, the present application can also involve related network training in the early stage, which also corresponds to the case that high-precision, high-stability and efficient three-dimensional point cloud completion effect can be obtained under the network structure design of the present application.

[0035] For this, the method of the present application can further include, in specific operation: Obtain the incomplete point cloud sample and its corresponding fine-grained complete point cloud true value; Train the three-dimensional point cloud completion network based on the incomplete point cloud sample and its corresponding fine-grained complete point cloud true value.

[0036] Among them, in addition to the above-mentioned acquisition method of the incomplete point cloud to be completed, the configuration of the virtual incomplete point cloud can also be involved, that is, the virtual incomplete point cloud that can be put into the network training work can be obtained by modifying the real incomplete point cloud, or the virtual incomplete point cloud that can be put into the network training work can also be directly constructed, so as to enrich the diversity of training samples and guarantee the generalization ability of the network after training.

[0037] The fine-grained complete point cloud true value corresponding to the incomplete point cloud sample is usually completed by manual close-range multi-view point cloud graph cutting and splicing, or is the fine-grained complete point cloud sampled during the construction of the virtual incomplete point cloud, or is obtained by searching the incomplete point cloud sample in a large-scale fine-grained complete data set.

[0038] In the specific network training process, it can be understood that the network training scheme (such as 80% training set and 20% validation set) can be specifically adopted, and the loss function (such as the chamfer distance loss function and the earth moving distance loss function) used in the training process. Considering that it is not the focus of the present application scheme, related ready-made schemes can be adopted, therefore, no further expansion is made here.

[0039] Of course, it can also be understood that both of them can be further optimized and improved on the basis of the existing scheme or even adopt a brand-new self-developed scheme, which is also possible in some cases.

[0040] Step S103, extract the fine-grained complete point cloud output by the three-dimensional point cloud completion network as the point cloud completion result.

[0041] It is easy to understand that after the current incomplete point cloud to be completed is processed by the three-dimensional point cloud completion network, the point cloud completion result output by the three-dimensional point cloud completion network can be extracted, that is, the fine-grained complete point cloud generated by the fine-grained point cloud decoder based on feature layer-by-layer fusion correction in the last layer of the three-dimensional point cloud completion network, to complete a three-dimensional point cloud completion task.

[0042] Further, it can be understood that the three-dimensional point cloud can be collected and configured for related point cloud application services. In this regard, point cloud completion can also be a data preprocessing means for improving the quality of specific point cloud application services such as digital twinning, target recognition, detection and tracking. Behind it can involve specific application scenarios such as three-dimensional modeling, robots and autonomous driving.

[0043] Correspondingly, after the fine-grained complete point cloud output by the three-dimensional point cloud completion network is extracted, it can be locally stored, stored remotely, displayed, pushed, output to complete the completion prompt or used for further point cloud application services.

[0044] From Figure 1 As shown in the embodiments, for the three-dimensional point cloud completion target, the present application specially designs a novel three-dimensional point cloud completion network structure. Through the design and training of the high-precision completion network, a mapping relationship is established between the input and the output, so as to maximize the use of the correlation in the point cloud based on in-depth analysis of the correlation, and timely correct the features extracted at each level and the different granularity point clouds generated in the point cloud completion process, and finally form a high-precision, high-stability and efficient three-dimensional point cloud completion effect.

[0045] Next, the three-dimensional point cloud completion network specially designed by the present application will be further described.

[0046] From the overall level, the input of the high-precision point cloud completion network is the three-dimensional coordinate matrix of the incomplete point cloud to be completed, and the output is the three-dimensional coordinate matrix of the fine-grained complete point cloud. That is, the goal of the high-precision three-dimensional point cloud completion network is to establish a mapping relationship between the input and the output through the design and training of the high-precision completion network, so as to achieve high-precision, high-stability and efficient completion of the incomplete point cloud.

[0047] For this purpose, in combination with Figure 3 Another working schematic diagram of the three-dimensional point cloud completion network of the present application is shown. In the point cloud completion process, there are: First, in the first stage, the dimensional residual defect cloud three-dimensional coordinate matrix (i.e. containing The residual defect cloud three-dimensional coordinate matrix of the points is input into the residual defect cloud encoder based on the global feature fusion point completion network to obtain dimensional residual defect cloud feature vector (i.e. residual defect cloud feature); Then, in the second stage, the The residual point cloud feature vector is further input into the coarse-grained point cloud decoder based on residual fine-tuning to generate dimensional coarse-grained point cloud three-dimensional coordinate matrix (i.e. coarse-grained complete point cloud); Then, in the third stage, the dimensional coarse-grained point cloud three-dimensional coordinate matrix, dimensional residual defect cloud eigenvector and The three-dimensional coordinate matrix of the residual defect cloud is input into the medium-granularity point cloud decoder based on feature layer-by-layer fusion correction to obtain 3D coordinate matrix of medium-granularity point cloud (i.e., medium-granularity complete point cloud); Finally, in the fourth stage, 3D coordinate matrix of medium-sized point cloud, dimensional residual defect cloud eigenvector and The three-dimensional coordinate matrix of the residual defect cloud is input into the fine-grained point cloud decoder based on feature layer-by-layer fusion correction to obtain dimensional fine-grained point cloud 3D coordinate matrix (i.e. fine-grained complete point cloud).

[0048] Regarding the overall operational processing, it can be learned that compared with existing completion networks, the high-precision 3D point cloud completion network designed by itself has the following two main characteristics: (1) The residual point cloud encoder directly uses PCN as the backbone network instead of PointNet++ or DGCNN, and does not use the attention mechanism to ensure the stability of point features. At the same time, the accuracy of the residual point cloud features is further improved by fusing low-level global features with high-level global features. (2) The medium / fine-grained point cloud decoder directly reuses the residual point cloud features and the coarse / medium-grained point cloud features for layer-by-layer fusion, which maximizes the correction of the deviation of the point features produced in the dimensionality upgrade process. At the same time, the feature correction network based on transformation-fine-tuning is used to correct the fused point features layer by layer, which minimizes the error of the point features produced in the fusion process.

[0049] The above measures have effectively improved the completion accuracy. Experiments on multiple benchmark test sets have confirmed that this application solution surpasses existing mainstream methods in multiple accuracy indicators. At the same time, the training and inference speeds, as well as the memory usage, have been significantly reduced, meeting users' needs for cost reduction and quality improvement.

[0050] Next, then continue according to the above-mentioned overall processing architecture involving specific specification parameters, respectively, on the basis of the global feature fusion point completion network of incomplete point cloud encoder, based on the residual fine-tuning coarse-grained point cloud decoder, based on the feature layer-by-layer fusion correction of medium-grained point cloud decoder and based on the feature layer-by-layer fusion correction of fine-grained point cloud decoder four of them are individually described.

[0051] Reference Figure 4 The application based on the global feature fusion point completion network of incomplete point cloud encoder is shown in a working schematic diagram, for the global feature fusion point completion network of incomplete point cloud encoder corresponding to the first stage, specifically can include one-dimensional convolution module , maximum pooling module , copy module , splicing module , one-dimensional convolution module , maximum pooling module , splicing module and one-dimensional convolution module ; On the basis of the above-mentioned component modules, the working process of the global feature fusion point completion network of incomplete point cloud encoder can be: 1.1) input the dimensional incomplete point cloud three-dimensional coordinate matrix to the one-dimensional convolution module , get dimensional low-level feature matrix (i.e. 256-dimensional low-level feature of incomplete point cloud points), wherein the network input point cloud is specifically recorded as dimensional incomplete point cloud three-dimensional coordinate matrix, is the number of points in the incomplete point cloud; 1.2) input the dimensional low-level feature matrix to the maximum pooling module , get dimensional low-level feature vector; 1.3) by copy module copy the dimensional low-level feature vector in the row direction times, and then by splicing module splicing the copy result of copy module , and the dimensional low-level feature matrix obtained in 1.1) in the column direction, get dimensional splicing feature matrix; 1.4) input the dimensional splicing feature matrix to the one-dimensional convolution module , get dimensional high-level feature matrix; 1.5) input the dimensional fusion feature matrix input to the maximum pooling module ,get dimensional high-level feature vector; 1.6) By splicing module Will dimensional high-level feature vector, obtained with 1.2) The low-dimensional feature vectors are concatenated in the column direction to obtain dimensional concatenated feature vector; 1.7) The concatenated feature vector is input into the one-dimensional convolution module ,get The residual defect cloud feature vector is used as the residual defect cloud feature extracted by the residual defect cloud encoder based on the global feature fusion point completion network.

[0052] Under the very specific network structure settings provided above, it can be understood that compared with the existing completion network, the residual point cloud encoder based on the global feature fusion point completion network specially designed for the first layer of the 3D point cloud completion network in this application has the following characteristics: (1) Affected by factors such as the resolution of the acquisition device, the acquisition distance, and the acquisition angle, the local neighborhood distribution of the point cloud actually collected changes dramatically, and the neighborhood correlation is weak. Using the neighborhood features of the point to correct the point features often leads to greater errors in the point features. Therefore, the residual point cloud encoder uses the point completion network encoder based on global features as the backbone network, rather than PointNet++ based on multi-scale neighborhood features or DGCNN based on dynamic neighborhood features, to ensure the stability of point features, thereby laying the foundation for high-precision global feature extraction; (2) In addition to the acquisition factors, the actual point cloud is also affected by the occlusion factors, which causes the acquired point cloud to have various parts and details missing, and the global and local correlations are weak. The use of the attention mechanism to correct the point features often leads to greater errors in the point features. Therefore, the residual point cloud encoder avoids the use of the attention mechanism to eliminate the influence of the above errors on the feature extraction accuracy. (3) The low-level feature vector contains the basic geometric information of the point cloud, and the high-level feature vector contains the global structural information of the point cloud. The fusion of the two is conducive to better balancing the basic geometric information and the global structural information, and further improving the accuracy of residual point cloud feature extraction.

[0053] Next, refer to Figure 5 The schematic diagram of the coarse-grained point cloud decoder based on residual fine-tuning of the present application is shown. For the coarse-grained point cloud decoder based on residual fine-tuning corresponding to the second stage, it can specifically include a transposed convolution module. , Copy Module , splicing module , one-dimensional convolution module , shape reshaping module , residual module and one-dimensional convolution module .

[0054] Based on the above-mentioned component modules, the working process of the coarse-grained point cloud decoder based on residual fine-tuning can be: 2.1) Convert 1.7) to dimensional residual defect cloud feature vector input transposed convolution module ,get dimensional low-level transposed convolution feature matrix; 2.2) By the copy module 1.7) The residual defect cloud feature vector is replicated 128 times in the row direction and then connected by the splicing module The module will be copied The replication results are the same as those obtained in 2.1) The low-dimensional transposed convolution feature matrix is ​​spliced ​​in the column direction to obtain dimensional splicing feature matrix; 2.3) Dimensional concatenated feature matrix input to the one-dimensional convolution module ,get dimensional fusion feature matrix; 2.4) Reshape module right The dimensional fusion feature matrix is ​​reshaped to obtain Dimensionally reshape the feature matrix; 2.5) Dimensionally reshaped feature matrix input to the residual module ,get Dimensional reshape residual feature matrix; 2.6) The reshaped residual feature matrix is ​​input into the one-dimensional convolution module ,get The 3D coordinate matrix of the coarse-grained point cloud is used as the coarse-grained complete point cloud generated by the coarse-grained point cloud decoder based on residual fine-tuning.

[0055] Under the very specific network structure settings provided above, it can be understood that compared with existing completion networks, the residual point cloud encoder based on residual fine-tuning specifically designed for the second layer of the 3D point cloud completion network in this application has the following characteristics: (1) In the process of generating a coarse-grained complete point cloud, the points are sparsely distributed and the correlation between point features is weak. Therefore, using the attention mechanism to correct the point features in the process of generating a coarse-grained point cloud often leads to larger errors in the point features. Therefore, the coarse-grained point cloud decoder based on residual fine-tuning here avoids the use of the attention mechanism to eliminate the influence of the above errors on the accuracy of feature extraction; (2) The low-dimensional transposed convolution feature matrix contains the basic geometric information of the point cloud, and the residual point cloud feature vector contains the global structural information of the point cloud. The fusion of the two is conducive to better balancing the basic geometric information and global structural information, and improving the accuracy of coarse-grained complete point cloud generation. (3) Shape reshaping causes changes in the feature dimension space. This change will introduce dimension conversion errors, so the residual module is introduced. After the remodeling The feature matrix is ​​reshaped and fine-tuned to reduce the dimension conversion error and further improve the generation accuracy of coarse-grained complete point cloud.

[0056] Next, refer to Figure 6 A working diagram of a medium / fine-grained point cloud decoder based on feature layer-by-layer fusion correction of the present application is shown, wherein the subscripted letters after the forward slash and the number of dimensions in the brackets correspond to the fine-grained point cloud decoder based on feature layer-by-layer fusion correction. When focusing on the medium-grained point cloud decoder based on feature layer-by-layer fusion correction here, the content after the forward slash and in the brackets is directly ignored.

[0057] In the above case, the medium-granularity point cloud decoder corresponding to the third stage based on feature layer-by-layer fusion correction can specifically include a merging module , farthest point sampling module , one-dimensional convolution module , Feature Correction Module , one-dimensional convolution module , one-dimensional convolution module , one-dimensional convolution module , Copy Module , splicing module , one-dimensional convolution module , Feature Correction Module , Copy Module , splicing module , one-dimensional convolution module , Feature Correction Module , Copy Module , splicing module , one-dimensional convolution module , shape reshaping module , one-dimensional convolution module and the copy module .

[0058] Based on the above-mentioned modules, the working process of the medium-granularity point cloud decoder based on the layer-by-layer fusion correction of features can be as follows: 3.1) By merging modules The same as 1.1) input The three-dimensional coordinate matrix of the residual defect cloud is obtained by 2.6) The three-dimensional coordinate matrix of the coarse-grained point cloud is merged to obtain Merge point cloud 3D coordinate matrix; 3.2) Merge point cloud 3D coordinate matrix input farthest point sampling module ,get dimensional sampling point cloud three-dimensional coordinate matrix, where Select 512 or 1024; 3.3) The three-dimensional coordinate matrix of the sampling point cloud is input into the one-dimensional convolution module ,get dimensional initial feature matrix; 3.4) Dimensional initial feature matrix input feature correction module ,get dimensional initial modified feature matrix; 3.5) Convert 1.7) to The residual defect cloud feature vector is input into the one-dimensional convolution module ,get dimensional mid-layer residual defect cloud feature vector, and then The residual cloud feature vector of the 1D middle layer is input into the 1D convolution module ,get dimensional low-level residual cloud feature vector, and then The low-dimensional residual cloud feature vector is input into the one-dimensional convolution module ,get dimensional initial residual defect cloud feature vector; 3.6) By the copy module Will The initial residual defect cloud eigenvector is replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as those obtained in 3.4) The initial modified feature matrix of the dimension is spliced ​​in the column direction to obtain dimensional low-level concatenated feature matrix; 3.7) The low-level concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional low-level feature matrix; 3.8) dimensional low-level feature matrix input feature correction module ,get dimensional low-level corrected feature matrix; 3.9) By the copy module 3.5) to obtain The eigenvectors of the residual defect cloud at the low level are replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as those obtained in 3.8) The dimensional low-level corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional mid-layer concatenation feature matrix; 3.10) The concatenated feature matrix of the middle layer is input into the one-dimensional convolution module ,get dimensional mid-level feature matrix; 3.11) Dimensional mid-level feature matrix input feature correction module ,get dimensional mid-level corrected feature matrix; 3.12) By the copy module 3.5) to obtain The residual defect cloud feature vector of the dimensional middle layer is replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as those obtained in 3.10) The dimensional middle layer corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional high-level splicing feature matrix; 3.13) Dimensional high-level concatenated feature matrix input to the one-dimensional convolution module ,get dimensional high-level feature matrix; 3.14) Dimensional high-level feature matrix input shape reshaping module ,get Dimensionally reshape the feature matrix; 3.15) Dimensionally reshape the feature matrix and input it into the one-dimensional convolution module , get the three-dimensional incremental coordinates of 2048 points; 3.16) By the copy module 3.2) The 3D coordinate matrix of the sampling point cloud is replicated in the row direction The second layer is added to the three-dimensional incremental coordinates of 2048 points, and the three-dimensional coordinates matrix of the medium-granularity point cloud in the second dimension is obtained The three-dimensional coordinates matrix of the medium-granularity point cloud in the second dimension is taken as the medium-granularity complete point cloud generated by the medium-granularity point cloud decoder based on feature layer-by-layer fusion and correction.

[0059] Under the above very specific network structure setting, it can be understood that, compared with the existing completion network, the medium-granularity point cloud decoder based on feature layer-by-layer fusion and correction specially designed for the third layer of the three-dimensional point cloud completion network has the following characteristics: (1) In the process of generating the medium-granularity complete point cloud, the missing point cloud features of the corresponding layer are fused by layer-by-layer splicing from the low-dimensional initial feature of the point to the high-dimensional high-layer feature, and the fusion of global features is throughout the entire feature extraction process, which helps to strengthen the continuous guidance of global features to local features and correct the deviation generated in the feature dimensioning process in time. (2) In the process of converting point features from a low-dimensional space to a high-dimensional space, conversion errors are introduced, and the points in the medium-granularity complete point cloud are densely distributed, and the correlation between point features is strong. Therefore, the features are corrected by layer-by-layer correction from the low-dimensional initial feature of the point to the high-dimensional high-layer feature, and the utilization of point feature correlation is throughout the entire feature extraction process, which helps to eliminate conversion errors in time and further improve the accuracy of point feature extraction.

[0060] Continue to focus on the fine-granularity point cloud decoder based on feature layer-by-layer fusion and correction corresponding to the fourth stage, still refer to Figure 6 but in details, the subscripted letter after the forward slash and the dimension number in the brackets are used, that is, the subscripted letter after the forward slash is used instead of the subscripted letter before the forward slash, and the dimension number in the brackets is used instead of the dimension number outside the brackets, for example, it should be the three-dimensional coordinates matrix of the medium-granularity point cloud in the second dimension, not the three-dimensional coordinates matrix of the coarse-granularity point cloud in the second dimension; the input is the merging module , not the merging module .

[0061] Specifically, the fine-granularity point cloud decoder based on feature layer-by-layer fusion and correction includes a merging module , a farthest point sampling module , a one-dimensional convolution module , a feature correction module , a one-dimensional convolution module , a one-dimensional convolution module , a one-dimensional convolution module , a copying module , a splicing module , a one-dimensional convolution module , feature correction module , copy module , splicing module , one-dimensional convolution module , feature correction module , copy module , splicing module , one-dimensional convolution module , shape remodeling module , one-dimensional convolution module , and copy module .

[0062] Based on the above component modules, the working process of the fine-grained point cloud decoder based on feature layer-by-layer fusion correction can be: 4.1) by merging module merge the 3D coordinate matrix of the incomplete point cloud with 1.1) input dimension and the 3D coordinate matrix of the medium-grained point cloud obtained in 3.16) dimension, to obtain a 3D coordinate matrix of a merged point cloud dimension; 4.2) input the 3D coordinate matrix of the merged point cloud dimension into the farthest point sampling module , to obtain a 3D coordinate matrix of a sampled point cloud dimension; 4.3) input the 3D coordinate matrix of the sampled point cloud dimension into the one-dimensional convolution module , to obtain an initial feature matrix dimension; 4.4) input the initial feature matrix dimension into the feature correction module , to obtain an initial corrected feature matrix dimension; 4.5) input the incomplete point cloud feature vector obtained in 1.7) dimension into the one-dimensional convolution module , to obtain a medium-layer incomplete point cloud feature vector dimension, and then input the medium-layer incomplete point cloud feature vector dimension into the one-dimensional convolution module , to obtain a low-layer incomplete point cloud feature vector dimension, and then input the low-layer incomplete point cloud feature vector dimension into the one-dimensional convolution module , to obtain an initial incomplete point cloud feature vector dimension; 4.6) by copy module copy the initial incomplete point cloud feature vector dimension in the row direction Second, the splicing module The module will be copied The replication results are the same as those obtained in 4.4) The initial modified feature matrix of the dimension is spliced ​​in the column direction to obtain dimensional low-level concatenated feature matrix; 4.7) The low-level concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional low-level feature matrix; 4.8) dimensional low-level feature matrix input feature correction module ,get dimensional low-level corrected feature matrix; 4.9) By the copy module 4.5) The eigenvectors of the residual defect cloud at the low level are replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as those obtained in 4.8) The dimensional low-level corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional mid-layer concatenated feature matrix; 4.10) The concatenated feature matrix of the middle layer is input into the one-dimensional convolution module ,get dimensional mid-level feature matrix; 4.11) Dimensional mid-level feature matrix input feature correction module ,get dimensional mid-level corrected feature matrix; 4.12) By the copy module 4.5) The residual defect cloud feature vector of the dimensional middle layer is replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as those obtained in 4.11) The dimensional middle layer corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional high-level splicing feature matrix; 4.13) Dimensional high-level concatenated feature matrix input to the one-dimensional convolution module ,get dimensional high-level feature matrix; 4.14) Dimensional high-level feature matrix input shape reshaping module , get reconstruct the feature matrix; 4.15) input the reconstructed feature matrix into a one-dimensional convolution module , get a three-dimensional incremental coordinate of 16384 points; 4.16) copy the three-dimensional coordinate matrix of the 4.2) obtained dimensional sampling point cloud in the row direction times, and add the three-dimensional incremental coordinate of 16384 points to obtain a three-dimensional coordinate matrix of the 4.2) obtained dimensional sampling point cloud in the row direction times, and add the three-dimensional incremental coordinate of 16384 points to obtain a three-dimensional coordinate matrix of the 4.2) obtained dimensional sampling point cloud in the row direction

[0063] And in the above-mentioned very specific network structure setting, it can be understood that, compared with the existing completion network, the fourth layer of the three-dimensional point cloud completion network specially designed based on the feature layer-by-layer fusion correction of the fine-grained point cloud decoder, similar to the third layer of the feature layer-by-layer fusion correction of the medium-grained point cloud decoder, has the following characteristics: (1) In the process of generating fine-grained complete point cloud, the low-dimensional initial feature of the point to the high-dimensional high-level feature extraction adopts the way of layer-by-layer splicing and fusing the corresponding level of incomplete point cloud feature, which integrates the fusion of global feature throughout the feature extraction, which helps to strengthen the continuous guidance of global feature to local feature and correct the deviation generated in the feature upgrading process in time; (2) In the process of converting point feature from low-dimensional space to high-dimensional space, conversion error will be introduced, and the point distribution in the fine-grained complete point cloud is dense, and the correlation between point features is strong, therefore, in the process of extracting low-dimensional initial feature of the point to high-dimensional high-level feature, the way of layer-by-layer correction of feature is adopted, which integrates the use of point feature correlation throughout the feature extraction, which helps to eliminate conversion error in time and further improve the accuracy of point feature extraction.

[0064] In addition, from the above content, it can be seen that the medium-grained point cloud decoder based on feature layer-by-layer fusion correction and the fine-grained point cloud decoder based on feature layer-by-layer fusion correction both contain corresponding feature correction modules based on transformation fine-tuning to improve feature extraction accuracy.

[0065] For this purpose, referring to Figure 7 a structural schematic diagram of the feature correction module of the present application, the module structure of the feature correction module in the medium-grained point cloud decoder based on feature layer-by-layer fusion correction and the fine-grained point cloud decoder based on feature layer-by-layer fusion correction can specifically include a self-attention sub-module and a residual sub-module ; On the basis of the above composition module, the working process of the feature correction module can be: 5.1) input the original feature matrix of dimension D into the self-attention sub-module to transform and obtain a transformed feature of dimension D'; 5.2) input the transformed feature of dimension D' into the residual sub-module to fine-tune and obtain a corrected feature matrix of dimension D.

[0066] And under the above very specific network structure setting, it can be understood that the feature correction module adopts a transformation-fine-tuning idea to eliminate the conversion error of features from a low-dimensional space to a high-dimensional space.

[0067] On the one hand, the use of the self-attention mechanism can fully utilize the global correlation to transform the features and eliminate the conversion error as a whole. On the other hand, the use of the residual mechanism can further utilize the global correlation to fine-tune the features and further eliminate the residual error.

[0068] In this way, transformation and fine-tuning complement each other and together improve the accuracy of feature extraction.

[0069] Finally, for the specific configuration content of the three-dimensional point cloud completion network at each level, in general, it can be learned that based on in-depth analysis of the point feature correlation in the medium / fine-grained complete point cloud, the application designs a medium / fine-grained point cloud decoder by using feature layer-by-layer fusion correction, maximizes the correlation of point features at each level of medium / fine-grained, and realizes high-precision point cloud completion.

[0070] At the same time, based on in-depth analysis of the point feature correlation in the incomplete point cloud and the coarse-grained complete point cloud, the application carefully selects and improves the incomplete point cloud encoder and the coarse-grained point cloud decoder backbone network, and pre-excludes the influence of improper module selection on the accuracy of point feature extraction, thereby improving the accuracy of point cloud completion while ensuring the stability of point cloud completion performance.

[0071] In this way, compared with existing completion networks, on the one hand, the application avoids time-consuming neighborhood calculation throughout the process, and on the other hand, it avoids using time-consuming attention mechanisms in the incomplete point cloud encoding and coarse-grained point cloud decoding stages, so the training, inference time and memory consumption are significantly reduced, which can well meet the user's requirement for cost reduction and quality improvement.

[0072] The above is an introduction to the improved three-dimensional point cloud completion method provided by the application. In order to better implement the improved three-dimensional point cloud completion method provided by the application, the application also provides an improved three-dimensional point cloud completion device from the perspective of functional modules.​​​​

[0073] See Figure 8 , Figure 8 This is a schematic diagram of the structure of the improved three-dimensional point cloud completion device of the present application. In the present application, the improved three-dimensional point cloud completion device 800 may specifically include the following structure: An acquiring unit 801 is configured to acquire a defect cloud to be completed, wherein the defect cloud to be completed is specifically a three-dimensional point cloud; The completion unit 802 is configured to input the residual defect cloud to be completed into a pre-configured three-dimensional point cloud completion network, wherein the three-dimensional point cloud completion network sequentially includes a residual defect cloud encoder based on a global feature fusion point completion network, a coarse-grained point cloud decoder based on residual fine-tuning, a medium-grained point cloud decoder based on feature layer-by-layer fusion correction, and a fine-grained point cloud decoder based on feature layer-by-layer fusion correction, the residual defect cloud encoder extracts residual defect cloud features based on the point cloud input by the network, the coarse-grained point cloud decoder generates a coarse-grained complete point cloud based on the residual defect cloud features, the medium-grained point cloud decoder generates a medium-grained complete point cloud based on the point cloud input by the network, the residual defect cloud features, and the coarse-grained complete point cloud, and the fine-grained point cloud decoder generates a fine-grained complete point cloud based on the point cloud input by the network, the residual defect cloud features, and the medium-grained complete point cloud; The extraction unit 803 is used to extract the fine-grained complete point cloud output by the 3D point cloud completion network as the point cloud completion result.

[0074] In an exemplary embodiment, the residual cloud encoder includes a one-dimensional convolution module , maximum pooling module , Copy Module , splicing module , one-dimensional convolution module , maximum pooling module , splicing module and one-dimensional convolution module ; The working process of the residual cloud encoder is as follows: 1.1) The three-dimensional coordinate matrix of the residual defect cloud is input to the one-dimensional convolution module ,get dimensional low-level feature matrix, where the point cloud input by the network is recorded as dimensional residual defect cloud three-dimensional coordinate matrix, is the number of points in the residual cloud; 1.2) dimensional low-level feature matrix input to the maximum pooling module ,get dimensional low-level feature vector; 1.3) By the copy module Will copying the 3D low-level feature vector in the row direction , and then splicing the copying result of the copying module with the 3D low-level feature matrix in the column direction to obtain a 3D spliced feature matrix ; 2.3) inputting the 3D spliced feature matrix into a one-dimensional convolution module to obtain a 3D high-level feature matrix ; 2.4) inputting the 3D high-level feature matrix into a maximum pooling module to obtain a 3D high-level feature vector ; 2.5) splicing the 3D high-level feature vector with the 3D low-level feature vector in the column direction by a splicing module to obtain a 3D spliced feature vector ; 2.6) inputting the 3D spliced feature vector into a one-dimensional convolution module to obtain a 3D incomplete point cloud feature vector as the incomplete point cloud feature ; In another exemplary embodiment, the coarse-grained point cloud decoder comprises a transpose convolution module , a copying module , a splicing module , a one-dimensional convolution module , a shape remodeling module , a residual module and a one-dimensional convolution module

[0075] ; The working process of the coarse-grained point cloud decoder is as follows: 2.1) inputting the 3D incomplete point cloud feature vector into the transpose convolution module to obtain a 3D low-level transpose convolution feature matrix ; 2.2) copying the 3D incomplete point cloud feature vector 128 times in the row direction by the copying module , and then splicing the copying result of the copying module with the 3D low-level transpose convolution feature matrix in the column direction to obtain a 3D spliced feature matrix ; 2.3) inputting the 3D spliced feature matrix into the one-dimensional convolution module to obtain a 3D high-level feature matrix ; 2.4) inputting the 3D high-level feature matrix into the maximum pooling module to obtain a 3D high-level feature vector ; 2.5) splicing the 3D high-level feature vector with the 3D low-level feature vector in the column direction by the splicing module to obtain a 3D spliced feature vector ; 2.6) inputting the 3D spliced feature vector into the one-dimensional convolution module to obtain a 3D incomplete point cloud feature vector as the incomplete point cloud feature ; The splicing feature matrix input one-dimensional convolution module , get dimension fusion feature matrix; 2.4) by shape remodeling module to dimension fusion feature matrix shape remodeling, get dimension remodeling feature matrix; 2.5) input dimension remodeling feature matrix residual module , get dimension remodeling residual feature matrix; 2.6) input dimension remodeling residual feature matrix one-dimensional convolution module , get dimension coarse-grained point cloud three-dimensional coordinate matrix, as coarse-grained complete point cloud.

[0076] In another exemplary embodiment, the medium-grained point cloud decoder includes merging module , the farthest point sampling module , one-dimensional convolution module , feature correction module , one-dimensional convolution module , one-dimensional convolution module , one-dimensional convolution module , copy module , splicing module , one-dimensional convolution module , feature correction module , copy module , splicing module , one-dimensional convolution module , feature correction module , copy module , splicing module , one-dimensional convolution module , shape remodeling module , one-dimensional convolution module and copy module ; The working process of the medium-grained point cloud decoder is: 3.1) by merging module merge dimension incomplete point cloud three-dimensional coordinate matrix and dimension coarse-grained point cloud three-dimensional coordinate matrix, get dimension merging point cloud three-dimensional coordinate matrix; 3.2) input dimension merging point cloud three-dimensional coordinate matrix farthest point sampling module , get dimensional sampling point cloud three-dimensional coordinate matrix, where Select 512 or 1024; 3.3) The three-dimensional coordinate matrix of the sampling point cloud is input into the one-dimensional convolution module ,get dimensional initial feature matrix; 3.4) Dimensional initial feature matrix input feature correction module ,get dimensional initial modified feature matrix; 3.5) The residual defect cloud feature vector is input into the one-dimensional convolution module ,get dimensional mid-layer residual defect cloud feature vector, and then The residual cloud feature vector of the 1D middle layer is input into the 1D convolution module ,get dimensional low-level residual cloud feature vector, and then The low-dimensional residual cloud feature vector is input into the one-dimensional convolution module ,get dimensional initial residual defect cloud feature vector; 3.6) By the copy module Will The initial residual defect cloud eigenvector is replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as The initial modified feature matrix of the dimension is spliced ​​in the column direction to obtain dimensional low-level concatenated feature matrix; 3.7) The low-level concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional low-level feature matrix; 3.8) dimensional low-level feature matrix input feature correction module ,get dimensional low-level corrected feature matrix; 3.9) By the copy module Will The eigenvectors of the residual defect cloud at the low level are replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as The dimensional low-level corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional mid-layer concatenated feature matrix; 3.10) The concatenated feature matrix of the middle layer is input into the one-dimensional convolution module ,get dimensional mid-level feature matrix; 3.11) Dimensional mid-level feature matrix input feature correction module ,get dimensional mid-level corrected feature matrix; 3.12) By the copy module Will The residual defect cloud feature vector of the dimensional middle layer is replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as The dimensional middle layer corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional high-level splicing feature matrix; 3.13) Dimensional high-level concatenated feature matrix input to the one-dimensional convolution module ,get dimensional high-level feature matrix; 3.14) Dimensional high-level feature matrix input shape reshaping module ,get Dimensionally reshape the feature matrix; 3.15) Dimensionally reshape the feature matrix and input it into the one-dimensional convolution module , get the three-dimensional incremental coordinates of 2048 points; 3.16) By the copy module Will The 3D coordinate matrix of the sampling point cloud is replicated in the row direction times, and add them to the three-dimensional incremental coordinates of 2048 points to get The three-dimensional coordinate matrix of the medium-granularity point cloud is used as the medium-granularity complete point cloud.

[0077] In another exemplary embodiment, the fine-grained point cloud decoder includes a merging module , farthest point sampling module , one-dimensional convolution module , Feature Correction Module , one-dimensional convolution module , one-dimensional convolution module , one-dimensional convolution module , Copy Module , splicing module , one-dimensional convolution module , Feature Correction Module , Copy Module , splicing module , one-dimensional convolution module , Feature Correction Module , Copy Module , splicing module , one-dimensional convolution module , shape reshaping module , one-dimensional convolution module and the copy module ; The working process of the fine-grained point cloud decoder is: 4.1) By merging modules Will The three-dimensional coordinate matrix of the residual defect cloud and The three-dimensional coordinate matrix of the granular point cloud is merged to obtain Merge point cloud 3D coordinate matrix; 4.2) Merge point cloud 3D coordinate matrix input farthest point sampling module ,get Dimensional sampling point cloud three-dimensional coordinate matrix; 4.3) The three-dimensional coordinate matrix of the sampling point cloud is input into the one-dimensional convolution module ,get dimensional initial feature matrix; 4.4) Dimensional initial feature matrix input feature correction module ,get dimensional initial modified feature matrix; 4.5) The residual defect cloud feature vector is input into the one-dimensional convolution module ,get dimensional mid-layer residual cloud feature vector, and then The residual cloud feature vector of the 1D middle layer is input into the 1D convolution module ,get dimensional low-level residual cloud feature vector, and then The low-dimensional residual cloud feature vector is input into the one-dimensional convolution module ,get dimensional initial residual defect cloud feature vector; 4.6) By the copy module Will The initial residual defect cloud eigenvector is replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as The initial modified feature matrix of the dimension is spliced ​​in the column direction to obtain dimensional low-level concatenated feature matrix; 4.7) The low-level concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional low-level feature matrix; 4.8) dimensional low-level feature matrix input feature correction module ,get dimensional low-level corrected feature matrix; 4.9) By the copy module Will The eigenvectors of the residual defect cloud at the low level are replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as The dimensional low-level corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional mid-layer concatenation feature matrix; 4.10) The concatenated feature matrix of the middle layer is input into the one-dimensional convolution module ,get dimensional mid-level feature matrix; 4.11) Dimensional mid-level feature matrix input feature correction module ,get dimensional mid-level corrected feature matrix; 4.12) By the copy module Will The residual defect cloud feature vector of the dimensional middle layer is replicated in the row direction Second, the splicing module The module will be copied The replication results are the same as The dimensional middle layer corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional high-level splicing feature matrix; 4.13) Dimensional high-level concatenated feature matrix input to the one-dimensional convolution module ,get dimensional high-level feature matrix; 4.14) Dimensional high-level feature matrix input shape reshaping module ,get Dimensionally reshape the feature matrix; 4.15) Dimensionally reshape the feature matrix and input it into the one-dimensional convolution module , and obtain the three-dimensional incremental coordinates of 16384 points; 4.16) By the copy module copying the 3D coordinate matrix of the 2D point cloud in the row direction the 3D coordinate matrix of the 2D point cloud in the row direction the 3D coordinate matrix of the 2D point cloud in the row direction

[0078] In another exemplary embodiment, the module structure of the feature correction module in both the medium-grained point cloud decoder and the fine-grained point cloud decoder includes a self-attention sub-module and a residual sub-module ; The working process of the feature correction module is as follows: 5.1) input the 2D original feature matrix into the self-attention sub-module to perform transformation and obtain a 2D transformed feature; 5.2) input the 2D transformed feature into the residual sub-module to fine-tune and obtain a 2D corrected feature matrix. In another exemplary embodiment, the apparatus further includes a training unit 804 configured to: obtain the incomplete point cloud sample and its corresponding fine-grained complete point cloud ground truth; train the 3D point cloud completion network based on the incomplete point cloud sample and its corresponding fine-grained complete point cloud ground truth.

[0079] The present application also provides a processing device from the hardware structure, referring to , Fig. 1 shows a structural schematic diagram of the processing device of the present application. Specifically, the processing device of the present application can include a processor 901, a memory 902 and an input / output device 903. The processor 901 is configured to execute the computer program stored in the memory 902 to realize the steps of the improved 3D point cloud completion method in the corresponding embodiments; or the processor 901 is configured to execute the computer program stored in the memory 902 to realize the functions of the units in the corresponding embodiments. The memory 902 is configured to store the computer program required by the processor 901 to execute the improved 3D point cloud completion method in the corresponding embodiments.

[0080] The present application also provides a processing device from the hardware structure, referring to Figure 9 , Figure 9 Fig. 1 shows a structural schematic diagram of the processing device of the present application. Specifically, the processing device of the present application can include a processor 901, a memory 902 and an input / output device 903. The processor 901 is configured to execute the computer program stored in the memory 902 to realize the steps of the improved 3D point cloud completion method in the corresponding embodiments; or the processor 901 is configured to execute the computer program stored in the memory 902 to realize the functions of the units in the corresponding embodiments. The memory 902 is configured to store the computer program required by the processor 901 to execute the improved 3D point cloud completion method in the corresponding embodiments. Figure 1 Figure 8 Figure 1

[0081] ​​​​​For example, the computer program can be divided into one or more modules / units, one or more modules / units are stored in the memory 902 and executed by the processor 901 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device.

[0082] The processing device can include, but is not limited to, the processor 901, the memory 902, and the input / output device 903. Those skilled in the art can understand that the diagram is only an example of the processing device and does not constitute a limitation on the processing device, which can include more or fewer components than the diagram, or combine certain components, or different components, for example, the processing device can also include a network access device, a bus, etc., and the processor 901, the memory 902, and the input / output device 903 are connected through the bus.

[0083] The processor 901 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or other conventional processors, and the processor is the control center of the processing device, which connects various parts of the device through various interfaces and lines.

[0084] The memory 902 can be used to store computer programs and / or modules, and the processor 901 realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 902, and calling the data stored in the memory 902. The memory 902 mainly includes a storage program area and a storage data area, wherein the storage program area can store operating systems, application programs required to realize at least one function, etc.; the storage data area can store data created during use of the processing device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory card, a smart media card (SMC), a secure digital (SD) card, a flash card, a memory device with at least one disk, a flash memory device, or other volatile solid-state memory device.

[0085] The processor 901 is used to execute the computer program stored in the memory 902, and can specifically implement the following functions: Obtain a to-be-completed incomplete point cloud, wherein the to-be-completed incomplete point cloud is specifically a three-dimensional point cloud; Input the to-be-completed incomplete point cloud into a pre-configured three-dimensional point cloud completion network, wherein the three-dimensional point cloud completion network sequentially includes an incomplete point cloud encoder based on a global feature fusion point completion network, a coarse-grained point cloud decoder based on residual fine-tuning, a medium-grained point cloud decoder based on feature layer-by-layer fusion correction, and a fine-grained point cloud decoder based on feature layer-by-layer fusion correction, the incomplete point cloud encoder extracts incomplete point cloud features according to the input point cloud of the network, the coarse-grained point cloud decoder generates a coarse-grained complete point cloud according to the incomplete point cloud features, the medium-grained point cloud decoder generates a medium-grained complete point cloud according to the input point cloud of the network, the incomplete point cloud features, and the coarse-grained complete point cloud, and the fine-grained point cloud decoder generates a fine-grained complete point cloud according to the input point cloud of the network, the incomplete point cloud features, and the medium-grained complete point cloud; Extract the fine-grained complete point cloud output by the three-dimensional point cloud completion network as a point cloud completion result.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the improved three-dimensional point cloud completion device, the processing device and the corresponding units thereof described above can be referred to as Figure 1 The description of the improved three-dimensional point cloud completion method in the corresponding embodiments will not be repeated here.

[0087] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0088] Therefore, the present application provides a computer readable storage medium, which stores a plurality of instructions capable of being loaded by a processor to execute the present application as Figure 1 The steps of the improved three-dimensional point cloud completion method in the corresponding embodiments can be specifically implemented by referring to Figure 1 The description of the improved three-dimensional point cloud completion method in the corresponding embodiments will not be repeated here.

[0089] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0090] Due to the instructions stored in the computer readable storage medium, the present application as Figure 1The improved three-dimensional point cloud completion method in the corresponding embodiment corresponds to the steps, and thus the application can be implemented as Figure 1 The beneficial effects that can be achieved by the improved three-dimensional point cloud completion method in the corresponding embodiment are described in detail in the foregoing description, and will not be repeated here.

[0091] The improved three-dimensional point cloud completion method, device, processing equipment, and computer readable storage medium provided by the application are described in detail above, and the principles and implementation modes of the application are described in this paper. The above description of the embodiments is only used to help understand the core idea of the application; at the same time, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the application.

Claims

1. An improved 3D point cloud completion method, characterized in that: The method comprises: Obtaining a defect cloud to be completed, wherein the defect cloud to be completed is specifically a three-dimensional point cloud; Input the residual defect cloud to be completed into a preconfigured three-dimensional point cloud completion network, wherein the three-dimensional point cloud completion network sequentially includes a residual defect cloud encoder based on a global feature fusion point completion network, a coarse-grained point cloud decoder based on residual fine-tuning, a medium-grained point cloud decoder based on feature layer-by-layer fusion correction, and a fine-grained point cloud decoder based on the feature layer-by-layer fusion correction. The residual defect cloud encoder extracts residual defect cloud features according to the point cloud input by the network, the coarse-grained point cloud decoder generates a coarse-grained complete point cloud according to the residual defect cloud features, the medium-grained point cloud decoder generates a medium-grained complete point cloud according to the point cloud input by the network, the residual defect cloud features and the coarse-grained complete point cloud, and the fine-grained point cloud decoder generates a fine-grained complete point cloud according to the point cloud input by the network, the residual defect cloud features and the medium-grained complete point cloud; The fine-grained complete point cloud output by the three-dimensional point cloud completion network is extracted as a point cloud completion result.

2. The method according to claim 1, characterized in that The residual cloud encoder includes a one-dimensional convolution module , maximum pooling module , Copy Module , splicing module , one-dimensional convolution module , maximum pooling module , splicing module and one-dimensional convolution module ; The working process of the residual cloud encoder is as follows: Will The three-dimensional coordinate matrix of the residual defect cloud is input to the one-dimensional convolution module ,get dimensional low-level feature matrix, where the point cloud input by the network is recorded as dimensional residual defect cloud three-dimensional coordinate matrix, is the number of points in the residual cloud; The The low-dimensional feature matrix is ​​input into the maximum pooling module ,get dimensional low-level feature vector; By the copy module The The low-level feature vectors are replicated in the row direction Second, the splicing module The replication module The replication results are the same as those described The low-level feature matrix of the dimension is spliced ​​in the column direction to obtain dimensional splicing feature matrix; The The concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional high-level feature matrix; The The high-level feature matrix is ​​input into the maximum pooling module ,get dimensional high-level feature vector; By the splicing module The dimensional high-level feature vector, and The low-dimensional feature vectors are concatenated in the column direction to obtain dimensional concatenated feature vector; The The concatenated feature vector is input into the one-dimensional convolution module ,get The residual defect cloud feature vector is used as the residual defect cloud feature.

3. The method according to claim 2, characterized in that The coarse-grained point cloud decoder includes a transposed convolution module , Copy Module , splicing module , one-dimensional convolution module , shape reshaping module , residual module and one-dimensional convolution module ; The working process of the coarse-grained point cloud decoder is as follows: The dimensional residual defect cloud feature vector input transposed convolution module ,get dimensional low-level transposed convolution feature matrix; By the copy module The The residual defect cloud feature vector is replicated 128 times in the row direction and then the concatenation module The replication module The replication results are the same as those described The low-dimensional transposed convolution feature matrix is ​​spliced ​​in the column direction to obtain dimensional splicing feature matrix; The The concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional fusion feature matrix; The shape reshaping module Regarding the The dimensional fusion feature matrix is ​​reshaped to obtain Dimensionally reshape the feature matrix; The The dimensionally reshaped feature matrix is ​​input to the residual module ,get Dimensional reshape residual feature matrix; The The reshaped residual feature matrix is ​​input into the one-dimensional convolution module ,get The three-dimensional coordinate matrix of the coarse-grained point cloud is used as the coarse-grained complete point cloud.

4. The method according to claim 3, characterized in that The medium-granularity point cloud decoder includes a merging module , farthest point sampling module , one-dimensional convolution module , Feature Correction Module , one-dimensional convolution module , one-dimensional convolution module , one-dimensional convolution module , Copy Module , splicing module , one-dimensional convolution module , Feature Correction Module , Copy Module , splicing module , one-dimensional convolution module , Feature Correction Module , Copy Module , splicing module , one-dimensional convolution module , shape reshaping module , one-dimensional convolution module and the copy module ; The working process of the medium-granularity point cloud decoder is as follows: By the merge module The The three-dimensional coordinate matrix of the residual defect cloud is The three-dimensional coordinate matrix of the coarse-grained point cloud is merged to obtain Merge point cloud 3D coordinate matrix; The The three-dimensional coordinate matrix of the merged point cloud is input into the farthest point sampling module ,get dimensional sampling point cloud three-dimensional coordinate matrix, where Select 512 or 1024; The The three-dimensional coordinate matrix of the sampling point cloud is input into the one-dimensional convolution module ,get dimensional initial feature matrix; The The initial feature matrix is ​​input into the feature correction module ,get dimensional initial modified feature matrix; Will The residual defect cloud feature vector is input into the one-dimensional convolution module ,get dimensional mid-layer residual cloud feature vector, and then the The residual defect cloud feature vector of the dimensional middle layer is input into the one-dimensional convolution module ,get dimensional low-level residual cloud feature vector, and then the The low-dimensional residual cloud feature vector is input into the one-dimensional convolution module ,get dimensional initial residual defect cloud feature vector; By the copy module The The initial residual defect cloud eigenvector is replicated in the row direction Second, the splicing module The replication module The replication results are the same as described The initial modified feature matrix of the dimension is spliced ​​in the column direction to obtain dimensional low-level concatenated feature matrix; The The low-level concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional low-level feature matrix; The The low-level feature matrix is ​​input into the feature correction module ,get dimensional low-level corrected feature matrix; By the copy module The The eigenvectors of the residual defect cloud at the low level are replicated in the row direction Second, the splicing module The replication module The replication results are the same as described The dimensional low-level corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional mid-layer concatenated feature matrix; The The concatenated feature matrix of the middle layer is input into the one-dimensional convolution module ,get dimensional mid-level feature matrix; The The mid-dimensional feature matrix is ​​input into the feature correction module ,get dimensional mid-level corrected feature matrix; By the copy module The The residual defect cloud feature vector of the dimensional middle layer is replicated in the row direction Second, the splicing module The replication module The replication results are the same as described The dimensional middle layer corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional high-level splicing feature matrix; The The high-level concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional high-level feature matrix; The dimensional high-level feature matrix is ​​input into the reshape module ,get Dimensionally reshape the feature matrix; The The reshaped feature matrix is ​​fed into the one-dimensional convolution module. , get the three-dimensional incremental coordinates of 2048 points; By the copy module The The 3D coordinate matrix of the sampling point cloud is replicated in the row direction times, and add them to the three-dimensional incremental coordinates of the 2048 points to obtain The three-dimensional coordinate matrix of the medium-granularity point cloud is used as the medium-granularity complete point cloud.

5. The method according to claim 4, characterized in that The fine-grained point cloud decoder includes a merging module , farthest point sampling module , one-dimensional convolution module , Feature Correction Module , one-dimensional convolution module , one-dimensional convolution module , one-dimensional convolution module , Copy Module , splicing module , one-dimensional convolution module , Feature Correction Module , Copy Module , splicing module , one-dimensional convolution module , Feature Correction Module , Copy Module , splicing module , one-dimensional convolution module , shape reshaping module , one-dimensional convolution module and the copy module ; The working process of the fine-grained point cloud decoder is as follows: By the merge module The The three-dimensional coordinate matrix of the residual defect cloud is The three-dimensional coordinate matrix of the granular point cloud is merged to obtain Merge point cloud 3D coordinate matrix; The The three-dimensional coordinate matrix of the merged point cloud is input into the farthest point sampling module ,get Dimensional sampling point cloud three-dimensional coordinate matrix; The The three-dimensional coordinate matrix of the sampling point cloud is input into the one-dimensional convolution module ,get dimensional initial feature matrix; The The initial feature matrix is ​​input into the feature correction module ,get dimensional initial modified feature matrix; Will The residual defect cloud feature vector is input into the one-dimensional convolution module ,get dimensional mid-layer residual cloud feature vector, and then the The residual defect cloud feature vector of the dimensional middle layer is input into the one-dimensional convolution module ,get dimensional low-level residual cloud feature vector, and then the The low-dimensional residual cloud feature vector is input into the one-dimensional convolution module ,get dimensional initial residual defect cloud feature vector; By the copy module The The initial residual defect cloud eigenvector is replicated in the row direction Second, the splicing module The replication module The replication results are the same as described The initial modified feature matrix of the dimension is spliced ​​in the column direction to obtain dimensional low-level concatenated feature matrix; The The low-level concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional low-level feature matrix; The The low-level feature matrix is ​​input into the feature correction module ,get dimensional low-level corrected feature matrix; By the copy module The The eigenvectors of the residual defect cloud at the low level are replicated in the row direction Second, the splicing module The replication module The replication results are the same as described The dimensional low-level corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional mid-layer concatenated feature matrix; The The concatenated feature matrix of the middle layer is input into the one-dimensional convolution module ,get dimensional mid-level feature matrix; The The mid-dimensional feature matrix is ​​input into the feature correction module ,get dimensional mid-level corrected feature matrix; By the copy module The The residual defect cloud feature vector of the dimensional middle layer is replicated in the row direction Second, the splicing module The replication module The replication results are the same as described The dimensional middle layer corrected feature matrix is ​​spliced ​​in the column direction to obtain dimensional high-level splicing feature matrix; The The high-level concatenated feature matrix is ​​input into the one-dimensional convolution module ,get dimensional high-level feature matrix; The dimensional high-level feature matrix is ​​input into the reshape module ,get Dimensionally reshape the feature matrix; The The reshaped feature matrix is ​​fed into the one-dimensional convolution module. , and obtain the three-dimensional incremental coordinates of 16384 points; By the copy module The The 3D coordinate matrix of the sampling point cloud is replicated in the row direction times, and adding them to the three-dimensional incremental coordinates of the 16384 points, we get The three-dimensional coordinate matrix of the fine-grained point cloud is used as the fine-grained complete point cloud.

6. The method according to claim 5, characterized in that The module structure of the feature correction module in both the medium-grained point cloud decoder and the fine-grained point cloud decoder includes the self-attention submodule and residual submodule ; The working process of the feature correction module includes: Will The original feature matrix is ​​input into the self-attention submodule Transform and get Dimensional transformation features; The The dimensional transformed features are input into the residual submodule Fine-tune and obtain Maintenance positive feature matrix.

7. The method according to claim 1, characterized in that The method further comprises: Obtain residual point cloud samples and their corresponding fine-grained complete point cloud truth values; The three-dimensional point cloud completion network is trained based on the residual point cloud samples and their corresponding fine-grained complete point cloud ground truths.

8. An improved 3D point cloud completion device, characterized in that: The device comprises: an acquiring unit, configured to acquire a defect cloud to be completed, wherein the defect cloud to be completed is specifically a three-dimensional point cloud; A completion unit, configured to input the residual defect cloud to be completed into a preconfigured three-dimensional point cloud completion network, wherein the three-dimensional point cloud completion network sequentially includes a residual defect cloud encoder based on a global feature fusion point completion network, a coarse-grained point cloud decoder based on residual fine-tuning, a medium-grained point cloud decoder based on feature layer-by-layer fusion correction, and a fine-grained point cloud decoder based on the feature layer-by-layer fusion correction, the residual defect cloud encoder extracts residual defect cloud features according to the point cloud input by the network, the coarse-grained point cloud decoder generates a coarse-grained complete point cloud according to the residual defect cloud features, the medium-grained point cloud decoder generates a medium-grained complete point cloud according to the point cloud input by the network, the residual defect cloud features and the coarse-grained complete point cloud, and the fine-grained point cloud decoder generates a fine-grained complete point cloud according to the point cloud input by the network, the residual defect cloud features and the medium-grained complete point cloud; An extraction unit is used to extract the fine-grained complete point cloud output by the three-dimensional point cloud completion network as a point cloud completion result.

9. A processing device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the method according to any one of claims 1 to 7.

Images