A point cloud completion method, system, device and medium based on self-supervised learning

Abstract

The present application belongs to the technical field of computer vision, in order to solve the problem of point cloud data scarcity at present, a point cloud completion method, system, device and medium based on self-supervised learning are proposed, including: different patch regions obtained by farthest point sampling on point cloud original data, the region perception distance between the point cloud completed by the point cloud completion network model and the different patch regions is taken as the loss function, and the point cloud completion network model is self-supervised trained; the point cloud data to be completed is input into the trained point cloud completion network model, the global feature is obtained by performing different resolution upsampling operations on the point cloud data to be completed; by performing parallel decoding prediction operation on the global feature, and the prediction result is combined to obtain complete point cloud data, the point cloud completion can be carried out without knowing the real point cloud, and the limitation of supervised point cloud completion is avoided.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a point cloud completion method, system, device and medium based on self-supervised learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the rapid development of today's technology, three-dimensional point cloud data plays an increasingly important role in the fields of computer vision, robotics, autonomous driving, etc. However, it has to be admitted that due to the limitations of the performance of the acquisition device itself and the interference of complex and variable environmental factors, and for many reasons, the point cloud data obtained in actual application scenarios inevitably has incomplete conditions. Such incomplete point cloud data not only affects subsequent data analysis and processing, but also may cause serious consequences in some key applications, such as misjudgment in autonomous driving, inaccurate operation of robots, etc. Therefore, how to effectively solve the problem of incompleteness of point cloud data has become an important topic in current related field research.

[0004] Point cloud completion technology aims to restore the missing part of point cloud data to obtain a complete three-dimensional model. Traditional point cloud completion methods usually rely on supervised learning, although supervised learning has achieved certain results in the field of point cloud completion, but it relies on real point cloud data for training. This dependence requires a large amount of labeled data, which not only consumes a lot of manpower and time, but also may face the problem of data scarcity in practical applications, thereby limiting the wide application of supervised learning methods in point cloud completion. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the present application provides a point cloud completion method, system, device and medium based on self-supervised learning, which can complete point cloud without knowing the real point cloud, avoiding the limitations of supervised point cloud completion, and having higher universality.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a point cloud completion method based on self-supervised learning, comprising:

[0008] obtaining point cloud original data;

[0009] obtaining different patch regions of the point cloud original data through farthest point sampling, taking the point cloud data completed through the point cloud completion network model and the region perception distance of the different patch regions as a loss function, and performing self-supervised training on the point cloud completion network model;

[0010] The point cloud data to be completed is input into the trained point cloud completion network model, a global feature is obtained by performing down-sampling operations of different resolutions on the point cloud data to be completed through an encoder module, and a complete point cloud data of the point cloud data to be completed is obtained by performing parallel decoding prediction operations on the global feature through a decoder module and combining the prediction results.

[0011] In a second aspect, the present application provides a point cloud completion system based on self-supervised learning, comprising:

[0012] The training point cloud acquisition module is configured to acquire point cloud original data.

[0013] The training module is configured to perform self-supervised training on the point cloud completion network model by taking the region perception distance of the point cloud data completed through the point cloud completion network model and different patch regions obtained by farthest point sampling on the point cloud original data as a loss function.

[0014] The point cloud completion module is configured to input the point cloud data to be completed into the trained point cloud completion network model, obtain a global feature by performing down-sampling operations of different resolutions on the point cloud data to be completed through an encoder module, and obtain a complete point cloud data of the point cloud data to be completed by performing parallel decoding prediction operations on the global feature through a decoder module and combining the prediction results.

[0015] In a third aspect, the present application provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein when the computer instructions are run by the processor, the method of the first aspect is completed.

[0016] In a fourth aspect, the present application provides a computer readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method of the first aspect is completed.

[0017] The above one or more technical solutions have the following beneficial effects:

[0018] The point cloud completion network model proposed in the present application outputs complete point clouds through self-supervised generation learning, and compared with the supervised learning of existing completion methods, the present application can complete point clouds without knowing the real point clouds, avoids the limitations of supervised point cloud completion, has higher universality, and has similar completion effect to supervised point cloud completion.

[0019] The point cloud completion network model of the present application can capture point cloud features from multiple angles through the encoder module, improve the accuracy of feature extraction, and make the generated point clouds more perfect and uniform through the parallel decoding architecture of the decoder module.

[0020] Advantages of additional aspects of the application will be in part apparent to those of ordinary skill in the art, and others will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which constitute a part of this specification, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification. The embodiments of the application, together with its description, serve to explain the application.

[0022] Figure 1 A flow chart of a point cloud completion method based on self-supervised learning in an embodiment of the application;

[0023] Figure 2 A workflow diagram of an encoder in an embodiment of the application;

[0024] Figure 3 A network structure diagram of a geometric detail perception enhancement module in an embodiment of the application;

[0025] Figure 4 A parallel decoding flow chart of a decoder module in an embodiment of the application;

[0026] Figure 5 A comparison chart of the effect of an embodiment of the application and a prior art point cloud completion method. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed description is merely exemplary and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0028] It should be noted that the terms used herein are merely intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the application.

[0029] In the case of no conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

[0030] The point cloud data of the application can be aircraft point cloud data. The point cloud data is completed by the acquired aircraft point cloud data, and then the aircraft shape structure is analyzed according to the completed point cloud data. Of course, it can also be point cloud data in other fields, such as acquired cabinets, chairs, cars, tables, etc., which are not limited here.

[0031] Embodiment I

[0032] The following will be described by taking the completion of the acquired aircraft point cloud data as an example. The embodiment discloses a point cloud completion method based on self-supervised learning, comprising:

[0033] obtain point cloud raw data;

[0034] The point cloud raw data is sampled by the farthest point sampling to obtain different patch regions, and the point cloud data completed by the point cloud completion network model and the region perception distance of the different patch regions are taken as a loss function to perform self-supervised training on the point cloud completion network model.

[0035] The to-be-completed point cloud data is input into the trained point cloud completion network model, the to-be-completed point cloud data is subjected to down-sampling operation of different resolutions by the encoder module to obtain global features, the global features are subjected to parallel decoding and prediction operation by the decoder module, and the prediction results are combined to obtain complete point cloud data of the to-be-completed point cloud data.

[0036] In order to more clearly illustrate the scheme, the point cloud completion network model is first described in detail. The point cloud completion network model adopts a transformer structure, including an encoder module and a decoder module. In order to realize self-supervised learning of the point cloud completion network model, a grouping module is designed.

[0037] In this embodiment, the point cloud completion network model is trained by using a point cloud raw data set. The point cloud raw data set can come from a public 3D-EPN data set.

[0038] The incomplete point cloud in the point cloud raw data set is taken as the point cloud raw data, and the specific implementation process includes the following steps:

[0039] The encoder module is composed of four sequentially connected geometric detail perception enhancement modules. The incomplete point cloud is input into the encoder module, and the global features are obtained by feature extraction through the cascaded geometric detail perception enhancement modules in the encoder module .

[0040] Suppose the dimension of the incomplete point cloud is , the output of the first layer of the geometric detail perception enhancement module is , the output of the second layer of the geometric detail perception enhancement module is , the output of the third layer of the geometric detail perception enhancement module is , and the output of the fourth layer of the geometric detail perception enhancement module is Then, the maximum pooling operation is performed on the point cloud features output by the last layer of the geometric detail perception enhancement module to obtain the global features .

[0041] For example, the dimension of the incomplete point cloud is ​, after the first layer of geometric detail perception enhancement module, the feature dimension increases, the point cloud point number reduces, and becomes , the output dimensions of the subsequent three layers of geometric detail perception enhancement modules are as follows: , , , the maximum pooling operation is performed on the output of the last layer of geometric detail perception enhancement module, and the global feature with dimension is obtained .

[0042] Next, taking the operation of the first layer of geometric detail perception enhancement module on the input point cloud, which is the incomplete point cloud , as an example, the operation process of the geometric detail perception enhancement module on the input point cloud is explained. The geometric detail perception enhancement module includes a feature extraction module and a feature enhancement module. The feature extraction module is used to learn features from the input point cloud, which is the incomplete point cloud , using a multi-head cross-attention residual form, and process the learned features using a feedforward network to obtain extracted features. The feature enhancement module is used to enhance the extracted features using a multi-head self-attention residual form, and process the enhanced features using a feedforward network to obtain the output of the geometric detail perception enhancement module.

[0043] Specifically, the farthest point sampling is performed on the input point cloud, which is the incomplete point cloud , and a multi-head cross-attention residual form is used to learn features :

[0044]

[0045] wherein is the incomplete point cloud, is the multi-head attention layer, is the normalization layer, and the incomplete point cloud has points, and after farthest point sampling, it obtains points, denoted as .

[0046] The advantage of using farthest point sampling in point cloud completion is that it can achieve uniform sampling, preserve geometric features, reduce redundancy, enhance robustness, and optimize computing resources. By preferentially selecting points far from the sampled points, farthest point sampling ensures that the point cloud is uniformly distributed in space, captures key geometric features, reduces redundant points, improves computational efficiency, and enhances robustness to noise and outliers, thereby improving completion effect and algorithm stability.

[0047] Through residual connection, the geometric features of the original incomplete point cloud are preserved, and then a feedforward network is used for further updating. The output F of the final feature extraction module is:

[0048]

[0049] wherein, FFN denotes a feed-forward network layer, Concat denotes a connection.

[0050] The feature enhancement module applies multi-head self-attention in the form of residual to the output features of the feature extraction module to perform feature enhancement:

[0051]

[0052] Then further update using again a feed-forward network, the output of the final geometry detail-aware enhancement module is:

[0053]

[0054] wherein, FFN denotes a feed-forward network layer.

[0055] The decoder module of the embodiment adopts a parallel multi-stage decoding strategy, in order to avoid ignoring the local details of the global features in the process of generating point clouds, the decoder module first performs a global feature down-sampling through two linear fully connected layers, and the first local feature , the second local feature and the global feature are generated through the decoding operation of the corresponding branch, the first group of point cloud data, the second group of point cloud data and the third group of point cloud data are generated, and the first group of point cloud data, the second group of point cloud data and the third group of point cloud data are fused to obtain complete predicted point cloud data.

[0056] Specifically, the first group of point cloud data, the second group of point cloud data and the third group of point cloud data correspond to the first branch, the second branch and the third branch in the decoder module respectively, the first branch, the second branch and the third branch are all composed of linear layers and convolutional layers connected in turn, and the first group of point cloud data, the second group of point cloud data and the third group of point cloud data are generated through the first branch, the second branch and the third branch respectively, and the corresponding point cloud points are , and , three groups of point clouds are fused in parallel to generate the final predicted point cloud , once the decoder module generates the predicted point cloud , the predicted point cloud is resampled to obtain the occlusion-like region corresponding to the patch region .

[0057] The local features can also be well expressed by the partial decoding of the decoder module in this embodiment.

[0058] The decoder module, for example, first decodes the global feature with a dimension of to obtain a first local feature with a dimension of and a second local feature with a dimension of . The local features of the point cloud are better expressed after the downsampling. The first local feature , the second local feature and the global feature are respectively input into three branches of the decoder module to respectively generate predicted point clouds with dimensions of , and . The outputs of the three branches of the decoder are linearly superimposed to obtain a complete point cloud with a dimension of .

[0059] In this embodiment, in order to realize the self-supervised learning of the point cloud completion network model, a grouping module is designed. Specifically, in the grouping module, the input incomplete point cloud is subjected to a farthest point sampling operation to obtain points. The points are taken as the center points of each patch, that is, . Based on the Euclidean distance, the nearest neighbor points of each center point are found and collected in the incomplete point cloud, so that the incomplete point cloud is divided into the following set of patch regions:

[0060]

[0061] wherein represents a patch region with the center and containing points, and represents a point in the patch region.

[0062] These patches are randomly divided into three regions with a patch number ratio of , that is, , , .

[0063] For example, the number of center points obtained by farthest point sampling is 64, each region is composed of 32 nearest neighbor points of the center point, and the grouping module divides the input incomplete point cloud into three groups, and the patch number ratio of each group is 40:20:4.

[0064] The three patch regions obtained above , , , the predicted point cloud of the point cloud completion network model is calculated self-supervised learning loss function of point cloud completion.

[0065] The embodiment uses region-aware distance as loss function, set and are two sets of point clouds, which are divided into several patch regions and , then the region-aware distance is defined as follows:

[0066]

[0067] Wherein, , , indicates the first neighborhood of point in point cloud , indicates the first neighborhood of point in point cloud , , indicates a point in point cloud, , indicates a patch region set belonging to , respectively, indicates the total number of patch regions in or . Region-aware distance is used to calculate the distance between the regions divided by two point clouds, R represents the region.

[0068] Chamfer distance is defined as follows:

[0069]

[0070] Wherein, and are two sets of point clouds, is the point of point cloud set , is the point of point cloud set .

[0071] The loss function designed in this embodiment is defined as:

[0072]

[0073] Wherein, , and are loss weights, is an input incomplete point cloud, , , are three groups of region point clouds obtained by proportioning the incomplete point cloud, and the predicted point cloud is resampled to obtain a class occlusion region corresponding to the patch region .

[0074] Exemplary, , , The initial learning rate is 0.001, the weight decay is 0.001, and a total of 270 rounds of training are performed.

[0075] The embodiment adopts end-to-end iterative training, minimizes the loss function in the training process, calculates the loss function value and performs back propagation in each iteration, until the network converges, and the completed complete point cloud is obtained.

[0076] Wherein, the training process of the whole network is as shown in Figure 1 The software parameters of the training platform used in the embodiment can be Pytorch1.11.0 and CUDA11.3, and the operating system can be ubuntu20.0.4, of course, in other embodiments, different software parameters and operating systems can be selected according to the situation. After the network training is completed, the input incomplete point cloud is completed to generate the complete point cloud according to the test process of Figure 1 .

[0077] Table 1 shows the quantitative comparison results of the current popular point cloud completion methods PCN and P2C and the present application on the test set of 3D-EPN dataset, and the chamfer distance is used as the evaluation index of the test results, and the smaller the chamfer distance, the closer the completed point cloud is to the real point cloud. From the table, it can be seen that the present method exceeds the P2C network which is also a self-supervised learning, and the completion effect can be comparable to the supervised network.

[0078] Table 1: Quantitative comparison results of the present method and other methods on 3D-EPN dataset

[0079]

[0080] As shown in Figure 5As shown, the first row is the input incomplete point cloud, the second row is the completed point cloud of the method PCN, the third row is the completed point cloud of the method P2C, the fourth row is the completed point cloud of the method proposed in the application, and the fifth row is the real complete point cloud in the data set. As can be seen from the figure, for the airplane in the first column, the wings and tail of the airplane completed by the application are closer to the wings and tail of the airplane of the reference point cloud, and the uniformity is better, and the airplane completed by the method P2C has a lot of noise. For the seat in the second column, the backrest and leg of the seat completed by the application are in good shape, and the point cloud distribution is more uniform, and the seats completed by other methods have many noise points.

[0081] The experimental results show that the application can realize point cloud completion without knowing the real point cloud, generate a predicted point cloud with complete and accurate structure, and the effect can be comparable to supervised learning, and has very promising application prospects in the fields of robots, autonomous driving and the like.

[0082] Embodiment Two

[0083] The purpose of this embodiment is to provide a point cloud completion system based on self-supervised learning, comprising:

[0084] The training point cloud acquisition module is configured to acquire point cloud original data.

[0085] The training module is configured to take the region perception distance of the point cloud data completed by the point cloud completion network model and the different patch regions obtained by farthest point sampling of the point cloud original data as a loss function, and perform self-supervised training on the point cloud completion network model.

[0086] The point cloud completion module is configured to input the point cloud data to be completed into the trained point cloud completion network model, perform down-sampling operations of different resolutions on the point cloud data to be completed by the encoder module to obtain global features, perform parallel decoding prediction operations on the global features by the decoder module, and obtain complete point cloud data of the point cloud data to be completed by jointing the prediction results.

[0087] In more embodiments, there are also provided:

[0088] An electronic device comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the method described in Embodiment One is completed. For brevity, this will not be repeated here.

[0089] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0090] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0091] A computer readable storage medium for storing computer instructions, the computer instructions being executed by a processor to complete the method described in embodiment one.

[0092] The method in embodiment one can be directly embodied as a hardware processor to complete, or be completed by a combination of hardware and software modules in the processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory to complete the steps of the above method in combination with the hardware. To avoid repetition, it will not be described in detail here.

[0093] A computer program product comprising a computer program, the computer program being executed by a processor to implement the method described in embodiment one.

[0094] The present application also provides at least one computer program product tangibly stored on a non-transitory computer readable storage medium. The computer program product includes computer executable instructions, such as instructions included in program modules, which are executed in devices on real or virtual processors of the target to perform processes / methods as described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In various embodiments, the functions of the program modules can be combined or divided as desired. Machine executable instructions for program modules can be executed within a local or distributed device. In a distributed device, program modules can be located in local and remote storage media.

[0095] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages. The computer program code can execute entirely on a computer, a special purpose computer, or other programmable apparatus to produce the functions / acts specified in the flow diagrams and / or block diagrams. The program code can execute entirely on a computer, a special purpose computer, or other programmable apparatus, as a stand-alone software package, partly on the computer and partly on a remote computer, or entirely on the remote computer or server.

[0096] In the context of the present application, the computer program code or related data can be carried by any suitable carrier to enable the device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, and the like. Examples of signals can include electrical, optical, radio, sound or other forms of propagated signals, such as carrier waves, infrared signals, and the like.

[0097] Those skilled in the art can understand that the units and algorithm steps of the examples described in conjunction with the embodiments can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0098] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without inventive labor are still within the scope of protection of the present application.

Claims

1. A point cloud completion method based on self-supervised learning, characterized in that, The method comprises the following steps: obtaining point cloud raw data; performing self-supervised training on a point cloud completion network model by taking the point cloud data completed by the point cloud completion network model and the region-aware distance of different patch regions obtained by farthest point sampling on the point cloud raw data as a loss function, wherein the loss function is specifically: wherein, , and are loss weights, is a chamfer distance, , and are patch regions, is a missing point cloud, denotes a region-aware distance, and patch regions corresponding class occluded regions ; The to-be-completed point cloud data is input into the trained point cloud completion network model, and a global feature is obtained by performing down-sampling operations of different resolutions on the to-be-completed point cloud data through an encoder module; specifically, the encoder module is composed of four sequentially connected geometric detail perception enhancement modules, and the incomplete point cloud data is input into the four sequentially connected geometric detail perception enhancement modules to obtain the global feature The input is input into the encoder module, and the global feature is obtained by performing feature extraction through the geometric detail perception enhancement modules cascaded in the encoder module ​ Set the missing point cloud The dimension of the point cloud is , the output of the first layer of geometric detail perception enhancement module , the output of the second layer of geometric detail perception enhancement module , the output of the third layer of geometric detail perception enhancement module , the output of the fourth layer of geometric detail perception enhancement module Then, the maximum pooling operation is performed on the point cloud features output by the last layer of geometric detail perception enhancement module to obtain global features ; The geometry detail perception enhancement module comprises a feature extraction module and a feature enhancement module. The feature extraction module learns features of the input point cloud by using a multi-head cross-attention in a residual form, and processes the learned features by using a feedforward network to obtain extracted features. The feature extraction module enhances the extracted features by using a multi-head self-attention in a residual form, and processes the enhanced features by using a feedforward network to obtain the output of the geometry detail perception enhancement module. The operation process of the geometry detail perception enhancement module in the first layer of the encoder module on the point cloud raw data is specifically: ； ； ； The feature enhancement module applies multi-head self-attention in the form of a residual to the output features of the feature extraction module performing feature enhancement: ； wherein, is a point cloud, is a multi-headed attention layer, is a normalization layer, is a point cloud, is a point obtained after farthest point sampling, FFN denotes a feed-forward network layer, and Concat denotes concatenation. performing parallel decoding prediction operations on the global features by using a decoder module, and combining the prediction results to obtain complete point cloud data of the point cloud data to be completed, wherein the operation process is specifically: performing up-sampling operations on the global features in different resolutions to obtain first local features and second local features; generating first group point cloud data, second group point cloud data and third group point cloud data by performing decoding operations on the global features, the first local features and the second local features through corresponding branches, respectively; the first group point cloud data, the second group point cloud data and the third group point cloud data correspond to the first branch, the second branch and the third branch in the decoder module, respectively; combining the first group point cloud data, the second group point cloud data and the third group point cloud data to obtain the complete point cloud data of the point cloud data to be completed.

2. The point cloud completion method based on self-supervised learning of claim 1, wherein, Each branch performs linear operations and convolution operations on the corresponding global features, first local features and second local features to obtain completed point cloud output by each branch.

3. The point cloud completion method based on self-supervised learning of claim 1, wherein, The different patch regions obtained by farthest point sampling on the point cloud raw data are specifically: obtaining a plurality of points of the point cloud raw data by farthest point sampling, and taking the plurality of points as center points of each patch; obtaining the nearest neighbor points of each center point based on the Euclidean distance, and dividing the point cloud raw data into different patch regions.

4. The point cloud completion system based on self-supervised learning of any one of claims 1-3, wherein, The method comprises the following steps: a training point cloud obtaining module configured to obtain point cloud raw data; a training module configured to perform self-supervised training on a point cloud completion network model by taking the point cloud data completed by the point cloud completion network model and the region-aware distance of different patch regions obtained by farthest point sampling on the point cloud raw data as a loss function; a point cloud completion module configured to input point cloud data to be completed into the trained point cloud completion network model, perform down-sampling operations on the point cloud data to be completed in different resolutions by using an encoder module to obtain global features; perform parallel decoding prediction operations on the global features by using a decoder module, and combine the prediction results to obtain complete point cloud data of the point cloud data to be completed.

5. An electronic device, comprising: A computer program product comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, which, when run by the processor, perform the method of any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, A computer program product for storing computer instructions, which, when executed by a processor, perform the method of any one of claims 1-3.

Images