Unsupervised multi-view 3D point cloud reconstruction method and device

Abstract

The present invention discloses an unsupervised multi-view three-dimensional point cloud reconstruction method and device. The method includes: constructing an interactive point cloud enhancement module for obtaining enhanced point clouds from multiple input rough point clouds using a point cloud enhancement subnet; extracting point cloud features from the multiple input enhanced point clouds using a point cloud feature extractor, cascading the multiple point cloud features and sending them into a multi-layer perceptron layer to reconstruct fine point clouds; utilizing multi-view consistency loss to minimize the pixel-by-pixel difference between the projected two-dimensional image and the corresponding pose input view to achieve unsupervised three-dimensional point cloud reconstruction; and optimizing the network by applying the constructed multi-view consistency loss in network training to constrain the consistency between the projected view and the input view, thereby constraining the reconstructed point cloud to have a reasonable geometric shape, and ultimately achieving unsupervised three-dimensional point cloud reconstruction. The device includes: a processor and a memory. The present invention effectively reduces the dependence on large-scale real three-dimensional point cloud data and improves the performance of unsupervised three-dimensional point cloud reconstruction.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and three-dimensional point cloud reconstruction, and in particular to an unsupervised multi-view three-dimensional point cloud reconstruction method and device. Background Art

[0002] With the application of 3D point clouds in fields such as autonomous driving, virtual reality, and robot navigation, research related to 3D point clouds has attracted widespread attention from scientific researchers. As one of the basic tasks in the field of 3D point cloud research, 3D point cloud reconstruction aims to infer the 3D geometric shape and structure of an object from a single or multiple 2D views. Existing 3D point cloud reconstruction methods mainly include: supervised 3D point cloud reconstruction and unsupervised 3D point cloud reconstruction. Supervised 3D point cloud reconstruction methods often require a large amount of real 3D point cloud data for training the reconstruction network. However, in practical applications, obtaining large-scale 3D point cloud data is costly and difficult, which greatly limits the scope of application and generalization ability of supervised methods. Unsupervised 3D point cloud reconstruction methods can effectively alleviate the above problems. They aim to reconstruct 3D point clouds without real 3D point clouds as constraints in the reconstruction process. They have become a hot research topic in the field of 3D point cloud reconstruction.

[0003] Depending on the number of input views, unsupervised 3D point cloud reconstruction methods can be divided into single-view-based unsupervised 3D point cloud reconstruction and multi-view-based unsupervised 3D point cloud reconstruction. Among them, single-view-based 3D point cloud reconstruction refers to the process of directly inferring the 3D point cloud of the object contained in it from a given single-view 2D view. For example, Navaneet et al. proposed a self-supervised single-view 3D point cloud reconstruction network, which uses cycle consistency loss to constrain the consistency between the projected image and the input image, as well as the consistency between the reconstructed point cloud and the point cloud reconstructed using the projected image. Han et al. proposed an unsupervised single-view 3D point cloud reconstruction network, which directly projects the generated 3D point cloud onto a 2D plane and matches it with discrete points randomly sampled from the true value mask.

[0004] However, a single view only describes a 3D object from a specific angle. Due to occlusions, a single view contains only limited visual information about the object and lacks depth information. This results in poor unsupervised 3D point cloud reconstruction based on a single view. Compared to a single view, multiple views not only provide more complete shape and structure information of the 3D object, but also more comprehensive spatial position and depth information, which facilitates more accurate reconstruction of the 3D point cloud in the absence of a real 3D point cloud as supervision. Therefore, constructing an unsupervised 3D point cloud reconstruction method based on multiple views has important practical significance and application value. Summary of the Invention

[0005] The present invention provides an unsupervised multi-view 3D point cloud reconstruction method and device. The present invention makes full use of the visual information and spatial information of multiple views to design an unsupervised 3D point cloud reconstruction network. The network enhances the information of the rough point cloud reconstructed from multiple views through an interactive point cloud enhancement module, and Figure 1 The network is trained using a consistency loss to reconstruct a detailed 3D point cloud from multiple views without real 3D point cloud supervision, effectively reducing the reliance on large-scale real 3D point cloud data and improving the performance of unsupervised 3D point cloud reconstruction. Detailed description is provided below:

[0006] An unsupervised multi-view three-dimensional point cloud reconstruction method, the method comprising:

[0007] Construct an interactive point cloud enhancement module to obtain enhanced point clouds using point cloud enhancement subnets for multiple input rough point clouds;

[0008] The point cloud feature extractor is used to extract point cloud features from multiple input enhanced point clouds, and the multiple point cloud features are cascaded and sent to a multi-layer perceptron layer to reconstruct the fine point cloud;

[0009] Using Multiview Figure 1 The consistency loss minimizes the pixel-by-pixel difference between the projected 2D image and the corresponding pose input view to achieve unsupervised 3D point cloud reconstruction;

[0010] By applying the constructed multi-view Figure 1 The network is optimized based on the consistency loss to constrain the consistency between the projection view and the input view, thereby constraining the reconstructed point cloud to have a reasonable geometric shape, and finally achieving unsupervised 3D point cloud reconstruction.

[0011] The point cloud enhancement subnet consists of a point cloud feature extractor, a point cloud feature interaction unit and a multi-layer perceptron.

[0012] Furthermore, the point cloud feature extractor adopts a three-dimensional point cloud feature extraction network based on deep learning, which takes a three-dimensional point cloud as input and extracts point cloud features containing spatial structure information of the point cloud;

[0013] The point cloud feature interaction unit consists of two cascade layers, V convolutional layers, and two activation layers. This unit takes point cloud features extracted from multiple rough point clouds as input, and through multi-branch information interaction, effectively utilizes the beneficial information in the point cloud features of other branches to enhance the point cloud features of the current branch;

[0014] The multi-layer perceptron is used to map the enhanced point cloud features to a three-dimensional space to reconstruct the enhanced point cloud.

[0015] The point cloud features extracted containing the spatial structure information of the point cloud are:

[0016] f i =Extractor(P i )

[0017] Among them, f i Represents the rough point cloud P i Extracted point cloud features, Extractor(·) represents the point cloud feature extractor;

[0018] The method of enhancing the point cloud features of the current branch by effectively utilizing the beneficial information in the point cloud features of other branches through multi-branch information interaction is as follows:

[0019] The j-th branch point cloud feature f j Send it to a convolutional layer for adaptive learning to obtain the adaptive point cloud feature f′ j , the i-th branch point cloud feature f i and f′ j Cascade, and send the cascaded features into a convolution layer and an activation layer to obtain the j-th branch point cloud feature f j The useful information in the point cloud feature f i enhancement;

[0020] The obtained j-th branch point cloud feature f j Useful information in j and the point cloud feature f of the i-th branch i Do dot product calculation to get enhanced features

[0021] Using the point cloud features of other branches The above enhancement method can be used to obtain the point cloud features f of the i-th branch respectively. i Enhanced features The enhanced features corresponding to each branch are cascaded and sent to a convolution layer and an activation layer for fusion, and then the fused features are sent to a multi-layer perceptron to obtain the rough point cloud P i Point cloud E after interactive enhancement i .

[0022] Furthermore, the two-dimensional image after the minimization projection is:

[0023] view′ i =Φ(F,θ i )

[0024] Where Φ represents the projection operation, view′ represents the true pose θ of the refined point cloud F according to the i-th input view i The two-dimensional image obtained by projection.

[0025] Wherein, the multi-view Figure 1 The causal loss is:

[0026]

[0027] Among them, L MVC Indicates multi-view Figure 1 Consistency loss, view i represents the i-th view of the input.

[0028] An unsupervised multi-view three-dimensional point cloud reconstruction device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the method steps described.

[0029] The beneficial effects of the technical solution provided by the present invention are:

[0030] 1. This paper designs an interactive point cloud enhancement module, which uses multi-view information to enhance the rough point cloud to obtain a 3D point cloud with richer information.

[0031] 2. The present invention designs a multi-view Figure 1 Consistency loss, which effectively constrains the reconstruction process of the 3D point cloud without the real 3D point cloud as supervision, thereby reconstructing a more refined 3D point cloud;

[0032] 3. Through the above methods, this method realizes the reconstruction of fine 3D point clouds from multiple views in an unsupervised manner, effectively reducing the dependence on large-scale real 3D point cloud data and improving the performance of unsupervised 3D point cloud reconstruction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flowchart of an unsupervised multi-view 3D point cloud reconstruction method. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0035] 1. Constructing a rough point cloud reconstruction module

[0036] First, a rough point cloud reconstruction module is constructed, which consists of multiple point cloud reconstruction subnets. Each point cloud reconstruction subnet uses an unsupervised single-view 3D point cloud reconstruction network SSPNet. The SSPNet network takes an image of a 3D object taken from a certain perspective as input, and reconstructs a rough 3D point cloud in an unsupervised manner by learning the visual information of each input image. For multiple input views, the constructed point cloud reconstruction subnet is used to reconstruct the rough point cloud P of the object contained in the corresponding view. i , where i = 1, 2, ..., V, where V represents the number of input views. The reconstruction of the coarse point cloud corresponding to each view can be regarded as an unsupervised single-view 3D point cloud reconstruction task, so the point cloud reconstruction subnet can be implemented by any unsupervised single-view 3D point cloud reconstruction model.

[0037] Among them, SSPNet is well known to those skilled in the art and will not be described in detail in the embodiments of the present invention.

[0038] 2. Building an interactive point cloud enhancement module

[0039] Then, an interaction-based point cloud enhancement module is constructed, which consists of multiple point cloud enhancement subnetworks. Use the point cloud enhancement subnet to obtain enhanced point clouds Specifically, each point cloud enhancement subnet consists of a point cloud feature extractor, a point cloud feature interaction unit, and a multi-layer perceptron (MLP). Specifically, the point cloud feature extractor uses a deep learning-based 3D point cloud feature extraction network (DGCNN). This DGCNN network takes a 3D point cloud as input and extracts point cloud features that contain information about the spatial structure of the point cloud. The point cloud feature interaction unit consists of two cascade layers, V convolutional layers, and two activation layers. This unit takes point cloud features extracted from multiple rough point clouds as input and, through multi-branch information interaction, effectively utilizes the beneficial information in the point cloud features of other branches to enhance the point cloud features of the current branch. Multi-layer perceptrons (MLPs) are used to map the enhanced point cloud features to 3D space to reconstruct the enhanced point cloud.

[0040] Among them, DGCNN and multi-layer perceptrons MLPs are well known to those skilled in the art and will not be described in detail in the embodiments of the present invention.

[0041] Taking the rough point cloud enhancement process of the i-th branch as an example, first the input rough point cloud P i Use the point cloud feature extractor to extract point cloud features. The feature extraction process formula is as follows:

[0042] f i =Extractor(P i )

[0043] Among them, f i Represents the rough point cloud P i Extracted point cloud features, Extractor(·) represents the point cloud feature extractor.

[0044] Then, for the extracted point cloud features f i , using the point cloud feature interaction unit to effectively utilize the beneficial information in other branch point cloud features to enhance the point cloud feature f i Specifically, using the j-th branch point cloud feature f j Enhance the point cloud feature f of the i-th branch i As an example, where j=1,2,...,V and j≠i. First, the j-th branch point cloud feature f j Send it to a convolutional layer for adaptive learning to obtain the adaptive point cloud feature f′ j Then, the i-th branch point cloud feature f i and f′ j Cascade, and send the cascaded features into a convolution layer and an activation layer to obtain the j-th branch point cloud feature f j The useful information in the point cloud feature f i The enhancement formula is as follows:

[0045] r j =relu(Conv[f i ,f′ j ])

[0046] Among them, r j Represents the point cloud feature f of the j-th branch j Relu(·) represents the relu activation function, Conv(·) represents the convolution layer with a convolution kernel size of 3×3, and [,] represents the cascade operation.

[0047] Afterwards, in order to realize the j-th branch point cloud feature f j For point cloud features f i The information enhancement is to obtain the j-th branch point cloud feature f j Useful information in j and the point cloud feature f of the i-th branch i Do dot product calculation to get enhanced features

[0048]

[0049] in, Represents dot product calculation.

[0050] Using the point cloud features of other branches The above enhancement method can be used to obtain the point cloud features f of the i-th branch respectively. i Enhanced features Then, the enhanced features corresponding to each branch are cascaded and sent to a convolution layer and an activation layer for fusion, and then the fused features are sent to a multi-layer perceptron MLPs to obtain the rough point cloud P i Point cloud E after interactive enhancement i , the formula is as follows:

[0051]

[0052] Where MLPs(·) represents a multi-layer perceptron layer, Indicates the use of the V-th branch point cloud feature f V For the i-th branch point cloud feature f i Features obtained after information enhancement.

[0053] Furthermore, for multiple rough point clouds input The interactive point cloud enhancement module adopts the above interactive enhancement method and can eventually output multiple enhanced point clouds corresponding to the rough point cloud.

[0054] 3. Building a Point Cloud Fusion Module

[0055] Multiple enhanced point clouds for input Point cloud features are extracted using a point cloud feature extractor. These features are then concatenated and fed into a multi-layer perceptron (MLP) layer to reconstruct the refined point cloud F. Specifically, the point cloud feature extractor uses a deep learning-based 3D point cloud feature extraction network (DGCNN). The reconstruction process for the refined point cloud F is expressed as follows:

[0056] F=MLPs([Extractor(E1),…,Extractor(E i ),…,Extractor(E V )])

[0057] Among them, E V is the enhanced point cloud obtained after information enhancement on the rough point cloud of the V-th branch.

[0058] 4. Build Multi-View Figure 1 Causative loss

[0059] In order to reconstruct a relatively accurate 3D point cloud without real 3D point cloud data as supervision, the embodiment of the present invention uses the real poses of multiple views as input to project the reconstructed fine point cloud F onto the corresponding 2D plane. The projection process is expressed as follows:

[0060] view′ i =Φ(F,θ i )

[0061] Where Φ represents the projection operation, view′ represents the true pose θ of the refined point cloud F according to the i-th input view i The two-dimensional image obtained by projection.

[0062] Then, using multi-view Figure 1 The consistency loss minimizes the pixel-by-pixel difference between the projected 2D image and the corresponding pose input view to achieve unsupervised 3D point cloud reconstruction. Figure 1 The formula for consistency loss is as follows:

[0063]

[0064] Among them, L MVC Indicates multi-view Figure 1 Consistency loss, view i represents the i-th view of the input.

[0065] By applying the constructed multi-view Figure 1 The network is optimized based on the consistency loss to constrain the consistency between the projection view and the input view, thereby constraining the reconstructed point cloud to have a reasonable geometric shape, and finally achieving unsupervised 3D point cloud reconstruction.

[0066] An unsupervised multi-view 3D point cloud reconstruction device includes: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to perform the following method steps:

[0067] Construct an interactive point cloud enhancement module to obtain enhanced point clouds using point cloud enhancement subnets for multiple input rough point clouds;

[0068] The point cloud feature extractor is used to extract point cloud features from multiple input enhanced point clouds, and the multiple point cloud features are cascaded and sent to a multi-layer perceptron layer to reconstruct the fine point cloud;

[0069] Using Multiview Figure 1 The consistency loss minimizes the pixel-by-pixel difference between the projected 2D image and the corresponding pose input view to achieve unsupervised 3D point cloud reconstruction;

[0070] By applying the constructed multi-view Figure 1 The network is optimized based on the consistency loss to constrain the consistency between the projection view and the input view, thereby constraining the reconstructed point cloud to have a reasonable geometric shape, and finally achieving unsupervised 3D point cloud reconstruction.

[0071] Among them, the point cloud enhancement subnet consists of a point cloud feature extractor, a point cloud feature interaction unit and a multi-layer perceptron.

[0072] Furthermore, the point cloud feature extractor adopts a three-dimensional point cloud feature extraction network based on deep learning, which takes a three-dimensional point cloud as input and extracts point cloud features containing spatial structure information of the point cloud;

[0073] The point cloud feature interaction unit consists of two cascade layers, V convolutional layers, and two activation layers. This unit takes point cloud features extracted from multiple rough point clouds as input and effectively utilizes the beneficial information in the point cloud features of other branches to enhance the point cloud features of the current branch through multi-branch information interaction.

[0074] A multi-layer perceptron is used to map the enhanced point cloud features into three-dimensional space to reconstruct the enhanced point cloud.

[0075] Unless otherwise specified, the embodiments of the present invention do not limit the models of the components. Any component that can perform the above functions may be used.

[0076] Those skilled in the art will understand that the accompanying drawings are only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An unsupervised multi-view 3D point cloud reconstruction method, characterized in that: The method comprises: Construct an interactive point cloud enhancement module to obtain enhanced point clouds using point cloud enhancement subnets for multiple input rough point clouds; The point cloud feature extractor is used to extract point cloud features from multiple input enhanced point clouds, and the multiple point cloud features are cascaded and sent to a multi-layer perceptron layer to reconstruct the fine point cloud; Multi-view consistency loss is used to minimize the pixel-by-pixel difference between the projected 2D image and the corresponding pose input view to achieve unsupervised 3D point cloud reconstruction. By applying the constructed multi-view consistency loss to network optimization during network training, the consistency between the projected view and the input view is constrained, so that the reconstructed point cloud can have a reasonable geometric shape, and finally unsupervised 3D point cloud reconstruction is achieved; The point cloud enhancement subnet consists of a point cloud feature extractor, a point cloud feature interaction unit and a multi-layer perceptron; The point cloud feature interaction unit consists of two cascade layers, V convolutional layers, and two activation layers. This unit takes point cloud features extracted from multiple rough point clouds as input, and through multi-branch information interaction, effectively utilizes the beneficial information in the point cloud features of other branches to enhance the point cloud features of the current branch; The method of enhancing the point cloud features of the current branch by effectively utilizing the beneficial information in the point cloud features of other branches through multi-branch information interaction is as follows: The j-th branch point cloud feature f j Send it into a convolutional layer for adaptive learning to obtain the adaptive point cloud feature f j ′, the i-th branch point cloud feature f i With f j ′ cascade, and send the cascaded features into a convolution layer and an activation layer to obtain the j-th branch point cloud feature f j The useful information in the point cloud feature f i enhancement; The obtained j-th branch point cloud feature f j Useful information in j and the point cloud feature f of the i-th branch i Do dot product calculation to get enhanced features Using the point cloud features of other branches The above enhancement method can be used to obtain the point cloud features f of the i-th branch respectively. i Enhanced features The enhanced features corresponding to each branch are cascaded and sent to a convolution layer and an activation layer for fusion, and then the fused features are sent to a multi-layer perceptron to obtain the rough point cloud P i Point cloud E after interactive enhancement i .

2. The unsupervised multi-view 3D point cloud reconstruction method according to claim 1, characterized in that: The point cloud feature extractor adopts a three-dimensional point cloud feature extraction network based on deep learning, which takes a three-dimensional point cloud as input and extracts point cloud features containing point cloud spatial structure information; The multi-layer perceptron is used to map the enhanced point cloud features to a three-dimensional space to reconstruct the enhanced point cloud.

3. The unsupervised multi-view 3D point cloud reconstruction method according to claim 2, characterized in that: The point cloud features extracted containing point cloud spatial structure information are: f i =Extractor(P i ) Among them, f i Represents the rough point cloud P i Extracted point cloud features, Extractor(·) represents the point cloud feature extractor.

4. The unsupervised multi-view 3D point cloud reconstruction method according to claim 1, characterized in that: The two-dimensional image after the minimization projection is: view i ′=Φ(F,θ i ) Where Φ represents the projection operation, view′ represents the true pose θ of the refined point cloud F according to the i-th input view i The two-dimensional image obtained by projection.

5. The unsupervised multi-view 3D point cloud reconstruction method according to claim 4, characterized in that: The multi-view consistency loss is: Among them, L MVC represents the multi-view consistency loss, view i represents the i-th view of the input.

6. An unsupervised multi-view 3D point cloud reconstruction device, characterized in that: The device includes: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps according to any one of claims 1 to 5.

Images