Sparse view angle three-dimensional reconstruction method based on point cloud generation and deformation prediction

Abstract

The invention relates to the field of three-dimensional reconstruction, in particular to a sparse visual angle three-dimensional reconstruction method based on point cloud generation and deformation prediction, computer equipment and a storage medium. Comprising the steps of obtaining training data including object true value point cloud, object sparse view angle images and camera internal and external parameters; the method comprises the following steps: generating a rough point cloud of an object by using a point cloud generation architecture based on adaptive centroid constraint, constraining a point cloud generation and fusion process by using centroid consistency of the point cloud under different perspectives, and decoupling spatial position information and shape features of the point cloud from the perspective of a loss function to obtain a more accurate intermediate representation; a refined network based on Transform is used for deforming rough point clouds, shallow image features and local geometric features of the point clouds are extracted to enhance the guiding effect of point-by-point features, self-attention layers stacked in series are used for completing global feature coding, and cross attention query is carried out on deformation position proposals to regress deformation residual errors. Based on the neural network model, the three-dimensional point cloud is automatically reconstructed according to the sparse view angle image of the object without depending on manpower. Experiments prove that the method can well consider the integrity of the overall shape of the three-dimensional reconstruction result and the accuracy of the local detail topology, thereby obtaining better reconstruction quality.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction, and in particular to a sparse viewpoint three-dimensional reconstruction method based on point cloud generation and deformation prediction. Background Art

[0002] Three-dimensional reconstruction, that is, using specific structures to build a three-dimensional model of the target object, is one of the important research directions in the field of computer vision. Three-dimensional reconstruction can enhance the understanding, expression and interaction capabilities of visual models, and is an important bridge for computer vision technology to move towards virtual reality. For applications in real scenes, three-dimensional reconstruction has very important value and significance in the fields of virtual reality, autonomous driving, film and television special effects, medical treatment, culture and education.

[0003] Existing 3D methods are mainly classified according to the number of viewpoints. A single image can provide limited information about the object's space, and reconstructing the object's 3D structure based on this information will lead to a series of problems such as blur, noise, and rough synthesis of occluded parts.

[0004] Reconstructing with multi-view images is an effective way to solve the above problems, because multi-view images can provide additional spatial information of the target object. Traditional SFM (Structure From Motion) and SLAM (Simultaneous Localization and Mapping) methods, as well as some deep learning methods based on multi-view stereo (MVS) can establish feature correspondences between multiple views to obtain more accurate reconstruction results. These methods usually require areas with significant overlap between viewpoints and a fixed image input order to obtain feature matching relationships for reconstruction. However, in practical applications, the need for a higher degree of overlap between viewpoints may lead to a significant increase in data collection costs and operational complexity, thereby limiting the generalization of these methods. On the other hand, these methods focus on depth estimation and it is difficult to reconstruct the complete three-dimensional shape of an object with only sparse view inputs (e.g., 2-5).

[0005] In order to cope with the limitations of these multi-view reconstruction methods and achieve sparse view 3D reconstruction, some works have integrated the idea of ​​"shape generation" to directly encode the image features under sparse view and use it to serve the reconstruction process, replacing the feature matching process between views. Such methods generally focus on reconstructing the surface mesh or voxel of the object, which can restore the overall shape of the object relatively well. However, due to the limitations of model principles or calculation and storage, they are difficult to guarantee the accuracy of local detail topology, and there is still room for improvement in reconstruction quality: the computational and storage overhead brought by voxel reconstruction will increase sharply with the increase of resolution, and it is difficult to achieve a balance between the whole and the details; surface mesh reconstruction relies more on the idea of ​​mesh deformation, and the form of mesh initialization determines that this type of method is not robust enough for complex topology. Compared with these two 3D expressions, the lightweight point cloud representation can achieve reasoning efficiency that voxel reconstruction cannot achieve and is not limited by resolution. The lack of connectivity between points also enables point clouds to adaptively adjust local detail topology and reconstruct 3D structures as accurately as possible. Summary of the invention

[0006] The main purpose of the present invention is to provide a sparse perspective 3D reconstruction method, computer equipment and storage medium based on point cloud generation and deformation prediction, aiming to overcome the problem that the current sparse perspective 3D reconstruction method cannot take into account both the overall shape integrity and local detail topological accuracy of the reconstruction result.

[0007] To achieve the above object, the present invention is implemented through the following technical solutions:

[0008] Acquire training samples, each training sample comprising a true point cloud corresponding to an object, images of the object at three sparse viewing angles, and intrinsic and extrinsic parameters of a camera at each viewing angle, wherein the images of the object at the three viewing angles correspond one-to-one to the intrinsic and extrinsic parameters of the camera;

[0009] By using a point cloud generation architecture based on adaptive centroid constraints, image features of three perspective images of the object are encoded to generate a rough point cloud corresponding to the object.

[0010] The rough point cloud of the object is feature encoded point by point by using a point cloud refinement position deformation network based on the Transformer architecture, and the rough point cloud of the object is deformed point by point according to the point by point feature encoding to obtain a result point cloud corresponding to the object.

[0011] Furthermore, the point cloud generation architecture based on adaptive centroid constraint includes the following steps:

[0012] Based on the images of the object under the three perspectives, regressing the single-view point clouds corresponding to the three perspective images of the object respectively using a single-view point cloud generation module based on an encoder-decoder architecture;

[0013] Based on the single-view point clouds corresponding to each view of the object and the internal and external parameters of the camera at each view, the single-view point clouds corresponding to each view of the object are fused using a multi-view point cloud fusion module based on adaptive centroid constraints to obtain a rough point cloud corresponding to the object.

[0014] Furthermore, the single-view point cloud generation module based on the encoder-decoder architecture includes the following steps:

[0015] Based on the images of the object under three viewing angles, a convolutional neural network is used to extract image features;

[0016] Based on the image features, a multi-layer perceptron is used to directly regress the single-view point clouds corresponding to each view of the object;

[0017] Output the single-view point cloud corresponding to each view of the object and the shallow image features in the image features of each view.

[0018] Furthermore, the multi-view point cloud fusion module based on adaptive centroid constraint includes the following steps:

[0019] Based on the camera extrinsics at the various viewing angles, the single-view point clouds corresponding to the various viewing angles of the object are transformed into a standard space;

[0020] Based on the single-view point clouds corresponding to each view of the object, the point clouds under each view are fused using the centroid consistency constraint between views: the centroid of the single-view point clouds corresponding to each view of the object is moved to the origin, and the same operation is performed on the true point cloud corresponding to the object; during the training process, the point cloud distance loss is used to constrain the shape consistency between the single-view point clouds corresponding to each view of the object and the true point cloud corresponding to the object, and the centroid position consistency is explicitly supervised; during the reasoning process, the centroid mean of the single-view point clouds corresponding to each view of the object is used to unify their position prediction results to achieve the constraint operation of adaptive centroid consistency;

[0021] Based on the fusion results of the single-view point clouds corresponding to each view of the object, a rough point cloud corresponding to the object is output.

[0022] Furthermore, the point cloud refined position deformation network based on the Transformer architecture includes the following steps:

[0023] Based on the shallow image features in the image features of each viewing angle, a shallow image feature extraction module is used to bind the shallow image features under each viewing angle to each point of the rough point cloud corresponding to the object;

[0024] Based on the rough point cloud corresponding to the object, using a local geometric feature extraction module to extract geometric features of local point cloud distribution for each point of the rough point cloud corresponding to the object;

[0025] splicing the point-by-point shallow image features and the point-by-point local geometric features to obtain point-by-point features;

[0026] Based on the point-by-point features of the rough point cloud, a point cloud global feature encoder is used to perform adaptive information exchange between points to obtain a point-by-point global feature encoding result;

[0027] Based on the rough point cloud corresponding to the object, using a deformation position proposal module to generate a possible deformation position for each point of the rough point cloud;

[0028] Based on the point-by-point global feature encoding result and the deformation position proposal, regressing the point-by-point deformation residual using a deformation position prediction module based on a cross-attention mechanism;

[0029] The point-by-point deformation residual is added to the rough point cloud corresponding to the object to obtain a result point cloud corresponding to the object.

[0030] Furthermore, the shallow image feature extraction module includes the following steps:

[0031] Transforming the rough point cloud corresponding to the object to each camera perspective based on the camera extrinsic parameters at each perspective;

[0032] Projecting the rough point cloud onto a camera imaging plane based on the camera intrinsic parameters at each viewing angle;

[0033] Binding shallow image features in each perspective image feature to each point of the rough point cloud by using a bilinear interpolation algorithm;

[0034] Based on the multi-view image features bound to each point of the rough point cloud, three statistics (maximum value, mean value, standard deviation) are used to fuse the features between viewpoints to obtain point-by-point shallow image features.

[0035] Furthermore, the local geometric feature extraction module comprises the following steps:

[0036] Based on the rough point cloud corresponding to the object, the KNN algorithm is used to find K nearest neighbors for each point, and the difference between the K nearest neighbors of each point is used as the relative position code of the point;

[0037] splicing the point-by-point relative position code with the coordinates of the rough point cloud to obtain a point-by-point local context;

[0038] Based on the point-by-point local context, a multi-layer perceptron and a maximum pooling operation are used to obtain point-by-point local geometric features.

[0039] Furthermore, the point cloud global feature encoder includes three self-attention encoder sublayers stacked in series, and the point-by-point features of the rough point cloud pass through the self-attention encoder sublayers in turn to obtain point-by-point features at four different interaction levels; the point-by-point features at the four different interaction levels are spliced ​​together in the feature dimension, and a multilayer perceptron is used to obtain the point-by-point features after interaction.

[0040] The self-attention encoder sublayer is characterized by including a multi-head self-attention module and a feature feedforward network. The multi-head self-attention module captures the relationship between different input elements, and the feature feedforward network converts element features through a multi-layer perceptron.

[0041] Furthermore, the deformation position proposal module provides points with fixed distances in six directions of up, down, left, right, front and back as deformation position proposals for the rough point cloud corresponding to the object, that is, expands the rough point cloud by 7 times.

[0042] Furthermore, the deformation position prediction module based on the cross attention mechanism includes the following steps:

[0043] Based on the deformation position proposals of the rough point cloud, extracting features of each deformation position using the shallow image feature extraction module;

[0044] Based on the deformed position feature, using a cross attention layer to perform a cross query from the point-by-point features of the rough point cloud to obtain an updated deformed position feature;

[0045] Based on the updated deformed position features, a local attention module including multiple self-attention layers is used to perform a self-attention operation on each group of 7 deformed position proposals to obtain the deformed position features after intra-group interaction;

[0046] Based on the deformation position features after the intra-group interaction, a multi-layer perceptron and SoftMax operation are used to decode and obtain the possibility weight of each deformation position;

[0047] Based on the deformed position proposals of the rough point cloud and the likelihood weights of the deformed positions, a weighted sum operation is used to obtain the deformation residual of each point in the rough point cloud.

[0048] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the steps of any of the above methods.

[0049] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic diagram of the steps of a sparse view 3D reconstruction method according to an embodiment of the present invention;

[0051] Figure 2 1 is a schematic diagram of the overall process of a sparse view 3D reconstruction method according to an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of a single-view point cloud generation module in one embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of a point cloud refinement position deformation network in one embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of an input object main view image, an object true value point cloud and a three-dimensional reconstruction result in one embodiment of the present invention.

[0055] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] Reference Figure 1 In one embodiment of the present invention, a sparse view 3D reconstruction method based on point cloud generation and deformation prediction is provided, wherein a 3D point cloud corresponding to an object is generated from a sparse view image by training a point cloud reconstruction network, wherein the sparse view point cloud reconstruction network comprises the following steps:

[0058] Step S1, obtaining training samples, each training sample comprising a true point cloud corresponding to an object, images of the object at three sparse viewing angles, and intrinsic and extrinsic parameters of a camera at each viewing angle, wherein the images of the object at three viewing angles correspond one to one to the intrinsic and extrinsic parameters of the camera;

[0059] Step S2, using a point cloud generation architecture based on adaptive centroid constraints, encoding image features of the three perspective images of the object to generate a rough point cloud corresponding to the object.

[0060] Step S3, using a point cloud refinement position deformation network based on a Transformer architecture, performs point-by-point feature encoding on the rough point cloud of the object, performs point-by-point deformation prediction on the rough point cloud of the object according to the point-by-point feature encoding, and obtains a result point cloud corresponding to the object.

[0061] Reference Figure 2 , the left and right parts respectively show the point cloud generation architecture based on adaptive centroid constraint described in step S2 and the point cloud refined position deformation network based on Transformer architecture described in step S3.

[0062] As described in step S1 above, the true point cloud corresponding to the object is given in the form of a coordinate file, the external parameters of the three viewing angle cameras are given by the transformation matrix with respect to the standard space, and the internal parameters are given by the projection matrix. The above files can be recognized by a computer.

[0063] As described in step S2 above, the point cloud generation architecture based on adaptive centroid constraint is obtained by deep neural network training, and is used to reconstruct a rough point cloud corresponding to the object from the sparse view image of the object. On the basis of single-view point cloud reconstruction, the generation results of different view angles are fused, and the generation and fusion process is adaptively constrained through the consistency of the point cloud centroid to improve the reconstruction quality.

[0064] As described in step S3 above, the Transformer-based point cloud refinement position deformation network is trained by a deep neural network and is used to optimize the point-by-point deformation of the rough point cloud. Shallow image features and local geometric features of the point cloud are extracted to enhance the guiding role of point-by-point features, and global feature encoding is completed using a series of stacked self-attention layers. Cross-attention queries are performed on the deformed position proposals to regress the deformation residuals to improve the accuracy of the point cloud reconstruction results.

[0065] In the above process, a deep neural network is used to perform the above-mentioned sparse perspective 3D reconstruction work without relying on manual labor, making the 3D reconstruction process more lightweight and improving modeling efficiency.

[0066] In one embodiment, step S2 of reconstructing a rough point cloud from a sparse view image of the object using a point cloud generation architecture based on an adaptive centroid constraint includes:

[0067] Step S21, based on the images of the object under three viewing angles, using a single-view point cloud generation module based on an encoder-decoder architecture to regress the single-view point clouds corresponding to the three viewing angle images of the object respectively;

[0068] Step S22, based on the single-view point clouds corresponding to each view of the object and the internal and external parameters of the camera at each view, the single-view point clouds corresponding to each view of the object are fused using a multi-view point cloud fusion module based on adaptive centroid constraints to obtain a rough point cloud corresponding to the object.

[0069] Reference Figure 3 In one embodiment, step S21 of regressing an object point cloud from a single image using a single-view point cloud generation module based on an encoder-decoder architecture includes:

[0070] Step S211, extracting image features using a convolutional neural network ResNet based on the images of the object at three viewing angles;

[0071] Step S212: Based on the image features, a multi-layer perceptron is used to directly regress the single-view point cloud corresponding to each view of the object. The point cloud corresponding to each view is recorded as , M is the number of viewing angles;

[0072] Step S213: outputting the single-view point clouds corresponding to each view of the object and the shallow image features in the image features of each view.

[0073] In one embodiment, step S22 of fusing the single-view point clouds corresponding to each view of the object using a multi-view point cloud fusion module based on adaptive centroid constraint includes:

[0074] Step S221, based on the camera extrinsics at each viewing angle, that is, a three-dimensional transformation function from each viewing angle to the standard space is given , then the point cloud generated under each viewing angle Can be Convert to standard space, where yes Representation in standard space.

[0075] Step S222, based on the single-view point clouds corresponding to each view of the object, the point clouds under each view are fused using the centroid consistency constraint between views. Before fusion, the centroid of the single-view point clouds corresponding to each view of the object needs to be moved to the origin, and the same operation is performed on the true value point clouds corresponding to the object. When fusion occurs, the training and reasoning processes require different processing methods:

[0076] Step S222a, during the training process, using point cloud distance loss to constrain the shape consistency between the single-view point cloud corresponding to each view of the object and the true point cloud corresponding to the object, and explicitly supervising the consistency of the center of mass position;

[0077] Step S222b, during the reasoning process, the centroid mean of the single-view point clouds corresponding to each view of the object is used to unify their position prediction results to achieve an adaptive centroid consistency constraint operation.

[0078] Step S223: Based on the fusion results of the single-view point clouds corresponding to each view of the object, a rough point cloud corresponding to the object is output, which is recorded as , where N is the number of points.

[0079] Reference Figure 4 In one embodiment, step S3 of performing point-by-point deformation optimization on the rough point cloud of the object using a point cloud refinement position deformation network based on a Transformer architecture includes:

[0080] Step S31, based on the shallow image features in the image features of each viewing angle, use a shallow image feature extraction module to bind the shallow image features under each viewing angle to each point of the rough point cloud corresponding to the object ;

[0081] Step S32: based on the rough point cloud corresponding to the object, using the local geometric feature extraction module to extract the geometric features of the local point cloud distribution for each point of the rough point cloud corresponding to the object ;

[0082] Step S33, concatenating the point-by-point shallow image features and the point-by-point local geometric features to obtain point-by-point features ,Right now ;

[0083] Step S34, based on the point-by-point features of the rough point cloud , using the point cloud global feature encoder to perform point-by-point adaptive information exchange and obtain point-by-point global feature encoding results ;

[0084] Step S35, based on the rough point cloud corresponding to the object , using the deformation position proposal module to generate possible deformation positions for each point in the rough point cloud ;

[0085] Step S36, based on the point-by-point global feature encoding result And the deformation position proposal , using the deformation position prediction module based on the cross-attention mechanism to regress the point-by-point deformation residual ;

[0086] Step S37: transform the point-by-point deformation residual The rough point cloud corresponding to the object is added , get the result point cloud corresponding to the object ,Right now .

[0087] In one embodiment, step S31 of extracting point-by-point image features using a shallow image feature extraction module includes:

[0088] Step S311, based on the camera external parameters at each viewing angle, that is, based on the three-dimensional transformation function from each viewing angle to the standard space The inverse function of The rough point cloud corresponding to the object is transformed to each camera perspective, where is the rough point cloud of the object Representation in each camera coordinate system;

[0089] Step S312, projecting the rough point cloud onto a camera imaging plane based on the camera intrinsic parameters at each viewing angle;

[0090] Step S313, using a bilinear interpolation algorithm to bind shallow image features in each perspective image feature in step S213 to each point of the rough point cloud;

[0091] Step S314, based on the multi-view image features bound to each point of the rough point cloud, three statistics (maximum value, mean value, standard deviation) are used to perform feature fusion between views to obtain point-by-point shallow image features. .

[0092] In one embodiment, step S32 of extracting point-by-point geometric features using a local geometric feature extraction module includes:

[0093] Step S321: rough point cloud of the object Each point in Search k neighbors using KNN algorithm , thus obtaining the domain matrix , represents the neighborhood information of each point in the coordinate space, through Perform relative position encoding, where Indicates that One dimension of is replicated k times;

[0094] Step S322: concatenate the point-by-point relative position code with the coordinates of the rough point cloud to obtain a point-by-point local context. ;

[0095] Step S323, based on the point-by-point local context, using a multi-layer perceptron and maximum pooling operation, obtain point-by-point local geometric features ,Right now .

[0096] In one embodiment, step S34 of obtaining a point-by-point global feature encoding result by using a point cloud global feature encoder, wherein the point cloud global feature encoder comprises three self-attention encoder sublayers stacked in series, and the point-by-point feature of the rough point cloud is Passing through the self-attention encoder sublayers in turn, we get point-by-point features at four different interaction levels. ; The point-by-point features at the four different interaction levels are spliced ​​together in the feature dimension, and the point-by-point features after interaction are obtained using a multi-layer perceptron ,Right now The self-attention encoder sublayer includes a multi-head self-attention module and a feature feed-forward network. The multi-head self-attention module captures the relationship between different input elements, and the feature feed-forward network transforms element features through a multi-layer perceptron.

[0097] In one embodiment, a step S35 of generating a possible deformed position for each point of the rough point cloud using a deformed position proposal module, wherein the deformed position proposal module is a rough point cloud corresponding to the object. Provides points with fixed distances in six directions: up, down, left, right, front, and back as deformation position proposals , that is, expanding the rough point cloud to 7 times.

[0098] In one embodiment, step S36 of regressing point-by-point deformation residuals using a deformation position prediction module based on a cross-attention mechanism includes:

[0099] Step S361: Deformed position proposal based on the rough point cloud , using the shallow image feature extraction module in step S31 to extract the features of each deformation position ;

[0100] Step S362: based on the deformation position feature , use the cross attention layer to cross-query the point-by-point features of the rough point cloud to obtain the updated deformed position features ;

[0101] Step S363: based on the updated deformation position feature , a local attention module containing multiple self-attention layers is used to perform self-attention operations on each group of 7 deformation position proposals to obtain the deformation position features after intra-group interaction ;

[0102] Step S364: based on the deformation position features after the intra-group interaction , use multi-layer perceptron and SoftMax operation decoding to get the probability weight of each deformation position ;

[0103] Step S365: Deformed position proposal based on the rough point cloud and the probability weight of the deformation position , the deformation residual of each point in the rough point cloud is obtained by weighted summation operation .

[0104] In summary, the sparse perspective 3D reconstruction method based on point cloud generation and deformation prediction provided in the embodiment of the present invention generates a rough point cloud of the object based on a given sparse perspective image of the object using a point cloud generation architecture based on adaptive centroid constraints, and deforms the rough point cloud using a Transformer-based refinement network to obtain a result point cloud. The present invention is based on a neural network model and automatically performs sparse perspective 3D reconstruction without relying on manual labor.

[0105] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0106] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0107] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A sparse view 3D reconstruction method based on point cloud generation and deformation prediction, characterized in that: Generate a 3D point cloud corresponding to an object from a sparse viewpoint image by training a point cloud reconstruction network, wherein the sparse viewpoint point cloud reconstruction network comprises the following steps: Acquire training samples, each training sample comprising a true point cloud corresponding to an object, images of the object at three sparse viewing angles, and intrinsic and extrinsic parameters of a camera at each viewing angle, wherein the images of the object at the three viewing angles correspond one-to-one to the intrinsic and extrinsic parameters of the camera; Using a point cloud generation architecture based on adaptive centroid constraints, encoding image features of three perspective images of the object, and generating a rough point cloud corresponding to the object; The rough point cloud of the object is feature encoded point by point by using a point cloud refinement position deformation network based on the Transformer architecture, and the rough point cloud of the object is deformed point by point according to the point by point feature encoding to obtain a result point cloud corresponding to the object.

2. The method according to claim 1, characterized in that The point cloud generation architecture based on adaptive centroid constraint includes the following steps: Based on the images of the object under the three perspectives, regressing the single-view point clouds corresponding to the three perspective images of the object respectively using a single-view point cloud generation module based on an encoder-decoder architecture; Based on the single-view point clouds corresponding to each view of the object and the internal and external parameters of the camera at each view, the single-view point clouds corresponding to each view of the object are fused using a multi-view point cloud fusion module based on adaptive centroid constraints to obtain a rough point cloud corresponding to the object.

3. The method according to claim 2, characterized in that The single-view point cloud generation module based on the encoder-decoder architecture includes the following steps: Based on the images of the object under three viewing angles, extracting image features using a convolutional neural network; Based on the image features, a multi-layer perceptron is used to directly regress the single-view point clouds corresponding to each view of the object; Output the single-view point cloud corresponding to each view of the object and the shallow image features in the image features of each view.

4. The method according to claim 2 and claim 3, characterized in that The multi-view point cloud fusion module based on adaptive centroid constraint includes the following steps: Based on the camera extrinsics at the various viewing angles, the single-view point clouds corresponding to the various viewing angles of the object are transformed into a standard space; Based on the single-view point clouds corresponding to each view of the object, the point clouds under each view are fused using the centroid consistency constraint between views: the centroid of the single-view point clouds corresponding to each view of the object is moved to the origin, and the same operation is performed on the true point cloud corresponding to the object; during the training process, the point cloud distance loss is used to constrain the shape consistency between the single-view point clouds corresponding to each view of the object and the true point cloud corresponding to the object, and the centroid position consistency is explicitly supervised; During the reasoning process, the centroid mean of the single-view point clouds corresponding to each view of the object is used to unify their position prediction results to achieve an adaptive centroid consistency constraint operation; Based on the fusion results of the single-view point clouds corresponding to each view of the object, a rough point cloud corresponding to the object is output.

5. The method according to claim 1, characterized in that The point cloud refined position deformation network based on the Transformer architecture includes the following steps: Based on the shallow image features in the image features of each viewing angle, a shallow image feature extraction module is used to bind the shallow image features under each viewing angle to each point of the rough point cloud corresponding to the object; Based on the rough point cloud corresponding to the object, using a local geometric feature extraction module to extract geometric features of local point cloud distribution for each point of the rough point cloud corresponding to the object; splicing the point-by-point shallow image features and the point-by-point local geometric features to obtain point-by-point features; Based on the point-by-point features of the rough point cloud, a point cloud global feature encoder is used to perform adaptive information exchange between points to obtain a point-by-point global feature encoding result; Based on the rough point cloud corresponding to the object, using a deformation position proposal module to generate a possible deformation position for each point of the rough point cloud; Based on the point-by-point global feature encoding result and the deformation position proposal, regressing the point-by-point deformation residual using a deformation position prediction module based on a cross-attention mechanism; The point-by-point deformation residual is added to the rough point cloud corresponding to the object to obtain a result point cloud corresponding to the object.

6. The method according to claim 5, characterized in that The shallow image feature extraction module comprises the following steps: Transforming the rough point cloud corresponding to the object to each camera perspective based on the camera extrinsic parameters at each perspective; Projecting the rough point cloud onto a camera imaging plane based on the camera intrinsic parameters at each viewing angle; Binding shallow image features in each perspective image feature to each point of the rough point cloud by using a bilinear interpolation algorithm; Based on the multi-view image features bound to each point of the rough point cloud, three statistics (maximum value, mean value, standard deviation) are used to fuse the features between viewpoints to obtain point-by-point shallow image features.

7. The method according to claim 5, characterized in that The local geometric feature extraction module comprises the following steps: Based on the rough point cloud corresponding to the object, the KNN algorithm is used to find K nearest neighbors for each point, and the difference between the K nearest neighbors of each point is used as the relative position code of the point; splicing the point-by-point relative position code with the coordinates of the rough point cloud to obtain a point-by-point local context; Based on the point-by-point local context, a multi-layer perceptron and a maximum pooling operation are used to obtain point-by-point local geometric features.

8. The method according to claim 5, characterized in that The point cloud global feature encoder includes three self-attention encoder sublayers stacked in series, and the point-by-point features of the rough point cloud are sequentially passed through the self-attention encoder sublayers to obtain point-by-point features at four different interaction levels; the point-by-point features at the four different interaction levels are spliced ​​together in the feature dimension, and the point-by-point features after interaction are obtained by using a multi-layer perceptron; The self-attention encoder sublayer is characterized by including a multi-head self-attention module and a feature feedforward network. The multi-head self-attention module captures the relationship between different input elements, and the feature feedforward network converts element features through a multi-layer perceptron.

9. The method according to claim 5, characterized in that The deformation position proposal module provides points with fixed distances in six directions of up, down, left, right, front and back as deformation position proposals for the rough point cloud corresponding to the object, that is, expands the rough point cloud by 7 times.

10. The method according to claim 5, characterized in that The deformation position prediction module based on the cross attention mechanism includes the following steps: Based on the deformation position proposals of the rough point cloud, extracting features of each deformation position using the shallow image feature extraction module; Based on the deformed position features, using a cross attention layer to perform cross query from the point-by-point features of the rough point cloud to obtain updated deformed position features; Based on the updated deformed position features, a local attention module including multiple self-attention layers is used to perform a self-attention operation on each group of 7 deformed position proposals to obtain the deformed position features after intra-group interaction; Based on the deformation position features after the intra-group interaction, a possibility weight of each deformation position is obtained by decoding using a multi-layer perceptron and a SoftMax operation; Based on the deformed position proposals of the rough point cloud and the likelihood weights of the deformed positions, a weighted sum operation is used to obtain the deformation residual of each point in the rough point cloud.

Images