Self-supervised three-dimensional human body reconstruction method and system based on point cloud data

Abstract

The invention discloses a self-supervised three-dimensional human body reconstruction method and system based on point cloud data, and belongs to the field of three-dimensional reconstruction and the field of deep learning of artificial intelligence. The method comprises the following steps: acquiring human body point cloud data; processing the human body point cloud data based on a self-supervised regression reconstruction module to obtain posture parameters and shape parameters; and inputting the posture parameters and the shape parameters into an SMPL human body parameterized model to obtain a human body reconstruction result. According to the method, the point cloud semantics can be estimated from the human body point cloud data in a self-supervised manner so as to guide the deep regression network to reconstruct the three-dimensional human body, self-lifting is realized through fusion iteration between the deep regression network and the optimization algorithm, and accurate and efficient three-dimensional human body reconstruction is realized.

Description

Technical Field

[0001] This invention belongs to the fields of 3D reconstruction and deep learning in artificial intelligence, and specifically relates to a self-supervised 3D human body reconstruction method and system based on point cloud data. Background Technology

[0002] In recent years, with the growing popularity of smart homes and the Internet of Things, understanding and interpreting human behavior and states, and providing human-centered intelligent services, has become a crucial technology essential for various intelligent applications, the development of the digital economy, and the construction of an intelligent social ecosystem. Specifically, researchers hope to reconstruct the human body in the environment, monitor, calculate, and infer changes in it, understand human behavior and intentions, guide intelligent systems to make rational decisions, and thus provide better services to people.

[0003] While traditional visible light cameras offer high reconstruction accuracy, they are easily constrained by lighting conditions and pose privacy risks. In contrast, 3D sensors such as depth sensors and LiDAR can operate stably in dark environments, and the point cloud data they generate not only poses no privacy risks but also naturally contains rich 3D spatial information, making them more conducive to efficient reconstruction.

[0004] Most existing 3D human reconstruction methods are based on deep learning and rely heavily on high-quality labeled data. Acquiring this data is time-consuming and costly, and models are prone to overfitting and long-tail problems when labeled data is insufficient. In contrast, the cost of acquiring unlabeled point cloud data is extremely low, thus necessitating the development of self-supervised solutions to reduce reliance on large-scale labeling.

[0005] To achieve self-supervised 3D human reconstruction, the following challenges need to be addressed. In real-world scenarios, deploying multi-view 3D sensors to achieve blind-spot-free coverage of the target user is extremely costly and introduces additional problems such as time synchronization. Consequently, the quality of the acquired human point cloud data is low, often exhibiting sparsity, incompleteness, and missing semantic information. Under these circumstances, it is difficult to extract reliable self-supervised signals. Furthermore, the symmetry of the human body structure makes it highly susceptible to causing the self-supervised training model to get trapped in local optima, resulting in unrealistic human reconstruction results.

[0006] In existing technologies, some methods first allow the model to perform semantic prediction on the input human point cloud, and then guide the 3D human reconstruction based on this (Bhatnagar BL, Sminchisescu C, et al. Loopreg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration[J]. Advances in Neural Information Processing Systems, 2020, 33: 12909-12922). However, these methods usually rely on datasets with real labels to pre-train the semantic prediction network, resulting in limited generalization ability. Other methods, such as HMRN, use Gaussian mixture models to model the semantics of human point clouds and complete the reconstruction through cascaded neural networks and optimization modules (Zuo X, Wang S, Sun Q, et al. Self-supervised 3d humanmesh recovery from noisy point clouds[J]. arxiv preprint arxiv:2107.07539, 2021). However, this approach is prone to misestimating human orientation and incurs huge computational costs.

[0007] To date, some research has been conducted to address the challenge of self-supervised 3D human reconstruction from point cloud data. However, no method has been found that can efficiently process point cloud data in a self-supervised manner while simultaneously addressing interferences such as human orientation and point cloud data quality, thus preventing predictions from falling into local optima. Summary of the Invention

[0008] This invention provides a self-supervised 3D human body reconstruction method and system based on point cloud data. It can self-supervisively estimate the semantics of point clouds from human body point cloud data, thereby guiding a deep regression network to reconstruct a 3D human body. Through the fusion and iteration between the deep regression network and the optimization algorithm, it can achieve self-improvement and realize accurate and efficient 3D human body reconstruction.

[0009] To achieve the above objectives, the technical solution of the present invention includes the following:

[0010] A self-supervised 3D human body reconstruction method based on point cloud data, the method comprising:

[0011] Acquire human body point cloud data;

[0012] The human point cloud data is processed using a self-supervised regression reconstruction module to obtain pose parameters. and shape parameters The self-supervised regression reconstruction module is used to extract features from human point cloud data and predict pose parameters based on these features. and shape parameters

[0013] pose parameters and shape parameters Input the data into the SMPL human body parametric model to obtain the human body reconstruction results.

[0014] Furthermore, the process of training the self-supervised regression reconstruction module includes:

[0015] Obtain human body point cloud data sample X;

[0016] The pose parameters are preset to all zeros. and shape parameters Input into the SMPL human body parametric model to obtain the template human body point cloud T;

[0017] Preprocess the human point cloud data sample X and the template human point cloud T;

[0018] Calculate the relationship matrix between human point cloud data sample X and template human point cloud T, and optimize the relationship matrix using a bidirectional softmax function to obtain the relationship matrix S. XT ;

[0019] Combining relation matrix S XT Self-reconstruction and cross-reconstruction are performed on human point cloud data samples X and template human point cloud T to obtain the loss L. re ;

[0020] A reconstructed human body mesh is generated based on human body point cloud data sample X and the same human body point cloud data sample X. And calculate the human point cloud data sample X and the reconstructed human body mesh. The semantic consistency-guided human regression reconstruction loss L SC ;

[0021] Calculate the semantic loss L between human point cloud data and human reconstruction results. pse ;

[0022] According to the loss L re Loss L SC and semantic loss L pse Generate the total loss L, and adjust the parameters of the self-supervised regression reconstruction module based on the total loss L.

[0023] Furthermore, the human point cloud data sample X and the template human point cloud T are preprocessed, including:

[0024] Calculate the centroids of human point cloud data sample X and template human point cloud T respectively, and subtract the corresponding centroids from the points in human point cloud data sample X and template human point cloud T to obtain human point cloud data sample X and template human point cloud T centered on the origin of the coordinate system.

[0025] A fixed number of points N is set as the upper bound of the number of points in the point cloud. If the number of points in the human point cloud data sample X or the template human point cloud T is greater than this upper bound, uniform downsampling is used to reduce the number of points in the human point cloud data sample X or the template human point cloud T to the number of points N. If the number of points in the human point cloud data sample X or the template human point cloud T is less than this upper bound, zero-value coordinate points are added to the end of the point set to supplement the number of points in the human point cloud data sample X or the template human point cloud T to the number of points N.

[0026] Furthermore, combining the relation matrix S XT Self-reconstruction and cross-reconstruction are performed on human point cloud data samples X and template human point cloud T to obtain the loss L. re ,include:

[0027] In the case that the human point cloud data sample X is complete human point cloud data, based on the relation matrix S XT Human point cloud data sample X and template human point cloud T are used to obtain the self-reconstruction result of human point cloud data sample X. With cross-reconstruction results And the self-reconstruction results of the template human body point cloud T With cross-reconstruction results Based on human point cloud data sample X, template human point cloud T, and self-reconstruction results Self-reconstruction results Cross reconstruction results and cross-reconstruction results Loss L re ;

[0028] When the human point cloud data sample X is incomplete, a subset T′ semantically aligned with the human point cloud data sample X is constructed from the template human point cloud T, based on the relation matrix S. XT Human point cloud data sample X, template human point cloud T, and subset T′ are used to obtain the self-reconstruction result of human point cloud data sample X. With cross-reconstruction results And the self-reconstruction results of the template human body point cloud T With cross-reconstruction results Based on human point cloud data sample X, template human point cloud T, subset T′, and self-reconstruction results Self-reconstruction results Cross reconstruction results and cross-reconstruction results Loss L re .

[0029] Furthermore, based on the relation matrix S XT Human point cloud data sample X and template human point cloud T are used to obtain the self-reconstruction result of human point cloud data sample X. With cross-reconstruction results And the self-reconstruction results of the template human body point cloud T With cross-reconstruction results include:

[0030] Extracting the midpoint x of human point cloud data sample X i Point features a i Midpoint t of the human body point cloud template T j Point features b j ;

[0031] For each point x i In the human point cloud data sample X, select the point feature a. i The nearest neighbor points are used to obtain the self-reconstruction result of the human body point cloud data sample X.

[0032] For each point t j In the template human point cloud T, select the point feature b. j The nearest neighbor points are used to obtain the self-reconstruction result of the template human body point cloud T.

[0033] According to the relation matrix S XT Obtain point x from the template human body point cloud T i The corresponding point t j And by selecting point features b in the template human point cloud T. j The closest neighbor points are used to obtain the cross-reconstruction results of the human body point cloud data sample X.

[0034] According to the relation matrix S XT Obtain point t from human point cloud data sample X. j The corresponding point x i And by selecting point features a in the template human point cloud T. i The closest neighbor points are used to obtain the cross-reconstruction result of the template human body point cloud T.

[0035] Furthermore, a subset T′ semantically aligned with the human point cloud data sample X is constructed from the template human point cloud T, including:

[0036] Based on relation matrix SXT Calculate the value of each point t in the template human point cloud T. j semantic score;

[0037] Select several points t with the highest semantic scores j , to generate a subset T′.

[0038] Furthermore, a reconstructed human body mesh is generated based on the human body point cloud data sample X and the human body point cloud data sample X. And calculate the human point cloud data sample X and the reconstructed human body mesh. The semantic consistency-guided human regression reconstruction loss L SC ,include:

[0039] A mesh restorer is constructed using convolutional neural networks, downsampling neural networks, and multiple fully connected layers;

[0040] The human body point cloud data sample X and its features are input into the mesh restorer to obtain the reconstructed human body mesh.

[0041] Based on the semantic invariance of the SMPL human body parameterization model, and based on the relation matrix S XT The human body point cloud data sample X and the reconstructed human body mesh were obtained. Relationship matrix

[0042] Based on relation matrix Calculate human point cloud data sample X and reconstruct human body mesh The distance matrix between them is D = [d ij ]; where d ij Point x in human point cloud data sample X represents i With Reconstructing the Human Body Grid point m in j The distance between them;

[0043] Based on relation matrix Let x be the point i Choosing to reconstruct the human body mesh The highest number of neighboring points with the highest feature similarity And the corresponding weights w are obtained using the softmax function. ij :

[0044] In reconstructing the human body grid Extract a subset semantically aligned with the human point cloud data sample X.

[0045] Based on distance d ij Weight w ij Human point cloud data sample X and subset The loss L based on semantic consistency-guided human regression reconstruction is obtained. SC .

[0046] Furthermore, the semantic loss L between the human point cloud data and the human reconstruction result is calculated. pse ,include:

[0047] Obtain the attitude parameters for this iteration Among them, the attitude parameters of the initial wheel

[0048] Regarding the attitude parameters Perform data augmentation to generate multiple sets of differentiated SMPL parameters;

[0049] The algorithm is used to fit the multiple sets of differentiated SMPL parameters to generate multiple candidate solutions, and the loss value generated during the optimization process is obtained.

[0050] Based on the loss value generated during the optimization process, a better solution M is selected from multiple candidate solutions. opt ;

[0051] For all points x in the human body point cloud data sample X i By calculating the optimal solution M opt Upper distance point x i The index k of the nearest point is used to generate point x. i pseudo-semantic tag c i ;

[0052] According to the relation matrix S XT Prediction point x i semantic tags

[0053] Based on pseudo-semantic tags c i and semantic tags The semantic loss L between human point cloud data and human reconstruction results is obtained. pse .

[0054] A self-supervised 3D human body reconstruction system based on point cloud data, the system comprising:

[0055] The data processing module is used to acquire human body point cloud data;

[0056] The parameter calculation module is used to process the human point cloud data based on the self-supervised regression reconstruction module to obtain pose parameters. and shape parameters The self-supervised regression reconstruction module is used to extract features from human point cloud data and predict pose parameters based on these features. and shape parameters

[0057] The results generation module is used to generate pose parameters. and shape parameters Input the data into the SMPL human body parametric model to obtain the human body reconstruction results.

[0058] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the self-supervised three-dimensional human body reconstruction method based on point cloud data as described above.

[0059] Compared with the prior art, the present invention has the following beneficial effects.

[0060] This invention directly reconstructs a 3D human body from point cloud data through self-supervised learning, achieving efficient and accurate 3D reconstruction without any pose labels. Simultaneously, the self-supervised regression reconstruction method fully utilizes the predefined human body structure and semantic consistency features of the human body parameterized model, realizing neural network-based self-supervised human body reconstruction. The proposed optimized regression method, employing an integrated optimization strategy of multiple initializations and dynamic optimal result selection, effectively alleviates the problem of optimization results easily getting trapped in local optima due to missing point cloud information and the symmetry of human body structure, leading to incorrect orientation and unrealistic poses in the fitted human body. The iterative self-improvement technique organically integrates regression and optimization methods, transferring knowledge from the optimization method to the regression network from both a data-driven perspective and a pre-designed optimization loss perspective, avoiding the problems of inaccurate regression networks and slow optimization methods, thus achieving efficient and accurate self-supervised 3D human body reconstruction. The self-supervised 3D human body reconstruction method proposed in this invention can provide rich user modeling and behavioral information, thereby supporting a wider range of intelligent sensing applications. This invention is easy to promote and applicable to fields such as intelligent sensing and 3D reconstruction. Attached Figure Description

[0061] Figure 1 This is a flowchart of a self-supervised 3D human body reconstruction method based on point cloud data.

[0062] Figure 2 This is a schematic diagram of the training process for the self-supervised regression reconstruction module and the SMPL human body parameterization model.

[0063] Figure 3 This is a visualization experiment of the system of the present invention on the Behave dataset. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0065] The self-supervised 3D human body reconstruction method based on point clouds of the present invention, such as Figure 1 As shown, the process includes: acquiring human point cloud data; processing the human point cloud data based on a self-supervised regression reconstruction module to obtain pose parameters and shape parameters; inputting the pose parameters and shape parameters into the SMPL human parametric model to obtain human reconstruction results; wherein, the self-supervised regression reconstruction module is used to extract features from the human point cloud data and predict pose parameters and shape parameters based on the features.

[0066] Specifically, for the aforementioned self-supervised regression reconstruction module and SMPL human body parameterization model, this invention first designs a self-supervised human body regression reconstruction method based on the semantic consistency of the human body parameterization model; then it designs a self-supervised optimization method based on multiple initializations, iterative fitting, and adaptive result selection; next, it designs an iterative self-improvement strategy to organically integrate the regression method and the optimization method, preventing the entire system from falling into overfitting results when there is no supervision and the point cloud lacks information. The two promote each other, ultimately achieving efficient and accurate self-supervised human body reconstruction.

[0067] Specifically, such as Figure 2 As shown, the training process of the self-supervised regression reconstruction module and the SMPL human body parameterized model in this invention mainly includes three steps: self-supervised regression reconstruction, optimized reconstruction, and iterative self-improvement. Self-supervised regression reconstruction mainly includes feature extraction, similarity calculation, point cloud self-reconstruction, point cloud cross-reconstruction, and human body 3D reconstruction. Optimized reconstruction includes initial point cloud enhancement, point cloud fitting, and adaptive fitting result selection. Finally, iterative self-improvement mainly improves the accuracy of self-supervised regression reconstruction by fusing regression reconstruction and optimized reconstruction methods. The specific steps are as follows:

[0068] Step S1: Self-supervised regression reconstruction.

[0069] Step S1-1: Obtain the human body point cloud X, and obtain the human body template point cloud through the SMPL human body parametric model.

[0070] The present invention first deploys three-dimensional sensors such as depth cameras or lidar in the environment, and then obtains human point cloud X through background subtraction, target detection and other means.

[0071] The SMPL human parametric model is one of the most widely used parametric human models currently available, and it provides a pre-trained mapping function. Different reconstructed human bodies M can be obtained using different pose parameters θ and shape parameters β. M is a triangular mesh containing 6890 vertices and 13776 faces. This invention utilizes pose parameters θ, which are preset to all zeros. zero and attitude parameter β zero Obtain template human body And the corresponding 6890 template human body point cloud T.

[0072] Step S1-2: After preprocessing the human point cloud X and the template point cloud T, the feature extractor is used to extract the point-by-point features of X and T respectively, and the correspondence between the input point cloud and the template point cloud is obtained.

[0073] The centroids of the human point cloud X and the template point cloud T are calculated. The coordinates of the centroids of each point cloud are subtracted from the centroids of all points, ensuring that both point clouds are centered at the origin of the coordinate system. A fixed number of points N is set as an upper bound for the number of points in the point cloud. For point clouds larger than this upper bound, this invention uses uniform downsampling to reduce the number of points to this threshold. For point clouds smaller than this upper bound, the total number of points is supplemented to the threshold by adding zero-value coordinate points (0,0,0) to the end of the point set, resulting in the processed X and T.

[0074] Furthermore, this invention utilizes a parameter-shared feature extractor to extract pointwise features of X and T respectively to obtain F. X ,F Y Therefore, the correspondence matrix S between two point clouds can be calculated using cosine similarity. XT =[s ij ],

[0075]

[0076] Among them, s ij Represents x i ∈X and t h Similarity between ∈T Represents x i The corresponding feature, g(·), is a projection layer network consisting of two fully connected layers. ij The larger the value, the greater the value of x. i and t j The higher the feature similarity, the greater the probability that they correspond. Points that correspond have the same semantics.

[0077] Since the correspondence matrix calculated in this way may exhibit severe many-to-one or one-to-many phenomena, this invention utilizes a bidirectional softmax function to adjust S. XT Optimize:

[0078] s ij =Softmax(S XT (:,j)) i Softmax(S XT (i,:)) j (2)

[0079] If x i and tj If the correspondence score between them is higher than their scores with other points in X and T, then s ij It will be larger. Bidirectional softmax encourages one-to-one correspondences in the correspondence matrix.

[0080] Within a self-supervised learning framework, to optimize the parameters of the feature extractor and obtain a more accurate correspondence matrix, this invention designs a loss function based on self-reconstruction and aligned cross-reconstruction to guide the effective optimization of the neural network. The self-reconstruction process prompts the feature extractor to extract similar features from spatially adjacent points in the point cloud, thus ensuring local consistency. The cross-reconstruction process further constrains the feature extractor, enabling it to extract similar features from points with similar structures across different point clouds, thereby improving its cross-point cloud structural correspondence capability.

[0081] Specifically, for any two point sets A and B, the reconstruction result The formula for single-point reconstruction is as follows:

[0082]

[0083] Where A, B∈{X,T},N B (a i ) refers to B and a i The index of the 10 nearest neighbor points. By using different combinations of sets, this invention can obtain different reconstruction results. in and It is self-reconstruction. and This is the result of cross-reconstruction.

[0084] Because this invention has fewer restrictions on X, it can simultaneously accommodate complete point clouds and incomplete human point clouds. Suppose X represents an incomplete human body, while T depicts a complete human body. In this case, directly performing cross-reconstruction is unreasonable because some structures in T do not have corresponding points in X. To solve this problem, this invention proposes an aligned cross-reconstruction method. This method adaptively calculates the cross-reconstruction loss from a subset of T that is semantically aligned with X using a correspondence matrix. Specifically, let t... j semantic score of ∈T Then, the top N / 2 points with the highest semantic scores are selected as the semantic alignment template subset T′. If X is a complete point cloud, the above steps are unnecessary, and the semantic alignment template subset T′ = T. Thus, this method achieves compatibility with both complete and incomplete human point clouds.

[0085] The final loss function is as follows:

[0086]

[0087] By utilizing the chamfer distance loss CD(·,·) to minimize the direct loss between the reconstruction result and the original point cloud, joint constraints and optimizations of the feature extractor are achieved.

[0088] Step S1-3: Guide human regression reconstruction based on semantic consistency.

[0089] The above steps allow us to obtain a reasonable correspondence between points X and T, which is semantic information. This invention aligns the reconstructed human body with the input point cloud based on a semantic consistency-guided model. Semantic consistency means that point pairs with similar semantics should remain close in Euclidean space.

[0090] This invention utilizes a mesh restorer designed with convolutional neural networks, downsampling neural networks, and multiple fully connected layers. Its inputs are X and F. X The output is the predicted SMPL parameters. and The resulting reconstructed human body mesh In the After performing the same uniform downsampling process as mentioned in S2-2, due to the semantic invariance of the SMPL model, it can be guaranteed that... Therefore, the correspondence between X and T can be easily transferred to X and T. Up. Then calculate x. i ∈X and The distance matrix D = [d ij ], where d ij =‖x i -m j ‖2.

[0091] This invention is based on For x i Choose in The 10 neighboring points with the highest feature similarity The corresponding weights are then obtained using the softmax function:

[0092]

[0093] Similar to the steps in S2-2, the present invention extracts Subsets semantically aligned with X And design the loss function:

[0094]

[0095] Step S2: Self-supervised optimization reconstruction.

[0096] This invention adopts the optimization function L designed in HMRN. opt(X,θ,β), this function is characterized by introducing several artificially designed penalty terms. The difference between X and X is used to iteratively update parameters θ and β during backpropagation, such that... Ultimately, a match can be achieved with X. However, this optimization process is very sensitive to initial values ​​and can easily get trapped in local optima. To address this issue, this invention proposes a dynamic initialization ensemble strategy.

[0097] Step S2-1: Initialize the diverse human body mesh.

[0098] At this stage, the present invention first determines an initial attitude parameter θ. init Initially it was θ zero Subsequently, data augmentation was performed on this parameter to generate multiple sets of differentiated SMPL parameter initializations {(θ1,β)}. zero ),(θ2,β zero By using the formulas ), ...,}, we can obtain human body meshes with different orientations. Therefore, after fitting the meshes using an optimization algorithm, multiple candidate solutions will be generated. This strategy transforms the difficult problem of "escaping local optima" into the more manageable problem of "selecting a better solution from a set of diverse local minima".

[0099] Step S2-2: Adaptive selection of the optimal solution.

[0100] This invention directly uses the loss value generated during the optimization process for judgment.

[0101]

[0102] This allows us to obtain a better solution.

[0103] Step S3: Iterative self-promotion.

[0104] The iterative self-improvement phase organically integrates the self-supervised regression module and the self-supervised optimization module during the model training phase.

[0105] Step S3-1: Obtain the initial optimization results and calculate the pseudo-semantic labels.

[0106] Let θ only at the beginning of training init =θ zero In subsequent iterations, let Then, M is obtained according to step S3. opt This invention is based on M opt For all x i ∈X calculate M opt Upper distance x i The index k of the nearest point is used to generate the pseudo-semantic label c of the one-hot vector. i ci It is a Boolean vector of length N, containing only c. i,k =1, all others are 0.

[0107] Step S3-2: Self-supervised regression reconstruction, incorporating semantic loss and iterating for a certain number of rounds.

[0108] Calculate the predicted semantic labels based on the correspondence matrix. This invention calculates semantic loss according to the following formula:

[0109]

[0110] Where CE(·,·) is the cross-entropy loss, and the formula is:

[0111] The loss of the entire self-supervised regression reconstruction module is as follows:

[0112] L = L re +L SC +L pse (11)

[0113] Step S3-3: Iterative update.

[0114] Repeat steps S4-1 and S4-2 until convergence or the preset number of iterations is reached.

[0115] Example Description

[0116] In this embodiment, the Python programming language is used, which can run on mainstream computer system platforms. The operating system used in this embodiment is Ubuntu 16.04, requiring an Intel Core i7 CPU, 8GB or more of memory, 16GB or more of hard disk space, and an NVIDIA GeForce 3090 GPU with 24GB of video memory.

[0117] This invention is implemented based on the PyTorch 1.7.0 framework. The Adam optimization algorithm (Kingma DP, Ba J. 2014. Adam: A method for stochastic optimization[J]. arXiv preprint arXiv:1412.6980.) is used as the gradient descent algorithm to solve for all the parameters to be learned in the neural network. The learning rate is 0.0001, the number of iterations during training is set to 200, the batch size is 32, and the upper bound of the number of point cloud points is set to N = 1024.

[0118] The data in this embodiment of the invention comes from the open-source dataset Behave (Bhatnagar BL, Xie X, Petrovi A, et al. Behave: Dataset and method for tracking human object interactions[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2022:15935-15946.), which contains RGB-D videos from four perspectives, recording the interaction process between humans and various objects. The experiment is limited to evaluation under single-view conditions, where the obtained point clouds often have severe occlusion. This invention uses the officially divided training and test sets and downsamples the dataset, selecting one frame every other frame. The final division results include 16 different objects, 70 scenes, 22,840 training samples, and 9,180 test samples.

[0119] To evaluate the performance of the method, this embodiment uses two metrics, mean per joint position error (MPJPE) and mean per vertex error (PVE), to assess the human body reconstruction effect.

[0120] The final performance of this embodiment on the test set is as follows: average joint error of 9.21 cm and average vertex error of 11.96 cm. Visualization results are shown below. Figure 3 As shown

[0121] This invention, through the above embodiments, verifies that the system possesses accurate self-supervised 3D human body reconstruction capabilities and exhibits strong robustness. This invention argues that self-supervised 3D human body reconstruction based on point cloud data is a fundamental capability that can provide behavioral information of target users for intelligent sensing applications, thereby supporting a wider range of intelligent sensing scenarios.

[0122] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A self-supervised 3D human body reconstruction method based on point cloud data, characterized in that, The method includes: Acquire human body point cloud data; The human point cloud data is processed using a self-supervised regression reconstruction module to obtain pose parameters. and shape parameters The self-supervised regression reconstruction module is used to extract features from human point cloud data and predict pose parameters based on these features. and shape parameters pose parameters and shape parameters Input the data into the SMPL human body parametric model to obtain the human body reconstruction results.

2. The method according to claim 1, characterized in that, The process of training the self-supervised regression reconstruction module includes: Obtain human body point cloud data sample X; The pose parameters are preset to all zeros. and shape parameters Input into the SMPL human body parametric model to obtain the template human body point cloud T; Preprocess the human point cloud data sample X and the template human point cloud T; Calculate the relationship matrix between human point cloud data sample X and template human point cloud T, and optimize the relationship matrix using a bidirectional softmax function to obtain the relationship matrix S. XT ; Combining relation matrix S XT Self-reconstruction and cross-reconstruction are performed on human point cloud data samples X and template human point cloud T to obtain the loss L. re ; A reconstructed human body mesh is generated based on human body point cloud data sample X and the human body point cloud data sample X. And calculate the human point cloud data sample X and the reconstructed human body mesh. The semantic consistency-guided human regression reconstruction loss L SC ; Calculate the semantic loss L between human point cloud data and human reconstruction results. pse ; According to the loss L re Loss L SC and semantic loss L pse Generate the total loss L, and adjust the parameters of the self-supervised regression reconstruction module based on the total loss L.

3. The method according to claim 2, characterized in that, Preprocessing is performed on human point cloud data sample X and template human point cloud T, including: Calculate the centroids of human point cloud data sample X and template human point cloud T respectively, and subtract the corresponding centroids from the points in human point cloud data sample X and template human point cloud T to obtain human point cloud data sample X and template human point cloud T centered on the origin of the coordinate system. A fixed number of points N is set as the upper bound of the number of points in the point cloud. If the number of points in the human point cloud data sample X or the template human point cloud T is greater than this upper bound, uniform downsampling is used to reduce the number of points in the human point cloud data sample X or the template human point cloud T to the number of points N. If the number of points in the human point cloud data sample X or the template human point cloud T is less than this upper bound, zero-value coordinate points are added to the end of the point set to supplement the number of points in the human point cloud data sample X or the template human point cloud T to the number of points N.

4. The method according to claim 2, characterized in that, Combining relation matrix S XT Self-reconstruction and cross-reconstruction are performed on human point cloud data samples X and template human point cloud T to obtain the loss L. re ,include: In the case that the human point cloud data sample X is complete human point cloud data, based on the relation matrix S XT Human point cloud data sample X and template human point cloud T are used to obtain the self-reconstruction result of human point cloud data sample X. With cross-reconstruction results And the self-reconstruction results of the template human body point cloud T With cross-reconstruction results Based on human point cloud data sample X, template human point cloud T, and self-reconstruction results Self-reconstruction results Cross reconstruction results and cross-reconstruction results Loss L re ; When the human point cloud data sample X is incomplete, a subset T′ semantically aligned with the human point cloud data sample X is constructed from the template human point cloud T, based on the relation matrix S. XT Human point cloud data sample X, template human point cloud T, and subset T′ are used to obtain the self-reconstruction result of human point cloud data sample X. With cross-reconstruction results And the self-reconstruction results of the template human body point cloud T With cross-reconstruction results Based on human point cloud data sample X, template human point cloud T, subset T′, and self-reconstruction results Self-reconstruction results Cross reconstruction results and cross-reconstruction results Loss L re .

5. The method according to claim 4, characterized in that, Based on relation matrix S XT Human point cloud data sample X and template human point cloud T are used to obtain the self-reconstruction result of human point cloud data sample X. With cross-reconstruction results And the self-reconstruction results of the template human body point cloud T With cross-reconstruction results include: Extracting the midpoint x of human point cloud data sample X i Point features a i Midpoint t of the human body point cloud template T j Point features b j ; For each point x i In the human point cloud data sample X, select the point feature a. i The nearest neighbor points are used to obtain the self-reconstruction result of the human body point cloud data sample X. For each point t j In the template human point cloud T, select the point feature b. j The nearest neighbor points are used to obtain the self-reconstruction result of the template human body point cloud T. According to the relation matrix S XT Obtain point x from template human body point cloud X i The corresponding point t j And by selecting point features b in the template human point cloud X. j The closest neighbor points are used to obtain the cross-reconstruction results of the human body point cloud data sample X. According to the relation matrix S XT Obtain point t from human point cloud data sample X. j The corresponding point x i And by selecting point features a in the template human point cloud T. i The closest neighbor points are used to obtain the cross-reconstruction result of the template human body point cloud T.

6. The method according to claim 4, characterized in that, From the template human point cloud T, construct a subset T′ that is semantically aligned with the human point cloud data sample X, including: Based on relation matrix S XT Calculate the value of each point t in the template human point cloud T. j semantic score; Select several points t with the highest semantic scores j , to generate a subset T′.

7. The method according to claim 2, characterized in that, A reconstructed human body mesh is generated based on human body point cloud data sample X and the human body point cloud data sample X. And calculate the human point cloud data sample X and the reconstructed human body mesh. The semantic consistency-guided human regression reconstruction loss L SC ,include: A mesh restorer is constructed using convolutional neural networks, downsampling neural networks, and multiple fully connected layers; The human body point cloud data sample X and its features are input into the mesh restorer to obtain the reconstructed human body mesh. Based on the semantic invariance of the SMPL human body parameterization model, and based on the relation matrix S XT The human body point cloud data sample X and the reconstructed human body mesh were obtained. Relationship matrix Based on relation matrix Calculate human point cloud data sample X and reconstruct human body mesh The distance matrix between them is D = [d ij ]; where d ij Point x in human point cloud data sample X represents i With Reconstructing the Human Body Grid point m in j The distance between them; Based on relation matrix Let x be the point i Choosing to reconstruct the human body mesh The highest number of neighboring points with the highest feature similarity And the corresponding weights w are obtained using the softmax function. ij : In reconstructing the human body grid Extract a subset semantically aligned with the human point cloud data sample X. Based on distance d ij Weight w ij Human point cloud data sample X and subset The loss L based on semantic consistency-guided human regression reconstruction is obtained. SC .

8. The method according to claim 2, characterized in that, Calculate the semantic loss L between human point cloud data and human reconstruction results. pse ,include: Obtain the attitude parameters for this iteration Among them, the attitude parameters of the initial wheel Regarding the attitude parameters Perform data augmentation to generate multiple sets of differentiated SMPL parameters; The algorithm is used to fit the multiple sets of differentiated SMPL parameters to generate multiple candidate solutions, and the loss value generated during the optimization process is obtained. Based on the loss value generated during the optimization process, a better solution M is selected from multiple candidate solutions. opt ; For all points x in the human body point cloud data sample X i By calculating the optimal solution M opt Upper distance point x i The index k of the nearest point is used to generate point x. i pseudo-semantic tag c i ; According to the relation matrix S XT Prediction point x i semantic tags Based on pseudo-semantic tags c i and semantic tags The semantic loss L between human point cloud data and human reconstruction results is obtained. pse .

9. A self-supervised 3D human body reconstruction system based on point cloud data, characterized in that, The system includes: The data processing module is used to acquire human body point cloud data; The parameter calculation module is used to process the human point cloud data based on the self-supervised regression reconstruction module to obtain pose parameters. and shape parameters The self-supervised regression reconstruction module is used to extract features from human point cloud data and predict pose parameters based on these features. and shape parameters The results generation module is used to generate pose parameters. and shape parameters Input the data into the SMPL human body parametric model to obtain the human body reconstruction results.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the self-supervised three-dimensional human body reconstruction method based on point cloud data as described in any one of claims 1-8.

Images