A 3D human body modeling and motion acquisition method based on multi-view synchronous shooting

Abstract

The present invention discloses a three-dimensional human body modeling and motion acquisition method based on multi-view synchronous shooting. A tracking video of a person shot synchronously from multiple perspectives is obtained, and two-dimensional human body key points and two-dimensional hand key points are extracted from each frame; then, the arm parameters of the human body parametric model are optimized using the two-dimensional human body key points. Afterwards, the parameters of the hand model are optimized based on the two-dimensional hand key points; according to the wrist position, the arm position is corrected to obtain the arm parameters of the human body parametric model. By arranging the hand model point cloud and the human body parametric model point cloud, an index of points and triangular facets is established to obtain a fusion model, and the parameters of the human body parametric model and the hand model are imported into the fusion model to obtain the result of human body reconstruction. The present invention can use multi-view images to perform human body modeling and motion acquisition including hands, and does not require a sophisticated multi-camera configuration and motion capture clothing.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional human body reconstruction, and in particular relates to a three-dimensional human body modeling and motion acquisition method based on multi-view synchronous shooting. Background Art

[0002] Human digitization primarily involves reconstructing a target human body and storing it in a computer. Human reconstruction has long been a challenging research topic in computer vision and graphics, primarily due to the body's complex geometry. In recent years, virtual human technology has been increasingly used in industries such as livestreaming and live news broadcasts. Furthermore, with the gradual development of AR and VR technologies, as well as advancements in holographic communication technology, the way people live and communicate may undergo radical changes in the near future. Against this backdrop, human digitization, particularly 3D human modeling and motion capture, has become an increasingly important research topic.

[0003] The geometric structure of the human body is influenced by factors such as body shape, posture, and gender. Changes in posture, in particular, can produce complex, large-scale, non-rigid deformations. This makes linear deformation methods, such as principal component analysis (PCA) for body shape and blending (BlendShape) for faces, incapable of directly applying to posture deformation. Furthermore, since the human body wears clothing in the real world, clothing naturally obscures the body's true form. The complex interaction between clothing and the human body significantly increases the complexity of human shape modeling. Furthermore, some human reconstruction efforts also involve modeling clothing, and extracting and reconstructing texture information from a wide variety of clothing items is a challenging task.

[0004] In fields like film and gaming, complex acquisition systems are commonly used to reconstruct the human body. These acquisition devices typically employ sophisticated multi-camera configurations, requiring precise camera calibration and synchronization. The subject also needs to wear specialized motion capture suits or be marked with key points. After data collection, complex and time-consuming data processing is required to achieve the desired high-precision digital image. These stringent acquisition equipment requirements limit the application of this technology.

[0005] Another method for human body reconstruction relies on RGB-D data. This method typically allows the gradual reconstruction of a person's shape during the scanning process using an RGB-D camera. However, professional 3D scanners are expensive, and compared to readily available RGB data, RGB-D data is more difficult to acquire, requiring more stringent data acquisition equipment and environments. Therefore, how to reconstruct a person's body from RGB images remains a question worth considering.

[0006] Unlike RGB-D images, RGB images lack depth information. Due to ambiguities introduced by perspective projection, they can lead to incorrect reconstructions of human poses. Furthermore, while RGB images are readily available, they are mostly monocular, and datasets with true 3D annotations are very limited. This prevents many advanced deep learning algorithms, which rely on large amounts of data for training, from fully realizing their original performance. Summary of the Invention

[0007] In view of the above, the purpose of the present invention is to provide a three-dimensional human body modeling and motion acquisition method based on multi-view synchronous shooting, which avoids complex acquisition systems and special motion capture clothing, and realizes three-dimensional human body reconstruction at low cost.

[0008] To achieve the above-mentioned object of the invention, an embodiment further provides a method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting, comprising the following steps:

[0009] Step 1: Obtain a tracking video of a person shot synchronously from multiple perspectives;

[0010] Step 2: Extract 2D human body key points and 2D hand key points from each frame of the tracking video;

[0011] Step 3, based on the two-dimensional human body key points and by optimizing the multi-view human body constraint function, obtaining the arm parameters of the human body parametric model, wherein the arm parameters include arm shape parameters and arm posture parameters;

[0012] Step 4: Optimize the hand constraint function based on the two-dimensional hand key points to obtain the parameters of the hand model;

[0013] Step 5: Determine the shoulder joint position and the elbow joint position based on the arm parameters of the human body parametric model. After determining the wrist joint position based on the parameters of the hand model, determine the actual elbow joint position that matches the wrist joint position using a geometric method based on the shoulder joint position, the elbow joint position, and the wrist joint position.

[0014] Step 6: Calculate the swing rotation matrix of the shoulder joint based on the actual elbow and shoulder joint positions. Use neural inverse kinematics to obtain the torsion rotation matrix of the shoulder joint based on the initial and final images of the shoulder joint motion. Update the swing rotation matrix and torsion rotation matrix into the arm posture parameters of the human body parametric model.

[0015] Step 7: Align the human body parametric model with updated arm posture parameters with the hand model to complete 3D human body modeling and motion acquisition.

[0016] In one embodiment, obtaining arm parameters of a human body parametric model based on human body key points and by optimizing a multi-view human body constraint function includes:

[0017] Calculate the two-dimensional human key points ∏ of the human body parametric model projection under each viewing angle K (R θ (J(β)) i ), calculate the three-dimensional human key points R of the human body parameter model θ (J(β)) i , using cyclic epipolar geometry to calculate the key points of the 3D human body J 3d,i , then construct the two-dimensional key point constraint L for human body reconstruction 2d and 3D key point constraint L 3d , expressed as:

[0018]

[0019]

[0020] Among them, R θ (J(β)) i It represents the coordinates of the three-dimensional human key points obtained by regression based on the hybrid linear skin deformation, which is obtained by the joint action of the posture parameters θ and shape parameters β of the human body parametric model. K (·) represents the parameterized model R of the human body using the camera parameter K θ (J(β)) i Projected to the camera's two-dimensional pixel coordinate system, ρ(·) represents the function describing the difference between the two coordinates, γ i ,ω i Represent the weight and confidence of the human key point i, J 2d,i Represents two-dimensional human body key points;

[0021] Based on L 3d and L 2d Constructing multi-view human body constraint function L multi , expressed as:

[0022]

[0023] Based on L multi Construct the total loss function L total , expressed as:

[0024] L total =min(L multi +A p L prior )

[0025] Among them, v represents the camera perspective index, which is 0-n, λ v ,λ w and λ p Both represent loss weights, L 2d,vrepresents the two-dimensional key point loss under the camera view v, L prior Represents the prior loss of human joints. For example, the human elbow joint cannot bend in the opposite direction. This loss term is set based on the human's subjective understanding of the body structure.

[0026] Using the total loss function L total Optimize arm parameters of a parametric human model.

[0027] In one embodiment, the total loss function L is used total Optimizing the arm parameters of the human body parametric model includes: optimizing the global displacement parameters and global rotation parameters of the human body parametric model in the first stage, optimizing the shape parameters of the human body parametric model in the second stage, optimizing the torso posture parameters by ignoring the key influence of the hand in the third stage, and refining the arm posture parameters by adding the influence of the hand key points in the fourth stage. Finally, the arm parameters of the human body parametric model are obtained. The arm parameters refer to a part of the torso posture parameters in the model, specifically the arm posture parameters.

[0028] In one embodiment, cyclic epipolar geometry is used to calculate the key points J of the 3D human body. 3d ,include:

[0029]

[0030] Among them, the cycle uses the epipolar geometry E for the two-camera combination v1 and v2 g Calculate the three-dimensional coordinates of the corresponding key points, and finally calculate the weight γ v1,v2 and confidence ω v1,v2 Weighted sum of three-dimensional coordinates to obtain the final three-dimensional human body key point coordinates J 3d , where λ v1,v2 is the proportional coefficient, J 2d,v1 , J 2d,v2 are the coordinates of the two-dimensional human key points under the two cameras, K v1 , K v2 Represent the parameters of the two cameras respectively.

[0031] In one embodiment, obtaining parameters of the hand model based on the hand key points and by optimizing the hand constraint function includes:

[0032] Based on the 2D hand key points and combined with the camera parameters, the 3D hand key points are calculated using cyclic epipolar geometry. With this The spatial position of is taken as the target, and the shape parameter β determined in step 4 is fixed to fit the posture parameter Θ of the hand model so that the three-dimensional hand key points regressed by the model are consistent with For registration, the hand constraint function used is:

[0033]

[0034] Where i represents the index of the hand keypoint, M is the number of hand keypoints, ‖·‖ represents the two-norm, γ represents the global displacement, α represents the scaling factor, and p(θ i ) is the pose prior of the hand model.

[0035] In one embodiment, determining the actual elbow joint position that matches the wrist joint position based on the shoulder joint position, the elbow joint position, and the wrist joint position using a geometric method includes:

[0036] Determine the upper arm length b based on the shoulder joint position and the actual elbow joint position e′ u , determine the forearm length b based on the actual elbow joint position e′ and wrist joint position f , determine another length sw′ based on the shoulder joint position and the wrist joint position, determine a unique triangle based on the three lengths, and rotate the triangle one circle in space with sw′ as the axis to determine the unique trajectory circle of e′ in space. After determining the trajectory circle and radius, the point closest to e on the trajectory circle is obtained based on the known original elbow joint position e, which is the actual elbow joint position e′.

[0037] In one embodiment, the step of calculating the swing rotation matrix of the shoulder joint based on the actual elbow joint position and the shoulder joint position, and obtaining the torsion rotation matrix of the shoulder joint using neural inverse kinematics based on the initial image and the end image of the shoulder joint movement, includes:

[0038] Determine the target rotation vector according to the actual elbow joint position and the shoulder joint position, and calculate the swing rotation matrix according to the target rotation vector and the initial rotation vector;

[0039] The twist rotation angle is calculated according to the initial image and the end image of the shoulder joint movement, and the twist rotation matrix is ​​calculated according to the twist rotation angle and the initial rotation vector.

[0040] In one embodiment, updating the swing rotation matrix and the twist rotation matrix into the arm posture parameters of the human body parameterized model includes:

[0041] Convert the original posture parameters of the shoulder joint into the original rotation matrix, and multiply the original rotation matrix by the twist rotation matrix R in sequence. tw and the swing rotation matrix R sw , get the new rotation matrix of the shoulder joint after correcting the elbow joint, convert the new rotation matrix into the axis angles of the three axes x, y, and z to cover the posture parameters of the shoulder joint, and get the updated arm posture parameters of the human body parametric model.

[0042] In one embodiment, aligning the human body parameterized model with updated arm posture parameters with the hand model includes:

[0043] The missing wrist patch index file is created by fusing the human body parametric model with updated arm parameters and the surface point cloud of the hand model;

[0044] Based on the missing wrist patch index file, the hand model is aligned with the human body parameterized model in the default hand posture. The aligned model can realize motion acquisition because it contains posture parameters.

[0045] In one embodiment, the method of fusing the updated arm parameter human body parametric model and the surface point cloud of the hand model to create a missing wrist patch index file includes:

[0046] Processing the surface point cloud of the human body parametric model, removing all the point clouds of the hands and their associated triangular facets, and obtaining a human body parametric model without the hands;

[0047] Arrange the point cloud of the hand model in sequence after the parametric model of the human body with both hands removed;

[0048] Organize the patch index file of the hand model so that the surfaces of the hand model and the human body parametric model can coexist;

[0049] The missing wrist point cloud is stitched together to create a missing wrist patch index file.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention adopts a model decoupling and registration optimization scheme based on the divide-and-conquer concept, decomposing the complete body model into independent left and right hand models, and retaining the torso part of the human body parametric model, so that the model reconstruction has a higher degree of freedom of expression. When the human body parametric model was originally used for human body reconstruction, there was only a single shape parameter to control the human body parametric model, which could not reconstruct the real body shape well, especially in the scenario where high-precision reconstruction of both hands is required. The present invention makes up for the problem that the hand shape and bone length are limited by the shape parameters when the human body parametric model is generally used for posture reconstruction. Secondly, the present invention uses multi-view constraints to eliminate the ambiguity of the three-dimensional reconstruction results of the human body. General human body reconstruction requires a sophisticated acquisition system, and the characters need to wear special motion capture clothing. The present invention only requires the actors to wear casual clothes, which reduces the difficulty of character selection during the motion acquisition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 This is a flow chart of a method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting provided by an embodiment;

[0054] Figure 2 is a flow chart of human body reconstruction provided by an embodiment;

[0055] Figure 3 is a schematic diagram of decoupling of a whole-body model provided in an embodiment;

[0056] Figure 4 is a flowchart of hand key point registration provided by an embodiment;

[0057] Figure 5 is a flowchart of body trunk reconstruction provided by an embodiment;

[0058] Figure 6 It is an abstract schematic diagram of solving arm mathematical problems provided by an embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0060] Figure 1 This is a flowchart of a method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting provided by an embodiment. Figure 2 This is a flow chart of human body reconstruction provided by the embodiment. Figure 1 As shown, the embodiment provides a method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting, comprising the following steps:

[0061] Step 1: Obtain a tracking video of a person shot synchronously from multiple perspectives.

[0062] In the embodiment, a person wears casual clothes and stands in a shooting area, and then four cameras distributed around the shooting area are used to record the person's movement process from various perspectives to obtain a corresponding number of tracking videos.

[0063] Step 2: Extract 2D human body key points and 2D hand key points from each frame of the tracking video.

[0064] In the embodiment, Figure 3As shown, the parametric human model (i.e., the body model) is decoupled into a hand model and a torso model. Based on the tracking video, the pixel-level positions of the human body key points and hand key points in each two-dimensional image frame of the tracking video are extracted. Specifically, any publicly available human body key point extraction method can be used. In this embodiment, a two-dimensional human pose estimator is used to estimate 2D human body key points and 2D hand key points.

[0065] Next, iterate the operations from step 3 to step 6 starting from the first frame of the video until every frame in the video has been iterated.

[0066] Step 3: Based on the two-dimensional human body key points of the current frame and by optimizing the multi-view human body constraint function, the arm parameters of the human body parametric model are obtained, wherein the arm parameters include arm shape parameters and arm posture parameters.

[0067] In the embodiment, when optimizing the arm parameters of the human body parametric model, first, the two-dimensional human body key points Π of the human body parametric model projection at each viewing angle are calculated. K (R θ (J(β)) i ), calculate the three-dimensional human key points R of the human body parameter model θ (J(β)) i , using cyclic epipolar geometry to calculate the key points of the 3D human body J 3d,i , the specific calculation formula is:

[0068]

[0069] Among them, the cycle uses the epipolar geometry E for the two-camera combination v1 and v2 g Calculate the three-dimensional coordinates of the corresponding key points, and finally calculate the weight γ v1,v2 and confidence ω v1,v2 Weighted sum of three-dimensional coordinates to obtain the final three-dimensional human body key point coordinates J 3d , where λ v1,v2 is the proportional coefficient, J 2d,v1 , J 2d,v2 are the coordinates of the two-dimensional human key points under the two cameras, K v1 , K v2 Represent the parameters of the two cameras respectively.

[0070] ∏ of the human body parametric model projection at various viewing angles K (R θ (J(β)) i ) and J 2d,i The two-dimensional key point constraint L between 2d , R of the human body parameterized model θ (J(β)) iJ calculated with cyclic epipolar geometry 3d,i The three-dimensional key point constraint L between 3d , respectively expressed as:

[0071]

[0072]

[0073] Among them, R θ (J(β)) i It represents the coordinates of the three-dimensional human key points obtained by regression based on the hybrid linear skin deformation, which is obtained by the joint action of the posture parameters θ and shape parameters β of the human body parametric model. K (·) represents the parameterized model R of the human body using the camera parameter K θ (J(β)) i Projected to the camera's two-dimensional pixel coordinate system, ρ(·) represents the function describing the difference between the two coordinates, γ i ,ω i Represent the weight and confidence of the human key point i, J 2d,i Represents two-dimensional human body key points;

[0074] Based on L 3d and L 2d Constructing multi-view human body constraint function L multi , expressed as:

[0075]

[0076] Based on L multi Construct the total loss function L total , expressed as:

[0077]

[0078] Among them, v represents the camera perspective index, which is 0-n, λ v ,λ w and λ p Both represent loss weights, L 2d,v represents the two-dimensional key point loss under the camera view v, L prior Represents the prior loss of human joints. For example, the human elbow joint cannot bend in the opposite direction. This loss term is set based on the human's subjective understanding of the body structure.

[0079] Using the total loss function L total Optimize the parameters of the human body parametric model, such as Figure 5As shown, the optimization is specifically divided into stages, including: stage one, optimizing the global displacement parameters and global rotation parameters of the human body parametric model; stage two, optimizing the body parameters (shape parameters) of the human body parametric model; stage three, optimizing the torso posture parameters by ignoring the key influence of the hand; stage four, increasing the influence of the hand key points to refine the arm posture parameters; and finally, obtaining the arm parameters of the human body parametric model, which refer to a part of the torso posture parameters in the model, specifically the arm posture parameters.

[0080] Step 4: Based on the two-dimensional hand key points of the current frame, the parameters of the hand model are obtained by optimizing the hand constraint function.

[0081] In the embodiment, based on the two-dimensional hand key points and combined with the camera parameters, the three-dimensional hand key points are calculated using cyclic epipolar geometry. Among them, M represents the number of key points of the hand, 3 represents the three-dimensional coordinates, and The spatial position of is taken as the target, and the state parameter β determined in step 4 is fixed to fit the posture parameter of the hand model The three-dimensional hand key points regressed by the model are consistent with For registration, the hand constraint function used is:

[0082]

[0083] Where i represents the index of the hand key point, ‖·‖ represents the two-norm, γ represents the global displacement, α represents the scaling factor, and p(θ i ) is the hand model’s posture prior, J(θ i ,β,γ,α) represents the finger key point positions calculated by the hand model based on the posture parameters of the current time step.

[0084] Step 5: Determine the shoulder joint position and elbow joint position according to the arm parameters of the human body parametric model. After determining the wrist joint position according to the parameters of the hand model, use a geometric method to determine the actual elbow joint position that matches the wrist joint position.

[0085] In the embodiment, Figure 6 As shown, after obtaining the arm parameters and hand model parameters of the human body parameterized model, the shoulder joint s position (x s ,y s ,z s ) and elbow joint position (x e ,y e ,z e ), the length of the upper arm is b u , the forearm length is b f The wrist joint w′ position (x w′ ,y w′,z w′ ), in order to adjust the arm posture and align the spatial position of the wrist key point, it is necessary to calculate the spatial coordinates (x e′ ,y e′ ,z e′ ). Because s(x s ,y s ,z s ), w′(x w′ ,y w′ ,z w′ ) is known in space, that is, the length and position of sw′ are known, and it is also known that se′=b u ,e′w′=b f , when the lengths of the three sides of a triangle are known, the unique shape of the triangle can be determined. Then, by rotating the triangle in space around side sw′ as the axis, the unique trajectory of e′ in space can be determined. The following formula can be obtained:

[0086]

[0087]

[0088]

[0089]

[0090] Due to the constraints of human joint angles, Must be greater than or equal to zero, o is defined as the origin of the world coordinate system, then the coordinates of the center of the circle c(x c ,y c ,z c ), can be obtained by space vector It can be concluded that the u(·) function is the normalization function of the vector:

[0091]

[0092]

[0093]

[0094] After determining the trajectory center and radius of e′, we need to calculate the original elbow key point e(x e ,y e ,z e ), the point closest to e on the trajectory circle is the actual elbow joint position e′(x e′ ,y e′ ,z e′ ),in, for direction vector, is a vector exist The projection on the vector, the final space coordinate of e′ (x e′ ,y e′ ,z e′ ), can be represented by the space vector It turns out that:

[0095]

[0096]

[0097]

[0098]

[0099] Step 6: Calculate the swing rotation matrix of the shoulder joint based on the actual elbow joint position and shoulder joint position. Use neural inverse kinematics to obtain the torsion rotation matrix of the shoulder joint based on the initial image and the end image of the shoulder joint movement. Update the swing rotation matrix and the torsion rotation matrix to the arm posture parameters of the human body parametric model.

[0100] In the embodiment, the posture parameter represents the original rotation matrix. The process of converting the original rotation matrix to the target rotation matrix requires a posture rotation. The posture rotation matrix can be expressed as R, which can be decomposed into the twist rotation matrix R tw and the swing rotation matrix R sw . From the initial rotation vector To target rotation vector Initial rotation vector In the initial state, the vector with the shoulder joint as the starting point and the elbow joint as the end point is a representation of the original rotation matrix. The target rotation vector It is a vector starting from the shoulder joint and ending at the elbow joint in the target state, which is a representation of the target rotation matrix. The attitude rotation matrix R can be expressed as:

[0101]

[0102] Where φ is the twist rotation angle estimated by the neural network, using Hybrik’s method, from the pixels of the initial image of the initial state and the ending image of the target state. sw (·) is the function used to solve the swing rotation matrix, D tw (·) is the function that converts the twist rotation angle φ into the twist rotation matrix. For the swing rotation, there is a perpendicular and Axis

[0103]

[0104] The swing angle α satisfies:

[0105]

[0106] Therefore, the swing rotation matrix R sw The solution of can be derived from the Rodrigues rotation formula, where yes is a skew-symmetric matrix, and I is the identity matrix.

[0107]

[0108] Twisting rotation is around The rotation of The axis itself is the axis, φ is the torsion angle, and the torsion rotation matrix R can be determined tw for:

[0109]

[0110] In the obtained R sw and R tw After that, the posture parameters of the human body parametric model are updated, specifically including: converting the original posture parameters of the shoulder joint into the original rotation matrix, multiplying the original rotation matrix by the twist rotation matrix R in sequence tw and the swing rotation matrix R sw , get the new rotation matrix of the shoulder joint after correcting the elbow joint, convert the new rotation matrix into the axis angles of the three axes x, y, and z to cover the posture parameters of the shoulder joint, and get the updated arm posture parameters of the human body parametric model.

[0111] In step 7, the human body parametric model with updated arm posture parameters is aligned with the hand model to complete the 3D human body modeling and motion acquisition.

[0112] In this embodiment, when aligning a human parametric model with updated posture parameters with a hand model, the surface point clouds of the human parametric model with updated arm parameters and the hand model are fused to create a patch index file for the missing wrist portion. Then, based on the patch index file for the missing wrist portion, the hand model and the human parametric model are aligned in the default hand posture to obtain a fused model. The parameters of the human parametric model and the hand model are imported into the fused model to obtain a human body reconstruction result. The aligned fused model, which contains posture parameters, enables motion capture.

[0113] Among them, the surface point cloud of the human body parametric model and the hand model with updated arm parameters is integrated to establish a patch index file of the missing wrist part, including: processing the surface point cloud of the human body parametric model, removing all the point clouds of the two hands and their associated triangular patches, and obtaining the human body parametric model without the hands; arranging the point cloud of the hand model in sequence after the human body parametric model without the hands; sorting the patch index file of the hand model so that the surfaces of the hand model and the human body parametric model coexist; stitching the missing wrist point cloud to establish a patch index file of the missing wrist part.

[0114] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting, characterized in that: The following steps are involved: Step 1: Obtain a tracking video of a person shot synchronously from multiple perspectives; Step 2: Extract 2D human body key points and 2D hand key points from each frame of the tracking video; Step 3: Based on the two-dimensional human body key points and by optimizing the multi-view human body constraint function, the arm parameters of the human body parametric model are obtained, wherein the arm parameters include arm shape parameters and arm posture parameters, including: Calculate the two-dimensional human key points ∏ of the human body parametric model projection under each viewing angle K (R θ (J(β)) i ), calculate the three-dimensional human key points R of the human body parameter model θ (J(β)) i , using cyclic epipolar geometry to calculate the key points of the 3D human body J 3d,i , then construct the two-dimensional key point constraint L for human body reconstruction 2d and 3D key point constraint L 3d , expressed as: Among them, R θ (J(β)) i It represents the coordinates of the three-dimensional human key points obtained by regression based on the hybrid linear skin deformation, which is obtained by the joint action of the posture parameters θ and shape parameters β of the human body parametric model. K (·) represents the parameterized model R of the human body using the camera parameter K θ (J(β)) i Projected to the camera's two-dimensional pixel coordinate system, ρ(·) represents the function describing the difference between the two coordinates, γ i ,ω i Represent the weight and confidence of the human key point i, J 2d,i Represents two-dimensional human body key points; Based on L 3d and L 2d Constructing multi-view human body constraint function L multi , expressed as: Based on L multi Construct the total loss function L total , expressed as: L total =min(L multi +λ p L prior ) Among them, v represents the camera perspective index, which is 0-n, λ v ,λ w and λ p Both represent loss weights, L 2d,v represents the two-dimensional key point loss under the camera view v, L prior Represents the prior loss of human joints; Using the total loss function L total Optimize arm parameters of human parametric model; Step 4: Optimize the hand constraint function based on the two-dimensional hand key points to obtain the parameters of the hand model, including: Based on the 2D hand key points and combined with the camera parameters, the 3D hand key points are calculated using cyclic epipolar geometry. With this The spatial position of is taken as the target, and the shape parameter β determined in step 4 is fixed to fit the posture parameter Θ of the hand model so that the three-dimensional hand key points regressed by the model are consistent with For registration, the hand constraint function used is: Where i represents the index of the hand keypoint, M is the number of hand keypoints, ‖·‖ represents the two-norm, γ represents the global displacement, α represents the scaling factor, and p(θ i ) is the hand model’s posture prior, J(θ i ,β,γ,α) represents the finger key point positions calculated by the hand model based on the posture parameters of the current time step; Step 5: Determine the shoulder joint position and the elbow joint position based on the arm parameters of the human body parametric model. After determining the wrist joint position based on the parameters of the hand model, determine the actual elbow joint position that matches the wrist joint position using a geometric method based on the shoulder joint position, the elbow joint position, and the wrist joint position. Step 6: Calculate the swing rotation matrix of the shoulder joint based on the actual elbow and shoulder joint positions. Use neural inverse kinematics to obtain the torsion rotation matrix of the shoulder joint based on the initial and final images of the shoulder joint motion. Update the swing rotation matrix and torsion rotation matrix into the arm posture parameters of the human body parametric model. In step 7, the human body parametric model with updated arm posture parameters is aligned with the hand model to complete the 3D human body modeling and motion acquisition.

2. The method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting according to claim 1, characterized in that: The total loss function L total Optimizing the arm parameters of the human body parametric model includes: optimizing the global displacement parameters and global rotation parameters of the human body parametric model in the first stage, optimizing the shape parameters of the human body parametric model in the second stage, optimizing the torso posture parameters by ignoring the key influence of the hand in the third stage, and refining the arm posture parameters by adding the influence of the hand key points in the fourth stage. Finally, the arm parameters of the human body parametric model are obtained. The arm parameters refer to a part of the torso posture parameters in the model, specifically the arm posture parameters.

3. The method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting according to claim 1, characterized in that: Calculate the key points of 3D human body using cyclic epipolar geometry 3d ,include: Among them, the cycle uses the epipolar geometry E for the two-camera combination v1 and v2 g Calculate the three-dimensional coordinates of the corresponding key points, and finally calculate the weight γ v1,v2 and confidence ω v1,v2 Weighted sum of three-dimensional coordinates to obtain the final three-dimensional human body key point coordinates J 3d , where λ v1,v2 is the proportional coefficient, J 2d,v1 , J 2d,v2 are the coordinates of the two-dimensional human key points under the two cameras, K v1 , K v2 Represent the parameters of the two cameras respectively.

4. The method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting according to claim 1, characterized in that: The method of determining the actual elbow joint position that matches the wrist joint position based on the shoulder joint position, the elbow joint position, and the wrist joint position by using a geometric method includes: Based on the shoulder joint position and actual elbow joint position ′ Determine the upper arm length b u , based on the actual elbow joint position e ′ and wrist joint position to determine the forearm length b f , determine another length sw based on the position of the shoulder joint and the wrist joint ′ , determine the unique triangle based on the three lengths, and use sw ′ As the axis, rotate the triangle in space for one circle, and you can determine e ′ The only trajectory circle in space. After determining the trajectory circle and radius, the point closest to e on the trajectory circle is obtained based on the known original elbow joint position e, which is the actual elbow joint position e. ′ .

5. The method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting according to claim 1, characterized in that: The step of calculating the swing rotation matrix of the shoulder joint according to the actual elbow joint position and the shoulder joint position, and obtaining the torsion rotation matrix of the shoulder joint by using neural inverse kinematics according to the initial image and the end image of the shoulder joint movement, includes: Determine the target rotation vector according to the actual elbow joint position and the shoulder joint position, and calculate the swing rotation matrix according to the target rotation vector and the initial rotation vector; The twist rotation angle is calculated according to the initial image and the end image of the shoulder joint movement, and the twist rotation matrix is ​​calculated according to the twist rotation angle and the initial rotation vector.

6. The method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting according to claim 1, characterized in that: The updating of the swing rotation matrix and the twist rotation matrix into the arm posture parameters of the human body parameterized model includes: Convert the original posture parameters of the shoulder joint into the original rotation matrix, and multiply the original rotation matrix by the twist rotation matrix R in sequence. tw and the swing rotation matrix R sw , get the new rotation matrix of the shoulder joint after correcting the elbow joint, convert the new rotation matrix into the axis angles of the three axes x, y, and z to cover the posture parameters of the shoulder joint, and get the updated arm posture parameters of the human body parametric model.

7. The method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting according to claim 1, characterized in that: The method of aligning the human body parameterized model with the hand model after updating the arm posture parameters includes: The missing wrist patch index file is created by fusing the human body parametric model with updated arm parameters and the surface point cloud of the hand model; Based on the missing wrist patch index file, the hand model is aligned with the human body parameterized model in the default hand posture. The aligned model can realize motion acquisition because it contains posture parameters.

8. The method for 3D human body modeling and motion acquisition based on multi-view synchronous shooting according to claim 7, characterized in that: The surface point cloud of the human body parametric model and the hand model updated by fusing the arm parameters is used to establish the missing wrist part patch index file, including: Processing the surface point cloud of the human body parametric model, removing all the point clouds of the hands and their associated triangular facets, and obtaining a human body parametric model without the hands; Arrange the point cloud of the hand model in sequence after the parametric model of the human body with both hands removed; Organize the patch index file of the hand model so that the surfaces of the hand model and the human body parametric model can coexist; The missing wrist point cloud is stitched together to create a missing wrist patch index file.

Images