A method for generating virtual human poses for group photos

Abstract

This invention discloses a method for generating virtual human poses for group photos, belonging to the field of software development technology. The method includes: employing a dual-camera capture system to collect user facial expression and body posture data respectively, and synchronizing the user's facial expression and body posture data over time to generate user posture and expression data; training the body data using an SMPL human model to generate virtual training data, generating posture feature vectors, and obtaining posture influence factors by performing logistic regression on the influence of posture feature vectors and posture and expression data on posture; incorporating user posture and expression data and posture influence factors as constraints into a 3D posture prediction model to fit an objective function, optimizing the objective function, dynamically adjusting the virtual human's posture, and achieving the best group photo between the user and the virtual human.

Description

Technical Field

[0001] This invention relates to the field of software development technology, and more specifically, to a method for generating virtual human poses for group photos. Background Technology

[0002] Virtual reality technology is increasingly being used in our daily lives. Users have higher requirements for the acquisition and realism of virtual characters, hoping to have a good virtual character image in a three-dimensional virtual environment. In many cases, there is also a need for interaction between the user image and the virtual character.

[0003] In some 3D virtual application scenarios, there is a need for photos of virtual humans and users together. The main process involves dynamically matching the poses of a digitized virtual human with the user's pose using computer graphics technology. Existing technologies primarily employ a solution that involves acquiring pre-existing virtual human characters within the application, which have several built-in poses. Then, an image recognition algorithm automatically matches these characters with the optimal solution for the user's pose, ultimately synthesizing a virtual image.

[0004] However, in actual use, the fixed poses cannot match the diverse group photo needs of users. Especially after adding surrounding light, shadow, and other environmental scenery, the fit curve between the virtual human's pose and the user's pose drops even more significantly. Even with standardized pose guidance, due to the diversity of user factors and the environment, it is still impossible to present group photos with coordinated movements and artistic beauty, thus failing to meet users' high-end experience requirements. Summary of the Invention

[0005] 1. Technical problems to be solved

[0006] To address the problems existing in the prior art, the purpose of this invention is to provide a method for generating virtual human poses for group photos. This invention samples the user's pose in real time, and can train the model based on the sampling. It then generates pose influencing factors through model regression, and dynamically adjusts the virtual human's pose through a 3D pose prediction model to enable users to take photos with virtual human figures.

[0007] 2. Technical Solution

[0008] To solve the above problems, the present invention adopts the following technical solution:

[0009] A method for generating virtual human poses for group photos. The method uses a dual-camera capture system to collect user facial expression and body pose data respectively, and synchronizes the user's facial expression and body pose data in time to generate user pose expression data.

[0010] Body data is used to train the SMPL human body model to generate virtual training data and generate posture feature vectors. Then, by performing logistic regression on the influence of posture feature vectors and posture expression data on the degree of posture influence, the posture influence factor is obtained.

[0011] By incorporating user posture and facial expression data and posture influencing factors as constraints into the 3D posture prediction model to fit the objective function, the objective function is optimized, and the virtual character's posture is dynamically adjusted to achieve the best photo of the user and the virtual character.

[0012] As a preferred embodiment of the present invention, the dual-camera capture system includes two Azure Kinect cameras, which respectively capture facial expressions and body postures in real time. The body posture data captured by the Azure Kinect cameras includes: three-dimensional coordinates of human joints and quaternions of joint rotation; the facial expression data captured by the Azure Kinect cameras includes: facial feature point data and tracked facial data.

[0013] As a preferred embodiment of the present invention, the facial expression data is obtained by extracting facial feature data through the CLNF model (Constrained Local Neural Field), obtaining facial labels, and constructing a facial model using the Shape model and Patch model to obtain expression classification and regression.

[0014] As a preferred embodiment of the present invention, the time synchronization of user facial expression and body posture data specifically includes the following steps:

[0015] S11: Spatial calibration of the dual cameras, and timestamps added to the captured facial images, encapsulating them into a facial expression data structure;

[0016] S12: Add timestamps to the collected body poses and encapsulate them into body pose data.

[0017] S13: Match the timestamps corresponding to facial expression data and body posture data, and recombine them according to the order of the timestamps to generate facial expression and body posture data with a time slot error of less than T.

[0018] As a preferred embodiment of the present invention, in generating facial expression and body posture data with a time slot error of less than T, T is 0.5-1.5μs.

[0019] As a preferred embodiment of the present invention, body data is trained using the SMPL human body model to generate virtual training data, specifically including the following steps:

[0020] S21: Extract joint information from body data and obtain SMPL model through HMR algorithm; obtain β and θ parameters of human body using HMR algorithm, then obtain 3D joint position of SMPL through θ, and then fit the pose of SMPL model to Kinect joint.

[0021] S22: The spatial position of each joint is described by three values, X, Y and Z, through the ICP algorithm. The X and Y values ​​are the positions relative to the bone plane space. The bone coordinate points are transformed into the corresponding depth data image and the SMPL model is optimized.

[0022] S23: By projecting the optimized SMPL model onto the color image, virtual training data is generated to obtain virtual human posture imaging.

[0023] As a preferred embodiment of the present invention, a posture feature vector is generated, and a posture influence factor is obtained by performing logistic regression on the posture feature vector and posture expression data to assess the degree of posture influence. This specifically includes the following steps:

[0024] S31: Transfer the joint information R(β) from the body data i Form a set and calculate the attitude angle θ between all adjacent joints in the set. i The attitude feature vector I(θ) is calculated using the Rodriguez formula. i ), the β i This represents the human posture parameters, specifically the posture angle θ. i Indicates the axial angle of rotation between adjacent joints;

[0025] S32: The Holt two-parameter exponential smoothing algorithm is used to smoothly predict facial mood parameters, and the matched facial mood Q is obtained by matching the pose. i ;

[0026] S33: By determining the changes in pose features between the user's pose and the virtual human's pose, a matching degree result Y is output. A matching degree of 1 is defined as a match, and a matching degree of 0 is defined as a mismatch.

[0027] S34: Construct a pose feature vector I(θ) i ) and facial expressions Q i Let X be the independent variable j The matching result Y is a binary logistic regression model with the dependent variable. The main influencing factors affecting the matching degree Y are identified, and factor analysis is performed to obtain the component matrix.

[0028] As a preferred embodiment of the present invention, the user's posture and facial expression data φ nThe posture influencing factor is added as a constraint term to the 3D posture prediction model to fit the objective function, thereby optimizing the objective function and dynamically adjusting the virtual character's posture to achieve the best photo of the user and the virtual character. The specific steps include:

[0029] S41: By transforming user gesture and facial expression data from the original feature space to a new feature space through feature engineering;

[0030] S42: Combining attitude influence factors, high-frequency and low-frequency detail features are extracted from feature data using multi-scale wavelet analysis.

[0031] S43: Combining high-frequency and low-frequency detail features, a seq2seq pose prediction model is used to predict and adjust the pose of virtual characters.

[0032] In a preferred embodiment of the present invention, the seq2seq attitude prediction model predicts attitude data within the next 200-500ms. In other embodiments, attitude prediction models with longer prediction times may also be used.

[0033] As a preferred embodiment of the present invention, the determination of the change in posture features between the user's posture and the virtual human's posture is achieved by dividing the posture feature vector I(θ) into... i The model's 3D key locations X(θ,β) are projected onto a color image, and a fusion and matching evaluation system is learned through neural network training. The neural network model uses a three-layer network to classify the training samples and obtain the matching results, which include "matching" and "non-matching" virtual human poses, and assign values ​​to the matching scores. In the neural network sample training, 70% of the data is used as training samples and 30% as test samples. The neural network outputs 256, 128, and 2 from bottom to top, respectively. The final output matching scores are 2, which are assigned values ​​of 0 for non-matching and 1 for matching.

[0034] 3. Beneficial effects

[0035] Compared with the prior art, the advantages of this invention are:

[0036] This invention acquires facial expressions and body postures using dual cameras and synchronizes them to generate user posture and expression data. By training the body data with an SMPL human model, posture feature vectors are obtained. Logistic regression is then performed on the influence of posture feature vectors and posture and expression data on posture to obtain a posture influence factor. Finally, the user posture and expression data and the posture influence factor are incorporated as constraints into a 3D posture prediction model for optimal photo capture between the user and a virtual character. This method achieves good fitting results, and the virtual character's posture can dynamically adjust in real-time according to the user's posture, expanding and enriching the application scenarios of virtual characters. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating the steps of a virtual human pose generation method for group photos according to the present invention;

[0038] Figure 2 These are the joints of the SMPL human model after training in the virtual human pose generation method for group photos of the present invention.

[0039] Figure 3 This invention relates to a binary logistic regression model in a virtual human pose generation method for group photos.

[0040] Figure 4 This invention uses a binary logistic regression model to identify some of the main influencing factors.

[0041] Figure 5 This invention uses a binary logistic regression model to identify another major influencing factor.

[0042] Figure 6 This is a schematic diagram illustrating the effect of the method of the present invention in a practical application scenario.

[0043] The reference numerals used in the above figures are explained as follows:

[0044] 10. Dual cameras

[0045] 20. Facial Data

[0046] 30. Body Data

[0047] 40. Match timestamps

[0048] 50. Posture and Facial Expression Data

[0049] 60. SMPL Human Body Model

[0050] 70. Attitude Feature Vector

[0051] 80. Attitude Influence Factor

[0052] 90. 3D Attitude Prediction Model

[0053] 1. Skull

[0054] 2. Neck joint

[0055] 3. Spinal joints

[0056] 4. Left shoulder joint

[0057] 5. Left elbow joint

[0058] 6. Left wrist joint

[0059] 7. Left hand joint

[0060] 8. Right shoulder joint

[0061] 9. Right elbow joint

[0062] 11. Right wrist joint

[0063] 12. Right hand joint

[0064] 13. Lumbar joint

[0065] 14. Pelvic joint

[0066] 15. Left femoral joint

[0067] 16. Left knee joint

[0068] 17. Left ankle joint

[0069] 18. Left toe joint

[0070] 19. Right femoral joint

[0071] 21. Right knee joint

[0072] 22. Right ankle joint

[0073] 23. Right toe joint. Detailed Implementation

[0074] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0075] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0076] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0077] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0078] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0079] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0080] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0081] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0082] Unless otherwise expressly specified or limited, the terms "installation," "connection," "linking," "fixing," and "setting," as used in the description of the embodiments of this application, should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two components or the interaction between two components. For those skilled in the art to which this application pertains, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0083] Please see Figure 1-6 This invention provides a method for generating virtual human poses for group photos. The method employs a dual-camera capture system to separately acquire user facial expression and body posture data, and synchronizes the user's facial expression and body posture data over time to generate user pose and expression data φ. n ;

[0084] Body data is used to train a parametric human body model using SMPL to generate virtual training data, and a classifier is trained to obtain the posture influence factor.

[0085] User gesture and facial expression data φ n The posture influencing factor is added as a constraint term to the 3D posture prediction model to fit the target number of rows, thereby optimizing the objective function and dynamically adjusting the virtual character's posture to achieve the best photo of the user and the virtual character.

[0086] In a specific embodiment of the present invention, the dual-camera capture system includes two Azure Kinect cameras, which respectively capture facial expressions and body postures in real time. The body posture data captured by the Azure Kinect cameras includes: three-dimensional coordinates of human joints and joint rotation quaternions; the facial expression data captured by the Azure Kinect cameras includes: facial feature point data and tracked facial data. The facial expression data is extracted using a CLNF (Constrained Local Neural Field) model to obtain facial labels, and a Shape model and Patch model are used to construct a facial model to obtain expression classification and regression. This CLNF model can update the virtual character's facial expressions in real time according to the frame rate to obtain more realistic and vivid facial animation effects.

[0087] In a specific embodiment of the present invention, time synchronization of user facial expressions and body posture data can significantly reduce errors in subsequent data fusion processes. This allows for a better virtual character effect in three-dimensional space.

[0088] Specifically, the following steps are included:

[0089] S11: Spatial calibration of the dual cameras, and timestamps added to the captured facial images, encapsulating them into a facial expression data structure;

[0090] S12: Add timestamps to the collected body poses and encapsulate them into body pose data.

[0091] S13: Match the timestamps corresponding to the facial expression data and body posture data, and recombine them according to the order of the timestamps to generate facial expression and body posture data with a time slot error of less than T. In the generated facial expression and body posture data with a time slot error of less than T, T is 0.5-1.5μs.

[0092] The aforementioned timestamps are synchronized using a TCP network transmission timestamp synchronization method to synchronize facial expression and body posture data. During TCP transmission, a standard data table encapsulation mode is used to avoid packet loss and delays, improving data transmission efficiency and ensuring the real-time display quality of the image.

[0093] In a specific embodiment of the present invention, body data is trained using the SMPL human body model to generate virtual training data, specifically including the following steps:

[0094] S21: Extract joint information from body data. The SMPL model binds each vertex to several joints and assigns weights to each joint. The SMPL model is obtained through the HMR algorithm. The β and θ parameters of the human body are obtained using the HMR algorithm. Then, the 3D joint positions of the SMPL are obtained through θ. Finally, the pose of the SMPL model is fitted to the Kinect joints.

[0095] S22: The spatial position of each joint is described by three values, X, Y and Z, through the ICP algorithm. The X and Y values ​​are the positions relative to the bone plane space. The bone coordinate points are transformed into the corresponding depth data image and the SMPL model is optimized.

[0096] S23: By projecting the optimized SMPL model onto the color image, virtual training data is generated to obtain virtual human pose imaging.

[0097] For details, please refer to the following: Figure 3 As shown, in a specific embodiment of the present invention, a posture feature vector is generated, and a posture influence factor is obtained by performing logistic regression on the posture feature vector and posture expression data to assess the degree of influence of posture. This specifically includes the following steps:

[0098] S31: Transfer the joint information R(β) from the body data i Form a set and calculate the attitude angle θ between all adjacent joints in the set. i The attitude feature vector I(θ) is calculated using the Rodriguez formula. i ), the β i This represents the human posture parameters, specifically the posture angle θ. i Indicates the axial angle of rotation between adjacent joints;

[0099] S32: The Holt two-parameter exponential smoothing algorithm is used to smoothly predict facial mood parameters, and the matched facial mood Q is obtained by matching the pose. i ;

[0100] S33: By determining the changes in pose features between the user's pose and the virtual human's pose, a matching degree result Y is output. A matching degree of 1 is defined as a match, and a matching degree of 0 is defined as a mismatch.

[0101] S34: Construct a pose feature vector I(θ) i ) and facial expressions Q i Let X be the independent variable j The matching result Y is a binary logistic regression model of the dependent variable. The main influencing factors affecting the matching degree Y are identified, and factor analysis is performed to obtain the component matrix. In this embodiment, the independent variables X1-X5 are combinations of partial joint information, and the independent variable X6 is facial expression.

[0102] In a specific embodiment of the present invention, the user's gesture and facial expression data φ n The posture influencing factor is added as a constraint term to the 3D posture prediction model to fit the objective function, thereby optimizing the objective function and dynamically adjusting the virtual character's posture to achieve the best photo of the user and the virtual character. The specific steps include:

[0103] S41: By transforming user gesture and facial expression data from the original feature space to a new feature space through feature engineering;

[0104] S42: Combining attitude influence factors, high-frequency and low-frequency detail features are extracted from feature data using multi-scale wavelet analysis.

[0105] S43: Combining high-frequency and low-frequency detail features, a seq2seq pose prediction model is used to predict and adjust the pose of virtual characters.

[0106] In a preferred embodiment of the present invention, the seq2seq attitude prediction model predicts attitude data within the next 200-500ms. In other embodiments, attitude prediction models with longer prediction times may also be used.

[0107] As a preferred embodiment of the present invention, the determination of the change in posture features between the user's posture and the virtual human's posture is achieved by dividing the posture feature vector I(θ) into... i The model's 3D key locations X(θ,β) are projected onto a color image, and a fusion and matching evaluation system is learned through neural network training. The neural network model uses a three-layer network to classify the training samples and obtain the matching results, which include "matching" and "non-matching" virtual human poses, and assign values ​​to the matching scores. In the neural network sample training, 70% of the data is used as training samples and 30% as test samples. The neural network outputs 256, 128, and 2 from bottom to top, respectively. The final output matching scores are 2, which are assigned values ​​of 0 for non-matching and 1 for matching.

[0108] This invention acquires facial expressions and body postures using dual cameras, and synchronizes these to generate user posture and expression data. By training a SMPL human model on the body data, posture feature vectors are obtained. Logistic regression is then performed on the influence of posture feature vectors and posture and expression data on posture to obtain a posture influence factor. Finally, the user posture and expression data and the posture influence factor are used as constraints in a 3D posture prediction model to achieve optimal photo capture between the user and a virtual character. This method exhibits good fitting results, and the virtual character's posture can dynamically adjust in real-time according to the user's posture, expanding and enriching the application scenarios of virtual characters. (Reference) Figure 6As shown, the left side is the user's posture, and the right side is the virtual human's posture. The method of this invention can effectively demonstrate the effect. The virtual human's posture can be adjusted in real time according to the user's posture, resulting in a good group photo effect.

[0109] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A virtual human pose generation method for group photos, characterized in that, The method comprises the following steps: The method adopts a double camera capturing system to respectively collect user facial expression data and body posture data, and performs time synchronization on the user facial expression data and body posture data to generate user posture expression data; The body data is trained through an SMPL human body model to generate virtual training data, generate a posture feature vector, and perform logistic regression on the posture feature vector and the posture expression data to obtain a posture influence factor; The user posture expression data and the posture influence factor are added as constraint terms to a 3D posture prediction model fitting target function to realize optimization of the target function, dynamically adjust a virtual character posture, and realize best photo shooting of the user and the virtual character; The body data is trained through an SMPL human body model to generate virtual training data, and the specific steps include the following steps: S21: Extract the joint information in the body data, and obtain the SMPL model through an HMR algorithm; S22: The ICP algorithm is used to describe the spatial position of each joint node by X, Y and Z values, and the X and Y values are the positions relative to the bone plane space. The bone coordinate points are converted into the corresponding depth data image to optimize the SMPL model; S23: The optimized SMPL model is superimposed on the color image through projection to generate virtual training data and obtain a virtual human body posture image; The posture feature vector is generated, and logistic regression is performed on the posture feature vector and the posture expression data to obtain a posture influence factor, and the specific steps include the following steps: S31: Transfer joint information from body data Form a set and calculate the attitude angles between all adjacent joints in the set. The attitude feature vector is calculated using the Rodriguez formula. The Represents human posture parameters, the posture angles Indicates the axis angle of rotation between adjacent joints; S32: Adopt Holt double-parameter exponential smoothing algorithm to smooth and predict the facial expression parameters, and match the gestures to obtain the matched facial expressions ; S33: output a matching degree result by judging the change of the posture feature between the user posture and the virtual human posture ; S34: construct with attitude feature vector and facial expression for the independent variable , matching degree results binary logistic regression model with dependent variable, get the main influencing factors affecting matching degree and factor analysis, get the component matrix; User pose expression data is obtained And the pose influence factor is added to the 3D pose prediction model fitting target function as a constraint term, the target function optimization is realized, the virtual character pose is dynamically adjusted, the best photo of the user and the virtual character is realized, and the specific steps include the following steps: S41: The user posture expression data is converted from the original space to the new feature space through feature engineering; S42: High-frequency detail features and low-frequency detail features are extracted from the feature data by using a multi-scale wavelet analysis method in combination with the posture influence factor; S43: The high-frequency detail features and the low-frequency detail features are combined, and a seq2seq posture prediction model is used to predict and adjust the virtual character posture; The seq2seq posture prediction model predicts the posture data within 200-500 ms in the future; The method further comprises: determining the posture feature changes between the user posture and the virtual human posture, and determining the posture feature vector with the model three-dimensional key positions projected onto the color image, and learning the fusion degree and matching degree evaluation system through neural network training.

2. The virtual human pose generation method for group photo according to claim 1, wherein, The double camera capturing system comprises two Azure Kinect cameras, which are used to collect facial expression data and body posture data in real time. The body posture data collected by the Azure Kinect camera includes human joint three-dimensional coordinates and joint rotation quaternions. The facial expression data collected by the Azure Kinect camera includes facial feature point data and tracked facial data. 3.The virtual human pose generation method for group photo according to claim 1, wherein, The facial expression data is extracted through a CLNF model to obtain facial labels, and a Shape model and a Patch model are used to construct a facial model to obtain expression classification and regression.

4. The virtual human pose generation method for group photo according to claim 3, characterized in that, The specific steps of time synchronization of the user facial expression data and the body posture data include the following steps: S11: Space calibration is performed on the double cameras, and the collected facial images are added with time stamps and encapsulated into facial expression data structures; S12: The collected body posture is added with a time stamp and encapsulated into body posture data, S13: Match the time stamps corresponding to the facial expression data and the body posture data, recombine according to the time stamps in chronological order, and generate facial expression and body posture data with time slot error less than T.

5. The virtual human pose generation method for group photo according to claim 4, wherein, In the generated facial expression and body posture data with time slot error less than T, T is 0.5-1.5µs.

6. The virtual human pose generation method for group photo according to claim 1, wherein, The neural network model adopts a three-layer network, classifies the training samples, and obtains a matching degree result.

Images