High-fidelity hand avatar generation method based on feature embedding and smoothing strategy
Through the method based on feature embedding and partial perception smoothing strategies, the problem of fine reconstruction and rendering of hand geometry is solved, real-time high-fidelity rendering and animation effects of hand avatars is realized, suitable for fields such as virtual reality, and enhanced user immersion and interactivity.
Patent Information
- Application Number
- CN202411055064.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art has difficulties in fine reconstruction and high-fidelity rendering of hand geometry, making it difficult to capture hand details, resulting in a lack of realistic rendering effect, and high-quality rendering methods are not feasible in real-time applications and are difficult to generalize to the end. Seen postures and shapes.
Using a method based on feature embedding and partially perceptual smoothing strategy, the feature embedding module and neural network renderer are trained through a multi-view hand image dataset to generate high-fidelity hand rendered images, real-time response and animation effects of hand postures are realized, combined with Spherical harmonics and Laplace regularization, optimize the hand mesh model to maintain detail and reduce artifacts.
Real-time rendering of hand avatars is realized, fine geometric details such as muscles and skin folds are captured, and high-fidelity visual effects are maintained. It is suitable for application scenarios that require fast response and high-quality images, and has high generalization capabilities.
Smart Images

Figure CN119006708A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a hand avatar generation method in the field of three-dimensional reconstruction, and in particular to a high-fidelity hand avatar generation method based on feature embedding and part-aware smoothing strategy. Background Art
[0002] Hand avatars play a key role in digital applications, especially in the fields of virtual reality, digital entertainment, and human-computer interaction, where they enhance users' immersion and facilitate natural interactions in virtual environments.
[0003] Although previous studies have focused on realistic hand rendering, little attention has been paid to the fine reconstruction of hand geometry, which is critical for high-quality rendering. Existing techniques typically rely on mesh-based representations, which are limited by fixed topology and limited resolution and have difficulty accurately representing details. Emerging neural implicit representation techniques have shown strong capabilities in synthesizing new views of static scenes, and some studies have extended these methods to the field of movable objects such as the human body to facilitate realistic rendering. However, these neural implicit representation techniques require a lot of time resource consumption to achieve realistic rendering. In the field of extended reality and gaming, the requirement for real-time rendering is very important, which requires technology to be able to achieve fast response while maintaining high rendering quality.
[0004] In general, the existing technologies have the following defects in the field of high-fidelity hand avatar generation:
[0005] 1. Existing techniques have difficulty in recovering the detailed geometry of the hand, which plays a key role in realistic rendering.
[0006] 2. Although some methods can achieve real-time rendering, they usually fail to capture the fine details of the hand mesh, resulting in a lack of realism in the rendering effect. Some high-quality rendering methods require a lot of time during training, which limits their feasibility in real-time applications.
[0007] 3. Existing methods have difficulty generalizing to unseen poses and shapes, which limits their scope of application. Summary of the invention
[0008] In order to solve the problems existing in the background technology, the present invention provides a high-fidelity hand avatar generation method based on feature embedding and part-aware smoothing strategy, which is used to obtain hand geometry structures with detailed textures and high-fidelity hand rendering images. At the same time, it can achieve real-time response to various hand postures and generate real-time animation effects of various hand postures. It can be used in virtual reality, augmented reality, games, human-computer interaction and other digital environments to enhance user immersion and achieve natural interaction.
[0009] The technical solution adopted by the present invention comprises the following steps:
[0010] 1. A high-fidelity hand avatar generation method based on feature embedding and smoothing strategy
[0011] S1: Obtain a multi-view hand image dataset, which contains multiple hand image samples from different viewpoints;
[0012] S2: construct an initial hand mesh model based on a hand image sample and generate mean mesh model features corresponding to three feature embedding modules;
[0013] S3: Based on the initial hand mesh model and the mean mesh model features corresponding to the three feature embedding modules, the three feature embedding modules and the neural network renderer are trained separately using the multi-view hand image dataset to obtain the optimized three feature embedding modules and the final neural network renderer as well as the updated hand mesh model;
[0014] S4: For the actual hand image sequence, combined with the current hand mesh model, the three current feature embedding modules are iteratively trained according to a hand image to obtain the optimized three feature embedding modules and the updated hand mesh model; then the updated hand mesh model is rendered using the final neural network renderer to obtain a high-fidelity 2D hand rendering image corresponding to the hand image;
[0015] S5: Repeat S4 to render other hand images in the actual hand image sequence in sequence to obtain corresponding high-fidelity 2D hand rendering images, thereby obtaining a hand rendering image sequence.
[0016] The S2 is specifically:
[0017] S2.1: constructing an initial hand mesh model according to the masked image and posture parameters in the current hand image sample, and using the linear mixed skinning weights of the initial hand mesh model as the mean mesh model features of the first feature embedding module;
[0018] S2.2: Combining the spherical harmonic function and the camera parameters under the current viewing angle, the initial hand mesh model is used to construct the mean mesh model features corresponding to the second feature embedding module and the mean mesh model features corresponding to the third feature embedding module.
[0019] The S3 is specifically:
[0020] S3.1: According to the current hand image sample, the mean grid model features corresponding to the three feature embedding modules, and the current hand grid model, the three feature embedding modules are trained iteratively respectively to obtain three trained feature embedding modules and update the hand grid model;
[0021] S3.2: training the current neural network renderer iteratively according to the current hand mesh model and the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module to obtain an updated neural network renderer;
[0022] S3.3: Based on the latest feature embedding module, hand mesh model, and neural network renderer, repeat S3 and S4, use other hand image samples in the multi-view hand image dataset to train the three feature embedding modules, hand mesh model, and neural network renderer, and obtain the final three feature embedding modules, hand mesh model, and neural network renderer.
[0023] The S2.1 is specifically:
[0024] According to the masked image under a viewing angle and its corresponding posture parameters, the original mesh model is constructed using the MANO model. The resolution of the original mesh model is then improved through the subdivision method to generate an initial hand mesh model. The linear mixed skinning weights of the initial hand mesh model are used as the mean mesh model features of the first feature embedding module.
[0025] The S2.2 is specifically:
[0026] Initialize the feature vector of hand vertex offset and the feature vector of vertex albedo, and then render the initial hand mesh model in combination with spherical harmonics and camera parameters at the current viewing angle to obtain a first 2D hand image, and iterate the feature vector of hand vertex offset and the feature vector of vertex albedo according to the first loss value calculated from the first 2D hand image and the masked image at the current viewing angle until the training is completed to obtain the optimal feature vector of hand vertex offset and the feature vector of vertex albedo, and use the optimal feature vector of hand vertex offset as the mean mesh model feature of the second feature embedding module, and the optimal feature vector of vertex albedo as the mean mesh model feature of the third feature embedding module.
[0027] The S3.1 is specifically:
[0028] S3.1.1: According to the hand posture at the current viewpoint, the mean mesh model features and the current latent space feature vector, three feature embedding modules are used to predict the hand vertex offset, vertex albedo and linear blend skin weight respectively, and then the current hand mesh model is optimized according to the predicted hand vertex offset, vertex albedo and linear blend skin weight to obtain the optimized hand mesh model;
[0029] S3.1.2: After rendering the optimized hand mesh model, obtain a second 2D hand image;
[0030] S3.1.3: Calculate a second loss value according to the second 2D hand image and the masked image at the current viewing angle, and adjust and update the latent space feature vectors of the three feature embedding modules according to the second loss value;
[0031] S3.1.4: Repeat S3.1.1-S3.1.3 to iteratively train the latent space feature vectors of the three feature embedding modules until the training is completed, and obtain the optimal latent space feature vectors and the optimal hand mesh model corresponding to the three feature embedding modules and update them.
[0032] In S3.1, each feature embedding module includes a posture decoder, which concatenates the hand posture under the current viewing angle and the corresponding latent space feature vector Q and then inputs them into the posture decoder. The output of the posture decoder is multiplied by the mapping matrix and then added to the corresponding mean grid model feature. The result is used as the predicted output of the current feature embedding module.
[0033] The S3.2 is specifically:
[0034] S3.2.1: Use the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module to construct a neural network renderer. The rendering process of the neural network renderer is as follows:
[0035] C(π i )=c(x,N,h,d)
[0036] h=concat(Q D , Q ρ , Q render )
[0037] Among them, C(π i ) is the camera parameter π from the i-th viewpoint i The synthesized image, c() is the rendering function, x and N are the position and normal obtained by interpolating the triangle patch vertices at the intersection, d is the ray direction, h is the fused feature vector, concat() represents the splicing operation, Q D and Q ρ are the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module respectively; Q render is the feature vector for neural rendering;
[0038] S3.2.2: The neural network renderer is trained iteratively based on the loss of the 2D hand rendering image obtained by the neural network renderer and the masked image at the current viewing angle, and an updated neural network renderer is obtained after the training is completed.
[0039] In S3.1.3, the second loss value includes a Laplace smoothing loss value The calculation formula is as follows:
[0040]
[0041] A=L*f=[A1,A2,...,A i ]
[0042]
[0043] Where A is the Laplace characteristic matrix, φ pLap is the layer weight, L is the Laplacian matrix, f is the feature of the vertex attribute, A1, A2, A i are the 1st, 2nd and i-th eigenvalues of the Laplace characteristic matrix, p1 and p2 are the first and second thresholds, γ1, γ2 and γ3 are the first to third balancing coefficients, respectively.
[0044] In S3.2.2, the loss between the 2D hand rendering image obtained by the neural network renderer and the masked image at the current viewing angle The specific formula is as follows:
[0045]
[0046] Among them, C(π i ) is the camera parameter π from the i-th viewpoint i Synthesized image, ‖‖1 represents the L1 norm, represents the image similarity loss term, w is the third weight, I i It is the masked image at the current viewing angle.
[0047] 2. A computer device
[0048] The device comprises a memory and a processor, wherein the memory stores a computer program, and the steps of the method are implemented when the processor executes the computer program.
[0049] 3. A computer-readable storage medium
[0050] The medium stores a computer program, which implements the steps of the method when executed by a processor.
[0051] The beneficial effects of the present invention are:
[0052] By adopting the above technical scheme, the method of the present invention can capture the fine geometric details of the hand, including muscles, skin wrinkles, etc., and can realize real-time rendering of hand avatars while maintaining high-fidelity visual effects, which is crucial for application scenarios that require fast response and high-quality images.
[0053] The present invention introduces part-aware Laplace regularization to eliminate unwanted artifacts while maintaining details through regularization at different levels. Under the training of hand image data based on multi-view video input, the present invention can generate hand avatars in different postures with high practicality and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is an overall flow chart of a high-fidelity hand avatar generation method based on feature embedding and part-aware smoothing strategy according to an embodiment of the present invention.
[0055] Figure 2 It is a feature embedding module of an embodiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the above-mentioned objects, features and advantages of the present invention simpler and easier to understand, the present invention is described in detail below in conjunction with the drawings in the embodiments of the present invention, and the technical solutions in the embodiments of the present invention are clearly and completely described, but it does not constitute a limitation of the present invention.
[0057] The hardware platform of this embodiment uses an Intel i9 12900X CPU and an NVIDIA RTX 3090 graphics card. The system program is written in Python and uses pytorch, numpy, opencv, and nvdiffrast libraries.
[0058] like Figure 1 As shown, the embodiments of the present invention and their implementation process are as follows:
[0059] S1: Use cameras and other devices to collect and obtain a multi-view hand image dataset, which contains multiple hand image samples from different perspectives. Each hand image sample contains the original hand image and the corresponding mask image, masked image, hand posture and posture parameters; the white in the mask image is the object, the black is the background, and the masked image is aligned with the corresponding original image. The camera parameters are the camera intrinsic parameter matrix K and the rotation matrix R and translation vector t of the camera extrinsic parameter.
[0060] S2: construct an initial hand mesh model based on a hand image sample and generate mean mesh model features corresponding to three feature embedding modules;
[0061] S2 is specifically:
[0062] S2.1: constructing an initial hand mesh model according to the masked image and posture parameters in the current hand image sample, and using the linear mixed skinning weights of the initial hand mesh model as the mean mesh model features of the first feature embedding module;
[0063] S2.1 specifically:
[0064] According to the masked image under a viewing angle and its corresponding posture parameters, the original mesh model is constructed using the MANO model. The MANO model is used to initialize the hand geometry, and then the resolution of the original mesh model is improved by the subdivision method to generate a high-resolution initial hand mesh model. The linear blending skin weight of the initial hand mesh model (the linear blending skin weight is a property of the MANO model) is used as the mean mesh model feature of the first feature embedding module. Among them, the parameterized hand model has a unified topological structure, which makes it easier to achieve dynamic deformation effects. The present invention uses a parameterized MANO hand model as a basic model to initialize the hand geometry. The MANO model includes a set of predefined bones and weights, and the posture parameters and shape parameters of the skeleton key points can be mapped and adjusted to adjust the shape of the hand mesh through the linear blending skinning technology. The specific deformation formula is as follows:
[0065]
[0066] in, is the deformed mesh model, is the template mesh model, W is the weight value, θ is the pose parameter, β is the shape parameter, and Ω is the linear blend skin map.
[0067] Since the number of vertices of the basic MANO model is not enough to reflect fine texture details, the resolution of the hand model is improved through the subdivision method to generate a high-resolution mesh model. The subdivided mesh model has more vertices and three-dimensional faces, which is helpful for the subsequent fine-grained hand reconstruction. The subdivision algorithm divides the face of a mesh model into more small faces, thereby increasing the number of vertices and faces of the mesh model to achieve a fine representation of hand details. The specific subdivision formula is as follows:
[0068]
[0069] in, is the high-resolution template mesh model after subdivision, W′ is the linear blend skin weight after subdivision, represents the subdivision algorithm, is the initial template mesh model, and W is the initial linear blend skin weight.
[0070] S2.2: Combining the spherical harmonic function and the camera parameters under the current viewing angle, the initial hand mesh model is used to construct the mean mesh model features corresponding to the second feature embedding module and the mean mesh model features corresponding to the third feature embedding module.
[0071] S2.2 specifically:
[0072] The masked image of the first frame of the video sequence is used as the reference image, and the eigenvector D of the hand vertex offset and the eigenvector i of the vertex albedo are initialized. The eigenvector D of the hand vertex offset and the eigenvector ρ of the vertex albedo are both eigenvectors with a value of 0 and a dimension of (N, 3). N is the number of vertices in the hand mesh model. The eigenvectors of the hand vertex offset and the eigenvectors of the vertex albedo are both learnable parameters. Then, the initial hand mesh model is rendered in combination with the spherical harmonic function and the camera parameters at the current viewing angle to obtain the first 2D hand image. Specifically, the Lambertian reflection model is used to calculate the rendered image determined by the vertex albedo, vertex normal and ambient lighting coefficient. The ambient lighting is represented by the spherical harmonic function, and the specific formula is as follows:
[0073] B(π i ) = ρ·SH(G, N)
[0074] Among them, B(π i ) is the camera parameter π from the i-th viewpoint i The resulting rendered image, ρ is the vertex albedo, SH() is the third-order spherical harmonic function, G is the ambient lighting coefficient, and N is the vertex normal.
[0075] Then, according to the first loss value calculated from the first 2D hand image and the masked image at the current viewing angle, the feature vector of the hand vertex offset and the feature vector of the vertex albedo are trained iteratively until the training is completed, that is, the difference between the rendered image and the reference image is minimized, and the optimal feature vector of the hand vertex offset and the feature vector of the vertex albedo are obtained. The optimal feature vector of the hand vertex offset is used as the mean grid model feature of the second feature embedding module, and the optimal feature vector of the vertex albedo is used as the mean grid model feature of the third feature embedding module. The specific formula is as follows:
[0076]
[0077] Where L is the Laplacian matrix, D is the vertex offset, ρ is the vertex albedo, and B(π i ) is the camera parameter π from the i-th viewpoint i The resulting rendered image, I i It is the masked image at the current viewing angle.
[0078] S3: Based on the initial hand mesh model and the mean mesh model features corresponding to the three feature embedding modules, the three feature embedding modules and the neural network renderer are trained separately using the multi-view hand image dataset to obtain the optimized three feature embedding modules and the final neural network renderer as well as the updated hand mesh model;
[0079] S3.1: According to the hand posture at the current viewing angle, the mean grid model features corresponding to the three feature embedding modules, and the current hand grid model, the three feature embedding modules are trained and iterated respectively, and three trained feature embedding modules are obtained and updated, as well as the hand grid model;
[0080] S3.1 specifically:
[0081] S3.1.1: When training the first masked image, the latent space feature vectors of the three feature embedding modules need to be initialized, namely, the latent space feature vectors corresponding to the vertex offset, vertex albedo, and linear blend skin weight are Q D , Q ρ , Q lbs , the length is 10 and the initial value is 0. The latent space feature vector is a learnable parameter. When training the second and subsequent masked images, the previous optimal latent space feature vector is inherited. According to the hand posture, mean mesh model features and current latent space feature vector at the current viewing angle, three feature embedding modules are used to predict the hand vertex offset, vertex albedo and linear blended skin weight respectively, and then the current hand mesh model is further optimized according to the predicted hand vertex offset, vertex albedo and linear blended skin weight to obtain the optimized hand mesh model; that is, the vertex offset, linear blended skin weight and vertex albedo of the hand mesh model can be converted through the feature embedding module to realize the encoding and acquisition of posture-dependent features, thereby optimizing the geometric structure of the hand mesh model. The specific formula is:
[0082]
[0083] in, is the optimized hand mesh model, that is, the deformable hand mesh model with detailed textures. is the hand mesh model before optimization, D t is the vertex offset at the corresponding time, W′ t is the linear blend skin weight at the corresponding time, θ t is the hand pose parameter at the corresponding moment, β is the shape parameter, Ω() is the linear blend skinning map, is the mean vertex offset feature, is the mean vertex albedo feature, is the mean linear blend skin weight feature, and The mapping matrices of the mean grid model features corresponding to the three feature embedding modules are
[0084] S3.1.2: After rendering the optimized hand mesh model, obtain a second 2D hand image;
[0085] S3.1.3: Calculate a second loss value according to the second 2D hand image and the masked image at the current viewing angle, and adjust and update the latent space feature vectors of the three feature embedding modules according to the second loss value;
[0086] S3.1.4: Repeat S3.1.1-S3.1.3 to iteratively train the latent space feature vectors of the three feature embedding modules. The optimization process continuously reduces the loss of the rendered image and the true image until the training is completed. The optimal latent space feature vectors and optimal posture decoder corresponding to the three feature embedding modules and the optimal hand mesh model are obtained and updated, and recorded as the deformable hand mesh model with detailed texture.
[0087] In S3.1.3, in order to effectively maintain geometric details and eliminate artifacts during training, further reduce the excessive roughness of the mesh surface caused by albedo and vertex offset, and at the same time not affect the performance of fine details, a partially perceived Laplace smoothing strategy is proposed. This smoothing strategy is applied to the loss regularization term, and the optimized high-resolution hand mesh model is rendered. The loss is compared with the masked image at the current perspective, that is, the second loss value contains the Laplace smoothing loss value using the smoothing strategy. The calculation formula is as follows:
[0088]
[0089] A=L*f=[A1,A2,...,A i ]
[0090]
[0091] Where A is the Laplace characteristic matrix, φ pLap is a hierarchical weight used to balance the effect of regularization at different smoothness levels. The hierarchical weight is defined by a set of thresholds and corresponding balance coefficients to adjust the smoothing effect of the Laplacian matrix. L is the Laplacian matrix, f is the feature of the vertex attribute, specifically vertex offset or vertex albedo, A1, A2, A i are the 1st, 2nd and i-th eigenvalues of the Laplace characteristic matrix, p1 and p2 are the first and second thresholds, γ1, γ2 and γ3 are the first to third balancing coefficients, respectively.
[0092] The total regularization loss is calculated as follows:
[0093]
[0094] in, is the total regularization loss, and are the partial-aware Laplace smoothing terms of vertex albedo and vertex offset value, i.e., the first Laplace smoothing loss value and the second Laplace smoothing loss value, respectively. is the mask loss, is the Euclidean distance between adjacent vertices, is the constraint of the vertex offset value, and α1, α2 and α3 are the weights of the corresponding items respectively.
[0095] The loss of the optimized high-resolution hand mesh model after rendering is the image perception loss. The SSIM term is introduced into the image perception loss. The specific formula is as follows:
[0096]
[0097] Among them, B(π i ) is the camera parameter π from the i-th viewpoint i The resulting rendered image, I i is the true value image, and λ is the loss function weight.
[0098] Therefore, the calculation formula for the second loss value is as follows:
[0099]
[0100] in, It is the total loss function for feature embedding module training.
[0101] Among them, Figure 2 As shown in the figure, each feature embedding module includes a posture decoder, which concatenates the hand posture at the current perspective and the corresponding latent space feature vector Q and then inputs it into the posture decoder. The output of the posture decoder is multiplied by the mapping matrix and then added to the corresponding mean grid model feature. The result is used as the predicted output of the current feature embedding module. The feature embedding module is to obtain posture-dependent grid features. The average feature can be combined with the posture-dependent feature offset to generate features corresponding to specific hand postures. The specific formula is as follows:
[0102]
[0103] in, is the output grid feature of the feature embedding module, θ t is the hand posture parameter, is the mean grid model feature, Q represents the latent space feature vector, s() is a posture decoder composed of a multi-layer perceptron, is the mapping matrix.
[0104] Among them, the posture decoder maps the hand posture and latent space feature vector Q under the current perspective to the implicit space. The features obtained by the posture decoder need to be aligned, and the implicit space features obtained by the posture decoder are mapped to features of the same dimension as the mean grid model features through the mapping matrix. The specific formula of the mapping matrix is as follows:
[0105]
[0106] in, is the value of the i-th row and j-th column of the mapping matrix, m is the dimension of the pose decoder output feature, and n is the dimension of the mean grid model feature.
[0107] S3.2: training the current neural network renderer iteratively according to the current hand mesh model and the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module to obtain an updated neural network renderer;
[0108] S3.2 specifically states:
[0109] S3.2.1: A neural network renderer implemented by a multi-layer perceptron is constructed using the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module. The neural network renderer uses vertex-related features such as position, color, and normal. The rendering process of the neural network renderer is as follows:
[0110] C(π i ) = c(x, N, h, d)
[0111] h=concat(Q D , Q ρ , Q render )
[0112] Among them, C(π i ) is the camera parameter π from the i-th viewpoint i The synthesized image, c() is the rendering function, which is a neural network renderer implemented by a multi-layer perceptron; x and N are the position and normal obtained by interpolating the triangle patch vertices at the intersection, d is the ray direction, h is the fused feature vector, concat() represents the splicing operation, Q D and Q ρ are the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module, namely, the vertex offset latent space feature vector and the vertex albedo latent space feature vector; Q render is a feature vector for neural rendering, specifically a feature vector of length 20 and an initial value of 0, which is a learnable parameter;
[0113] S3.2.2: The neural network renderer is trained iteratively based on the loss of the 2D hand rendering image obtained by the neural network renderer and the masked image at the current viewing angle, which also includes training and optimization of the feature vector used for neural rendering. After the training is completed, an updated neural network renderer is obtained.
[0114] Among them, the loss of the 2D hand rendering image obtained by the neural network renderer and the masked image at the current perspective The specific formula is as follows:
[0115]
[0116] Among them, C(π i ) is the camera parameter π from the i-th viewpoint i Synthesized image, ‖‖1 represents the L1 norm, represents the image similarity loss term, w is the third weight, I i It is the masked image under the current viewing angle, that is, the true value image.
[0117] S4: For the actual hand image sequence, combined with the current hand mesh model, the three current feature embedding modules are iteratively trained according to a hand image to obtain the optimized three feature embedding modules and the updated hand mesh model; then the updated hand mesh model is rendered using the final neural network renderer to obtain a high-fidelity 2D hand rendering image corresponding to the hand image;
[0118] S5: Repeat S4, and render other hand images in the actual hand image sequence in sequence to obtain corresponding high-fidelity 2D hand rendering images, thereby obtaining a hand rendering image sequence and sending it to a display device for display, so as to realize the simulation of hand avatars, enhance the user's immersion and promote natural interaction in the virtual environment.
[0119] On the one hand, the present invention also proposes a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a high-fidelity hand avatar generation method based on feature embedding and part-aware smoothing strategy.
[0120] On the one hand, the present invention also proposes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a method for generating a high-fidelity hand avatar based on feature embedding and part-aware smoothing strategy are implemented.
[0121] The embodiment of the present invention proposes a high-fidelity hand avatar generation method based on feature embedding and part-aware smoothing strategy, which is used to achieve effects such as fine hand geometry reconstruction, real-time high-fidelity rendering, and hand animation in different postures. The present invention obtains a high-detail mesh that accurately corresponds to the gesture by utilizing an efficient feature embedding module to distinguish between average features and posture-dependent features. At the same time, in order to ensure the high quality of hand synthesis, the present invention adopts a grid-based neural rendering technology, taking into account the topological consistency of the mesh. During the training process, the present invention introduces part-aware Laplace regularization to eliminate unnecessary artifacts, while maintaining details through regularization at different levels. After a large number of experiments, it is proved that the method of the present invention can generate high-fidelity hand animation geometry and textures in a wide range of gestures.
[0122] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.
Claims
1. A high-fidelity hand avatar generation method based on feature embedding and smoothing strategy, It is characterized in that The following steps are involved: S1: Obtain a multi-view hand image dataset, which contains multiple hand image samples from different viewpoints; S2: construct an initial hand mesh model based on a hand image sample and generate mean mesh model features corresponding to three feature embedding modules; S3: Based on the initial hand mesh model and the mean mesh model features corresponding to the three feature embedding modules, the three feature embedding modules and the neural network renderer are trained separately using the multi-view hand image dataset to obtain the optimized three feature embedding modules and the final neural network renderer as well as the updated hand mesh model; S4: For the actual hand image sequence, combined with the current hand mesh model, the three current feature embedding modules are iteratively trained according to a hand image to obtain the optimized three feature embedding modules and the updated hand mesh model; then the updated hand mesh model is rendered using the final neural network renderer to obtain a high-fidelity 2D hand rendering image corresponding to the hand image; S5: Repeat S4 to render other hand images in the actual hand image sequence in sequence to obtain corresponding high-fidelity 2D hand rendering images, thereby obtaining a hand rendering image sequence.
2. A high-fidelity hand avatar generation method based on feature embedding and smoothing strategy according to claim 1, It is characterized in that The S2 is specifically: S2.1: constructing an initial hand mesh model according to the masked image and posture parameters in the current hand image sample, and using the linear mixed skinning weights of the initial hand mesh model as the mean mesh model features of the first feature embedding module; S2.2: Combining the spherical harmonic function and the camera parameters under the current viewing angle, the initial hand mesh model is used to construct the mean mesh model features corresponding to the second feature embedding module and the mean mesh model features corresponding to the third feature embedding module.
3. The method for generating a high-fidelity hand avatar based on feature embedding and smoothing strategy according to claim 1, It is characterized in that The S3 is specifically: S3.1: According to the current hand image sample, the mean grid model features corresponding to the three feature embedding modules, and the current hand grid model, the three feature embedding modules are trained iteratively respectively to obtain three trained feature embedding modules and update the hand grid model; S3.2: training the current neural network renderer iteratively according to the current hand mesh model and the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module to obtain an updated neural network renderer; S3.3: Based on the latest feature embedding module, hand mesh model, and neural network renderer, repeat S3 and S4, use other hand image samples in the multi-view hand image dataset to train the three feature embedding modules, hand mesh model, and neural network renderer, and obtain the final three feature embedding modules, hand mesh model, and neural network renderer.
4. The method for generating a high-fidelity hand avatar based on feature embedding and smoothing strategy according to claim 2, It is characterized in that The S2.1 is specifically: According to the masked image under a viewing angle and its corresponding posture parameters, the original mesh model is constructed using the MANO model. The resolution of the original mesh model is then improved through the subdivision method to generate an initial hand mesh model. The linear mixed skinning weights of the initial hand mesh model are used as the mean mesh model features of the first feature embedding module.
5. The method for generating a high-fidelity hand avatar based on feature embedding and smoothing strategy according to claim 2, It is characterized in that The S2.2 is specifically: Initialize the feature vector of hand vertex offset and the feature vector of vertex albedo, and then render the initial hand mesh model in combination with spherical harmonics and camera parameters at the current viewing angle to obtain a first 2D hand image, and iterate the feature vector of hand vertex offset and the feature vector of vertex albedo according to the first loss value calculated from the first 2D hand image and the masked image at the current viewing angle until the training is completed to obtain the optimal feature vector of hand vertex offset and the feature vector of vertex albedo, and use the optimal feature vector of hand vertex offset as the mean mesh model feature of the second feature embedding module, and the optimal feature vector of vertex albedo as the mean mesh model feature of the third feature embedding module.
6. The method for generating a high-fidelity hand avatar based on feature embedding and smoothing strategy according to claim 3, It is characterized in that The S3.1 is specifically: S3.1.1: According to the hand posture at the current viewpoint, the mean mesh model features and the current latent space feature vector, three feature embedding modules are used to predict the hand vertex offset, vertex albedo and linear blend skin weight respectively, and then the current hand mesh model is optimized according to the predicted hand vertex offset, vertex albedo and linear blend skin weight to obtain the optimized hand mesh model; S3.1.2: After rendering the optimized hand mesh model, obtain a second 2D hand image; S3.1.3: Calculate a second loss value according to the second 2D hand image and the masked image at the current viewing angle, and adjust and update the latent space feature vectors of the three feature embedding modules according to the second loss value; S3.1.4: Repeat S3.1.1-S3.1.3 to iteratively train the latent space feature vectors of the three feature embedding modules until the training is completed, and obtain the optimal latent space feature vectors and the optimal hand mesh model corresponding to the three feature embedding modules and update them.
7. The method for generating a high-fidelity hand avatar based on feature embedding and smoothing strategy according to claim 3, It is characterized in that In S3.1, each feature embedding module includes a posture decoder, which concatenates the hand posture under the current viewing angle and the corresponding latent space feature vector Q and then inputs them into the posture decoder. The output of the posture decoder is multiplied by the mapping matrix and then added to the corresponding mean grid model feature. The result is used as the predicted output of the current feature embedding module.
8. The method for generating a high-fidelity hand avatar based on feature embedding and smoothing strategy according to claim 3, It is characterized in that The S3.2 is specifically: S3.2.1: Use the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module to construct a neural network renderer. The rendering process of the neural network renderer is as follows: C(πi))=c(x,N,h,d) h=concat(Qp,Qp,Qrender) Among them, C(π i ) is the camera parameter π from the i-th viewpoint i The synthesized image, c() is the rendering function, x and N are the position and normal obtained by interpolating the triangle patch vertices at the intersection, d is the ray direction, h is the fused feature vector, concat() represents the splicing operation, Q D and Q ρ are the optimal latent space feature vectors corresponding to the second feature embedding module and the third feature embedding module respectively; Q render is the feature vector for neural rendering; S3.2.2: The neural network renderer is trained iteratively based on the loss of the 2D hand rendering image obtained by the neural network renderer and the masked image at the current viewing angle, and an updated neural network renderer is obtained after the training is completed.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
