Monocular video Gaussian human body avatar method based on multi-resolution image-text fusion

Abstract

The invention discloses a monocular video Gaussian human body avatar method based on multi-resolution image-text fusion, and belongs to the field of computer vision and graphics. Comprising the following steps: firstly, carrying out self-defined text description on a human body video as model input in a text image feature processing module; and designing a multi-resolution feature enhancement time sequence consistency module. And based on the generated Gaussian point cloud and the fused dynamic features, rendering is carried out through a 3D Gaussian splash technology. According to the method, global and local dynamic details are captured by extracting and fusing features under different resolutions. The low-resolution path is used for enhancing the capturing capability of the dynamic global appearance, the high-resolution path focuses on recovering the fine features of the human body avatar, and by combining the time sequence consistency and high-resolution details, the capturing of tiny dynamic changes can be assisted, and the cross-frame jitter can be relieved, so that the generated human body avatar actions are more natural and coherent.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and graphics, and specifically relates to a monocular video Gaussian human avatar method based on multi-resolution image and text fusion, which is applied to industries such as virtual reality, augmented reality, and digital human reconstruction, and aims to generate a fine three-dimensional human model through a monocular video. Background Art

[0002] Monocular video human avatar technology has received extensive attention in the field of computer vision and graphics in recent years. Although traditional image-based geometric modeling and optimization methods can restore the shape of the human body to a certain extent, they have significant limitations in capturing the dynamic posture and details of the human body, making it difficult to achieve high-quality results. With the development of implicit representation methods, neural radiance field (NeRF) was proposed and applied to human avatar modeling. By mapping the posture space in monocular video to the canonical space, good modeling results were achieved. However, NeRF easily introduces fuzzy correspondences in the reverse mapping process, resulting in the loss of human surface details. Compared with implicit representation, explicit representation methods have a natural advantage in capturing human surface details, but early mesh and point cloud representations have high computational costs and challenges in maintaining details and accuracy. In recent years, 3D modeling methods based on 3D Gaussian splash (3DGS) have gradually attracted attention. 3DGS represents points in 3D space through Gaussian distribution, which can not only effectively reduce the demand for computing resources, but also significantly reduce the number of parameters while maintaining high accuracy. Human reconstruction tasks usually rely on the SMPL model to represent the geometric structure of the human body by adjusting shape and posture parameters. Existing methods input monocular video into the model to generate SMPL, then use SMPL to generate Gaussian points, and finally generate human images through 3DGS-based rendering. However, although 3DGS shows good performance in static human modeling, it is still difficult to extract dynamic details when modeling dynamic models, especially when dealing with complex and fast movements. To solve the above problems, by introducing the CLIP model into human avatar modeling, text information can be used as an additional signal to enrich the understanding of human posture and appearance. When dealing with dynamic scenes, a multi-resolution feature fusion-enhanced action temporal consistency constraint module is proposed. By extracting and fusing features at different resolutions, the low-resolution path enhances the model's ability to capture the global dynamic appearance of the human body, while the high-resolution path focuses on recovering fine dynamic details. Combined with temporal consistency loss, the detail recovery path can effectively reduce cross-frame jitter and inconsistency, ensuring that the generated human avatar is smoother during movement. Summary of the invention

[0003] The present invention proposes a method for Gaussian human avatar in monocular video based on multi-resolution image-text fusion. The text features describing the human body in the video are extracted through the CLIP model and fused with the visual features to enhance the model's understanding of the human appearance and posture, and overcome the dependence on single visual information. It uses the Contrastive Language-Image Pretraining (CLIP) model to obtain text information. The text information is combined with the global appearance features that capture the human avatar, so that the model has a more comprehensive understanding of the human posture and appearance. In order to further improve the performance of the model in dynamic scenes, the present invention designs a multi-resolution feature fusion-enhanced temporal consistency constraint module, which captures global and local dynamic details by extracting and fusing features at different resolutions. In this strategy, the low-resolution path is used to enhance the ability to capture the dynamic global appearance, and the high-resolution path focuses on restoring the fine features of the human avatar. By combining temporal consistency, the high-resolution details can help capture small dynamic changes and alleviate cross-frame jitter. This synergy not only ensures the stability of the overall structure, but also dynamically adjusts the details, making the generated human avatar movements more natural and coherent.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] Step 1: First, a customized text description of the human body video is made as the model input in the text image feature processing module; the text description includes aspects such as human body posture, clothing, and action; the CLIP model is used to extract text features related to human body posture and appearance. Subsequently, the extracted text features are fused with the visual features extracted from the video frames. Visual features are extracted from video frames through convolutional neural networks (CNN) or other deep learning models, and contain the spatial layout, texture information, and dynamic changes of the human body. Through the splicing operation, the text features and visual features are combined to form a multimodal feature representation. The fused features are used to further guide subsequent 3D modeling and human avatar generation.

[0006] Step 2: In order to improve the accuracy of dynamic human reconstruction, a multi-resolution feature enhanced temporal consistency module is designed. Low-resolution and high-resolution features are extracted from the input video frames. The low-resolution features are used to capture the large-scale motion changes of the human body, while the high-resolution features focus on modeling the details, such as joint movements and surface textures. Then, the features of the two resolutions are combined through weighted fusion or attention mechanism to fully capture the global and local dynamic details of the human body. In order to ensure temporal consistency, a temporal consistency constraint module is designed, which ensures the smooth transition of human motion by calculating the difference between consecutive frames to avoid jitter or unnatural changes across frames.

[0007] Step 3: Finally, based on the generated Gaussian point cloud and the fused dynamic features, rendering is performed using the 3D Gaussian splash (3DGS) technique. The 3DGS method processes each Gaussian point in three-dimensional space to achieve high-quality detail rendering with less computational overhead. Through 3DGS rendering, the generated Gaussian points not only retain the geometric information of the human body, but also render rich detail textures and dynamic effects. This step is completed in the rendering module.

[0008] The output of this stage is a high-quality dynamic human image that can reflect the fine features of the human body while ensuring consistency and stability across frames. Ultimately, the system can generate a high-precision, stable and natural dynamic human avatar image suitable for application scenarios such as virtual reality and augmented reality.

[0009] This paper proposes a 3D Gaussian human avatar reconstruction framework (MRTI-Avatar) based on multi-resolution image-text feature fusion and temporal consistency constraints. The overall process is as follows Figure 1 The input consists of three parts: SMPL obtained by video preprocessing, custom optimizable features, and custom text description Text of the human body in the video. input After extracting text features through the CLIP model, they are combined with optimizable features and UV mapping to generate preliminary input features for predicting dynamic attribute colors in Gaussian points. size and position offset At the same time, in order to use isotropic Gaussian points to maintain accurate 3D shapes and solve the problem of low reconstruction accuracy due to low transparency of points on the human body boundary, the rotation q and opacity α parameters are fixed. On this basis, the temporal consistency constraint module of multi-resolution feature enhancement takes into account local details and global dynamic appearance through the fusion of high and low resolution features, and uses the temporal consistency loss to constrain the smooth transition of Gaussian points across frames to ensure the natural and coherent reconstructed human body movements. Finally, the present invention combines multiple loss functions for training models. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is the overall network framework of the present invention.

[0011] Figure 2 This is the effect diagram of the temporal consistency module enhanced by multi-resolution feature fusion.

[0012] Figure 3 It is a schematic diagram of the qualitative rendering result of the present invention.

[0013] Figure 4 It is a schematic diagram of the new posture synthesis result of the present invention. DETAILED DESCRIPTION

[0014] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0015] The text image feature processing module processes the features, and after processing, it is input into the multi-resolution feature enhanced temporal consistency module for parameter prediction. The obtained parameters are then rendered in the rendering module, and a joint optimization strategy is used in training.

[0016] Text image feature processing module: In order to model the human appearance of the current frame, the present invention generates a corresponding SMPL model from the input image in preprocessing, and generates a standard UV position feature map by sampling the points on the surface of the SMPL model of the current frame. Where H and W are the height and width of the feature map, respectively, and C n is the number of channels, n = 1, 2, 3.... By defining an optimizable feature It is continuously optimized during training to capture the geometric information of the human body.

[0017] In order to improve the accuracy of detail capture in the process of human avatar reconstruction, the present invention introduces text features and uses the CLIP model to incorporate natural language descriptions as additional information into feature expressions. The present invention generates text input Text by a detailed custom description of the human appearance in the monocular video. input , describing key features of the human body, such as clothing, posture, color, and rotation angle, Figure 1 Text: A man wearing a red T-shirt, standing position, rotating at various angles under a single-view camera. This information can make up for the lack of appearance details extracted by visual features alone, thereby enriching the feature expression of the human body. The Clip model encoder converts the input text description into text features in Represents a d-dimensional real space, where d is the feature dimension. Define the text features extracted by the Clip encoder as:

[0018] f clip =Clip text (Text input ), (1)

[0019] Among them, Clip text For text encoder, Text input is the natural language description of the input.

[0020] In order to transform the text feature f Clip To combine with UV coordinates and optimizable features, first The dimension is transformed into Combine the transformed features to get richer features:

[0021]

[0022] in, It is a concatenation operation. It can optimize features to capture rough global appearance features while supplementing them with details provided by the text.

[0023] This method can supplement the visual features from the semantic level, so that the generated image is more in line with expectations. Predict the Gaussian parameters that are input to the next module.

[0024] Multi-resolution feature enhancement temporal consistency module: The purpose of multi-resolution feature fusion is to improve the model's ability to capture details by fusing feature representations of different resolutions. The high-resolution path can preserve the details in the input image, while the low-resolution path helps capture a wider range of structural information. Through the parallel convolution operation of the high- and low-resolution paths, the input feature f Con The information is processed at different scales:

[0025]

[0026] Among them, Conv represents the convolution operation, and the high and low resolution paths are respectively represented by Conv high (1×1) and Conv low (5×5) represents convolutions of different scales. After the convolution operation, the feature map extracted by the high-resolution path is the feature map extracted from the low-resolution path. high With f low Finally, the high-resolution and low-resolution features are fused by element-by-element addition to obtain the fused feature map. It maintains the information of the original input features and has the ability to express features at multiple resolutions.

[0027] Furthermore, for f multi Perform convolution operation to predict dynamic attribute color size and position offset These dynamic properties are output through the convolutional network, so that the generated 3D model can carefully reflect the appearance characteristics of the input image:

[0028]

[0029] To ensure that the model maintains visual coherence when processing dynamic human motion, the present invention introduces temporal consistency loss. Specifically, the loss constrains the change of position parameters of each Gaussian point between adjacent frames to ensure that the motion performance of each Gaussian point in different time steps is continuous and consistent, so that the model can achieve smooth transition between frames. Each frame in the video contains multiple Gaussian points. For the i-th Gaussian point, the position offset in the T-th frame is When video frame T≥2:

[0030]

[0031] in, represents the position offset of the i-th Gaussian point in the T-th frame, represents the position offset of the i-th Gaussian point in the T-1th frame, and N is the number of all Gaussian points in the frame. This loss function constrains the change of Gaussian position parameters between adjacent frames through the L2 norm to minimize the position offset difference of Gaussian points in adjacent frames, making the dynamic performance of the human body in multi-frame videos more natural and coherent.

[0032] After that, the present invention introduces the predicted parameters into the 3D Gaussian model of the canonical space and associates the predicted attributes with the corresponding 3D Gaussian model.

[0033] Rendering module: Based on this representation, the 3D Gaussian models of these canonical spaces (the positions of their Gaussian points are x) can be placed into the motion space for free view rendering:

[0034]

[0035] Where R represents the rendered image, Render(·) represents the rendering process of 3DGS, represents the position offset, D represents the dynamic parameters in the Gaussian point except the position, and S represents the fixed parameters in the Gaussian point.

[0036] Joint optimization training strategy: In the training method of the present invention, multiple loss functions are used to optimize the model performance, with the goal of achieving accurate and consistent human avatar reconstruction. In order to ensure the accuracy of the reconstruction details, the present invention jointly trains the parameter prediction loss, the temporal consistency loss, and the rendering loss.

[0037] The goal of this invention is to guide network learning by optimizing the following loss function:

[0038]

[0039] in, and They are the L2 norm in parameter prediction loss, predicted position offset and predicted size, and λ Lpips They are respectively the rendering loss loss, SSIM loss

[44] and LPIPS loss

[45] , This is the timing consistency loss. The present invention sets λ L2 =1,λ offset =10,λ scale =1,λ1=0.8,λ ssim =0.2,λ Lpips =0.2,λ tem =0.02, and the influence of each loss item in the total loss is adjusted by the corresponding weight to balance the optimization objectives of various aspects.

[0040] Experimental Section

[0041] Experimental Datasets: The present invention is evaluated and tested on the People-Snapshot and DynVideo datasets.

[0042] People-Snapshot dataset: This dataset is a human video shot from a single angle in front of a green screen and rotated in place in a fixed posture. It is usually used for human 3D reconstruction and modeling tasks under controlled conditions. It focuses on reconstructing only the human body itself to better exclude background interference and improve the accuracy of comparative evaluation.

[0043] DynVideo dataset: To enhance the challenge of model evaluation, this paper uses the DynVideo dataset. This dataset covers a variety of complex and rotating action videos recorded by male and female characters using mobile devices, including high-dynamic scenes such as waving and turning. Evaluation indicators:

[0044] PSNR: A measure of the quality of the reconstructed image relative to the original image.

[0045] SSIM: is a perceptual metric that evaluates the similarity between two images based on brightness, contrast, and structure.

[0046] LPIPS: It evaluates perceptual differences by comparing image feature activations.

[0047] Experimental setup: In the experiment, the present invention sampled about 200,000 points from the SMPL grid model for Gaussian initialization. The adaptive control module was not enabled in this experiment, so the number of Gaussian points remained unchanged during training. The entire network framework was trained on a single NVIDIA RTX 4090 GPU, and the training time ranged from 0.5 hours to 8 hours depending on the experimental configuration. In addition, the present invention also conducted multiple trainings with different initial conditions to ensure the stability and robustness of the model performance.

[0048] Comparative experiments: The present invention is compared with some baseline methods and the most advanced 3D human avatar reconstruction methods on the People-Snapshot and DynVideo datasets. Table 1 shows the quantitative results on the People-Snapshot dataset, and Table 2 shows the quantitative results on the DynVideo dataset. Both have achieved good results. Figure 3 Qualitative comparative experimental results are presented.

[0049] The present invention further verifies the performance of the model in a novel posture test. Figure 4 The new posture reconstruction effect on two data sets is demonstrated, wherein the Example on the left is the result of human avatar reconstruction, and the left and right sides of the dotted line are the comparisons between the method of the present invention and the existing method GaussianAvatar.

[0050] The method of the present invention demonstrates good detail restoration capability in new posture reconstruction, especially in areas such as the face and hands. This result further proves the robustness and adaptability of the method of the present invention in dealing with complex postures.

[0051] Table 1. Results on the People-Snapshot dataset. The best and second-best results are in bold and Underline Highlight.

[0052]

[0053] Table 2. Results on the DynVideo dataset. The best and second-best results are in bold and Underline Wire Highlight.

[0054]

[0055] Ablation experiment: The method of the present invention is subjected to an ablation study on the male-3-casual data in the People-Snapshot dataset, removing the Clip text module and the Multi-Tem module to verify the contribution of different components in the model of the method of the present invention. Table 3 shows the impact of each module on the model performance.

[0056] Table 3. Ablation experiments on different components of the model

[0057]

[0058]

Claims

1. A monocular video Gaussian human avatar method based on multi-resolution image and text fusion, characterized in that: The steps include: Step 1: First, a customized text description of the human body video is made in the text image feature processing module as the model input; the text description includes the human body posture, clothing, and action; the CLIP model is used to extract text features related to the human body posture and appearance; then, the extracted text features are fused with the visual features extracted from the video frame; the visual features are extracted from the video frame through the convolutional neural network CNN or other deep learning models, including the spatial layout, texture information and dynamic changes of the human body; the text features and visual features are combined through the splicing operation to form a multimodal fused feature representation; the fused feature representation is used to guide the subsequent 3D modeling and human avatar generation; Step 2: To improve the accuracy of dynamic human reconstruction, a multi-resolution feature enhanced temporal consistency module is designed; low-resolution and high-resolution features are extracted from the input video frames; low-resolution features are used to capture large-scale motion changes of the human body, while high-resolution features focus on modeling at the detail level; features of the two resolutions are combined through weighted fusion or attention mechanism to fully capture the global and local dynamic details of the human body; to ensure temporal consistency, a temporal consistency constraint module is designed, which ensures smooth transition of human motion by calculating the difference between consecutive frames to avoid jitter or unnatural changes across frames; Step 3: Finally, based on the generated Gaussian point cloud and the fused dynamic features, rendering is performed through 3D Gaussian splash 3DGS; 3DGS can achieve high-quality detail rendering by processing each Gaussian point in three-dimensional space with low computational overhead.

2. The monocular video Gaussian human avatar method based on multi-resolution image and text fusion according to claim 1, characterized in that: MRTI-Avatar is a 3D Gaussian human avatar reconstruction framework based on multi-resolution image and text feature fusion and temporal consistency constraints. The input includes three parts: SMPL obtained by video preprocessing, custom optimizable features, and custom text description Text of the video human body input ; After extracting text features through the CLIP model, combine them with optimizable features and UV mapping to generate preliminary input features for predicting dynamic attribute colors in Gaussian points size and position offset In order to use isotropic Gaussian points to maintain accurate 3D shapes and solve the problem of low reconstruction accuracy due to low transparency of points on the human body boundary, the rotation q and opacity α parameters are fixed.

3. The monocular video Gaussian human avatar method based on multi-resolution image and text fusion according to claim 1, characterized in that: The text image feature processing module processes the features, and after processing, it is input into the multi-resolution feature enhanced temporal consistency module for parameter prediction. The obtained parameters are rendered in the rendering module, and a joint optimization strategy is used in training.

4. The monocular video Gaussian human avatar method based on multi-resolution image and text fusion according to claim 1, characterized in that: Text image feature processing module: Model the human appearance of the current frame, generate the corresponding SMPL model from the input image in preprocessing, and generate a standard UV position feature map by sampling the points on the surface of the SMPL model of the current frame Where H and W are the height and width of the feature map, respectively, and C n is the number of channels, n = 1, 2, 3....; by defining an optimizable feature It is continuously optimized during training to capture human body geometry information.

5. The monocular video Gaussian human avatar method based on multi-resolution image and text fusion according to claim 4, characterized in that: In order to improve the accuracy of detail capture in the process of human avatar reconstruction, text features are introduced, and the CLIP model is used to incorporate natural language descriptions as additional information into feature expressions; text input Text is generated by a detailed custom description of the human appearance in the monocular video input , describing the key features of human clothing, posture, color, and rotation perspective; the Clip model encoder converts the input text description into text features in Represents a d-dimensional real number space, where d is the feature dimension; the text features extracted by the Clip encoder are defined as: f clip =Clip text (Text input ) (1) Among them, Clip text For text encoder, Text input is the natural language description of the input.

6. The method for monocular video Gaussian human avatar based on multi-resolution image and text fusion according to claim 5, characterized in that: In order to transform the text feature f Clip Combine with UV coordinate map and optimizable features. First, The dimension is transformed into Combine the transformed features to get richer features: in, is a splicing operation; it supplements visual features from the semantic level to make the generated image more in line with expectations; through the final Predict the Gaussian parameters that are input to the next module.

7. The monocular video Gaussian human avatar method based on multi-resolution image and text fusion according to claim 1, characterized in that: Multi-resolution feature enhancement temporal consistency module: The purpose of multi-resolution feature fusion is to improve the model's ability to capture details by fusing feature representations of different resolutions; through parallel convolution operations of high and low resolution paths, the input feature f Con The information is processed at different scales: Among them, Conv represents the convolution operation, and the high and low resolution paths are respectively represented by Conv high (1×1) and Conv low (5×5) represents convolutions of different scales; after the convolution operation, the feature map extracted by the high-resolution path is the feature map extracted by the low-resolution path; get f high With f low Finally, the high-resolution and low-resolution features are fused by element-by-element addition to obtain the fused feature map. It maintains the information of the original input features and has the ability to express features at multiple resolutions.

8. The method for monocular video Gaussian human avatar based on multi-resolution image and text fusion according to claim 7, characterized in that: f multi Perform convolution operation to predict dynamic attribute color size and position offset These dynamic properties are output through the convolutional network, so that the generated 3D model can carefully reflect the appearance characteristics of the input image: Ensure that the model maintains visual coherence when processing dynamic human body movements, and introduce temporal consistency loss; by constraining the change of position parameters of each Gaussian point between adjacent frames, ensure that the motion performance of each Gaussian point in different time steps is continuous and consistent, so that the model can achieve smooth transition between frames; each frame in the video contains multiple Gaussian points, and for the i-th Gaussian point, the position offset in the T-th frame is When video frame T≥2: in, represents the position offset of the i-th Gaussian point in the T-th frame, represents the position offset of the i-th Gaussian point in the T-1-th frame, and N is the number of all Gaussian points in the frame. This loss function constrains the change of Gaussian position parameters between adjacent frames through the L2 norm to minimize the position offset difference of Gaussian points in adjacent frames, making the dynamic performance of the human body in multi-frame videos more natural and coherent. The predicted parameters are introduced into the 3D Gaussian model in the canonical space, and the predicted attributes are associated with the corresponding 3D Gaussian model.

9. The method for monocular video Gaussian human avatar based on multi-resolution image and text fusion according to claim 1, characterized in that: Rendering module: Place the 3D Gaussian model of the standard space, i.e. the position of the Gaussian point is x, into the motion space for free view rendering: Where R represents the rendered image, Render(·) represents the rendering process of 3DGS, represents the position offset, D represents the dynamic parameters in the Gaussian point except the position, and S represents the fixed parameters in the Gaussian point.

10. The monocular video Gaussian human avatar method based on multi-resolution image and text fusion according to claim 1, characterized in that: Joint optimization training strategy: multiple loss functions are used to optimize model performance, aiming to achieve accurate and consistent human avatar reconstruction; parameter prediction loss, temporal consistency loss, and rendering loss are trained jointly; The network is guided by optimizing the following loss function: in, and They are the L2 norm in parameter prediction loss, predicted position offset and predicted size, and They are respectively the rendering loss loss, SSIM loss [44] and LPIPS loss [45], This is the loss of timing consistency.

Images