A Gaussian human avatar method based on multi-resolution image and text fusion in monocular video

Abstract

The application discloses a monocular video Gaussian human avatar method based on multi-resolution text and image fusion, and belongs to the fields of computer vision and graphics. The method comprises the following steps: firstly, in a text image feature processing module, a self-defined text description of a human video is taken as model input; a multi-resolution feature reinforcement time consistency module is designed. Based on the generated Gaussian point cloud and the fused dynamic features, rendering is performed through 3D Gaussian splashing technology. The application captures global and local dynamic details by extracting and fusing features at different resolutions. The low-resolution path is used to enhance the ability to capture dynamic global appearance, and the high-resolution path focuses on restoring the fine features of the human avatar. Through the combination of time consistency, the high-resolution details can assist in capturing minor dynamic changes and relieving cross-frame jitter, so that the generated human avatar motion is more natural and coherent.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and graphics, and specifically relates to a monocular video Gaussian human avatar method based on multi-resolution image and text fusion. The method is applied to industries such as virtual reality, augmented reality, and digital human reconstruction, and aims to generate a detailed three-dimensional human body model through monocular video. Background Art

[0002] Human avatar technology based on monocular video has garnered widespread attention in the fields of computer vision and graphics in recent years. While traditional image-based geometric modeling and optimization methods can recover the human body's shape to a certain extent, they have significant limitations in capturing dynamic poses and details, making it difficult to achieve high-quality results. With the development of implicit representation methods, Neural Radiance Fields (NeRFs) have been proposed and applied to human avatar modeling. By mapping the pose space in monocular video to a canonical space, they have achieved promising modeling results. However, NeRFs can easily introduce ambiguous correspondences during the reverse mapping process, resulting in a loss of surface details. Explicit representation methods have a natural advantage in capturing surface details compared to implicit representations, but early mesh and point cloud representations were computationally expensive and faced challenges in preserving detail and accuracy. In recent years, 3D modeling methods based on 3D Gaussian splatter (3DGS) have gained increasing attention. 3DGS represents points in 3D space using a Gaussian distribution, effectively reducing computational resources and significantly reducing the number of parameters while maintaining high accuracy. Human reconstruction tasks typically rely on the SMPL model, which represents human geometry by adjusting shape and pose parameters. Existing methods use a monocular video input model to generate SMPL, then use SMPL to generate Gaussian points, and finally generate human images through 3DGS-based rendering. However, while 3DGS demonstrates good performance in static human modeling, it still struggles to extract dynamic details when modeling dynamic human bodies, especially when dealing with complex and fast motion. To address this issue, by introducing the CLIP model into human avatar modeling, textual information can serve as an additional signal to enrich the understanding of human posture and appearance. When dealing with dynamic scenes, a motion temporal consistency constraint module enhanced by multi-resolution feature fusion is proposed. By extracting and fusing features at different resolutions, the low-resolution path enhances the model's ability to capture the global dynamic appearance of the human body, while the high-resolution path focuses on recovering fine dynamic details. Combined with a temporal consistency loss, the detail recovery path effectively mitigates jitter and inconsistencies across frames, ensuring smoother motion in the generated avatar. Summary of the Invention

[0003] The application provides a monocular video Gaussian human avatar method based on multi-resolution image-text fusion. The CLIP model is used to extract text features describing the human body in the video, which are combined with visual features to enhance the model's understanding of human appearance and posture, overcoming the dependence on single visual information. The Contrastive Language-Image Pretraining (CLIP) model is used to obtain text information. The text information is combined with the global appearance features of the human avatar, making the model more comprehensive in understanding human posture and appearance. To further improve the model's performance in dynamic scenes, the application designs a multi-resolution feature fusion reinforcement temporal consistency constraint module. The module extracts and fuses features at different resolutions to capture global and local dynamic details. In this strategy, the low-resolution path is used to enhance the ability to capture dynamic global appearance, while the high-resolution path focuses on restoring the fine features of the human avatar. By combining temporal consistency, high-resolution details can help capture subtle dynamic changes and reduce cross-frame jitter. This synergy not only ensures the stability of the overall structure, but also dynamically adjusts the details, making the generated human avatar movements more natural and coherent.

[0004] To achieve the above purpose, the application adopts the following technical solutions:

[0005] Step 1: First, the text image feature processing module is used to perform custom text description on the human video as model input; the text description includes human posture, clothing, action, etc.; the CLIP model is used to extract text features related to human posture and appearance. Then, the extracted text features are combined with visual features extracted from video frames. The visual features are extracted from video frames by convolutional neural network (CNN) or other deep learning models, including human spatial layout, texture information, and dynamic changes. The text features and visual features are combined through splicing operations to form a multi-modal feature representation. The fused features are used to further guide subsequent 3D modeling and human avatar generation.

[0006] Step 2: To improve the accuracy of dynamic human reconstruction, a multi-resolution feature reinforcement temporal consistency module is designed. Low-resolution and high-resolution features are extracted from input video frames. Low-resolution features are used to capture large-scale motion changes of the human body, while high-resolution features focus on detailed modeling, such as joint movements and surface textures. Then, the features of the two resolutions are combined through weighted fusion or attention mechanisms to capture global and local dynamic details of the human body. To ensure temporal consistency, a temporal consistency constraint module is designed. The module calculates the differences between consecutive frames to ensure smooth transitions of human movements and avoid cross-frame jitter or unnatural changes.

[0007] Step 3: Finally, based on the generated Gaussian point cloud and the fused dynamic features, rendering is performed using the 3D Gaussian splatter (3DGS) technique. By processing each Gaussian point in three-dimensional space, the 3DGS method achieves high-quality, detailed rendering with minimal computational overhead. Through 3DGS rendering, the generated Gaussian points not only preserve the geometric information of the human body but also render rich, detailed textures and dynamic effects. This step is completed in the rendering module.

[0008] The output of this stage is a high-quality dynamic human image that not only reflects the body's fine features but also ensures consistency and stability across frames. Ultimately, the system is able to generate a highly accurate, stable, and natural dynamic human avatar image suitable for applications such as virtual reality and augmented reality.

[0009] This paper proposes a 3D Gaussian human avatar reconstruction framework (MRTI-Avatar) based on multi-resolution image-text feature fusion and temporal consistency constraints. Figure 1 The input consists of three parts: SMPL obtained by video preprocessing, custom optimizable features, and custom text description of the human body in the video. input After extracting text features through the CLIP model, it is combined with optimizable features and UV mapping to generate preliminary input features for predicting dynamic attribute colors in Gaussian points. size and position offset At the same time, in order to use isotropic Gaussian points to maintain accurate 3D shapes and address the problem of low reconstruction accuracy due to low transparency of points at the human body boundary, the rotation q and opacity α parameters are fixed. On this basis, the temporal consistency constraint module for multi-resolution feature enhancement takes into account both local details and global dynamic appearance by fusing high- and low-resolution features. At the same time, the temporal consistency loss is used to constrain the smooth transition of Gaussian points across frames, ensuring the natural and coherent nature of the reconstructed human body movements. Finally, the present invention introduces the combination of multiple loss functions for model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is the overall network framework of the present invention.

[0011] Figure 2 This is the effect diagram of the temporal consistency module enhanced by multi-resolution feature fusion.

[0012] Figure 3 This is a schematic diagram of the qualitative rendering results of the present invention.

[0013] Figure 4 Schematic diagram of the new posture synthesis result of the present invention. DETAILED DESCRIPTION

[0014] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0015] The text image feature processing module processes the features, and after processing, it is input into the multi-resolution feature enhancement temporal consistency module for parameter prediction. The obtained parameters are then rendered in the rendering module, and a joint optimization strategy is used in training.

[0016] Text image feature processing module: In order to model the human appearance of the current frame, the present invention generates a corresponding SMPL model from the input image in preprocessing, and generates a standard UV position feature map by sampling the points on the surface of the SMPL model of the current frame. Where H and W are the height and width of the feature map respectively, C n is the number of channels, n=1, 2, 3.... By defining an optimizable feature It is continuously optimized during training to capture the geometric information of the human body.

[0017] In order to improve the accuracy of detail capture in the process of human avatar reconstruction, this paper introduces text features and uses the CLIP model to incorporate natural language descriptions as additional information into feature expression. This paper generates text input by a detailed custom description of the human appearance in the monocular video. input , describing key features of the human body such as clothing, posture, color, and rotation perspective, such as Figure 1 Text: A man wearing a red T-shirt, standing position, rotating at various angles under a single-view camera. This information can make up for the lack of appearance details extracted by visual features alone, thereby enriching the feature expression of the human body. The Clip model encoder converts the input text description into text features in Represents a d-dimensional real space, where d is the feature dimension. The text features extracted by the Clip encoder are defined as:

[0018] f clip =Clip text (Text input ), (1)

[0019] Among them, Clip text For text encoder, Text input is the natural language description of the input.

[0020] In order to transform the text feature f Clip To combine with UV coordinate map and optimizable features, first Dimension transformation to Combining the transformed features to obtain richer features:

[0021]

[0022] in, It is a splicing operation. It can optimize features to capture rough global appearance features while supplementing these features with details provided by the text.

[0023] This method can supplement the visual features from the semantic level, so that the generated image is more in line with expectations. Predict the Gaussian parameters input to the next module.

[0024] Multi-resolution feature enhancement temporal consistency module: The purpose of multi-resolution feature fusion is to improve the model's ability to capture details by fusing feature representations of different resolutions. The high-resolution path can preserve the details of the input image, while the low-resolution path helps capture a wider range of structural information. Through the parallel convolution operation of the high- and low-resolution paths, the input feature f is combined into a Con The information is processed at different scales:

[0025]

[0026] Among them, Conv represents the convolution operation, and the high and low resolution paths are respectively represented by Conv high (1×1) and Conv low (5×5) represents convolutions of different scales. After the convolution operation, the feature map extracted by the high-resolution path is the feature map extracted from the low-resolution path. high With f low Finally, the high and low resolution features are fused by element-by-element addition to obtain the fused feature map It maintains the information of the original input features while also having the ability to express features at multiple resolutions.

[0027] Furthermore, for f multi Perform convolution operation to predict dynamic attribute color size and position offset These dynamic properties are output through the convolutional network, so that the generated 3D model can carefully reflect the appearance characteristics of the input image:

[0028]

[0029] To ensure that the model maintains visual coherence when processing dynamic human motion, the present invention introduces temporal consistency loss. Specifically, the loss constrains the change of each Gaussian point position parameter between adjacent frames, ensuring that the motion performance of each Gaussian point in different time steps is continuous and consistent, so that the model can achieve smooth transition between frames. Each frame in the video contains multiple Gaussian points. For the i-th Gaussian point, the position offset in the T-th frame is When video frame T≥2:

[0030]

[0031] in, represents the position offset of the i-th Gaussian point in the T-th frame, represents the position offset of the i-th Gaussian point in the T-1th frame, and N is the number of all Gaussian points in the frame. This loss function constrains the change in Gaussian position parameters between adjacent frames using the L2 norm to minimize the position offset difference of Gaussian points in adjacent frames, making the dynamic performance of the human body in multi-frame videos more natural and coherent.

[0032] After that, the present invention introduces the predicted parameters into the 3D Gaussian model of the canonical space and associates the predicted attributes with the corresponding 3D Gaussian model.

[0033] Rendering module: Based on this representation, the 3D Gaussian models of these canonical spaces (the position of their Gaussian points is x) can be placed into the motion space for free view rendering:

[0034]

[0035] Where R represents the rendered image, Render(·) represents the rendering process of 3DGS, represents the position offset, D represents the dynamic parameters in the Gaussian point except the position, and S represents the fixed parameters in the Gaussian point.

[0036] Joint Optimization Training Strategy: In our training method, we employ multiple loss functions to optimize model performance, aiming to achieve accurate and consistent avatar reconstruction. To ensure the accuracy of reconstructed details, we jointly train the model using a parameter prediction loss, a temporal consistency loss, and a rendering loss.

[0037] The goal of this invention is to guide network learning by optimizing the following loss function:

[0038]

[0039] in, and They are the L2 norm in the parameter prediction loss, the predicted position offset and the predicted size, and λ Lpips They are respectively the rendering loss loss, SSIM loss

[44] and LPIPS loss

[45] , This is the loss of timing consistency. L2 =1,λ offset =10,λ scale =1,λ1=0.8,λ ssim =0.2,λ Lpips =0.2,λ tem =0.02, and the influence of each loss item in the total loss is adjusted by the corresponding weight to balance the optimization objectives of various aspects.

[0040] Experimental part

[0041] Experimental dataset: The present invention is evaluated and tested on the datasets People-Snapshot and DynVideo.

[0042] People-Snapshot Dataset: This dataset consists of videos of people rotating in a fixed pose, shot from a single angle in front of a green screen. It is commonly used for 3D human reconstruction and modeling tasks under controlled conditions. It focuses on reconstructing only the human body to better exclude background interference and improve the accuracy of comparative evaluation.

[0043] DynVideo dataset: To enhance the challenge of model evaluation, this paper uses the DynVideo dataset. This dataset contains videos of various complex and rotating movements recorded by male and female characters using mobile devices, including high-dynamic scenes such as waving and turning. Evaluation metrics:

[0044] PSNR: A measure of the quality of the reconstructed image relative to the original image.

[0045] SSIM: is a perceptual metric that evaluates the similarity between two images based on brightness, contrast, and structure.

[0046] LPIPS: It evaluates perceptual differences by comparing image feature activations.

[0047] Experimental Setup: In this experiment, we sampled approximately 200,000 points from the SMPL grid model for Gaussian initialization. The adaptive control module was not enabled in this experiment, so the number of Gaussian points remained constant during training. The entire network framework was trained on a single NVIDIA RTX 4090 GPU, with training time ranging from 0.5 to 8 hours depending on the experimental configuration. Furthermore, we performed multiple training runs with different initial conditions to ensure the stability and robustness of the model performance.

[0048] Comparative Experiments: The proposed method is compared with some baseline methods and the most advanced 3D human avatar reconstruction methods on the People-Snapshot and DynVideo datasets. Table 1 shows the quantitative results on the People-Snapshot dataset, and Table 2 shows the quantitative results on the DynVideo dataset. Both methods achieve good results. Figure 3 Qualitative comparative experimental results are presented.

[0049] The present invention further verifies the performance of the model in novel posture tests. Figure 4 The new posture reconstruction effect on two data sets is demonstrated. The example on the left is the result of human avatar reconstruction, and the left and right sides of the dotted line are the comparisons between the method of the present invention and the existing method GaussianAvatar.

[0050] The method of the present invention demonstrates good detail restoration capabilities in reconstructing new postures, especially in areas such as the face and hands. This result further demonstrates the robustness and adaptability of the method of the present invention in dealing with complex postures.

[0051] Table 1. Results on the People-Snapshot dataset. The best and second-best results are in bold and Underline Highlight.

[0052]

[0053] Table 2. Results on the DynVideo dataset. The best and second-best results are in bold and Underline Wire Highlight.

[0054]

[0055] Ablation experiments: We conducted an ablation study on the male-3-casual data from the People-Snapshot dataset, removing the Clip Text module and the Multi-Tem module to verify the contributions of different components in our approach. Table 3 shows the impact of each module on model performance.

[0056] Table 3. Ablation experiments on different components of the model

[0057]

[0058]

Claims

1. A monocular video Gaussian human avatar method based on multi-resolution image and text fusion, characterized in that: The steps include: Step 1: First, a customized text description of the human body video is generated in the text image feature processing module as model input. The text description includes the human body posture, clothing, and movements. The CLIP model is used to extract text features related to the human body posture and appearance. Subsequently, the extracted text features are fused with visual features extracted from the video frames. The visual features are extracted from the video frames using a convolutional neural network (CNN) or other deep learning model, and include the spatial layout, texture information, and dynamic changes of the human body. The text features and visual features are combined through a splicing operation to form a multimodal fused feature representation. This fused feature representation is used to guide subsequent 3D modeling and human avatar generation. Step 2: To improve the accuracy of dynamic human reconstruction, a multi-resolution feature-enhanced temporal consistency module is designed. Low-resolution and high-resolution features are extracted from the input video frames. The low-resolution features are used to capture large-scale motion changes of the human body, while the high-resolution features focus on modeling the details. The features of the two resolutions are combined through weighted fusion or attention mechanisms to fully capture the global and local dynamic details of the human body. To ensure temporal consistency, a temporal consistency constraint module is designed. The temporal consistency constraint module ensures smooth transitions of human motion by calculating the differences between consecutive frames, avoiding jitter or unnatural changes across frames. Step 3: Finally, based on the generated Gaussian point cloud and the fused dynamic features, rendering is performed through 3D Gaussian splash 3DGS; 3DGS processes each Gaussian point in three-dimensional space to achieve high-quality detail rendering with low computational overhead.

2. The method for Gaussian human avatar based on monocular video and multi-resolution image-text fusion according to claim 1, characterized in that: MRTI-Avatar is a 3D Gaussian human avatar reconstruction framework based on multi-resolution image and text feature fusion and temporal consistency constraints. Its input consists of three parts: SMPL obtained by video preprocessing, custom optimizable features, and custom text description of the human body in the video. input ; After extracting text features through the CLIP model, it is combined with optimizable features and UV mapping to generate preliminary input features for predicting dynamic attribute colors in Gaussian points size and position offset In order to use isotropic Gaussian points to maintain accurate 3D shapes and solve the problem of low reconstruction accuracy due to low transparency of points on the human body boundary, the rotation q and opacity α parameters are fixed.

3. The method for Gaussian human avatar based on monocular video and multi-resolution image-text fusion according to claim 1, characterized in that: The text image feature processing module processes the features, and after processing, it is input into the multi-resolution feature enhancement temporal consistency module for parameter prediction. The obtained parameters are rendered in the rendering module, and a joint optimization strategy is used in training.

4. The method for Gaussian human avatar based on monocular video and multi-resolution image-text fusion according to claim 1, characterized in that: Text image feature processing module: Model the human appearance of the current frame, generate the corresponding SMPL model from the input image in preprocessing, and generate a standard UV position feature map by sampling the points on the surface of the SMPL model of the current frame Where H and W are the height and width of the feature map respectively, C n is the number of channels, n = 1, 2, 3....; by defining an optimizable feature It is continuously optimized during training to capture human body geometry information.

5. The method for Gaussian human avatar based on monocular video and multi-resolution image-text fusion according to claim 4, characterized in that: In order to improve the accuracy of detail capture in the process of human avatar reconstruction, text features are introduced, and the CLIP model is used to incorporate natural language descriptions as additional information into feature expression; text input is generated by a detailed custom description of the human appearance in the monocular video. input , describing the key features of human clothing, posture, color and rotation perspective; Clip model encoder converts the input text description into text features in Represents a d-dimensional real space, where d is the feature dimension; the text features extracted by the Clip encoder are defined as: f clip =Clip text (Text input ) (1) Among them, Clip text For text encoder, Text input is the natural language description of the input.

6. The method for creating a Gaussian human avatar based on monocular video using multi-resolution image-text fusion according to claim 5, characterized in that: In order to transform the text feature f Clip Combine with UV coordinate map and optimizable features, first Dimension transformation to Combining the transformed features to obtain richer features: in, It is a splicing operation; it supplements visual features from the semantic level to make the generated image more in line with expectations; through the final Predict the Gaussian parameters input to the next module.

7. The method for creating a Gaussian human avatar based on monocular video using multi-resolution image and text fusion according to claim 1, wherein: Multi-resolution feature enhancement temporal consistency module: The purpose of multi-resolution feature fusion is to improve the model's ability to capture details by fusing feature representations of different resolutions; through parallel convolution operations of high and low resolution paths, the input feature f is Con The information is processed at different scales: Among them, Conv represents the convolution operation, and the high and low resolution paths are respectively represented by Conv high (1×1) and Conv low (5×5) represents convolutions of different scales; after the convolution operation, the feature map extracted by the high-resolution path is the feature map extracted from the low-resolution path; get f high With f low Finally, the high and low resolution features are fused by element-by-element addition to obtain the fused feature map It maintains the information of the original input features while having the ability to express features at multiple resolutions.

8. The method for creating a Gaussian human avatar from a monocular video based on multi-resolution image-text fusion according to claim 7, characterized in that: f multi Perform convolution operation to predict dynamic attribute color size and position offset These dynamic properties are output through the convolutional network, so that the generated 3D model can reflect the appearance characteristics of the input image in detail: To ensure that the model maintains visual coherence when processing dynamic human motion, temporal consistency loss is introduced; by constraining the changes in the position parameters of each Gaussian point between adjacent frames, the movement performance of each Gaussian point in different time steps is guaranteed to be continuous and consistent, so that the model can achieve smooth transition between frames; each frame in the video contains multiple Gaussian points, and for the i-th Gaussian point, the position offset in the T-th frame is When video frame T≥2: in, represents the position offset of the i-th Gaussian point in the T-th frame, represents the position offset of the i-th Gaussian point in the T-1th frame, and N is the number of all Gaussian points in the frame. This loss function constrains the change of Gaussian position parameters between adjacent frames through the L2 norm to minimize the position offset difference of Gaussian points in adjacent frames, making the dynamic performance of the human body in multi-frame videos more natural and coherent. The predicted parameters are introduced into the 3D Gaussian model in the canonical space, and the predicted attributes are associated with the corresponding 3D Gaussian model.

9. The method for creating a Gaussian human avatar based on monocular video using multi-resolution image and text fusion according to claim 1, wherein: Rendering module: Place the 3D Gaussian model of the standard space, i.e. the position of the Gaussian point is x, into the motion space for free view rendering: Where R represents the rendered image, Render(·) represents the rendering process of 3DGS, represents the position offset, D represents the dynamic parameters in the Gaussian point except the position, and S represents the fixed parameters in the Gaussian point.

10. The method for creating a Gaussian human avatar based on monocular video using multi-resolution image and text fusion according to claim 1, wherein: Joint optimization training strategy: Multiple loss functions are used to optimize model performance, aiming to achieve accurate and consistent human avatar reconstruction; parameter prediction loss, temporal consistency loss, and rendering loss are jointly trained; The network is guided by optimizing the following loss function: in, and They are the L2 norm in the parameter prediction loss, the predicted position offset and the predicted size, and They are respectively the rendering loss loss, SSIM loss [44] and LPIPS loss [45], This is the loss of timing consistency.

Images