Digital human modeling method and system combining gaussian sputtering image and gan model

Abstract

The application discloses a kind of digital human modeling method and system combined with Gaussian sputtering image and GAN model, belong to digital human generation technical field, including the following steps: conditional generation: from the monocular video of single person, the human key point sequence is extracted, using human key point sequence drive constructed Gaussian model generates Gaussian sputtering image sequence with the identity of person as the identity condition of person, also based on human key point sequence drawing generation hand image sequence, neural semantic image sequence and eye fixation image sequence, as action condition;Digital human modeling: generator in GAN model simultaneously as the driving condition of generator in GAN model with the identity of person condition and action condition as digital human video frame generation, constitute digital human video.This method and system, Gaussian sputtering image effectively introduces the identity condition control of person, improves the generalization ability of generation model, has wide application prospect in multiple fields.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital human generation, and particularly relates to a digital human modeling method and system combining a Gaussian sputtering image and a GAN model. BACKGROUND

[0002] In recent years, with the rapid development of digital economy, virtual digital human technology as a core carrier of new generation of human-computer interaction is profoundly changing the service mode and application scenarios of various industries. The all-weather service capability and high customizability of virtual digital human make it show great commercial value and social benefits in the fields of remote office, online education, etc. With the rapid development of artificial intelligence technology, the generation of high-fidelity and large-scale virtual digital human using generative artificial intelligence technology is gradually becoming a research hotspot.

[0003] With the rise of generative adversarial networks (GAN), people try to use generative adversarial networks to generate digital humans. StyleGAN series models, as an important branch in the field of GAN, adopt a unique style generation architecture, realizing the automatic separation of high-level semantics (such as face posture, identity) and low-level random details (such as skin texture, freckles) of generated images. For example, StyleAvatar extends StyleGAN2 from an encoder-only structure to a UNet encoding-decoding structure, making it a conditional generative adversarial network, which directly learns the mapping from 3DMM (3D Morphable Model) to real person image sequences; StyleHeat introduces an additional action encoder, and uses 3DMM coefficients or audio conditions to generate corresponding optical flow fields to distort the intermediate layer features of the StyleGAN2 network.

[0004] Three-dimensional Gaussian sputtering (3DGS for short) is an efficient three-dimensional representation method: it models three-dimensional geometry with a set of anisotropic Gaussian distributions, each Gaussian cell is described by a center position, a scale parameter and a quaternion rotation, and also contains opacity and appearance features encoded with spherical harmonics. When rendering, these voxel primitives are projected to two-dimensional screen space, and then the pixel color is combined by opacity blending in the order of the viewpoint. Since 3DGS naturally contains low-frequency shape and appearance information, it has become the mainstream representation of real-time photorealistic rendering.

[0005] LHM (Large Animatable Human Reconstruction Model) first introduces the large model idea into the reconstruction of a single image of a digital human that can be animated; the LHM model represents a portrait as a Gaussian spray, so that the model only needs to be forward propagated once to generate a 3D portrait that can be animated within a second-level time, and the 3D Gaussian set output by the model supports real-time, light-realistic rendering and pose control based on the SMPL-X skeleton.

[0006] In the above technical solution, in the traditional GAN generated digital human, the identity ID information of the characters cannot be well controlled in multi-character training, so that the model lacks generalization. SUMMARY

[0007] In view of the above, the purpose of the present application is to provide a digital human modeling method and system combining Gaussian spray images and GAN models, using Gaussian models constructed from single-frame character images to obtain Gaussian spray image sequences, and modeling multi-modal action image sequences from key point features extracted from character videos, to generate specific identity, high-fidelity digital human videos consistent with input expression actions. The method and system can reduce the development difficulty of digital human related applications, construct a multi-character pre-training, single-character fine-tuning training strategy, effectively introduce character identity condition control through Gaussian spray images, and improve the generalization ability of the generated model, which has a wide application prospect in many fields.

[0008] To achieve the above-mentioned purpose of the application, the embodiment provides a digital human modeling method combining Gaussian spray images and GAN models, comprising the following steps:

[0009] Conditional generation: extract the human key point sequence from the monocular video of a single character, use the human key point sequence to drive the constructed Gaussian model to generate a Gaussian spray image sequence with the character identity as the character identity condition, and also draw a hand image sequence, a neural semantic image sequence and an eye gaze image sequence based on the human key point sequence as the action condition;

[0010] Digital human modeling: the generator in the GAN model takes the character identity condition and the action condition as the driving condition of the generator in the GAN model to generate digital human video frames, and constitutes a digital human video.

[0011] Preferably, after extracting the human key point sequence, the human key point sequence is further calibrated, including: for monocular video, first, the Aios model is used to detect the key point sequence of the whole human body, and the HaMer model is used to detect the hand key point sequence more accurately; then, the hand gesture coefficients obtained by the HaMer model are replaced with the corresponding hand gesture coefficients in the Aios model to calibrate and update the two-dimensional projection position of the hand key point sequence of the human body detected by the Aios model, and to calculate the displacement bias between the hand key point sequence detected by the HaMer model and the hand key point sequence of the Aios model after gesture calibration in the two-dimensional projection space; based on the displacement bias, the camera displacement parameters corresponding to the Aios model human key point sequence after gesture calibration are further adjusted to align the hand key points of the two in the two-dimensional projection space, so as to obtain the calibrated human key point sequence with more accurate two-dimensional projection position and more reliable depth information for subsequent neural semantic image rendering and hand image generation.

[0012] Preferably, the Gaussian model adopts the LHM model, and the LHM model is driven by the human key point sequence to generate a Gaussian splash image sequence with the identity and action information of the person. Since Gaussian splash can provide strong constraints when modeling human geometry and low-frequency appearance information, the Gaussian splash image rendered thereby not only retains the identity and pose of the person, but also contains coarse-grained texture and lighting information. Therefore, the Gaussian splash image output by the LHM is introduced into the training of the GAN model as the identity condition of the generated control image, which can serve as a guide for the identity ID and coarse-grained image information. In this way, the introduction of the Gaussian splash image can make up for the defect that the existing GAN generation scheme cannot fully control the identity of the person, thereby indirectly increasing the size of the training sample, improving the generalization of the model and reducing the training cost, opening up new application space for digital human generation technology, and bringing a wide range of application scenarios and practical value.

[0013] Preferably, the generator includes a Gaussian splash image encoder, an action condition image encoder, and a decoder. The Gaussian splash image encoder is used to encode the Gaussian splash image sequence as the identity condition to obtain identity feature vectors of different dimensions. The action condition image encoder is used to encode the hand image sequence, the neural semantic image sequence, and the eye gaze image sequence as the action condition to obtain action feature vectors of different dimensions. The decoder is used to receive the action feature vectors and the identity feature vectors as driving conditions, to generate digital human video frames with actions based on the driving conditions by denoising and decoding the latent noise vectors, and to compose the digital human video.

[0014] The Gaussian sputtering image encoder and the motion condition image encoder have the same structure. The Gaussian sputtering image sequence is input to the Gaussian sputtering image encoder. The hand image sequence, the neurosemantic image sequence, and the eye gaze image sequence are spliced ​​together and then input to the motion condition image encoder. The identity feature vector and the motion feature vector output by the two modules are spliced ​​together in different dimensions and then input to the decoder. The decoder residual connection receives the spliced ​​feature vector output by the motion condition image encoder and the Gaussian sputtering image encoder, introducing multi-dimensional driving conditions for human actions and human identities.

[0015] Preferably, the GAN model undergoes adversarial training with multi-person sample data to optimize generator parameters. During adversarial training, the projection discriminator included in the GAN model is used to determine the authenticity of input video frames. The projection discriminator structure includes a pre-trained encoder (such as the vit_small_patch16_224_dino encoder) and a multi-head attention discriminator. The pre-trained encoder is used to generate feature vectors of different dimensions from the input video frames through a multi-layer transformer structure. The multi-head attention discriminator receives the multi-dimensional feature vectors and combines them with the CLIP feature vectors corresponding to the Gaussian sputtering image to determine the authenticity of the input video frames.

[0016] Preferably, during adversarial training, the loss function used includes generation loss and discrimination loss, wherein the generation loss includes reconstruction loss based on the difference between generated video frames and real video frames, image perception loss based on the difference between image features of generated video frames and image features of real video frames, and adversarial loss based on the projection discriminator. ;

[0017] Among them, combating losses , Indicates the first i Frame generation video frames With the i CLIP feature vector corresponding to a frame of Gaussian sputtering image In the true / false discrimination result of the projection discriminator, N represents the number of images;

[0018] Determine the loss , Indicates the first i Frame Real Frame With the i CLIP feature vector corresponding to a frame of Gaussian sputtering image The true / false judgment result of the projection discriminator.

[0019] To achieve the above-mentioned objectives, embodiments also provide a digital human modeling system combining Gaussian sputtering images and GAN models, comprising:

[0020] The conditional generation module extracts a human key point sequence from the single-person monocular video, generates a Gaussian splatter image sequence with the identity of the person as the identity condition of the person by using the Gaussian model driven by the human key point sequence, and draws a hand image sequence, a neural semantic image sequence and an eye gaze image sequence based on the human key point sequence as the action condition.

[0021] The digital human modeling module: the generator in the GAN model generates a digital human video frame with the identity condition and the action condition as the driving conditions of the generator in the GAN model.

[0022] To achieve the above-mentioned purposes, the embodiment also provides a digital human modeling system combining a Gaussian splatter image and a GAN model, which comprises a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to realize the above-mentioned digital human modeling method combining a Gaussian splatter image and a GAN model.

[0023] Compared with the prior art, the present application has at least the following beneficial effects:

[0024] The present application uses the Gaussian splatter image sequence with the identity of the person as the identity condition of the person, that is, uses the identity maintaining information of the Gaussian splatter image in the multi-person training sample to perform multi-person pre-training modeling, indirectly increases the size of the training sample, and enables the generator to improve the generalization generation capability of the generator on the multi-person pre-training. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0026] Figure 1 is a flowchart of the digital human modeling method combining a Gaussian splatter image and a GAN model provided by the embodiment;

[0027] Figure 2 is a schematic diagram of a Gaussian splatter image provided by the embodiment;

[0028] Figure 3 is a schematic diagram of a neural semantic image, an eye gaze image and a hand image provided by the embodiment;

[0029] Figure 4 is a structural schematic diagram of a GAN model provided by the embodiment;

[0030] Figure 5is a structural schematic diagram of a digital human modeling system provided by the embodiment and combining a Gaussian sputtering image and a GAN model. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0032] The inventive concept of the present application is that the existing digital human technology has some technical problems, such as lack of identity information in multi-person training, high cost in single-person training, lack of generalization of the model, and low quality of action generation outside the domain. Based on this, the present application provides a digital human modeling scheme combining Gaussian sputtering image and GAN model, introduces Gaussian sputtering technology in monocular video character image modeling, constructs Gaussian sputtering image image of the character, represents the character identity information, and provides coarse-grained character identity information modeling guidance for the model, so as to further improve the high fidelity of digital human video while introducing character identity information, so as to better meet the user demand and application scene requirement.

[0033] As shown in Figure 1 , the embodiment provides a digital human modeling method combining Gaussian sputtering image and GAN model, including the following steps:

[0034] S1, conditional generation: extracting human key point sequence from monocular video of single person, using human key point sequence to drive the constructed Gaussian model to generate Gaussian sputtering image sequence with character identity as character identity condition, and drawing and generating hand image sequence, neural semantic image sequence and eye gaze image sequence based on human key point sequence as action condition.

[0035] In the embodiment, the input monocular video sequence , wherein , represents the i video frame, is the video frame size, and 3 represents the number of channels. The generated frame contains human key point sequence including face key points, torso key points, hand key points and eye key points , wherein , is the number of key points, and 3 is the three-dimensional coordinates of the key points. represents the i key point sequence, and in actual extraction, , specifically contains 468 face key points, 8 torso key points, 21 hand key points per hand and 21 eye key points per eye.

[0036] Specifically, the Faceverse 3DMM model is used to extract 468 face key point sequences , wherein , the Mediapipe tool is used to extract 21 eye key point sequences for each eye , wherein , the Aios model is used to extract 8 trunk key point sequences and 21 key point sequences for each hand , wherein and , the HaMer model is used to extract more accurate 21 key point sequences for each hand , wherein ; then, the hand posture coefficient in is calibrated , and the displacement bias between and the posture calibrated in the two-dimensional projection space is calculated; based on the displacement bias, the camera displacement parameters corresponding to are further adjusted to align the hand key points in the two-dimensional projection space; after calibration, the human body key point sequence is constructed using the face key point sequence , the eye key point sequence , the trunk key point sequence , and the posture calibrated hand key point sequence .

[0037] In the embodiment, a monocular video sequence of frames is inputted, a single frame video image is selected, a Gaussian model is constructed using the LHM model, and the human body key point sequence is combined to perform Gaussian sputtering rendering after LBS deformation, so as to realize a Gaussian sputtering image sequence , as shown in . The obtained Gaussian sputtering image sequence has specific person identity information and action information, and is used as a person identity condition to drive video frame generation. Figure 2 In the embodiment, the Aios hand key point sequence

[0038] calibrated by frames is inputted, and the Pyrender is used as a rendering bottom layer to draw and generate a Mano hand image sequence .

[0039] In the embodiment, for the extracted human body key point sequence ​CCBR neural rendering semantic image sequences were plotted using OpenGL-based real-time neural semantic image mapping. Color-coded Body Representation (CCBR) models the semantic information of body posture and facial expressions through structure and color. (Eye keypoint sequence) The resulting sequence of eye gaze images like Figure 3 As shown in the figure, the hand image sequence, the neurosemantic image sequence, and the eye gaze image sequence together serve as action conditions to drive the generation of video frames.

[0040] S2. Digital Human Modeling: The generator in the GAN model uses both the person's identity and actions as driving conditions to generate digital human video frames, thus forming a digital human video.

[0041] In this embodiment, the GAN model includes a generator and a projection discriminator. The GAN model is subjected to two-stage adversarial training, which involves multi-person pre-training and single-person fine-tuning, to optimize the GAN model parameters.

[0042] like Figure 4 As shown, the generator includes a Gaussian sputtering image encoder, a motion-conditioning image encoder, and a decoder. The Gaussian sputtering image sequence, serving as the identity condition, is encoded by the Gaussian sputtering image encoder to obtain identity feature vectors of different dimensions. The hand image sequence, neurosemantic image sequence, and eye gaze image sequence, serving as the motion condition, are concatenated and input to the motion-conditioning image encoder, which encodes them into motion feature vectors of different dimensions. These motion feature vectors and identity feature vectors serve as driving conditions, and are concatenated separately for different dimensions before being input to the decoder. A CLIP encoder is also introduced. After the Gaussian sputtering image sequence is encoded into CLIP feature vectors, these vectors are projected through a mapping layer and an embedding layer to form projection vectors, which are then concatenated with latent noise variables on the channel before being input to the decoder. The decoder denoises and decodes the latent noise vectors based on the driving conditions and projection vectors to generate digital human video frames with motion, thus composing the digital human video.

[0043] Specifically, the N-frame Gaussian sputtering image sequence The input is fed into the Gaussian sputtering encoding module; N frames of neural semantic image sequence are processed. and N frames of eye gaze image sequence Hand images with N frames The sequences are concatenated along the channel dimension and uniformly represented as input action condition sequences. And input the motion condition image encoder; use the CLIP image encoder to process the Gaussian sputtering image sequence. Encoded as CLIP feature vector And projected into a projection vector through the mapping layer, along with the latent noise variables. Connect to the channel and input it to the decoder; receiving the above input, the decoder generates a sequence of N frames of generated video. ,in .

[0044] In this embodiment, a projection discriminator is used to determine the authenticity of input video frames. The projection discriminator's structure includes a pre-trained encoder (such as the vit_small_patch16_224_dino encoder, abbreviated as DINO encoder) and a multi-head attention discriminator. The pre-trained encoder generates feature vectors of different dimensions from the input video frames through a multi-layer transformer structure. The multi-head attention discriminator receives the multi-dimensional feature vectors and combines them with the CLIP feature vectors corresponding to the Gaussian sputtering image to determine the authenticity of the input video frames. Specifically, the pre-trained encoder extracts multi-dimensional feature vectors (such as local details and global semantics) from the input video frames through a multi-layer transformer structure, enabling the projection discriminator to simultaneously capture low-level texture realism (such as subtle hand joint movements) and high-level semantic consistency (such as the continuity of action sequences), significantly improving the fine-grained discrimination accuracy of the generated video frames.

[0045] In this embodiment, during adversarial training, the loss function used includes generation loss and discriminative loss, wherein the generation loss... Including reconstruction loss based on the difference between generated video frames and real video frames. Image perceptual loss based on the image features of generated video frames and image features of real video frames. and adversarial loss based on projection discriminator ;

[0046] Among them, reconstruction losses Using L1 loss, image perception loss The image features used can be those extracted by inputting video frames into the VGG network. Adversarial loss. , Indicates the first i Frame generation video frames With the i CLIP feature vector corresponding to a frame of Gaussian sputtering image In the true / false discrimination result of the projection discriminator, N represents the number of images. Then, the generation loss... Represented as:

[0047]

[0048] in, , and respectively represent the weights of the loss.

[0049] discrimination loss , represent the first i frame real frame and the first i frame Gaussian sputtering image corresponding to the CLIP feature vector The authenticity discrimination result of the projection discriminator, that is, the discrimination score.

[0050] For the adversarial learning of GAN, a multi-person training strategy is adopted, which includes:

[0051] Construct a multi-person sample data, for monocular video of different persons, construct different person's Gaussian sputtering image sequence and hand image sequence according to step S1, Gaussian sputtering image sequence, neural semantic image sequence, and eye gaze image sequence, form multi-person sample data

[0052] A multi-person training data loading module is constructed, so that the GAN model receives uniformly distributed multi-person training sample data in each round of training, avoiding the problem of uneven sample distribution caused by inconsistent number of different person samples;

[0053] Random translation and scaling enhancement settings are added to the multi-person training data loading module; when loading model training samples, according to the set random probability, the same translation and scaling strategy is performed on the hand image sequence, Gaussian sputtering image sequence, neural semantic image sequence and eye gaze image sequence, to improve the generalization ability of the model.

[0054] After the adversarial training of the sample data using the person identity information, the parameter-optimized generator is used as a video generation model for video frame generation. Specifically, the generation of digital human video based on the generator inference includes:

[0055] First, based on the human key point sequence constructed according to the target action, the target hand image sequence, the target Gaussian sputtering image sequence, the target neural semantic image sequence and the target eye gaze image sequence are drawn;

[0056] Then, the generator generates continuous target digital human images based on the target hand image sequence, the target Gaussian sputtering image sequence, the target neural semantic image sequence and the target eye gaze image sequence, and introduces random noise, and adds audio to the continuous digital human images to obtain a digital human video.

[0057] Based on the above invention concept, such as Figure 5As shown, the embodiment also provides a digital human modeling system 50 combining a Gaussian splat image and a GAN model, including a conditional generation module 51 and a digital human modeling module 52. Among them, the conditional generation module 51 is used to extract a human key point sequence from a monocular video of a single person, use the human key point sequence to drive a constructed Gaussian model to generate a Gaussian splat image sequence with a person identity as a person identity condition, and also draw a hand image sequence, a neural semantic image sequence and an eye gaze image sequence based on the human key point sequence as an action condition; the digital human modeling module 52: the generator in the GAN model generates a digital human video frame with the person identity condition and the action condition as the driving condition of the generator in the GAN model, and constitutes a digital human video.

[0058] It should be noted that the digital human modeling device combining a Gaussian splat image and a GAN model provided by the above embodiment should be illustrated by the division of the above functional modules when performing digital human modeling. The above functions can be completed by different functional modules as needed, i.e., the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the digital human modeling device combining a Gaussian splat image and a GAN model provided by the above embodiment and the digital human modeling method combining a Gaussian splat image and a GAN model embodiment belong to the same concept, and the specific implementation process is described in detail in the digital human modeling method combining a Gaussian splat image and a GAN model. Here, it will not be repeated.

[0059] Based on the same inventive concept, the embodiment also provides a digital human modeling system combining a Gaussian splat image and a GAN model, including a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the above-mentioned digital human modeling method combining a Gaussian splat image and a GAN model, specifically including the following steps:

[0060] S1, conditional generation: extract a human key point sequence from a monocular video of a single person, use the human key point sequence to drive a constructed Gaussian model to generate a Gaussian splat image sequence with a person identity as a person identity condition, and also draw a hand image sequence, a neural semantic image sequence and an eye gaze image sequence based on the human key point sequence as an action condition;

[0061] S2, digital human modeling: the generator in the GAN model generates a digital human video frame with the person identity condition and the action condition as the driving condition of the generator in the GAN model, and constitutes a digital human video.

[0062] The computing device provided by the embodiment comprises, in addition to the processor and the memory, internal buses, network interfaces, memories and other hardware required by the business. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the digital human modeling method combining the Gaussian sputtering image and the GAN model as described in S1-S2. Of course, in addition to the software implementation, the present application does not exclude other implementation manners, such as logic devices or the combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0063] The specific embodiments described above explain the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A digital human modeling method combining Gaussian sputtering images and GAN models, characterized in that, Includes the following steps: Conditional generation: Extract human key point sequences from monocular videos of a single person, use the human key point sequences to drive the construction of a Gaussian model to generate a Gaussian sputtering image sequence with the person's identity as a condition for the person's identity, and also generate hand image sequences, neurosemantic image sequences and eye gaze image sequences based on the human key point sequences as action conditions. Digital human modeling: The generator in the GAN model uses both the person's identity and actions as driving conditions to generate digital human video frames, thus forming a digital human video. The GAN model undergoes adversarial training with multi-person sample data to optimize generator parameters. During adversarial training, the loss function used includes generation loss and discriminative loss. The generation loss includes reconstruction loss based on the difference between generated and real video frames, image perception loss based on the difference between image features of generated and real video frames, and adversarial loss based on the projection discriminator. ; Among them, combating losses , Indicates the first i Frame generation video frames With the i CLIP feature vector corresponding to a frame of Gaussian sputtering image In the true / false discrimination result of the projection discriminator, N represents the number of images; Determine the loss , Indicates the first i Frame Real Frame With the i CLIP feature vector corresponding to a frame of Gaussian sputtering image The true / false judgment result of the projection discriminator.

2. The digital human modeling method combining Gaussian sputtering images and GAN models according to claim 1, characterized in that, The Gaussian model uses the LHM model and is driven by the human keypoint sequence to generate a Gaussian sputtering image sequence with human identity.

3. The digital human modeling method combining Gaussian sputtering images and GAN models according to claim 1, characterized in that, The generator includes a Gaussian sputtering image encoder, a motion condition image encoder, and a decoder. The Gaussian sputtering image encoder is used to encode the Gaussian sputtering image sequence, which serves as the identity condition, to obtain identity feature vectors of different dimensions. The motion condition image encoder is used to encode the hand image sequence, the neurosemantic image sequence, and the eye gaze image sequence, which serve as the motion condition, to obtain motion feature vectors of different dimensions. The decoder is used to receive the motion feature vector and the identity feature vector and use them as driving conditions. Based on the driving conditions, it denoises and decodes the latent noise vector to generate digital human video frames with motion, and then assembles the digital human video.

4. The digital human modeling method combining Gaussian sputtering images and GAN models according to claim 3, characterized in that, The Gaussian sputtering image encoder and the motion condition image encoder have the same structure. The output identity feature vector and motion feature vector are concatenated in different dimensions and then input into the decoder.

5. The digital human modeling method combining Gaussian sputtering images and GAN models according to claim 3, characterized in that, The generator also incorporates a CLIP encoder. After the Gaussian sputtering image sequence is encoded into a CLIP feature vector by the CLIP encoder, it is projected into a projection vector through a mapping layer and an embedding layer. This projection vector is then connected to the latent noise variables on the channel and input to the decoder for denoising and decoding.

6. The digital human modeling method combining Gaussian sputtering images and GAN models according to claim 1 or 3, characterized in that, During adversarial training, the GAN model includes a projection discriminator to determine the authenticity of the input video frames. The projection discriminator consists of a pre-trained encoder and a multi-head attention discriminator. The pre-trained encoder generates feature vectors of different dimensions from the input video frames through a multi-layer transformer structure. The multi-head attention discriminator receives the multi-dimensional feature vectors and combines them with the CLIP feature vectors corresponding to the Gaussian sputtering image to determine the authenticity of the input video frames.

7. The digital human modeling method combining Gaussian sputtering images and GAN models according to claim 1, characterized in that, After extracting the human keypoint sequence, the human keypoint sequence is also calibrated, including: For monocular video, the Aios model is first used to detect the keypoint sequence of the entire human body, while the HaMer model is used for more precise and accurate detection of the hand keypoint sequence. Then, the hand pose coefficients obtained from the HaMer model are used to replace the corresponding hand pose coefficients in the Aios model to calibrate and update the 2D projection position of the human hand keypoint sequence detected by the Aios model. The displacement offset between the hand keypoint sequence detected by the HaMer model and the pose-calibrated hand keypoint sequence of the Aios model in the 2D projection space is calculated. Based on this displacement offset, the camera displacement parameters corresponding to the pose-calibrated human keypoint sequence of the Aios model are further adjusted to align the hand keypoints in the 2D projection space, thereby obtaining a calibrated human keypoint sequence with more accurate 2D projection position and more reliable depth information.

8. A digital human modeling system combining Gaussian sputtering images and GAN models, characterized in that, include: Condition generation module: It is used to extract human key point sequences from monocular videos of a single person, and use the human key point sequences to drive the construction of a Gaussian model to generate a Gaussian sputtering image sequence with the person's identity as a condition for the person's identity. It also draws and generates hand image sequences, neurosemantic image sequences and eye gaze image sequences based on the human key point sequences as action conditions. Digital Human Modeling Module: The generator in the GAN model uses both the person's identity and actions as driving conditions to generate digital human video frames, thus composing the digital human video. The GAN model undergoes adversarial training with multi-person sample data to optimize generator parameters. During adversarial training, the loss function used includes generation loss and discriminative loss. The generation loss includes reconstruction loss based on the difference between generated and real video frames, image perception loss based on the difference between image features of generated and real video frames, and adversarial loss based on the projection discriminator. ; Among them, combating losses , Indicates the first i Frame generation video frames With the i CLIP feature vector corresponding to a frame of Gaussian sputtering image In the true / false discrimination result of the projection discriminator, N represents the number of images; Determine the loss , Indicates the first i Frame Real Frame With the i CLIP feature vector corresponding to a frame of Gaussian sputtering image The true / false judgment result of the projection discriminator.

9. A digital human modeling system combining Gaussian sputtering images and GAN models, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the digital human modeling method combining Gaussian sputtering images and GAN models as described in any one of claims 1-7.

Images