High-simulation virtual human generation method and system

Abstract

The invention provides a high-simulation virtual human generation method and system, and aims to gradually take a low-order virtual human as a generation source of a higher-order virtual human and carry out iterative approximation from basic image data. The technology comprises the following steps: firstly, obtaining a 2D image: step A, generating a virtual human head skeleton model file for preliminary three-dimensional face reconstruction, and then connecting the virtual human head skeleton model file to a body model bound with skeletons; b, training a neural network model for replacing the face in real time; in the step C, on one hand, the preliminary model generated in the step A is controlled through a program to realize changing actions, mouth shapes and expressions, and on the other hand, a low-order virtual human operation window is converted into virtual camera input to serve as video source input; and then performing real-time replacement with a higher simulation degree on the face by using the real-time replacement neural network model trained in the step B for the face so as to drive and form a virtual human with a higher simulation degree.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer software, in particular, a high-fidelity digital human development and production method and system. BACKGROUND

[0002] Virtual human is a product of the deep integration of technology and culture. Its concept can be traced back to 1989, but it did not really enter the public eye and attract widespread attention until a decade or so. With the continuous progress of key technologies such as computer graphics (CG), motion capture, and deep learning algorithms, virtual human technology has made significant strides. In particular, in the early 21st century, as these technologies continue to mature, virtual humans have shown great potential in the fields of film and television production, electronic games, and others. In 2007, Japan launched the virtual singer "Hatsune Miku", which pushed the concept of virtual idols to a new height and became the object of enthusiastic pursuit by young people around the world. Modern virtual human technology is mainly composed of multiple core modules, including character image design, speech synthesis, animation generation, audio and video synthesis and display, and interactive systems.

[0003] In recent years, with the continuous breakthroughs in artificial intelligence (AI) technology, the virtual human market has shown rapid growth. The wide application scenarios of virtual human technology have covered entertainment, education, finance, e-commerce, media, and other fields. In the entertainment industry, virtual idols and virtual anchors have become representatives of emerging forms of entertainment, enriching the public's entertainment life and creating huge economic value for related industries. In the field of education, virtual teachers can provide personalized and interactive teaching experiences, effectively improving teaching efficiency and quality. In the financial industry, service-oriented digital humans such as virtual customer service and virtual consultants can significantly reduce labor costs while improving service quality and customer experience. In addition, in the media, e-commerce, and other fields, virtual humans have shown great application potential and broad market prospects.

[0004] Although virtual human technology has made significant progress, its development still faces many challenges. Although the current technology can create highly realistic virtual human images, the production of highly realistic virtual humans is costly and the production cycle is relatively long. In addition to the high technical threshold, this centralized development mode also has many shortcomings. First, the current digital human production process is tedious and resource-intensive, but the fidelity still needs to be improved. The traditional digital human creation process involves complex modeling, rigging, and animation, which leads to high production costs and long production cycles. Even the products of professional institutions such as Microsoft's Xiaoice and Miha's Luming can still be clearly distinguished from real humans, which is a common feature of human NPC (non-player characters). Second, the current digital human also faces significant difficulties in updating and iterating. Once the digital human model is completed, its design is fixed and cannot be changed, and it is difficult to add functions. Due to the characteristics of centralized development, subsequent updates and function expansion face significant cost challenges. Most development projects rarely improve and optimize after initial development. In view of the problems existing in the prior art, the present application proposes a new virtual human production method. This method aims to significantly speed up the production time of high-fidelity virtual humans and facilitate subsequent upgrades and optimizations. SUMMARY

[0005] The present application proposes a method and system for generating high-fidelity virtual humans. It is important to note that all actions to obtain signals, information or data in this application are carried out in compliance with the data protection regulations and policies of the country where the device is located, and with the authorization of the owner of the corresponding device. The core of this method is to generate a high-fidelity 3D virtual human from a 2D image (photo or video) of a person. The preferred way to implement the present application is as follows: Step A, preliminary virtual human model construction: Based on the 2D image of the person, a 3D virtual human three-dimensional face reconstruction head model file is generated, and the generated head model is combined with a body model with a bound skeleton structure to form a preliminary low-fidelity virtual human model. In the specific implementation case of the present application, in step A, instead of reconstructing complete virtual human data from the image of the person, an algorithm based on a generative adversarial network is used to generate a three-dimensional head model without hair; the generated head model is combined with a three-dimensional human body model with a pre-bound skeleton, a pre-adapted hairstyle, and clothing.

[0006] In the specific implementation case of the present application, the method steps of creating a virtual digital human head in step A are as follows: first, a high-quality face dataset containing a large number of samples is constructed; then, the face is detected using the Dlib library, and the region containing only the face is cropped, and these processed images are used to train the StyleGan2 model; for a new face test image, the hair removal operation of the face image is performed through the HairMapper model, and the rembg algorithm is used to remove the image background; then, the trained model is used to map the image into the latent space to generate the image and its latent representation; then, the three-dimensional face reconstruction based on the generative adversarial network model is used, and the black background cutting of the model is performed through the Mesh Cutter plug-in of Unity, and finally the optimized three-dimensional hairless face model is obtained; and then combined with the pre-made hair and spliced with the pre-made body with bound skeleton.

[0007] Step B, based on the face image, a neural network model for subsequent real-time face replacement is trained. The occurrence order of step B can be adjusted, that is, it can be advanced before A, or it can be delayed at the same time as step C. The neural network model for real-time face replacement can be a specially trained refined model, which is based on the construction of a deepfake content based on an open source machine learning framework and a convolutional neural network (CNN) model. The pre-trained model such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once) is used to detect and locate the face in the image, and other pre-trained models (such as Mask R-CNN) are used for pixel-level face segmentation or replacement. Of course, it can also be an open source model such as the dfm model, and it can also be a model based on a single image for target face replacement (including but not limited to deep cam live or roop model).

[0008] Then in step C, on the one hand, the preliminary virtual human model generated in step A is controlled by the program to change the action, mouth shape and expression of the body and face skeleton, and on the other hand, the virtual human running window is converted into a virtual camera input as a video source input, and then through face recognition (including but not limited to using the YOLO model), the low-fidelity face of the virtual human in the window is recognized, and then the real-time replacement neural network model trained in step B for the face is used to replace the face with higher fidelity in real time to drive the formation of a virtual human with higher fidelity. Therefore, when the program controls the low-order virtual human to perform actions, dance and conversation, it appears as a virtual human with higher fidelity performing actions, dance and conversation, etc.

[0009] The procedure in step C includes five parts of a face detector, a face aligner, a face marker, a face blender, and a frame adjuster. The face detector functions to quickly and accurately detect the location of a face in the input video or image, effectively distinguish the face from non-face objects, further analyze the attributes of the face, and provide necessary data input for subsequent synthesis. The face aligner functions to roughly place different feature points of the face at the same relative position (within a certain range of the same position), and correct the posture according to a preset standard posture to avoid unnatural face replacement due to posture problems and improve the accuracy and stability of face replacement. The face marker functions to determine the key areas of the face, process the detailed features of the face, and improve the accuracy, authenticity, and naturalness of face alignment. The face blender blends the image features of the source face and the target face, and processes the expression features to achieve a natural transition of expression. The frame adjuster balances the adaptation of picture quality and face replacement effect in the entire process, and controls the smoothness and playing effect of the video.

[0010] The above flow is shown in the accompanying drawings of the specification. Figure 1 The completed virtual person produced by the method of the present application can be easily upgraded and modified due to the flexibility of changing parameters at each step.

[0011] Meanwhile, the system (server side) adopts a multi-opening, or multi-sandbox, or multi-virtual machine design method, so as to start N (more than 2) multi-opening video processing software in step C, wherein the Nth window takes the (N-1)th window as a video input source or a virtual camera, and so on, and the information of the (N-1)th window is input into the Nth window for further image processing, including but not limited to beautifying, editing, and special effects.

[0012] Alternatively, through a multi-opening, or multi-sandbox, or multi-virtual machine design method, M (more than 2) multi-opening OBS streaming software are started in step C, wherein one takes another as a video source, and further image processing including but not limited to beautifying, editing, and special effects is performed, so as to form a virtual person with higher fidelity, and the last Mth OBS source is used for OBS streaming to provide virtual person image services to the public.

[0013] In a specific implementation of this invention, different facial expressions and animations can be pre-assigned to the virtual human to build an expression library. For example, one method is as follows: 1. Design a video face feature point tracking algorithm. First, use a camera to capture images of the actor's facial expressions. Then, use a cascaded convolutional neural network algorithm for multi-task learning and a cascaded gradient-enhanced regression tree algorithm to detect faces and extract feature points. 2. Set facial animation control parameters. In the animation control parameters, use the EPnP algorithm to solve for head pose and generate corresponding head rotation matrices and translation vectors. Use a support vector machine to build an AU detector and an intensity value regressor. By establishing a mapping relationship between AU and the expression basis, the required expression parameters are obtained. 3. Then, transmit the facial expression control parameters to the low-level 3D virtual human software through a plugin to drive the low-level virtual human model to generate corresponding facial expression animations. 4. When the low-level virtual human model makes a corresponding facial expression, the face replacement in step C will track the expression that makes the same action.

[0014] Specific Implementation Examples of the Invention This invention has been successfully implemented in a specific embodiment. As a specific embodiment, based on the above basic architecture, this embodiment also makes the following settings: This specific implementation example utilizes a high-performance A800 AI computing server array (each server has 8 A800 GPUs, a 768GB RAM pool, and a 16TB hard drive for data storage, totaling 8 servers). A server-side kernel sandbox is installed to create independent user spaces, providing a dedicated management environment for each digital human model and ensuring independence and isolation between models. Sandbox isolation can also be implemented using virtual machine technologies (such as Docker or Kubernetes). The system is equipped with a content management system to support administrators' independent uploading, management, and distribution functions. This case study leverages sandbox isolation technology and decentralized blockchain storage. The three servers are connected to form an IPFS node network, and data is split and stored across all servers, ensuring high availability and security. No single server can form a complete dataset, enhancing the system's resistance to attacks and fault tolerance.

[0015] In a specific embodiment of the present invention, the system also allows for further iterations of the digital human to higher orders through a hierarchical structure in the following form: The system consists of a supply and distribution end (the end where steps A and B occur), a distributed reorganization and secondary training node (the end where step C occurs), a database (which configures system parameters and information, stores user data, model information, training records, and similar functions), and an electronic ledger of ownership. In this implementation case, the supply and distribution end is a centralized server, while the distributed reorganization and secondary training node can be the same server array or the data center of the next-level enterprise user. The system achieves automatic distributed evolution through the following mode: First, primary users on the supply and distribution sides create projects, upload image data, and set initial ownership allocation ratios. After primary users create low-level virtual humans according to the process described in steps A and B of claim 1, distributed nodes download the low-level virtual humans and trained facial replacement models (if any). Different distributed nodes can give different secondary modifications to the highly realistic virtual humans in step C (including but not limited to different styles of clothing, different route styles, different language models, and different facial processing models). After these nodes process the high-level virtual humans, they are used to interact with different audiences on relevant platforms, including but not limited to OBS streaming, dialogue, program interaction, etc., and different modifications are subsequently performed based on the feedback from their respective audiences.

[0016] In this example, the computing power requirement of the distributed nodes is therefore much smaller than that of the main server.

[0017] The steps for creating a virtual digital human head in this example are as follows: First, a high-quality face dataset containing a large number of samples is constructed. Next, the Dlib library is used to detect faces and crop out regions containing only faces; these processed images are used to train the StyleGAN2 model. A custom-trained StyleGAN2 model is then used to generate face images. The training process is based on the PyTorch framework. Before training begins, necessary Python libraries and functions are imported, including PyTorch and its submodules, data loading and processing tools, etc. Weights & Biases enables experimental tracking and visualization; the loss and other metrics during training are recorded on the W&B platform. Furthermore, image samples are generated and saved periodically to visualize the training progress. For data loading, the `data_sampler` function is used to process data sampling. Since distributed training is not used, this function will default to using `data.RandomSampler` (for randomly shuffled data) or `data.SequentialSampler` (for sequential data), using the `sample_data` generator function to infinitely loop through batches of data from the data loader. Then, based on the command-line arguments, the generator and discriminator models, along with their optimizers, are initialized. In this scenario, `channel_multiplier` is set to 2, increasing the number of channels in each convolutional layer of the model. During model training, a batch of real images is first loaded from the dataset. Then, the discriminator is trained, the generator parameters are fixed, the discriminator's loss for real and generated images is calculated, and backpropagation and iterative parameter updates are performed.

[0018] However, it's important to emphasize that in this specific example, a newly received face image is not processed directly. Instead, the HairMapper model is used to remove hair from the face image, and the rembg algorithm is used to remove the image background. See the attached manual. Figure 2 .

[0019] Hair Removal Method: First, pixel values ​​are normalized. Next, a pre-trained pSp (PyramidScene Parsing Network) deep learning model is used to encode them into a latent space representation. This step transforms the image into a form that is easier to manipulate and analyze in a computational model. Subsequently, the StyleGAN2-ADA generator, a face parsing network, and image fusion techniques are used to remove hair while preserving facial features. The StyleGAN2-ADA generator is initialized, and a level mapper is loaded to adjust the latent codes to achieve the desired hair removal effect. Simultaneously, the face parsing network is loaded to generate a hair mask. This step indicates the regions in the image containing hair. Next, images and their corresponding latent codes are read from a specified data directory. For each image, its format (PNG or JPG) is checked, and the corresponding latent code is loaded accordingly. For each image, based on its latent code, a level mapper is applied to generate an edited latent code. Then, the face parsing network is used to generate a hair mask, further identifying which regions in the image contain hair. In the hair removal and image editing steps, setting the `--remain_ear` parameter preserves the ear portion within the generated hair mask. Then, the StyleGAN2-ADA generator is used to convert the edited latent code into the edited image. Image dilation and blurring techniques are applied to adjust the hair mask for a more natural hair removal effect. (In the post-hair removal image compositing stage, enabling the `--diffuse` parameter performs an additional diffusion step, optimizing the hair removal result through blending with the original image.) Background Removal Method: This specific example uses the rembg algorithm to implement background removal. This algorithm can identify and separate foreground objects from the background in an image, thus automatically removing the background. First, the remove function from the rembg library is imported. Next, two variables, input_path and output_path, are specified to define the path of the image to be processed and the path where the processed image will be stored, respectively. Then, the image file to be processed is opened in binary read mode, and its contents are read. Following this, the read image data is passed to the remove function to perform the background removal operation. The remove function relies on a pre-trained deep learning model, which can automatically identify and separate foreground objects from complex backgrounds. After background removal, the image data is returned by the remove function, and this data is written to the previously defined output file path.

[0020] After completing the above steps, before face reconstruction, the trained model is needed to map the image into a latent space to generate the image and its latent representation. This process involves finding a set of latent vectors such that when these vectors are forward-propagated through a pre-trained generative model, they can reconstruct a specific given image. First, initialization and parameter settings are performed. PyTorch, torchvision, the image processing library PIL, the progress bar library tqdm, and the perceptual loss calculation library LPIPS are imported. Command-line arguments are parsed, including the type checkpoint path, image size, learning rate, and noise parameters. Then, the pre-trained generative and discriminative models are loaded and set to evaluation mode, and the input image is loaded and transformed through the preprocessing steps described above. Subsequently, in the optimization process, for each input image, a set of latent vectors and learnable noise parameters are initialized. The optimizer Adam is set for the latent vectors and noise parameters, and the latent vectors and noise parameters are iteratively updated by calculating the loss between the generated and target images. Adding random noise to the latent vectors increases the diversity of the generated images. This method introduces slight variations that help explore different regions of the latent space and prevent the optimization process from getting trapped in local minima, thus improving the accuracy and quality of the projection. Finally, after each iteration, the noise parameters are normalized. The optimized latent vectors, generated images, and any relevant noise parameters are saved to a .pt file for subsequent 3D face reconstruction operations.

[0021] 3D face reconstruction (excluding hair and the area below the neck) can be performed using the four-step method proposed in this example. The four-step model based on Generative Adversarial Networks (GANs) is used for 3D face reconstruction, and the black background is cropped using Unity's Mesh Cutter plugin to obtain an optimized 3D hairless face model. It is called a four-step method because this part of the 3D face reconstruction process in this example mainly includes the following four stages.

[0022] (1) Initial shape setting An ellipsoid is chosen as the prior for the object's shape. As a fundamental geometric shape, the ellipsoid provides a reasonable starting point for subsequent steps without unduly constraining the design.

[0023] (2) Shape and texture restoration stage This stage aims to recover the 3D shape (depth map) and surface texture from a given 2D image. Using a differentiable renderer, pseudo-samples are rendered under multiple viewpoints and lighting conditions based on the initial shape (ellipsoid). While these pseudo-samples are geometrically similar to real objects, they may deviate from the real image in terms of texture, detail, and lighting. The netD (depth prediction network) predicts the corresponding depth map from the input image, providing a foundation for 3D reconstruction based on the distances between objects in the scene and the observer. The netV (viewpoint estimation network) estimates the camera viewpoint from the input image. This includes the camera's position, orientation, and relative angle to the object, enabling the model to observe and reconstruct 3D objects from different angles. The netL (lighting estimation network) estimates the scene's lighting conditions, including the direction, intensity, and color of the light source, directly affecting the model's shadows, highlights, and overall lighting effects, which is crucial for generating realistic 3D reconstructed images. The netA (texture prediction network) is responsible for recovering the texture information of the object's surface from the input image, including color and texture, which helps to improve the visual realism and detail richness of the reconstructed model.

[0024] (3) Potential space projection stage This stage aims to learn the mapping from 2D images to a latent space defined by a pre-trained GAN model. The 2D image is projected into the latent space using netEnc (encoder network), and the corresponding 2D image is generated inversely by the GAN. The loss function for this process includes L1 loss, reconstruction loss, and regularization loss for the latent vector. The latent space projection stage enhances the model's expressive power within the latent space, laying the foundation for detailed recovery in subsequent steps.

[0025] (4) Comprehensive learning stage This stage combines the results of the previous two stages, refining information such as viewpoint, lighting, and texture through a comprehensive optimization process. In particular, it adjusts viewpoint and lighting estimates to more accurately simulate the rendering effect of 3D scenes. Finally, the loss function of the generator trained by this generative adversarial network is based on reconstruction loss, perceptual loss, and depth smoothing loss, while also considering the consistency between the reconstructed image and the projected image.

[0026] In this implementation, the generated 3D head model does not include the hair and the area below the neck; these parts will be customized and added by the distributed recombination and secondary training nodes. After downloading, the client combines the 3D head with the pre-processed body model bound to the skeletal structure. The effect can be seen in the attached manual. Figure 3 and attached Figure 4 .

[0027] Furthermore, in step C, the server constructs an independent dialogue model for each virtual digital human based on the content uploaded by the user. In this specific implementation example, the server also receives audio data uploaded by the user and trains the model using deep learning technology to generate a voice clone model with specific speaker characteristics. The server then mimics the timbre and intonation of the user-uploaded reference audio and uses Natural Language Processing (NLP) technology to derive the corresponding speech segments of the text to be generated. This enables the virtual digital human to engage in natural speech dialogue with others.

[0028] The inventiveness of this invention The inventiveness of this invention is mainly reflected in the following aspects.

[0029] 1. Similar to the approximation and iterative generation of mathematical derivatives, it is low-cost and fast: Innovation: Compared to traditional methods of creating virtual humans, this method does not directly target a single production result. Instead, it progresses from low to high dimensions, using lower-order virtual humans to gradually drive higher-order models. All of this can be achieved through computers, requiring only computing power from production to application without human intervention, thus resulting in low cost and high speed. Furthermore, the stepwise approximation method leads to higher accuracy.

[0030] 2. Excellent effect and high realism: Innovation: Different people often have different facial features and body types. In the past, even some computationally intensive artificial intelligence algorithms could not achieve a high degree of realism. For example, the quality of the virtual person created by deepfake, which has a certain influence in the industry, is closely related to the original actor's facial features, weight, face shape, body type, and skin color. However, with this technology, each stage of the virtual person is developed based on the previous stage, and the body characteristics are continuous. For example, in the stage of using 3D reconstructed virtual people to drive highly realistic virtual people, if random artificial actors are used, there will be differences in body type. Moreover, if the actor and the target person have very different expression habits, the expression after the facial replacement may appear unnatural. For example, if the actor habitually smiles with a large upward curve of the corners of the mouth, while the target person's smile is more subtle, strange expressions may appear.

[0031] This invention presents a method and system for driving highly realistic virtual humans using a 3D reconstructed virtual human. It employs a generative adversarial network (GAN) to first generate a virtual human with a skeleton and a preliminary 3D reconstructed face, already possessing a certain degree of realism. This virtual human then drives a further, more realistic model, enabling real-time face-swapping and interactive live-streaming digital humans without human intervention. On a computer, the 3D virtual human generated using the GAN can be run. The program can drive the body and facial skeletons to achieve movements and expressions. The running window is then used as a virtual camera input as the video source, instead of the traditional method of using a live camera. This gives it a more lifelike appearance and demeanor. (See appendix) Figure 5 As shown.

[0032] 3. High controllability: Innovation: Compared to real people, the facial and body skeletons of the 3D virtual human in this invention are program-controlled, making them more controllable. Then, by training a dialogue model to control the virtual human's responses to external conversations, and using that voice tone for input and output, it becomes a fully AI-controlled and lifelike virtual human.

[0033] For example, in a specific implementation of this invention, we use deep learning algorithms to generate a preliminary dialogue language model prototype for the virtual human on the server side, and assign a voice timbre to the digital human model based on reference audio. The overall architecture of the voice timbre model consists of a posterior encoder, a prior encoder, a decoder, a discriminator, and a random duration predictor. Then, the audio output of the dialogue model is generated through text-to-speech technology. The text that the dialogue model gives in response to the user is converted into audio after processing to achieve natural speech interaction. During the optimization process, an adversarial training framework is constructed to optimize the conditional variational autoencoder to improve the quality and realism of the generated speech. During training, a joint loss function is used, including reconstruction loss, KL divergence loss, and adversarial loss. The reconstruction loss and KL divergence loss are used to optimize the conditional variational autoencoder, and the adversarial loss is used to optimize the discriminator network.

[0034] 4. Continuous evolution and high degree of freedom: Innovation: When processing human body data, the system generates a 3D head model without hair and allows for the autonomous customization and addition of other parts on a distributed terminal, thereby accelerating processing speed and increasing flexibility.

[0035] At the same time, through such a distributed re-customization and re-training process, the system supports the continuous evolution and diversified development of virtual humans. Attached Figure Description

[0036] Appendix Figure 1 The architecture and flowchart of this system.

[0037] Appendix Figure 2 A technical flowchart for creating a separate 3D head, excluding hair and the area below the neck.

[0038] Appendix Figure 3 Header generation effect diagram.

[0039] Appendix Figure 4 A complete rendering of a low-level virtual human, showing the head pieced together from pre-made body, hair, and clothing libraries.

[0040] Appendix Figure 5 Low-order drive high-order test results. Note that the original image used in this test is the third image of a Western woman, not the previous two images of elderly Asian women. Detailed Implementation This invention has been successfully implemented in a specific embodiment. As a specific embodiment, based on the above basic architecture, this embodiment also makes the following settings: This specific implementation example utilizes a high-performance A800 AI computing server array (each server has 8 A800 GPUs, a 768GB RAM pool, and a 16TB hard drive for data storage, totaling 8 servers). A server-side kernel sandbox is installed to create independent user spaces, providing a dedicated management environment for each digital human model and ensuring independence and isolation between models. Sandbox isolation can also be implemented using virtual machine technologies (such as Docker or Kubernetes). The system is equipped with a content management system to support administrators' independent uploading, management, and distribution functions. This case study leverages sandbox isolation technology and decentralized blockchain storage. The three servers are connected to form an IPFS node network, and data is split and stored across all servers, ensuring high availability and security. No single server can form a complete dataset, enhancing the system's resistance to attacks and fault tolerance. In a specific implementation of the present invention, as a preferred method for realizing the invention, the system is also configured in a hierarchical manner to allow the digital human to continue to iterate to higher levels: the system is divided into a supply and distribution end (the end where steps A and B occur), a distributed reorganization and secondary training node (the end where step C occurs), a database (which configures system parameters and information, stores user data, model information and training records, and similar functions), and an electronic ledger of ownership. In this implementation, the supply and distribution end is a centralized server, while the distributed reorganization and secondary training node can be the same server array or the computer room of the next-level enterprise user. The system achieves automatic distributed evolution through the following mode: First, primary users on the supply and distribution sides create projects, upload image data, and set initial ownership allocation ratios. After primary users create low-level virtual humans according to the process described in steps A and B of claim 1, distributed nodes download the low-level virtual humans and trained facial replacement models (if any). Different distributed nodes can give different secondary modifications to the highly realistic virtual humans in step C (including but not limited to different styles of clothing, different route styles, different language models, and different facial processing models). After these nodes process the high-level virtual humans, they are used to interact with different audiences on relevant platforms, including but not limited to OBS streaming, dialogue, program interaction, etc., and different modifications are subsequently performed based on the feedback from their respective audiences.

Claims

1. A method and system for generating a highly realistic virtual human. The method first requires acquiring a 2D image (photo or video) of a person, and then performing the following steps: Step A, generating a 3D facial reconstruction head model file of the 3D virtual human, and then connecting it to a body model bound with a skeletal structure, thereby forming a preliminary virtual human with low realism; Step B, training a neural network model for real-time face replacement based on the 2D image. The order of Step B can be adjusted, i.e., it can be moved before Step A or postponed to be simultaneous with Step C. The neural network model for real-time face replacement can be a specially trained refined model or an open-source model such as a DFM model. It can also be a model that replaces the target face based on a single image; then, in step C, on the one hand, the preliminary 3D virtual human model generated in step A is controlled by the program to realize the changing movements, mouth shape and expression. On the other hand, the preliminary 3D virtual human running window is converted into a virtual camera input as a video source input, and then the low-fidelity virtual human face in the window is identified by face target recognition (including but not limited to using YOLO model, etc.). Then, the real-time replacement neural network model for the face trained in step B is used to replace the face in real time with higher fidelity. The 3D virtual human formed in the previous steps drives the formation of a virtual human with higher fidelity.

2. The system according to any of the preceding claims, wherein, The system adopts a design approach of multiple instances, multiple sandboxes, or multiple virtual machines, thereby launching N (more than 2) multiple video processing software instances in step C. The Nth instance uses the window of the N-1th instance as the video input source or virtual camera, and so on, to perform further image processing, including but not limited to beautification, editing, special effects, etc.

3. The system according to any of the preceding claims, wherein, The system adopts a design approach of multiple instances, multiple sandboxes, or multiple virtual machines, thereby launching M (more than 2) OBS streaming software instances in step C. One instance uses another as the video source, and then performs further image processing, including but not limited to beautification, editing, special effects, etc., to create a more realistic virtual human. At the same time, the final Mth OBS source is used for OBS streaming to provide virtual human interaction services to the outside world.

4. The system according to any of the preceding claims, wherein, The model described in step B is built on an open-source machine learning framework and convolutional neural network (CNN) for creating deepfake content. It uses pre-trained models such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once) to detect and locate faces in images, and then uses other pre-trained models (such as Mask R-CNN) to perform pixel-level face segmentation or replacement.

5. The system according to any of the preceding claims, wherein, Step C includes at least five parts: a face detector, a face aligner, a face marker, a face fusion unit, and a frame adjuster. The face detector quickly and accurately detects the location of faces in the input video or image, effectively distinguishing between faces and non-face objects, and further analyzing face attributes to provide necessary data input for subsequent synthesis. The face aligner positions different feature points of the face roughly in the same relative position (within a certain range of the same position), and corrects the pose according to a preset standard pose to avoid unnatural face-swapping effects caused by pose problems, improving the accuracy and stability of face-swapping. The face marker identifies key facial regions and processes facial details to improve the accuracy, realism, and naturalness of face alignment. The face fusion unit merges the image features of the source and target faces, and also fuses facial expression features to achieve a natural transition of expressions. The frame adjuster balances the adaptation between image quality and face-swapping effect throughout the process, while controlling the smoothness and playback effect of the video.

6. The system according to any of the preceding claims, wherein, In step A, the server does not reconstruct a complete 3D virtual human. Instead, it uses a generative adversarial network-based algorithm to generate only a 3D head model without hair (the generated 3D head model does not include hair and the part below the neck). Then, the generated head model is combined with a 3D human body with pre-bound skeleton and a pre-adapted hairstyle, and clothing that is pre-adapted to the above-mentioned 3D body shape is added.

7. The system according to any of the preceding claims, wherein, The server uses deep learning algorithms to generate a preliminary dialogue language model prototype for the virtual human on the server side, and assigns a voice timbre to the digital human model based on reference audio. The overall architecture of the voice timbre model consists of a posterior encoder, a prior encoder, a decoder, a discriminator, and a random duration predictor. Then, the audio output of the dialogue model will be generated through text-to-speech technology. The text that the dialogue model gives in response to the user is converted into audio after processing to achieve natural speech interaction. During the optimization process, an adversarial training framework is constructed to optimize the conditional variational autoencoder in order to improve the quality and realism of the generated speech. During training, a joint loss function is used, including reconstruction loss, KL divergence loss and adversarial loss. The reconstruction loss and KL divergence loss are used to optimize the conditional variational autoencoder, and the adversarial loss is used to optimize the discriminator network.

8. The system according to any of the preceding claims, wherein, The server-side method for creating a virtual digital human head in step A is as follows: First, a high-quality face dataset containing a large number of samples is constructed; then, the Dlib library is used to detect faces and crop out regions containing only faces. These processed images are used to train the StyleGan2 model; for a new face test image, the HairMapper model is used to perform hair removal on the face image, and the rembg algorithm is used to remove the image background. Subsequently, the trained model is used to map the image into the latent space to generate the image and its latent representation; then, a 3D face reconstruction is performed using a generative adversarial network-based model, and the black background is cropped using Unity's MeshCutter plugin to finally obtain an optimized 3D hairless face model.

9. The system according to any of the preceding claims, wherein, The system is structured in a hierarchical manner, consisting of a supply and distribution end (the origin of steps A and B), distributed reorganization and secondary training nodes (the origin of step C), a database (configuring system parameters and information), and an electronic ledger of ownership. The system achieves automatic distributed evolution through the following mode: First, a primary user on the supply and distribution end creates a project, uploads image data, and sets an initial ownership allocation ratio. After the primary user creates a low-level virtual human according to the process described in steps A and B of claim 1, the distributed nodes download the low-level virtual human and the trained facial replacement model (if any). These virtual humans on different distributed nodes give different secondary modifications to the high-fidelity virtual human in step C (including but not limited to different styles of clothing, different route styles, different language models, and different facial processing). Then, the high-level virtual human processed by these nodes is used to interact with different audiences on relevant platforms, including but not limited to OBS streaming, dialogue, program interaction, etc., and different modifications are subsequently performed based on the feedback of each audience.

10. The system according to any of the preceding claims, wherein, Users who create virtual humans can also give the virtual humans different facial expressions and different animation libraries. The method is as follows:

1. Design a video face feature point tracking algorithm. First, use a camera to capture images of facial expressions. Then, use a multi-task learning cascaded convolutional neural network algorithm and a cascaded gradient enhancement regression tree algorithm to detect faces and extract feature points.

2. Set the facial expression animation control parameters. In the animation control parameters, the EPnP algorithm is used to solve the head pose and generate the corresponding head rotation matrix and translation vector. The AU detector and intensity value regressor are constructed using support vector machine. By establishing the mapping relationship between AU and expression basis, the required facial expression parameters are obtained.

3. Next, design a virtual face expression driver, and use a plugin to transmit the face expression control parameters to a low-level 3D virtual human software to drive the low-level virtual human model to generate corresponding face expression animations; 4. When the low-level virtual human model makes the corresponding facial expression, the face replacement in step C will track the expression that makes the same action.

Images