Network training and digital human animation driving method, device, equipment and medium

Abstract

The embodiments of the present application relate to the field of digital media processing, and provide a network training and digital human animation driving method, apparatus, device and medium, which combine 3D face modeling with image restoration technology, and train an image restoration network so that the trained image restoration network can specifically process the key expression area of ​​the face, the mouth area, accurately reconstruct the mouth image, and realize dynamic and realistic 2D digital human facial animation, thereby fundamentally simplifying the production process of facial animation, which not only reduces the rendering cost and time in the generation process, but also improves the naturalness of the digital human facial animation and the accuracy of the expression.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of digital media processing, and more specifically, to a network training and digital human animation driving method, device, equipment and medium. Background Art

[0002] With the rapid development of virtual reality, augmented reality and video content production technology, the application of digital humans is becoming more and more extensive, especially in the fields of entertainment, news anchors, virtual education and video conferencing. In the traditional animation industry, high-quality 3D animation production usually requires expensive resources and time, which limits its application scope. Especially in real-time or near-real-time scenes, the generation of 3D animation often faces huge challenges.

[0003] In the field of facial driving of 3D digital humans, the authenticity of mouth animation plays a vital role in the naturalness of the overall expression. Accurate synchronization of mouth shapes is the core of achieving high-quality facial animation, especially when combined with speech. Unfortunately, existing technical solutions still have limitations. For example, 3D facial animation usually relies on complex morph targets or generates animation based on captured motion data, which is not only costly but also increases the complexity of the animation process.

[0004] In addition, traditional 2D facial animation generation methods rely on hand-made or semi-automatic software tools, which are inefficient and have limited scalability. Although machine learning and deep learning have made progress in this field in recent years, such as using generative adversarial networks (GAN) to simulate facial expressions, these methods usually require a large amount of training data and high-performance computing resources, and often lack fine control over the complex dynamics inside the mouth. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a network training and digital human animation driving method, device, equipment and medium, which can reduce the high rendering cost required for traditional 3D modeling and improve the generation efficiency and naturalness of digital human facial animation.

[0006] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a network training method, the method comprising:

[0008] Acquire an original video of a target individual, wherein the original video includes a plurality of video frames containing the target individual;

[0009] For each of the video frames, generating an occlusion image and a face rendering corresponding to the video frame, wherein the occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame, and the mouth area of ​​the face rendering lacks texture;

[0010] Inputting the occluded image and the face rendering image corresponding to each of the video frames into a pre-established image restoration network, and using the image restoration network to restore the mouth area in the face rendering image corresponding to each of the video frames to obtain a reconstructed image corresponding to each of the video frames;

[0011] The image restoration network is trained according to each of the video frames and the reconstruction images corresponding to each of the video frames to obtain a trained image restoration network.

[0012] Optionally, the step of generating a face rendering image corresponding to the video frame includes:

[0013] Acquire facial feature information of a target individual in the video frame;

[0014] Generating a facial texture map of the target individual in the video frame according to facial feature information of the target individual in the video frame;

[0015] The facial texture map of the target individual in the video frame is rendered, and the texture of the mouth area of ​​the target individual is removed during the rendering process to obtain a facial rendering corresponding to the video frame, wherein the facial rendering has an expression and head posture consistent with the target individual in the video frame and the mouth area lacks texture.

[0016] Optionally, the facial feature information includes facial geometry, facial expression and facial posture;

[0017] The step of obtaining facial feature information of the target individual in the video frame includes:

[0018] The video frame is input into a preset facial feature extraction model to obtain the facial geometry, facial expression and facial posture of the target individual in the video frame.

[0019] Optionally, the facial feature information includes facial geometry, facial expression and facial posture;

[0020] The step of generating a facial texture map of the target individual in the video frame according to the facial feature information of the target individual in the video frame comprises:

[0021] Performing three-dimensional reconstruction using the face geometry, face expression, and face posture of the target individual in the video frame to obtain a three-dimensional face model of the target individual in the video frame;

[0022] Texture mapping is performed on the video frame and the three-dimensional face model to generate a face texture map of the target individual in the video frame.

[0023] Optionally, the step of training the image restoration network according to each of the video frames and the reconstructed image corresponding to each of the video frames to obtain a trained image restoration network comprises:

[0024] Taking each of the video frames as a positive sample and the reconstructed image corresponding to each of the video frames as a negative sample, inputting them into a preset discriminator to obtain a discrimination result;

[0025] Calculating the loss value of the discriminator according to the discrimination result and the loss function corresponding to the discriminator;

[0026] Calculating the loss value of the image restoration network according to each of the video frames, the reconstructed image corresponding to each of the video frames, the loss value of the discriminator, and the loss function corresponding to the image restoration network;

[0027] According to the loss value of the discriminator and the loss value of the image restoration network, the parameters of the discriminator and the image restoration network are repeatedly optimized alternately using the back propagation algorithm, and after reaching the accuracy requirement or a preset number of iterations, the parameters of the image restoration network are fixed to obtain a trained image restoration network.

[0028] In a second aspect, the present application also provides a digital human animation driving method, the method comprising:

[0029] Acquire an original video of a target individual, wherein the original video includes a plurality of video frames containing the target individual;

[0030] Acquire an occluded image corresponding to each of the video frames, wherein the occluded image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame;

[0031] Based on a specific expression coefficient pre-set for each of the video frames, generating a specific face rendering corresponding to each of the video frames, wherein a mouth area of ​​the specific face rendering lacks texture;

[0032] Inputting the occluded image and the specific face rendering image corresponding to each of the video frames into the image restoration network trained by the above network training method to restore the mouth area in the specific face rendering image corresponding to each of the video frames, and obtaining a reconstructed video including the specific reconstruction image corresponding to each of the video frames;

[0033] The target individual in the specific reconstructed image has an expression represented by the specific expression coefficient associated with it, and the reconstructed video reflects the expression changes represented by all the specific expression coefficients.

[0034] In a third aspect, an embodiment of the present application further provides a network training device, the device comprising:

[0035] A first acquisition module is used to acquire an original video of a target individual, wherein the original video includes a plurality of video frames containing the target individual;

[0036] A first generating module is used to generate, for each of the video frames, an occlusion image and a face rendering corresponding to the video frame, wherein the occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame, and the mouth area of ​​the face rendering lacks texture;

[0037] A restoration module, used to input the occluded image and the face rendering image corresponding to each of the video frames into a pre-established image restoration network, and use the image restoration network to restore the mouth area in the face rendering image corresponding to each of the video frames to obtain a reconstructed image corresponding to each of the video frames;

[0038] The training module is used to train the image restoration network according to each of the video frames and the reconstruction image corresponding to each of the video frames to obtain a trained image restoration network.

[0039] In a fourth aspect, the present application also provides a digital human animation driving device, the device comprising:

[0040] A second acquisition module is used to acquire an original video of a target individual, wherein the original video includes a plurality of video frames containing the target individual; and acquire an occlusion image corresponding to each of the video frames, wherein the occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame;

[0041] A second generating module is used to generate a specific face rendering corresponding to each of the video frames based on a specific expression coefficient preset for each of the video frames, wherein a mouth area of ​​the specific face rendering lacks texture;

[0042] An image restoration module, used to input the occluded image and the specific face rendering image corresponding to each of the video frames into the image restoration network trained by the above method, to obtain a reconstructed video including the specific reconstruction image corresponding to each of the video frames;

[0043] The target individual in the specific reconstructed image has an expression represented by the specific expression coefficient associated with it, and the reconstructed video reflects the expression changes represented by all the specific expression coefficients.

[0044] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a program, and the processor is used to implement the network training method in the first aspect and / or the digital human animation driving method in the second aspect when executing the program.

[0045] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the network training method in the first aspect above, and / or the digital human animation driving method in the second aspect above.

[0046] Compared with the prior art, the embodiment of the present application provides a network training and digital human animation driving method, device, equipment and medium. When training a pre-established image restoration network, first, multiple video frames containing target individuals are obtained, and for each video frame, a corresponding occlusion image and a face rendering image are generated. The occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame, and the mouth area of ​​the face rendering image lacks texture; then, the occlusion image and the face rendering image corresponding to each video frame are input into the pre-established image restoration network, and the mouth area in the face rendering image corresponding to each video frame is restored by using the image restoration network to obtain a reconstruction image corresponding to each video frame; finally, each video frame is used as a label, and the image restoration network is trained in combination with the reconstruction image corresponding to each video frame to obtain a trained image restoration network. In this way, when the trained image restoration network is used for digital human animation driving, the mouth image can be accurately reconstructed to achieve dynamic and realistic 2D digital human facial animation, which fundamentally simplifies the production process of facial animation, not only reduces the rendering cost and time in the generation process, but also improves the naturalness of digital human facial animation and the accuracy of expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a network training method provided in an embodiment of the present application is shown. Figure 1 .

[0048] Figure 2 An example diagram of an occluded image provided by an embodiment of the present application is shown.

[0049] Figure 3 An example diagram of a face rendering provided by an embodiment of the present application is shown.

[0050] Figure 4 for Figure 1 FIG. 1 is a flow chart of step S102 in the network training method shown in FIG.

[0051] Figure 5 An example diagram of a three-dimensional face model provided in an embodiment of the present application is shown.

[0052] Figure 6 A schematic diagram of a network training method provided in an embodiment of the present application is shown. Figure 2 .

[0053] Figure 7 for Figure 1 FIG. 1 is a flow chart of step S104 in the network training method shown in FIG.

[0054] Figure 8 A schematic flow chart of a digital human animation driving method provided in an embodiment of the present application is shown.

[0055] Fig. 9 A block diagram of a network training device provided in an embodiment of the present application is shown.

[0056] Fig.10 A block diagram of a digital human animation driving device provided in an embodiment of the present application is shown.

[0057] Fig.11 A block diagram of an electronic device provided in an embodiment of the present application is shown.

[0058] Icons: 100 - network training device; 101 - first acquisition module; 102 - first generation module; 103 - repair module; 104 - training module; 200 - digital human animation driving device; 201 - second acquisition module; 202 - second generation module; 203 - image repair module; 10 - electronic device; 11 - processor; 12 - memory; 13 - bus. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application.

[0060] Currently, the generation and driving of 3D character animation usually requires complex 3D modeling and high-performance rendering processes. These processes are not only time-consuming and costly, but also difficult to generate facial expressions that match the characteristics of audio input in real time.

[0061] Traditional 2D facial animation generation methods rely on hand-made or semi-automatic software tools, which are inefficient and have limited scalability. Although machine learning and deep learning have made progress in this field in recent years, these methods usually require a large amount of training data and high-performance computing resources, and often lack fine control over the complex dynamics inside the mouth.

[0062] In order to overcome the above limitations, the embodiments of the present application provide a network training method and a digital human animation driving method, which trains an image restoration network so that the trained image restoration network can specifically process the key facial expression area, the mouth area, accurately reconstruct the mouth image, and realize dynamic and realistic 2D digital human facial animation, fundamentally simplifying the production process of facial animation, which not only reduces the rendering cost and time in the generation process, but also improves the naturalness of digital human facial animation and the accuracy of expression. The following is a detailed introduction.

[0063] Please refer to Figure 1 , Figure 1 A flow chart of a network training method provided in an embodiment of the present application is shown. The network training method is applied to an electronic device, which may be a server, such as a single server, a server cluster, etc., or a terminal, such as a smart phone, a laptop, a tablet computer, a desktop computer, etc., or a chip system, which may be composed of a chip, or may include a chip and other discrete devices, etc., and the embodiment of the present application does not impose any restrictions on this. The network training method may include the following steps:

[0064] S101, obtaining an original video of a target individual, where the original video includes multiple video frames containing the target individual.

[0065] In this embodiment, the target individual refers to the individual who needs to be driven by digital human animation, such as the user himself or the anchor in the live broadcast room. Each video frame in the original video must contain the target individual. The original video can be a video of only the user or a real live broadcast video of the anchor. The video length can be about 10 minutes, and this embodiment of the application does not impose any restrictions on this.

[0066] The original videos of the target individual need to be as diverse as possible to cover the various expressions and lip shapes of the target individual. At the same time, the quality of the original video will directly affect the effect of the final trained image restoration network, so high-resolution and high-frame-rate videos should be selected to ensure sufficient details and continuity.

[0067] S102, for each video frame, generating an occlusion image and a face rendering corresponding to the video frame, wherein the occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame, and the mouth area of ​​the face rendering lacks texture.

[0068] In this embodiment, after obtaining the original video of the target individual, it is necessary to generate an occlusion image and a face rendering corresponding to each video frame in the original video. The occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame, such as Figure 2As shown, a solid-color masking block can be used to mask the lower half of the face area of ​​the target individual in the video frame. In actual operation, a technician can first manually select the lower half of the face area of ​​the target individual in the video frame, and then set all the pixels in the selected frame to the same color to obtain a masked image.

[0069] The face rendering is obtained by reconstructing the face texture map of the target individual in the video frame and then rendering the face texture map. In addition, the face rendering needs to ensure that the mouth area of ​​the target individual has no missing texture. The mouth area here does not include lips, but specifically refers to the oral cavity and teeth. The reason is that the mouth area is the key expression area of ​​the face. For example, the difference between a person's expression when speaking in a gentle manner and when screaming and roaring is very large. More oral cavity and teeth will be shown when screaming and roaring. The training goal of the image restoration network is to accurately reconstruct the mouth image in order to achieve an effect that can be accurately driven. Therefore, the missing texture of the mouth area in the face rendering needs to be restored by the image restoration network. Figure 3 As shown, the face rendering includes a solid-color background and a face foreground, and the mouth area (i.e., the oral cavity and teeth) lacks texture, i.e., Figure 3 The black area in .

[0070] The process of generating a face rendering image corresponding to a video frame in step S102 is described in detail below.

[0071] Take any video frame as an example. Figure 1 Based on Figure 4 The process of generating a face rendering image corresponding to the video frame in step S102 may include sub-steps S1021 to S1023.

[0072] S1021, obtaining facial feature information of a target individual in a video frame.

[0073] In this embodiment, the facial feature information may include facial geometry, facial expression, and facial posture. Facial geometry refers to the outline and structure of the face, which is used to describe the basic outline of the face, the position and shape of the eyes, nose, mouth, etc., and can be represented by the coordinates of the key points of the face, which are usually the feature points of the face, such as the coordinates of the positions of the eyes, nose, mouth, etc.

[0074] Face geometry can include face frame, face mesh, and mouth region bounding box. Face frame refers to the outline of the face, which is used to locate the position and size of the face in the video frame in 3D reconstruction. Face mesh contains the 3D mesh structure of the face, which describes the shape and geometry of the face. It is used to build a realistic 3D face model in 3D reconstruction, and presents a realistic appearance by attaching a texture map to the face mesh. The mouth region bounding box refers to the area where the teeth and oral cavity are located (i.e. Figure 3 The outline of the mouth (black area in the figure) is used to locate the position and size of the mouth area in the video frame in 3D reconstruction. The face geometry is characterized by the face shape coefficient.

[0075] It should be pointed out that the face mesh does not include the mesh of the mouth area, so as to ensure that the mouth area lacks texture in the subsequent face rendering.

[0076] Facial expressions refer to the facial expressions of a person, such as smile, anger, surprise, etc. Facial expressions are represented by expression coefficients. As important data for describing changes in facial expressions, expression coefficients are extracted from videos through applicable posture tracking and expression analysis technologies. The extracted expression coefficients are not only used in the subsequent rendering process, but are also indispensable input parameters for training image restoration networks.

[0077] Facial posture refers to the direction and angle of the face in space, such as the upward or downward tilt angle of the head, the left and right rotation angle of the head, the rotation angle of the head around the vertical axis, etc. Facial posture is characterized by posture parameters.

[0078] In a possible implementation, the video frame may be input into a preset facial feature extraction model to obtain the facial geometry, facial expression, and facial posture of the target individual in the video frame.

[0079] In this embodiment, the facial feature extraction model can be an open source model of Mediapipe. By inputting the video frame into the open source model of Mediapipe, the facial shape coefficient, expression coefficient and posture parameter can be output, that is, the facial geometry, facial expression and facial posture of the target individual in the video frame can be obtained.

[0080] S1022, generating a facial texture map of the target individual in the video frame according to the facial feature information of the target individual in the video frame.

[0081] In this embodiment, the face texture map refers to the texture information of the face, which is used to describe the details of the skin color, spots, wrinkles, etc. of the face. The face texture map can be obtained by applying the expression coefficient and posture parameter to a pre-set standard average face model, matching the video frame closest to the front face from the original video, and extracting the corresponding texture from it. The quality of texture extraction will directly affect the realism of the generated 2D digital character, so accurate alignment and color fidelity must be ensured when extracting the face texture map.

[0082] In a possible implementation, the process of generating a facial texture map of a target individual in a video frame according to facial feature information of the target individual in the video frame in S1022 may include sub-steps S10221 to S10222.

[0083] S10221, performing three-dimensional reconstruction using the face geometry, face expression and face posture of the target individual in the video frame to obtain a three-dimensional face model of the target individual in the video frame.

[0084] In this embodiment, based on the face geometry, face expression and face posture extracted in step S1021, a 3D reconstruction algorithm is used to convert the 2D image into a 3D face model, thereby realizing 3D reconstruction of the real face in the video frame. For example, the reconstructed 3D face model is as follows: Figure 5 Optionally, Meidiapipe can be used to reconstruct the 3D face model.

[0085] S10222, perform texture mapping on the video frame and the three-dimensional face model to generate a face texture map of the target individual in the video frame.

[0086] In this embodiment, the three-dimensional face model is aligned to the original video frame, and the texture corresponding to each pixel can be obtained, thereby generating a face texture map.

[0087] S1023, rendering the facial texture map of the target individual in the video frame, and removing the texture of the mouth area of ​​the target individual during the rendering process to obtain a facial rendering image corresponding to the video frame, the facial rendering image has an expression and head posture consistent with the target individual in the video frame, and the mouth area lacks texture.

[0088] In this embodiment, for each video frame, after generating the facial texture map of the target individual in the video frame, professional 3D rendering software or self-developed rendering tools are used to render each frame. The rendering process should ensure that each frame accurately matches the expression and lip shape of the target individual in the original video, and during the rendering process, the texture of the mouth area is deliberately removed to form a blank area to be filled. The purpose of doing this is to simulate the occlusion or information missing that may exist in the mouth in actual application scenarios, create the necessary challenges for network training, and achieve higher generalization capabilities.

[0089] Rendering the face texture map is to use graphics and rendering technology to render the face texture map to achieve virtual presentation of the face, including processing of lighting, shadows, materials, etc., as well as realizing dynamic expressions and postures of the face. As mentioned above, the face mesh does not contain the mesh of the mouth area, so the mouth area will not be rendered during the rendering process, thereby ensuring that the mouth area in the final face rendering image lacks texture. The final face rendering image has the same expression and head posture as the target individual in the video frame, and the mouth area lacks texture. Optionally, the pytorch3D open source code can be used to render the face rendering image.

[0090] After the above steps S101 to S102, the preparation of training data can be completed. Compared with traditional technologies and existing data preparation processes, the above method provided in the embodiment of the present application can prepare training data, which can greatly improve the naturalness and personalized expression ability of 2D digital human rendering while improving efficiency, laying a solid foundation for realizing more widely used 2D digital human technology.

[0091] S103, inputting the occluded image and the face rendering image corresponding to each video frame into a pre-established image restoration network, and using the image restoration network to restore the mouth area in the face rendering image corresponding to each video frame to obtain a reconstructed image corresponding to each video frame.

[0092] In this embodiment, the core goal of the training is to build an image restoration network that can accurately restore the face rendering image with the mouth area removed. The network architecture used by the image restoration network is an hourglass network, which is suitable for deep feature extraction and long-distance transmission of context information, thereby providing more refined restoration results. The symmetry of the hourglass network can effectively transmit information between different levels of image resolution, and can integrate the global context while maintaining local details of the image.

[0093] like Figure 6 As shown in the figure, the input of the image restoration network is the occluded image and the face rendering corresponding to each video frame. These two parts together provide enough information for the image restoration network to not only learn to reconstruct the mouth area, but also understand the texture differences under different expressions and how to adapt to new expressions.

[0094] In this embodiment, the occluded image is also used as input to help the image restoration network understand the different expression states of the target individual during the training phase, increase the accuracy of the network output and improve the network's adaptability to new expressions. At the same time, the mouth area in the face rendering is missing to simulate possible information loss in actual applications.

[0095] S104, training an image restoration network according to each video frame and a reconstruction image corresponding to each video frame to obtain a trained image restoration network.

[0096] In this embodiment, the training goal is to minimize the difference between the output of the image restoration network and the real mouth image. Therefore, the loss function can select a pixel-level loss function, such as mean-square error (MSE) or structural similarity (SSIM), etc. The specific training process is similar to the prior art and will not be repeated here. In addition, in order to increase the coherence and natural transition between the generated mouth area and its surrounding area, the image restoration network can also be trained to learn to maintain the coherence of textures and edges. For example, by adding a specific layer in the image restoration network to optimize the high-frequency details of the reconstructed image.

[0097] The network is trained iteratively, with each step making small adjustments to the existing weights to better reproduce the texture features in the training dataset. This process may include a range of regularization strategies to prevent overfitting and keep the network generalizing to new expressions, such as dropout, weight decay, and data augmentation in image inpainting networks.

[0098] In addition, in order to further improve the efficiency and generalization ability of the network, it is also possible to combine traditional supervised learning methods with unsupervised or semi-supervised methods. For example, a part of the network can be pre-trained through an autoencoder (Unsupervised Learning), thereby reducing the cost and resource consumption of guided learning and alleviating the dependency problem of large-scale annotated data.

[0099] In one possible implementation, adversarial loss can be used in the training process to further improve the naturalness and realism of the texture. For example, through the application of GAN (Generative Adversarial Networks), the network is trained to not only generate an accurate-looking mouth area, but also its appearance is realistic enough that it cannot be easily distinguished from the real mouth by the human eye or other networks.

[0100] Therefore, in Figure 1 Based on Figure 7 In S104, the process of training the image restoration network according to each video frame and the reconstructed image corresponding to each video frame to obtain the trained image restoration network may include sub-steps S1041 to S1044.

[0101] S1041, taking each video frame as a positive sample and taking the reconstructed image corresponding to each video frame as a negative sample, inputting them into a preset discriminator to obtain a discrimination result.

[0102] S1042, calculating the loss value of the discriminator according to the discrimination result and the loss function corresponding to the discriminator.

[0103] S1043, calculating the loss value of the image restoration network according to each video frame, the reconstructed image corresponding to each video frame, the loss value of the discriminator, and the loss function corresponding to the image restoration network.

[0104] S1044, according to the loss value of the discriminator and the loss value of the image restoration network, the parameters of the discriminator and the image restoration network are repeatedly optimized by using the back propagation algorithm, and after reaching the accuracy requirement or the preset number of iterations, the parameters of the image restoration network are fixed to obtain the trained image restoration network.

[0105] That is, combined with Figure 6 The image restoration network is used as a generator and together with the discriminator, it forms an image restoration network. Each video frame in the original video is real data, and the reconstructed image corresponding to each video frame output by the image restoration network is generated data. The goal of the discriminator is to accurately distinguish between real data and generated data, and the goal of the generator is to strive to improve itself so as to generate data that can confuse the discriminator. The generator and the discriminator are trained alternately. Finally, after reaching the accuracy requirement or the preset number of iterations, the parameters of the image restoration network are fixed to obtain the trained image restoration network.

[0106] In this embodiment, the image restoration network is trained to achieve a high-precision 2D digital human expression driving effect for a single individual, while giving the network the ability to quickly learn and adapt to new expressions based on a small number of samples. During the training process, the focus is on improving the network's performance in generating natural, high-quality mouth area textures to meet the high standards of diverse application scenarios such as digital entertainment and virtual assistants. The above training process can achieve rapid migration and wide applicability of the network while ensuring personalization, laying a solid foundation for the realization of convenient and efficient 2D digital human applications.

[0107] Please refer to Figure 8 , Figure 8 The flowchart of the digital human animation driving method provided by the embodiment of the present application is shown. The digital human animation driving method is applied to an electronic device, which may be a terminal, such as a smart phone, a laptop, a tablet computer, a desktop computer, etc., or a development board, a chip system, etc. The digital human animation driving method may include the following steps:

[0108] S201, obtaining an original video of a target individual, where the original video includes multiple video frames containing the target individual.

[0109] S202, obtaining an occluded image corresponding to each video frame, where the occluded image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame.

[0110] In this embodiment, the process of obtaining the original video and the blocked image is similar to the process described in steps S101 to S102, and will not be described in detail here.

[0111] S203, generating a specific face rendering corresponding to each video frame based on a specific expression coefficient preset for each video frame, wherein a mouth region of the specific face rendering lacks texture.

[0112] In this embodiment, the specific expression coefficient can be an expression coefficient predicted based on an audio signal, where the audio signal can be the audio spoken by the target individual in the video to be reconstructed later, and the speech features in the audio can be identified and converted by high-precision audio analysis technology to generate accurate expression coefficients synchronized with the audio. The specific expression coefficient can also be a series of pre-set expression coefficient sequences, which are manually set and are usually used for non-speech driven expression changes in specific scenarios.

[0113] Taking any video frame as an example, a corresponding specific face rendering is generated according to the input specific expression coefficient. If there is a specific face rendering corresponding to the specific expression coefficient, it can be directly selected. In order to ensure the coherence and consistency of the output, the texture inside the mouth also needs to be removed in the rendering stage, so that the mouth area of ​​the specific face rendering lacks texture. The generation process of the specific face rendering is similar to the process of generating the face rendering in the above step S102, which will not be repeated here.

[0114] S204, inputting the occluded image and the specific face rendering image corresponding to each video frame into the image restoration network trained by the above network training method, so as to restore the mouth area in the specific face rendering image corresponding to each video frame, and obtain a reconstructed video including the specific reconstruction image corresponding to each video frame;

[0115] The target individual in the specific reconstructed image has an expression represented by a specific expression coefficient associated with it, and the reconstructed video reflects the expression changes represented by all the specific expression coefficients.

[0116] In this embodiment, the occluded image and the specific face rendering image corresponding to each video frame are input into the trained image restoration network, and the prediction ability of the trained image restoration network is used to automatically fill in the mouth area of ​​the specific face rendering image to generate a complete and natural facial expression image, and finally a reconstructed video that can reflect the expression changes represented by all specific expression coefficients is obtained.

[0117] For example, if the target individual in the original video is speaking in a gentle manner, and the target individual is required to roar out a piece of audio, accurate expression coefficients synchronized with the audio can be generated first, and then these expression coefficients can be used to generate a specific face rendering for each video frame in the original video. The occluded image and the specific face rendering corresponding to each video frame are then input into the trained image restoration network to output a reconstructed video of the target individual roaring out the audio.

[0118] In this embodiment, when the actual product is put into practical use, it is usually necessary to combine the audio signal with the reconstructed video to synthesize complete audio and video content. This audio and video synchronization technology enhances the realism of the digital character and improves the user experience.

[0119] This embodiment can quickly generate new expressions that the target individual has never shown in the original video by pre-setting or audio-driven generation of specific expression coefficients, while ensuring that the expressions are natural, smooth and coherent. The feasibility and efficiency of this method greatly reduce the complexity and cost of animation production through optimized networks and algorithms.

[0120] Compared with the prior art, the embodiment of the present application innovatively combines 3D face modeling with image restoration technology, significantly improving the naturalness and realism of 2D digital human-driven animation. While solving the problems of high-cost rendering and time consumption, it can also achieve a WYSIWYG animation effect, significantly improving the feasibility and flexibility of animation creation. It will promote the facial animation production of virtual digital assistants, online education, games and other media entertainment platforms to become more intelligent and efficient, and have a profound impact on the entire digital media industry.

[0121] In order to execute the corresponding steps in the above method embodiment and each possible implementation mode, the following provides a network training device and a digital human animation driving device for implementation.

[0122] Please refer to Fig. 9 , Fig. 9 The block diagram of the network training device 100 provided in the embodiment of the present application is shown. The network training device 100 is applied to an electronic device, and the network training device 100 includes: a first acquisition module 101, a first generation module 102, a repair module 103 and a training module 104.

[0123] The first acquisition module 101 is used to acquire an original video of a target individual, where the original video includes a plurality of video frames containing the target individual.

[0124] The first generating module 102 is used to generate an occlusion image and a face rendering corresponding to each video frame, wherein the occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame, and the mouth area of ​​the face rendering lacks texture.

[0125] The repair module 103 is used to input the occluded image and the face rendering image corresponding to each video frame into a pre-established image repair network, and use the image repair network to repair the mouth area in the face rendering image corresponding to each video frame to obtain a reconstructed image corresponding to each video frame.

[0126] The training module 104 is used to train the image restoration network according to each video frame and the reconstruction image corresponding to each video frame to obtain a trained image restoration network.

[0127] Optionally, the method of generating the face rendering image corresponding to the video frame in the first generating module 102 includes:

[0128] Obtain facial feature information of the target individual in the video frame;

[0129] Generate a facial texture map of the target individual in the video frame according to facial feature information of the target individual in the video frame;

[0130] The facial texture map of the target individual in the video frame is rendered, and the texture of the mouth area of ​​the target individual is removed during the rendering process to obtain a facial rendering corresponding to the video frame. The facial rendering has the same expression and head posture as the target individual in the video frame.

[0131] Optionally, the facial feature information includes facial geometry, facial expression and facial posture. The first generation module 102 acquires the facial feature information of the target individual in the video frame in a manner including:

[0132] The video frame is input into a preset facial feature extraction model to obtain the facial geometry, facial expression and facial posture of the target individual in the video frame.

[0133] Optionally, the first generating module 102 generates a facial texture map of a target individual in a video frame according to facial feature information of the target individual in the video frame, including:

[0134] The three-dimensional face model of the target individual in the video frame is obtained by using the face geometry, facial expression and facial posture of the target individual in the video frame for three-dimensional reconstruction;

[0135] Texture mapping is performed on the video frame and the three-dimensional face model to generate a face texture map of the target individual in the video frame.

[0136] Optionally, the training module 104 trains the image restoration network according to each video frame and the reconstructed image corresponding to each video frame to obtain a trained image restoration network, including:

[0137] Each video frame is used as a positive sample, and the reconstructed image corresponding to each video frame is used as a negative sample, and input into a preset discriminator to obtain a discrimination result;

[0138] According to the discrimination result and the loss function corresponding to the discriminator, the loss value of the discriminator is calculated;

[0139] Calculate the loss value of the image restoration network according to each video frame, the reconstructed image corresponding to each video frame, the loss value of the discriminator, and the loss function corresponding to the image restoration network;

[0140] According to the loss value of the discriminator and the loss value of the image restoration network, the back propagation algorithm is used to repeatedly and alternately optimize the parameters of the discriminator and the image restoration network. After reaching the accuracy requirement or the preset number of iterations, the parameters of the image restoration network are fixed to obtain the trained image restoration network.

[0141] Please refer to Fig.10 , Fig.10 The block diagram of the digital human animation driving device 200 provided in the embodiment of the present application is shown. The digital human animation driving device 200 is applied to electronic equipment, and the digital human animation driving device 200 includes: a second acquisition module 201, a second generation module 202 and an image restoration module 203.

[0142] The second acquisition module 201 is used to acquire the original video of the target individual, the original video includes multiple video frames containing the target individual; and acquire the occluded image corresponding to each video frame, the occluded image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame.

[0143] The second generating module 202 is used to generate a specific face rendering corresponding to each video frame based on a specific expression coefficient preset for each video frame, wherein the mouth area of ​​the specific face rendering lacks texture.

[0144] The image restoration module 203 is used to input the occluded image and the specific face rendering image corresponding to each video frame into the image restoration network trained by the above network training method, so as to restore the mouth area in the specific face rendering image corresponding to each video frame, and obtain a reconstructed video including the specific reconstruction image corresponding to each video frame;

[0145] The target individual in the specific reconstructed image has an expression represented by a specific expression coefficient associated with it, and the reconstructed video reflects the expression changes represented by all the specific expression coefficients.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the network training device 100 and the digital human animation driving device 200 described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0147] Please refer to Fig.11 , Fig.11 The electronic device 10 provided in the embodiment of the present application is shown in a block diagram. The electronic device 10 includes a processor 11, a memory 12 and a bus 13, and the processor 11 is connected to the memory 12 via the bus 13.

[0148] The memory 12 is used to store programs. After receiving the execution instruction, the processor 11 executes the program to implement the network training method and / or the digital human animation driving method disclosed in the above embodiments.

[0149] The memory 12 may include a high-speed random access memory (RAM) and may also include a non-volatile memory (NVM).

[0150] The processor 11 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 11 or an instruction in the form of software. The above processor 11 may be a general processor, including a central processing unit (CPU), a microcontroller unit (MCU), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), an embedded ARM and other chips.

[0151] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 11, the network training method and / or the digital human animation driving method disclosed in the above embodiment are implemented.

[0152] In summary, the network training and digital human animation driving method, device, equipment and medium provided by the embodiment of the present application, when training a pre-established image restoration network, first, obtain multiple video frames containing target individuals, and for each video frame, generate corresponding occlusion images and face rendering images, the occlusion images are obtained by occluding the lower half of the face area of ​​the target individual in the video frame, and the mouth area of ​​the face rendering image lacks texture; then, the occlusion images and face rendering images corresponding to each video frame are input into the pre-established image restoration network, and the mouth area in the face rendering image corresponding to each video frame is restored by using the image restoration network to obtain the reconstruction image corresponding to each video frame; finally, each video frame is used as a label, and the image restoration network is trained in combination with the reconstruction image corresponding to each video frame to obtain the trained image restoration network. In this way, when the trained image restoration network is used for digital human animation driving, the mouth image can be accurately reconstructed to achieve dynamic and realistic 2D digital human facial animation, which fundamentally simplifies the production process of facial animation, not only reduces the rendering cost and time in the generation process, but also improves the naturalness of digital human facial animation and the accuracy of expression.

[0153] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A network training method, characterized in that: The method comprises: Acquire an original video of a target individual, wherein the original video includes a plurality of video frames containing the target individual; For each of the video frames, generating an occlusion image and a face rendering corresponding to the video frame, wherein the occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame, and the mouth area of ​​the face rendering lacks texture; Inputting the occluded image and the face rendering image corresponding to each of the video frames into a pre-established image restoration network, and using the image restoration network to restore the mouth area in the face rendering image corresponding to each of the video frames to obtain a reconstructed image corresponding to each of the video frames; The image restoration network is trained according to each of the video frames and the reconstruction images corresponding to each of the video frames to obtain a trained image restoration network.

2. The method according to claim 1, characterized in that The step of generating a face rendering image corresponding to the video frame comprises: Acquire facial feature information of a target individual in the video frame; Generating a facial texture map of the target individual in the video frame according to facial feature information of the target individual in the video frame; The facial texture map of the target individual in the video frame is rendered, and the texture of the mouth area of ​​the target individual is removed during the rendering process to obtain a facial rendering corresponding to the video frame, wherein the facial rendering has an expression and head posture consistent with the target individual in the video frame and the mouth area lacks texture.

3. The method according to claim 2, characterized in that The facial feature information includes facial geometry, facial expression and facial posture; The step of obtaining facial feature information of the target individual in the video frame includes: The video frame is input into a preset facial feature extraction model to obtain the facial geometry, facial expression and facial posture of the target individual in the video frame.

4. The method according to claim 2, characterized in that The facial feature information includes facial geometry, facial expression and facial posture; The step of generating a facial texture map of the target individual in the video frame according to the facial feature information of the target individual in the video frame comprises: Performing three-dimensional reconstruction using the face geometry, face expression, and face posture of the target individual in the video frame to obtain a three-dimensional face model of the target individual in the video frame; Texture mapping is performed on the video frame and the three-dimensional face model to generate a face texture map of the target individual in the video frame.

5. The method according to claim 1, characterized in that The step of training the image restoration network according to each of the video frames and the reconstructed image corresponding to each of the video frames to obtain a trained image restoration network comprises: Taking each of the video frames as a positive sample and the reconstructed image corresponding to each of the video frames as a negative sample, inputting them into a preset discriminator to obtain a discrimination result; Calculating the loss value of the discriminator according to the discrimination result and the loss function corresponding to the discriminator; Calculating the loss value of the image restoration network according to each of the video frames, the reconstructed image corresponding to each of the video frames, the loss value of the discriminator, and the loss function corresponding to the image restoration network; According to the loss value of the discriminator and the loss value of the image restoration network, the parameters of the discriminator and the image restoration network are repeatedly optimized alternately using the back propagation algorithm, and after reaching the accuracy requirement or a preset number of iterations, the parameters of the image restoration network are fixed to obtain a trained image restoration network.

6. A digital human animation driving method, characterized in that: The method comprises: Acquire an original video of a target individual, wherein the original video includes a plurality of video frames containing the target individual; Acquire an occluded image corresponding to each of the video frames, wherein the occluded image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame; Based on a specific expression coefficient pre-set for each of the video frames, generating a specific face rendering corresponding to each of the video frames, wherein a mouth area of ​​the specific face rendering lacks texture; Inputting the occluded image and the specific face rendering image corresponding to each of the video frames into the image restoration network trained by the network training method according to any one of claims 1 to 5, so as to restore the mouth area in the specific face rendering image corresponding to each of the video frames, and obtain a reconstructed video including the specific reconstruction image corresponding to each of the video frames; The target individual in the specific reconstructed image has an expression represented by the specific expression coefficient associated with it, and the reconstructed video reflects the expression changes represented by all the specific expression coefficients.

7. A network training device, characterized in that: The device comprises: A first acquisition module is used to acquire an original video of a target individual, wherein the original video includes a plurality of video frames containing the target individual; A first generating module is used to generate, for each of the video frames, an occlusion image and a face rendering corresponding to the video frame, wherein the occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame, and the mouth area of ​​the face rendering lacks texture; A restoration module, used to input the occluded image and the face rendering image corresponding to each of the video frames into a pre-established image restoration network, and use the image restoration network to restore the mouth area in the face rendering image corresponding to each of the video frames to obtain a reconstructed image corresponding to each of the video frames; The training module is used to train the image restoration network according to each of the video frames and the reconstruction image corresponding to each of the video frames to obtain a trained image restoration network.

8. A digital human animation driving device, characterized in that: The device comprises: A second acquisition module is used to acquire an original video of a target individual, wherein the original video includes a plurality of video frames containing the target individual; and acquire an occlusion image corresponding to each of the video frames, wherein the occlusion image is obtained by occluding the lower half of the face area of ​​the target individual in the video frame; A second generating module is used to generate a specific face rendering corresponding to each of the video frames based on a specific expression coefficient preset for each of the video frames, wherein a mouth area of ​​the specific face rendering lacks texture; An image restoration module, used for inputting the occluded image and the specific face rendering image corresponding to each of the video frames into the image restoration network trained by the method according to any one of claims 1 to 5, to obtain a reconstructed video including the specific reconstruction image corresponding to each of the video frames; The target individual in the specific reconstructed image has an expression represented by the specific expression coefficient associated with it, and the reconstructed video reflects the expression changes represented by all the specific expression coefficients.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory is used to store a program, and the processor is used to implement the network training method described in any one of claims 1 to 5 and / or the digital human animation driving method described in claim 6 when executing the program.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the network training method as described in any one of claims 1 to 5 and / or the digital human animation driving method as described in claim 6 are implemented.

Images