Real-time animation avatar driving method and system

Abstract

The present invention discloses a real-time anime avatar driving method, belonging to the field of image processing technology. The method comprises: obtaining a real-time user image; performing a 3DMM model solution on the real-time user image to obtain a target expression vector and a target head posture vector; semantically mapping the target expression vector to generate an expression change partial graph; and identically mapping the target head posture vector to the expression change partial graph to generate a three-dimensional reconstructed image, thereby driving the avatar. The present invention also discloses a real-time anime avatar driving system. The present invention can mass-produce high-quality datasets for anime avatar driving. Generating datasets using this method can implement a data-driven anime character avatar driving method. The present invention can also achieve lightweight mobile avatar driving.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a real-time cartoon avatar driving method and system. Background Art

[0002] Anime avatar drivers transfer real-life user expressions and head postures onto anime characters, allowing the characters to mimic the user's expressions and movements, creating a "living photo" effect. With the rise of the metaverse and avatar technology, many people enjoy interacting through avatars, giving this technology great potential in media and entertainment. With the current surge in AIGC, anime avatar drivers can also serve as an auxiliary tool for artistic creation.

[0003] While real-life face image-based driving has already achieved very good results, related methods rely on high-quality face-based driving datasets such as VoxCeleb, as well as pre-trained models for facial feature point prediction and expression coefficient prediction. However, this kind of real-life face-based driving work is lacking in the animation field. Anime datasets collected from online images or videos, such as ICatoonFace, or GAN-generated anime face images, are difficult to annotate with expression coefficients. Furthermore, anime characters' expressions are more abstract than those of real people. Furthermore, there are no results regarding feature point detection or expression coefficient prediction for anime faces. Compared to real-life image-based driving, anime avatar image-based driving is not only a cross-identity problem, but also a cross-domain problem from the real-life domain to the animation domain, which further increases the difficulty of the problem. Summary of the Invention

[0004] The purpose of the present invention is to provide a real-time animation avatar driving method and system for large-scale production of high-quality data sets to achieve lightweight avatar driving.

[0005] To solve the above technical problems, the present invention provides a real-time cartoon avatar driving method, comprising the following steps:

[0006] Get real-time images of users;

[0007] Solve the 3DMM model of the user's real-time image to obtain the target expression vector and target head posture vector;

[0008] Perform semantic mapping on the target expression vector to generate a partial graph of expression changes;

[0009] The target head pose vector is mapped to the expression change part map to generate a 3D reconstructed image to drive the avatar.

[0010] Preferably, solving the 3DMM model for the user's real-time image to obtain the target expression vector and the target head posture vector includes the following steps:

[0011] According to the 3DMM model, the image to be mapped is obtained from the user's real-time image;

[0012] Get the target expression vector and target head pose vector from the image to be mapped.

[0013] Preferably, according to the 3DMM model, the image to be mapped is obtained from the user's real-time image, which specifically includes the following steps:

[0014] Construct an expression mapping dictionary based on the expression change names in the 3D model corresponding to the semantics of each dimension of the 3DMM model coefficient; the expression mapping dictionary includes a mapping relationship from the 3DMM model coefficient to multiple 3D model expression names;

[0015] According to the expression mapping dictionary, a mappable image corresponding to the expression name of the 3D model is obtained from the user's real-time image;

[0016] The mappable image is rendered to obtain an image to be mapped.

[0017] Preferably, obtaining the target expression vector and the target head posture vector from the image to be mapped specifically includes the following steps:

[0018] Generate weights of 3DMM model coefficients in the expression mapping dictionary;

[0019] Obtain facial expression data of the image to be mapped according to the weights of the 3DMM model coefficients;

[0020] Normalizing the facial expression data of the image to be mapped to obtain normalized facial expression data;

[0021] The image to be mapped is sampled through a probability distribution function to obtain head posture data;

[0022] The normalized facial expression data and head posture data are rendered to obtain a target expression vector and a target head posture vector.

[0023] Preferably, semantic mapping is performed on the target expression vector to generate an expression change partial graph, which specifically includes the following steps:

[0024] The identity encoder layer of the expression generation network extracts the identity features of the anime character from the base image;

[0025] At the minimum resolution, the morph decoder layer of the expression change generation network uses a multi-layer MobileOneBlock structure. When the resolution is increased, the morph decoder layer of the expression change generation network uses a single-layer MobileOneBlock structure. The morph decoder layer of the expression change generation network takes the identity features of the cartoon character extracted by the identity encoder layer and the target expression vector as input to generate the expression change partial map.

[0026] The error is calculated according to the expression change loss function, and the model parameters of the morphdecoder layer of the expression change generation network are adjusted according to the error.

[0027] Preferably, the target head posture vector is mapped identically to the expression change partial map to generate a three-dimensional reconstructed image, thereby driving the avatar, which specifically includes the following steps:

[0028] Sample the pixels of the expression change part of the map, and then generate a resampled map using a bilinear interpolation algorithm;

[0029] Obtain the identity features of the resampled image through the encoder layer of the head rotation generation network;

[0030] The decoder layer of the head rotation generation network takes the identity features of the resampled image obtained by the encoder layer and the target head pose vector as input to generate a 3D reconstructed image to drive the avatar;

[0031] The error is calculated according to the head rotation loss function, and the model parameters of the decoder layer of the head rotation generation network are adjusted according to the error.

[0032] Preferably, the method further comprises the following steps:

[0033] When generating the image to be mapped, the foreground and background binary images are saved, and the background is generated by making a cutout algorithm data set.

[0034] The present invention also provides a real-time cartoon avatar driving system, comprising:

[0035] Acquisition module, used to obtain real-time images of users;

[0036] Vector extraction module, used to solve the 3DMM model of the user's real-time image to obtain the target expression vector and target head posture vector;

[0037] The expression reconstruction module is used to perform semantic mapping on the target expression vector and generate a partial graph of expression changes;

[0038] The head posture reconstruction module is used to map the target head posture vector identity to the expression change part map to generate a three-dimensional reconstructed image, thereby driving the avatar.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention proposes a method for large-scale production of high-quality datasets for driving cartoon avatars. By generating datasets using this method, a data-driven cartoon character avatar driving method can be implemented.

[0041] This paper proposes a lightweight mobile avatar driving method that ensures expression fidelity and image quality, while reducing image artifacts. Because the human eye is highly sensitive to discontinuities in video, this method improves image generation stability without reference to temporal information. This method enables real-time processing on mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0043] Figure 1 It is a schematic diagram of different facial expressions;

[0044] Figure 2 is a schematic diagram of head posture;

[0045] Figure 3 It is a diagram of eye movement;

[0046] Figure 4 It is a schematic diagram of the probability density function of head posture sampling;

[0047] Figure 5 This is a schematic diagram of the sample composition of expression changes and head rotation;

[0048] Figure 6 This is a diagram of the background filling of the dataset;

[0049] Figure 7 It is a schematic diagram of the avatar driving process;

[0050] Figure 8 is a schematic diagram of network resolution;

[0051] Figure 9 This is the morph_decoder network structure diagram;

[0052] Figure 10 It is the network structure diagram generated by head rotation;

[0053] Figure 11 It is a schematic diagram of the flow field refinement process;

[0054] Figure 12This is a schematic diagram of reducing warp image artifacts.

[0055] Figure 13 It is a flow chart of a real-time cartoon avatar driving method of the present invention. DETAILED DESCRIPTION

[0056] The following description sets forth numerous specific details to facilitate a thorough understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific implementations disclosed below.

[0057] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0058] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0059] The present invention is described in further detail below with reference to the accompanying drawings:

[0060] like Figure 13 As shown, a real-time animation avatar driving method includes the following steps:

[0061] Get real-time images of users;

[0062] Solve the 3DMM model of the user's real-time image to obtain the target expression vector and target head posture vector;

[0063] Perform semantic mapping on the target expression vector to generate a partial graph of expression changes;

[0064] The target head pose vector is mapped to the expression change part map to generate a 3D reconstructed image, thereby driving the avatar.

[0065] Preferably, solving the 3DMM model for the user's real-time image to obtain the target expression vector and the target head posture vector includes the following steps:

[0066] According to the 3DMM model, the image to be mapped is obtained from the user's real-time image;

[0067] Get the target expression vector and target head pose vector from the image to be mapped.

[0068] Preferably, according to the 3DMM model, the image to be mapped is obtained from the user's real-time image, which specifically includes the following steps:

[0069] Construct an expression mapping dictionary based on the expression change names in the 3D model corresponding to the semantics of each dimension of the 3DMM model coefficient; the expression mapping dictionary includes a mapping relationship from the 3DMM model coefficient to multiple 3D model expression names;

[0070] According to the expression mapping dictionary, a mappable image corresponding to the expression name of the 3D model is obtained from the user's real-time image;

[0071] The mappable image is rendered to obtain an image to be mapped.

[0072] Preferably, obtaining the target expression vector and the target head posture vector from the image to be mapped specifically includes the following steps:

[0073] Generate weights of 3DMM model coefficients in the expression mapping dictionary;

[0074] Obtain facial expression data of the image to be mapped according to the weights of the 3DMM model coefficients;

[0075] Normalizing the facial expression data of the image to be mapped to obtain normalized facial expression data;

[0076] The image to be mapped is sampled through a probability distribution function to obtain head posture data;

[0077] The normalized facial expression data and head posture data are rendered to obtain a target expression vector and a target head posture vector.

[0078] Preferably, semantic mapping is performed on the target expression vector to generate an expression change partial graph, which specifically includes the following steps:

[0079] The identity encoder layer of the expression generation network extracts the identity features of the anime character from the base image;

[0080] At the minimum resolution, the morph decoder layer of the expression change generation network uses a multi-layer MobileOneBlock structure. When the resolution is increased, the morph decoder layer of the expression change generation network uses a single-layer MobileOneBlock structure. The morph decoder layer of the expression change generation network takes the identity features of the cartoon character extracted by the identity encoder layer and the target expression vector as input to generate the expression change partial map.

[0081] The error is calculated according to the expression change loss function, and the model parameters of the morphdecoder layer of the expression change generation network are adjusted according to the error.

[0082] Preferably, the target head posture vector is mapped identically to the expression change partial map to generate a three-dimensional reconstructed image, thereby driving the avatar, which specifically includes the following steps:

[0083] Sample the pixels of the expression change part of the map, and then generate a resampled map using a bilinear interpolation algorithm;

[0084] Obtain the identity features of the resampled image through the encoder layer of the head rotation generation network;

[0085] The decoder layer of the head rotation generation network takes the identity features of the resampled image obtained by the encoder layer and the target head pose vector as input to generate a 3D reconstructed image to drive the avatar;

[0086] The error is calculated according to the head rotation loss function, and the model parameters of the decoder layer of the head rotation generation network are adjusted according to the error.

[0087] Preferably, the method further comprises the following steps:

[0088] When generating the image to be mapped, the foreground and background binary images are saved, and the background is generated by making a cutout algorithm data set.

[0089] The present invention also provides a real-time cartoon avatar driving system, comprising:

[0090] Acquisition module, used to obtain real-time images of users;

[0091] Vector extraction module, used to solve the 3DMM model of the user's real-time image to obtain the target expression vector and target head posture vector;

[0092] The expression reconstruction module is used to perform semantic mapping on the target expression vector and generate a partial graph of expression changes;

[0093] The head posture reconstruction module is used to map the target head posture vector identity to the expression change part map to generate a three-dimensional reconstructed image, thereby driving the avatar.

[0094] In order to better illustrate the technical effects of the present invention, the present invention provides the following specific examples to illustrate the above technical process:

[0095] Example 1, a real-time cartoon avatar driving method, includes the following three parts:

[0096] 1. 3DMM;

[0097] The 3DMM (3DMorphableModel) is a universal 3D face model. Its core concept is that faces can be matched one-to-one in 3D space and can be derived by weighted addition of multiple orthogonal face bases. Each 3D face can be represented by a basis vector composed of all faces in a database. This allows the model of any 3D face to be solved, transforming 3D face reconstruction into a problem of solving the coefficients of each basis vector.

[0098] 3DMM coefficients are the bridge between real human faces and cartoon faces. We achieve driving of cartoon faces by mapping 3DMM coefficients with the same semantics to the expression coefficients of the cartoon model. In this invention, the expression vector expression∈R composed of the expression coefficients solved by 3DMM is required. 47 and head pose vector headpose∈R 3 .

[0099] 2. 3D anime avatar model;

[0100] The 3D anime avatar model has key components that enable cross-domain driving. One is facial expression deformation, which allows the model to generate facial changes. The degree of deformation for each expression varies continuously within the range [0, 1], enabling continuous control of the expression. Certain expressions, such as blinking, fully correspond to the semantics of the 3DMM expression coefficients. Another component is the skeleton. By controlling the skeleton of the head and eyeballs, head and eye movements can be achieved.

[0101] Facial expression deformation is divided into several regions, eyebrows, eyes, and mouth, and each region has a series of expressions, as shown in the figure below. We only select expressions that can correspond to 3DMM semantics for the following facial expression sampling, a total of 32 dimensions, which we record as e∈R 32 , and distinguish it from the expression coefficient of 3DMM. Figure 1 shown

[0102] We define the head movement posture using Euler angles, which correspond to the head pose vector headpose of 3DMM, namely pitch angle, yaw angle, and roll angle, as shown in Figure 2 As shown:

[0103] Eye movement is achieved by controlling the bones of the left and right eyes, such as Figure 3 As shown:

[0104] 3. Network reparameterization;

[0105] Reparameterization can play two main roles:

[0106] Improving network performance can achieve higher performance while maintaining the same network structure during deployment. This has been proven in ACNet, DDBNet, etc.

[0107] Change the network structure and convert it to a more efficient VGG-style structure during deployment.

[0108] The speed of network inference is mainly determined by three factors: computational complexity, memory access cost, and network parallelism. On many mobile devices, memory access cost is a major bottleneck limiting inference speed. Therefore, converting the residual or inception structure to a straight VGG structure can greatly reduce memory access cost, increase network parallelism, and thus reduce the time required for inference.

[0109] This paper mainly uses the basic network structure in "RepVGG: Making VGG-style ConvNets Great Again" and "AnImproved One millisecond Mobile Backbone" to build the network.

[0110] 1. Anime Avatar Driven Dataset Creation

[0111] 1.1) Model data preprocessing:

[0112] 1.1.1) Data model preprocessing mainly includes two stages:

[0113] (1) Search for the expression change names in the 3D model corresponding to the semantics of each dimension of the 3DMM model coefficient, and build a one-to-many expression mapping dictionary from the 3DMM model to the 3D model expression name. The dictionary has a priority.

[0114] (2) Model preprocessing and invalid expression annotation. Search for potential mappable expressions in the model according to the expression mapping dictionary, rename the mapped 3D model expression names to the standard names in the expression dictionary, and use the same rendering pipeline tool in the third step after the expression name standardization to render all the mappable expressions of the model into images to be annotated. Then perform data annotation, and mark the two cases of model production problems such as no expression change and expression names matching but not matching 3DMM semantics as invalid to prevent the generation of dirty data in the next step of sampling and affecting the final performance of the model.

[0115] 1.2) Sampling of facial expressions and head posture movements:

[0116] 1.2.1) Facial expression sampling;

[0117] Facial expression sampling first generates weights for each of the 32 expressions in the expression mapping dictionary. If the model has a corresponding expression, a random value in the interval [0, 1] is generated using a uniform distribution, as described below. If the model does not have a corresponding expression, the dimension is assigned a value of 0.

[0118] e i ~uniform[0,1],if model has e i

[0119] Where: e i Represents the expressions in the 3D model with the same semantics as the 3DMM coefficients, that is, each dimension in the expression mapping dictionary constructed in step (1) above. Uniform represents uniform distribution.

[0120] Expressions are divided into four categories based on facial regions: eyebrows, eyes, mouth, and eyeballs. Sampling between different regions does not affect each other. Expressions within the same region need to be normalized after sampling in each expression dimension to prevent model penetration. Taking the eye region as an example, ensure that the sum of the weights of all expressions related to the left eye is less than 1, and the sum of the weights of expressions related to the right eye is less than 1. The specific normalization process is as follows:

[0121]

[0122]

[0123] norm_weight_eye=max(Eye left ,Eye right )

[0124] e i =e i / (norm_weight_eye+eps),e i ∈Eye left or ei ∈Eye right

[0125] Where: Eye left represents the sum of all expression weights related to the left eye, and n represents e i Total dimension, in this embodiment n=32, norm_weight_eye represents the normalization coefficient of the eye used to prevent penetration, so that the sum of the relevant weights of the left eye is less than 1, and the sum of the relevant weights of the right eye is less than 1.

[0126] The normalization process of the eyebrow region is the same as that of the eyes. The mouth region is not distinguished between left and right, and the other processing processes are the same.

[0127] 1.2.2) Head posture sampling;

[0128] For head posture sampling, we define a probability distribution function f(x) = 0.6x 2 +0.3 instead of uniformly distributing the sampling in each dimension of the Euler angle, it can also be defined as other forms of probability distribution functions as long as it satisfies This increases the sampling probability of difficult samples and improves the performance of the final model. Figure 4 As shown;

[0129] In order to ensure the degree of expression restoration, we decompose the animation driver into two processes. The first process only performs expression changes, and the second process performs head movements. During the data production process, a GroundTruth image is generated for both processes, and finally a set of three sample images is generated, such as Figure 5 As shown, Figure 5 (a) is the basic image, (b) is the expression change, and (c) is the expression change + head rotation;

[0130] 1.3) Dataset rendering:

[0131] This process requires an automated rendering pipeline. The model data and sampling actions have already been obtained in the previous steps. Two other factors are required to complete the rendering: the camera position and lighting configuration. The camera position is calculated using the skeletal coordinates of the eyes and head, allowing for adaptive adjustments to the model's height and size. The lighting configuration ensures that the lighting configuration within a group of samples from the previous step is identical. Light direction and intensity are randomly assigned to each group of samples to enhance the model's robustness to lighting.

[0132] 1.4) Background filling data enhancement: We use the method of ImageMatting dataset to fill the background of the rendered anime portrait. When rendering the dataset, we save the binary images of the foreground and background at the same time, then randomly select a background image from the anime background dataset, randomly crop a patch, and resize it to the same resolution as the anime image. Figure 6 As shown, Figure 6 (a) is the foreground binary image, (b) is the mask image, (c) is the background binary image, and (d) is the fused image;

[0133] The fusion formula is:

[0134] I fusion =I fg *mask+(1-mask)*I bg .

[0135] Where: I fusion Represents the image after the foreground and background are combined, I fg Represents the foreground, i.e. the rendered cartoon character image, mask represents the foreground and background binary images saved during rendering, I bg Represents an anime background image.

[0136] 2. Mobile avatar driving method:

[0137] Our anime face driving network is generally divided into two parts. The first part is to change the expression of the frontal face image according to the target expression vector to obtain the anime image corresponding to the target expression. The second part is a head rotation generation network, which inputs the expression-changed image and the target head posture vector to generate the final image-driven result. Figure 7 As shown;

[0138] The expression generation network generates images of characters with altered expressions. It consists of two components: an identity encoder that extracts the character's identity features from a base image, and a morph decoder that takes the identity features and a target expression vector as input to generate images of the target expression. These two networks are trained together. During inference, the encoder performs a single forward inference pass on an image of an animated character to obtain and store the identity features. Later in inference, the identity features and the expression vector derived from the current real-life image are combined to generate the expression image. Our encoder and decoder employ an asymmetric design, using a computationally intensive encoder to improve network performance and a lightweight decoder to increase inference speed.

[0139] The head rotation generation network generates images after head rotation, which is also an encoder-decoder structure.

[0140] In addition, we adopted a strategy to save computational effort. The head rotation generation network uses full resolution, while the expression change network uses a smaller resolution. This is because the surrounding area is almost certainly the background area, and the middle part already covers the entire face of the anime character and can extract global facial features. This can also reduce FLOPs to about half of the full resolution or even lower. Figure 8 shown.

[0141] 2.1) Expression Change Generation Network:

[0142] 2.1.1) identity encoder

[0143] After reparameterization of the identity feature extraction network, the network structure is a stack of a series of convolutional layers. We use 4 convolutional layers at each scale to enhance the network's ability to extract features.

[0144] 2.1.2)morph decoder

[0145] The overall structure of the face deformation decoder is as follows Figure 9 As shown, we only use as many MobileOne blocks as possible at the minimum resolution. As the resolution increases, only one layer is used to reduce computational effort and increase inference speed. The network inputs identity features and expression vectors, and outputs a mask and an image of the expression change. Alpha fusion is then performed on the mask and the original image to enhance temporal stability. The formula is as follows:

[0146] I morph =I mask *I base +(1-I mask )*I morph_gen

[0147] Where: I morph The expression change generation network finally generates an image, I base For natural expression images, I mask is the expression change area mask learned by the network, I morph_gen Expression variation images generated by the network.

[0148] 2.1.3) Expression change loss function

[0149] The expression change loss function uses reconstruction loss and perception loss. In order to improve the performance of details, adversarial loss is added. The total loss is calculated as follows:

[0150] L G_total=λ r L r +λ vgg L vgg +λ adv L G_adv

[0151] The reconstruction loss L r =||I morph -I gt ||1, perceptual loss In our implementation, N=5, and the loss is calculated on a feature map before ReLU in each of the five stages of VGG19.

[0152] Generator’s adversarial loss L G_adv =(D(I morph )-1) 2

[0153] Discriminator’s adversarial loss L D_adv =(D(I gt )-1) 2 +(D(I morph )) 2

[0154] Where: L G_total is the total loss of the expression change generation network, L r is the reconstruction loss, λ vgg is the perceptual loss, L G_adv To combat losses;

[0155] λ r ,λ vgg ,λ adv They correspond to reconstruction loss weight, perception loss weight, and adversarial loss weight respectively;

[0156] I morph The expression change generation network finally generates an image, I gt Groundtruth image of expression changes;

[0157] VGG i () represents the features of the image extracted by the VGG network, VGG i (I gt ) is the feature extracted from the GroundTruth image, VGG i (I morph ) Generates features extracted from images for the expression change network;

[0158] D() represents the output result of the image through the discriminator, D(I gt ) is the output obtained by GroundTruth through the discriminator, D(I morph) is the output of the expression change network generated image through the discriminator.

[0159] 2.2) Head rotation generation network

[0160] 2.2.1) Based on flow field method

[0161] If we continue to use the direct generation method with the same expression changes, the generated image will be too blurry and fail to preserve the details of the original image. Therefore, we use a flow field method based on the swarm pathfinding algorithm. We predict a flow field. That is, the network predicts the position in the original image where each pixel in the generated image should be sampled. Then, bilinear interpolation is used to generate the resampled image.

[0162] 2.2.2) The overall network structure, such as Figure 10 As shown;

[0163] The decoder generates the final warp image in a coarse-to-fine process, with the resolution gradually increasing and the estimation of the flow field gradually becoming more accurate, such as Figure 11 As shown;

[0164] In addition, we also try to reduce the artifacts that are easy to be generated by the flow field method, so that the network can implicitly learn the template that needs to be repaired. mask and fill image I inpaint and compare it with the result of resampling the predicted flow field I warp On the other hand, when the head posture is very different from the original image, there will always be areas that cannot be resampled from the original image. These areas also need to be generated to achieve this, otherwise they will become blurry or artifact areas, such as Figure 12 As shown. It can be expressed as follows:

[0165] I rotate =I mask *I warp +(1-I mask )*I inpaint

[0166] Where: I rotate Generate the final image for the head rotation network, I mask is the inpainting mask implicitly learned by the network, I warp is the flow field pair I predicted by the network morph The image generated by resampling, I inpaint The inpainted image predicted by the network.

[0167] 2.2.3) Loss Function

[0168] The loss function consists of four parts, and the formula is as follows:

[0169] L total =λ warp L warp +λ r L r +λ adv L adv +λ init L init

[0170] The reconstruction loss and adversarial loss are the same as those of the expression change network. The loss of the warp resampled image is expanded to At n resolutions g(field i ,I morph ) represents the field i The reconstructed image, the corresponding I gt_i Representative I gt The result of downsampling to the corresponding resolution.

[0171] Initialize the loss L init Used to help network convergence in the early stages, init_field i Generate I for interpolation morph The flow field corresponding to the result of downsampling to each resolution, that is, the effect of no head rotation. init It will decrease as the number of steps increases during training, and will decrease to 0 after reaching the preset number of steps. The change calculation formula is as follows:

[0172] λ init =(1-step / step_end)*weight_init.

[0173] Where:

[0174] L total The total loss during training is the weighted sum of all losses, calculated according to the formula above.

[0175] L r is the image reconstruction loss, and the calculation formula is L r =||I rotate -I gt ||1, where I rotate As mentioned above, this is the final result of the head rotation network. gt A GroundTruth image of a head rotation.

[0176] L warp The image loss is resampled based on the flow field method, and the calculation formula is given above;

[0177] Ladv Adversarial loss L for head rotation generation network adv =(D(I rotate )-1) 2 , corresponding to the adversarial loss L of the discriminator network D_adv =(D(I gt )-1) 2 +(D(I rotate )) 2

[0178] L init Initialize the flow field loss. The details are given above, where step is the current iteration number in the training process, step_end is the iteration number at which the initial flow field loss is expected to be reduced to 0, and weight_init is the initial weight for initializing the flow field loss at the beginning of training. In this example, 10.0 is used.

[0179] λ r is the image reconstruction loss weight, λ warp Resample image loss weight based on flow field method, λ adv Adversarial loss weight, λ init Initialize the flow field loss weights.

[0180] In the several embodiments provided herein, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules, subassemblies, or units is merely a logical functional division. In actual implementation, other divisions may be used, such as combining or integrating multiple units, subassemblies, or components into another device, or some features may be omitted or not implemented.

[0181] The units may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0182] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0183] In particular, according to an embodiment disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.

[0184] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0185] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention shall be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A real-time cartoon avatar driving method, characterized in that: The following steps are involved: Get real-time images of users; Solve the 3DMM model of the user's real-time image to obtain the target expression vector and target head posture vector; The identity encoder layer of the expression generation network extracts the identity features of the anime character from the base image; At the minimum resolution, the morphdecoder layer of the expression change generation network uses a multi-layer MobileOneBlock structure; After the resolution is increased, the morphdecoder layer of the expression change generation network uses a MobileOneBlock structure. The morphdecoder layer of the expression change generation network takes the identity features of the cartoon character extracted by the identityencoder layer and the target expression vector as input to generate the expression change partial graph. Calculate the error based on the expression change loss function, and adjust the model parameters of the morphdecoder layer of the expression change generation network based on the error; The expression change generation network includes an identity encoder, a morph decoder, and an expression change loss function; The identity encoder: After the identity feature extraction network is reparameterized, the network structure is a stack of convolutional layers, using 4 convolutional layers at each scale; The morph decoder network input is identity features and expression vectors, and outputs a mask and a picture of the expression change. Alpha fusion is performed on the mask and the original image to enhance temporal stability. The formula is as follows: I morph =I mask *I base +(1-I mask )*I morph_gen Where: I morph The expression change generation network finally generates an image, I base For natural expression images, I mask is the expression change area mask learned by the network, I morph_gen Expression change images generated for the network; The expression change loss function: The expression change loss function uses reconstruction loss, perception loss, and adversarial loss. The total loss is calculated as follows: L G_total =λ r L r +λ vgg L vgg +λ adv L G_adv The reconstruction loss L r =||I morph -I gt ||1, perceptual loss N = 5, the loss is calculated on a feature map before ReLU in each of the five stages of VGG19; Generator’s adversarial loss L G_adv =(D(I morph )-1) 2 Discriminator’s adversarial loss L D_adv =(D(I gt )-1) 2 +(D(I morph )) 2 Where: L G_total is the total loss of the expression change generation network, L r To reconstruct the loss, L vgg is the perceptual loss, L G_adv To combat losses; λ r ,λ vgg ,λ adv They correspond to reconstruction loss weight, perception loss weight, and adversarial loss weight respectively; I morph The expression change generation network finally generates an image, I gt GroundTruth images with changing expressions; VGG i () represents the features of the image extracted by the VGG network, VGG i (I gt ) is the feature extracted from the GroundTruth image, VGG i (I morph ) Generates features extracted from images for the expression change network; D() represents the output result of the image through the discriminator, D(I gt ) is the output obtained by GroundTruth through the discriminator, D(I morph ) is the output of the expression change network generated image through the discriminator; Sample the pixels of the expression change part of the map, and then generate a resampled map using a bilinear interpolation algorithm; Obtain the identity features of the resampled image through the encoder layer of the head rotation generation network; The decoder layer of the head rotation generation network takes the identity features of the resampled image obtained by the encoder layer and the target head pose vector as input to generate a 3D reconstructed image to drive the avatar; Calculate the error based on the head rotation loss function and adjust the model parameters of the decoder layer of the head rotation generation network based on the error; The head rotation generation network is based on the flow field method, which uses a flow field method based on the group pathfinding algorithm to predict a flow field and then uses bilinear interpolation to generate a resampled image; The overall network structure of the head rotation generation network is: let the network implicitly learn the template I that needs to be repaired mask and fill image I inpaint and compare it with the result of resampling the predicted flow field I warp Perform alpha blending; the formula is as follows: I rotate =I mask *I warp +(1-I mask )*I inpaint Where: I rotate Generate the final image for the head rotation network, I mask is the inpainting mask implicitly learned by the network, I warp is the flow field pair I predicted by the network morph The image generated by resampling, I inpaint The image to be restored as predicted by the network; The loss function of the head rotation generation network consists of four parts, and the formula is as follows: L total =λ warp L warp +λ r L r +λ adv L adv +λ init L init The reconstruction loss and adversarial loss are the same as those of the expression change network; the loss of the warp resampled image is expanded to At n resolutions g(field i ,I morph ) represents the field i The reconstructed image, the corresponding I gt_i Representative I gt The result of downsampling to the corresponding resolution; Initialize the loss L init Used to help network convergence in the early stages, init_field i Generate I for interpolation morph The flow field corresponding to the result of downsampling to each resolution, that is, the effect of head not rotating; λ init It will decrease as the number of steps increases during training, and will decrease to 0 after reaching the preset number of steps. The change calculation formula is as follows: λ init =(1-step / step_end)*weight_init Where: L total The total loss during training is the weighted sum of all losses; L r is the image reconstruction loss, and the calculation formula is L r =||I rotate -I gt ||1, where I rotate is the final generated result of the head rotation network, I gt GroundTruth image for head rotation; L warp Resampling image loss based on flow field method; L adv Adversarial loss L for head rotation generation network adv =(D(I rotate )-1) 2 , corresponding to the adversarial loss L of the discriminator network D_adv =(D(I gt )-1) 2 +(D(I rotate )) 2 L init To initialize the flow field loss, step is the current iteration number in the training process, step_end is the iteration number at which the initialization flow field loss is expected to be reduced to 0, and weight_init is the initial weight for initializing the flow field loss at the beginning of training, which is 10.0; λ r is the image reconstruction loss weight, λ warp Resample image loss weight based on flow field method, λ adv Adversarial loss weight, λ init Initialize the flow field loss weights.

2. The real-time cartoon avatar driving method according to claim 1, characterized in that: Solve the 3DMM model of the user's real-time image to obtain the target expression vector and target head posture vector, including the following steps: According to the 3DMM model, the image to be mapped is obtained from the user's real-time image; Get the target expression vector and target head pose vector from the image to be mapped.

3. The real-time cartoon avatar driving method according to claim 2, characterized in that: According to the 3DMM model, the image to be mapped is obtained from the user's real-time image, which specifically includes the following steps: Construct an expression mapping dictionary based on the expression change names in the 3D model corresponding to the semantics of each dimension of the 3DMM model coefficient; the expression mapping dictionary includes a mapping relationship from the 3DMM model coefficient to multiple 3D model expression names; According to the expression mapping dictionary, a mappable image corresponding to the expression name of the 3D model is obtained from the user's real-time image; The mappable image is rendered to obtain an image to be mapped.

4. The real-time cartoon avatar driving method according to claim 3, characterized in that: Obtaining the target expression vector and target head pose vector from the image to be mapped includes the following steps: Generate weights of 3DMM model coefficients in the expression mapping dictionary; Obtain facial expression data of the image to be mapped according to the weights of the 3DMM model coefficients; Normalizing the facial expression data of the image to be mapped to obtain normalized facial expression data; The image to be mapped is sampled through a probability distribution function to obtain head posture data; The normalized facial expression data and head posture data are rendered to obtain a target expression vector and a target head posture vector.

5. The real-time cartoon avatar driving method according to claim 1, characterized in that: The following steps are also included: When generating the image to be mapped, the foreground and background binary images are saved, and the background is generated by making a cutout algorithm data set.

6. A real-time cartoon avatar driving system, used to implement the real-time cartoon avatar driving method according to any one of claims 1 to 5, characterized in that: include: Acquisition module, used to obtain real-time images of users; Vector extraction module, used to solve the 3DMM model of the user's real-time image to obtain the target expression vector and target head posture vector; The expression reconstruction module is used to perform semantic mapping on the target expression vector and generate a partial graph of expression changes; The head posture reconstruction module is used to map the target head posture vector identity to the expression change part map to generate a three-dimensional reconstructed image, thereby driving the avatar.

Images