A method, device, equipment and storage medium for determining expression drive parameters
Automatically generate expression parameter samples through the engine and use network models for semi-supervised training, solving the problem of manual labeling in the existing technology, realizing fast and accurate facial expression capture, reducing cost and time, improving efficiency and accuracy .
Patent Information
- Application Number
- CN202211017677.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-08-23
AI Technical Summary
The existing facial expression capture technology requires animators to manually mark three-dimensional expression marking points, which leads to long-term, low efficiency and high labor costs.
The engine automatically generates the annotation parameters required for the training stage, and uses the network model to automatically export two-dimensional and three-dimensional expression parameter samples to achieve semi-supervised training, reduce manual annotation, and thus quickly and accurately capture facial expressions.
It reduces training costs and time, improves the efficiency and accuracy of facial expression capture, avoids offset jitter, and achieves high-precision surface capture effect.
Smart Images

Figure CN115346262B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of facial capture technology, and in particular to a method, device, equipment and storage medium for determining expression drive parameters. Background Art
[0002] Facial expression capture refers to the process of using a marker-based optical motion capture system to record the facial expressions and movements of a physical character (such as a real person) and convert them into a series of parameters. Compared with artificially produced virtual character expressions, virtual characters generated using facial expression capture technology are more realistic.
[0003] In the related technology, optical identification points can be marked at preset positions on the face of a real actor, and the movements of the real actor can be recorded using a motion capture camera to obtain a video image, and then the video image can be three-dimensionally modeled using data processing equipment to generate a virtual character. On this basis, animators need to manually mark expression markers on the face of the virtual character based on the video image, and then, based on the principles of biokinematics, calculate the six-degree-of-freedom motion of the skeleton based on the three-dimensional expression markers to drive the facial expressions of the virtual character. However, in this process, animators need to manually mark three-dimensional expression markers on the face of the virtual character, which results in a long time-consuming, low-efficiency, and high labor cost for the marking of three-dimensional expression markers. Summary of the invention
[0004] According to one aspect of the present disclosure, a method for determining expression driving parameters is provided, comprising:
[0005] Collecting expression image samples of the virtual character having the first expression mark point in the engine;
[0006] Acquire a two-dimensional expression parameter sample and a three-dimensional expression parameter sample of a first expression marker point in the expression image sample from the engine;
[0007] Training a first network model based on expression image samples, two-dimensional expression parameter samples, and three-dimensional expression parameter samples;
[0008] Processing a target expression image of a target character having a second expression mark point and two-dimensional expression parameters of the second expression mark point in the target expression image based on the trained first network model to obtain three-dimensional expression parameters;
[0009] The three-dimensional expression parameters are converted into expression driving parameters based on the trained second network model.
[0010] According to another aspect of the present disclosure, a device for determining expression driving parameters is provided, comprising:
[0011] A construction module is used to collect expression image samples of a virtual character having a first expression mark point in the engine;
[0012] Acquire a two-dimensional expression parameter sample and a three-dimensional expression parameter sample of a first expression marker point in the expression image sample from the engine;
[0013] A training module, used for training a first network model based on expression image samples, two-dimensional expression parameter samples and three-dimensional expression parameter samples;
[0014] The inference module is used to process the target expression image of the target character with the second expression marker point and the two-dimensional expression parameters of the second expression marker point in the target expression image based on the trained first network model to obtain three-dimensional expression parameters; and convert the three-dimensional expression parameters into expression driving parameters based on the trained second network model.
[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0016] processor; and,
[0017] A memory for storing programs;
[0018] The program includes instructions, and when the instructions are executed by a processor, the processor executes the method according to the exemplary embodiment of the present disclosure.
[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method according to the exemplary embodiments of the present disclosure.
[0020] In one or more technical solutions provided in the exemplary embodiments of the present disclosure, when the first network model constructs a training set in the training phase, after collecting expression image samples of the virtual character with the first expression mark point in the engine, the two-dimensional expression parameter samples and three-dimensional expression parameter samples of the first expression mark point of the virtual character can be automatically exported directly from the engine, thereby completing the construction of the training set of the first network model. It can be seen that when the first network model constructs the training set in the training phase, there is no need to manually mark the two-dimensional expression parameter samples and the three-dimensional expression parameter samples, so that the first network model can achieve semi-supervised training, thereby saving training costs. On this basis, the target expression image and the two-dimensional expression parameters of the second expression mark point in the target expression image are input to the trained first network model, and the three-dimensional expression parameters can be automatically obtained, and the trained second network model is used to convert the three-dimensional expression parameters into expression driving parameters, so as to quickly and accurately capture the facial expressions of the actors and output the face capture effects of film and television. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0022] Figure 1 A schematic diagram showing an example system in which various methods described herein may be implemented according to some embodiments of the present disclosure;
[0023] Figure 2 A flow chart showing a method for determining expression drive parameters according to some embodiments of the present disclosure is shown;
[0024] Figure 3 A schematic diagram showing the distribution of first expression marker points in some embodiments of the present disclosure is shown;
[0025] Figure 4A to Figure 4E A schematic diagram showing the effect of marker tracking in some embodiments of the present disclosure is shown;
[0026] Figure 5 A schematic diagram showing the architecture of a first network model of some embodiments of the present disclosure is shown;
[0027] Fig. 6A A schematic diagram showing the architecture of a stacked hourglass network model according to some embodiments of the present disclosure is shown;
[0028] Figure 6B A schematic diagram showing the architecture of a sub-hourglass network according to some embodiments of the present disclosure is shown;
[0029] Figure 7 A schematic diagram showing the architecture of a second network model of some embodiments of the present disclosure is shown;
[0030] Figure 8A to Figure 8F A schematic diagram of a visualization space of three-dimensional expression parameter samples of six characters according to some embodiments of the present disclosure is shown;
[0031] Fig. 9 A schematic block diagram of modules of an apparatus for determining expression drive parameters in some embodiments of the present disclosure is shown;
[0032] Fig.10 A schematic block diagram of a chip according to some embodiments of the present disclosure is shown;
[0033] Fig.11 A structural block diagram of an electronic device that can be used to implement some embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0034] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0035] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0036] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0038] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0039] Before introducing the embodiments of the present disclosure, the following definitions are given for the terms involved in the embodiments of the present disclosure:
[0040] Digital human, in a narrow sense, is the product of the integration of information science and life science. It can be obtained by virtually simulating the human body's form and function at different levels through information science methods.
[0041] Entity characters are organisms in real life, such as humans, animals, etc., for example, characters in film and television dramas played by actors.
[0042] A virtual character is a living organism in the virtual world. It is a character created by animators through computer technology, such as virtual characters, virtual animals and other virtual characters in animations, games or movies.
[0043] A fully connected neural network consists of several layers of neurons, including an input layer, an output layer, and several hidden layers. The number of neurons in each layer and the number of hidden layers are freely defined by the model builder according to the task requirements. All neurons in each layer are connected to all neurons in another layer, and neurons in the same layer are not connected to each other.
[0044] The Stacked Hourglass Networks consists of multiple cascaded hourglass models, each of which can use multi-scale features to identify posture, thereby ensuring that the prediction results are complete and accurate.
[0045] Expression driving parameters, also known as RigValue, can represent the parameters of bone movement. It is a set of 168-dimensional vectors, and the value of each dimension represents the movement state (pulling and stretching) of each facial muscle.
[0046] Mesh vertex coordinates refer to the vertex coordinates in a three-dimensional mesh model (also known as Mesh). The surfaces of objects in the real world are intuitively composed of curved surfaces, but in the computer world, only discrete structures can be used to simulate continuous objects in reality. Therefore, the curved surfaces in the real world are actually composed of countless small polygonal meshes in the computer. These polygonal meshes are also called "Mesh", and the polygonal meshes can be triangular meshes. Taking triangular meshes as an example, Mesh includes at least information about point sets and triangular mesh sets. A Mesh vertex intersects with multiple triangular meshes. Therefore, the shapes of the multiple triangular meshes intersecting with it can be changed by modifying the Mesh vertex coordinates, thereby generating the corresponding three-dimensional network model.
[0047] The semi-supervised training method refers to the use of a large amount of unlabeled data to train artificial intelligence models, which can save a lot of manual labeling costs.
[0048] The loss function, also known as the objective function, is an important equation used to measure the difference between the predicted value and the target value. The higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of minimizing this loss as much as possible.
[0049] The model parameters include weight parameters and bias parameters. The weight parameters represent the slope of the hyperplane, and the bias parameters represent the intercept of the hyperplane.
[0050] The fully-connected layer is a classifier that can map the high-dimensional feature map obtained by feature extraction into a one-dimensional feature vector. This one-dimensional feature vector contains all feature information and can be converted into the probability of each category of the final classification.
[0051] The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model. In the process of training the neural network, the back propagation algorithm can be used to correct the size of the parameters in the initial neural network model, so that the reconstruction error loss of the neural network model becomes smaller and smaller. The forward transmission of the input signal to the output will generate error loss. The parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges.
[0052] In related technologies, facial markers can be placed on the faces of real actors (such as motion capture actors) to mark muscle details during facial movements, and then a motion capture camera can be used to record the movements of the real actors to obtain video images. Then, data processing equipment can be used to perform three-dimensional modeling on the video images to generate virtual characters. On this basis, animators need to manually mark expression markers on the faces of virtual characters based on video images, and then, based on the principles of biokinetics, calculate the six-degree-of-freedom motion of the skeleton based on the three-dimensional expression markers to drive the facial expressions of virtual characters.
[0053] In response to the above problems, an exemplary embodiment of the present disclosure provides a method for determining expression-driven parameters, which can utilize an engine to automatically generate the annotation parameters required in the training phase without manual annotation, thereby reducing training costs and enabling fast and accurate facial capture in the inference phase.
[0054] The method for determining expression driving parameters of the exemplary embodiment of the present disclosure can be applied to various scenarios that can be used to drive facial expressions of digital characters such as digital humans.
[0055] Figure 1 Schematic diagram of an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the system 100 of the exemplary embodiment of the present disclosure may include: a user device 110, a computing device 120, and a data storage system 130.
[0056] like Figure 1 As shown, the user equipment 110 can communicate with the computing device 120 through a communication network. The communication network can be a wired communication network or a wireless communication network. The wired communication network can be a communication network based on power line carrier technology, and the wireless communication network can be a local wireless network or a wide area wireless network. The local wireless network can be a WIFI wireless network, a Zigbee wireless network, a mobile communication network, a satellite communication network, etc.
[0057] like Figure 1As shown, the user device 110 may include a computer, a mobile phone, or an intelligent terminal such as an information processing center. The user device 110 may be used as the initiator of the model training operation or the expression drive parameter determination operation to initiate a request to the computing device 120. The computing device 120 may be a server with data processing functions such as a cloud server, a network server, an application server, and a management server, which is used to implement the training method and the generation method. A deep learning processor may be configured in the server. The deep learning processor may be a neuron of a single-core deep learning processor (Deep Learning Processor-Singlecore, abbreviated as DLP-S) or a multi-core deep learning processor (Deep Learning Processor-Multicore, abbreviated as DLP-M). DLP-M is a multi-core extension based on DLP-S. Multiple DLP-S are interconnected, multicasted, and inter-core synchronized through a network on chip (Network-on-chip, abbreviated as Noc) to communicate between cores to complete deep learning tasks and expression image generation tasks.
[0058] like Figure 1 As shown, the above-mentioned data storage system 130 can be a general term, including local storage and databases storing historical data. The database can be on the computing device 120, on other network servers, or on the data storage system 130. The data storage system 130 can be separate from the computing device 120 or integrated in the computing device 120. The data storage system 130 can not only input data uploaded by the user device, but also store program instructions, neuron data, weight data, etc. These neuron data and weight data can be trained data or untrained data, which is determined by the method implemented by the computing device 120. In addition, the data storage system 130 can also store the processing results (such as expression images, two-dimensional expression parameters of expression images, intermediate processing results or image quality) obtained by the computing device 120 into the data storage system.
[0059] In practical applications, such as Figure 1As shown, the user device 110 may have an image acquisition function, so that the user device can not only initiate a request to the execution device through the interactive interface, but also acquire images during the training phase and the expression drive parameter determination phase, and send the expression images to the computing device 120 through the communication network. Based on this, when the computing device 120 implements the training method, the sample images used can not only be obtained from the data storage system 130, but also from the user device 110 through the communication network. In addition, when the computing device 120 implements the generation method, the expression drive parameters generated by it can not only be fed back to the user device 110 through the communication network, but also be stored in the data storage system 130 as sample data for further training the model.
[0060] For example, Figure 1 As shown, the user device 110 may have a display interface, and image processing software such as Photoshop, MATLAB, Open CV, 3dmax, and MetaHumanCreator (Unreal Engine Face Pincher) may be installed locally on the user device. The user device 110 may respond to user operations, use these image processing software to pre-process the image offline, and then upload it to the computing device 120. Photoshop, MATLAB, Open CV, MetaHumanCreator (Unreal Engine Face Pincher) and other image processing software may also be installed in the computing device 120. The user may respond to user operations, remotely log in to the execution device, and use these image processing software to process the image.
[0061] The method for determining expression driving parameters provided by the exemplary embodiment of the present disclosure can obtain the three-dimensional expression parameters of the target expression image by using the trained first network model, and convert the three-dimensional expression parameters into expression driving parameters by using the trained second network model. In addition, the training set of the first network model in the training stage can be automatically generated by the engine, without the need for manual annotation of a large amount of character data, thereby reducing the manual annotation cost and training cost.
[0062] The method of the exemplary embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.
[0063] Figure 2 FIG. 1 is a flow chart showing a method for determining expression drive parameters in some embodiments of the present disclosure. Figure 2 As shown, the method for determining expression driving parameters of the exemplary embodiment of the present disclosure includes:
[0064] Step 201: Collect expression image samples of a virtual character with a first expression mark point in an engine. The engine may be various commercial or non-commercial engines, such as the metahuman creator in the Unreal Engine, or an engine developed by a user.
[0065] In practical applications, the desired digital human character can be selected from the digital character database as a virtual character, or a scanning technology (such as structured light scanning) can be used to build a virtual character in the engine, or the face pinching function of the engine can be used, such as manual carving, to build a virtual character. It should be understood that the virtual character of the embodiment of the present disclosure can be a digital image of an animal or a person. The expression image sample can be an expression image of a cat, dog, tiger, lion, etc., which will not be repeated here.
[0066] When the virtual character is built, you can draw expression markers on the face of the virtual character according to the preset reference. Figure 3 A schematic diagram showing the distribution of the first expression marker points in some embodiments is shown. Figure 3 As shown, preset positions on the face of the virtual character 300 can be marked with first expression markers, such as eyebrows, nose, lips, cheeks, dimples, and eyes are all marked with first expression markers. These markers are collectively referred to as first expression markers, with a total of 32 first expression markers.
[0067] After completing the construction of the virtual character, you can also bind expressions to the virtual character. Animators can use the engine to directly make the facial expressions of the virtual character based on expression needs, or use relevant tools to capture the expression-driven parameters of the real character and import them into the engine to bind facial expressions to the virtual character.
[0068] Exemplarily, when the facial expression is a dynamic facial expression, the facial expression bound to the virtual character is essentially a facial expression sequence. After the virtual character and the facial expression are bound, a virtual task expression animation can be generated; when the facial expression is a static facial expression, the facial expression bound to the virtual character is essentially a facial expression at a certain moment. After the virtual character and the facial expression are bound, the virtual character can display a virtual expression in a static state.
[0069] Considering the actual reasoning stage, it is necessary to use a real image acquisition device to capture the character's facial image, and obtain the two-dimensional expression parameters of the character's facial image based on the first expression mark point on the character's facial image, and obtain the three-dimensional expression parameters of the character's facial image with the help of the engine. Based on this, after binding facial expressions to the virtual character, the image acquisition parameters of the expression image samples can be set, that is, the parameters of the virtual image acquisition device that collects the expression image samples. If the number of expression image samples is relatively large, the image acquisition parameters of these expression image samples can be the same or the same.
[0070] Exemplarily, a virtual image acquisition device, such as a virtual video camera or a virtual camera, can be called, and the parameters of the virtual acquisition device, including the position and lens parameters of the virtual acquisition device, can be set so that the distance between the lens of the virtual acquisition device and the virtual character reaches the set distance. The lens parameters can include focal length, aperture, viewing angle, etc.
[0071] After setting up the image acquisition device of the expression image sample, the facial expression action displayed by the virtual character is recorded by the virtual image acquisition device, thereby obtaining the expression image sample. For example, if the expression animation of the virtual character is played in the engine, and the expression changes of the virtual character during the playing of the expression animation are recorded by the virtual camera, the expression video of the virtual character is obtained, that is, the expression sequence of the virtual character consisting of multiple frames of virtual character expressions.
[0072] Step 202: Obtain a two-dimensional expression parameter sample and a three-dimensional expression parameter sample of the first expression marker point in the expression image sample from the engine.
[0073] In this process, the two-dimensional expression parameter samples and three-dimensional expression parameter samples exported from the engine are actually corrected by image acquisition parameters, so that the obtained two-dimensional expression parameter samples and three-dimensional expression parameter samples and the expression image samples belong to the same spatial system.
[0074] Exemplarily, the two-dimensional expression parameter sample may refer to the two-dimensional coordinates of the first expression marker point, or may be the two-dimensional space parameters defining the two-dimensional coordinates, and the three-dimensional expression parameter sample may refer to the three-dimensional coordinates of the first expression marker point, or may be the three-dimensional space parameters defining the three-dimensional coordinates.
[0075] For example, when the expression image samples are multiple frames of virtual character expressions included in a virtual character expression video, the two-dimensional coordinates of the first expression marker point in each frame of the virtual character expression and the three-dimensional coordinates of each first expression marker point on the Mesh can be derived from the engine based on the image acquisition parameters of the expression image samples.
[0076] It can be seen that when the first network model constructs data in the training stage, after binding facial expressions for the virtual character in the engine, expression image samples can be directly collected through the virtual image acquisition device, and the two-dimensional expression parameter samples and three-dimensional expression parameter samples of the first expression marker point in the expression image samples can be automatically exported from the engine. The two-dimensional expression parameter samples and three-dimensional expression parameter samples of the first expression marker point can be automatically generated without a large amount of manually labeled data, thereby achieving semi-supervised training and saving a lot of training costs.
[0077] Step 203: training a first network model based on the expression image samples, the two-dimensional expression parameter samples and the three-dimensional expression parameter samples.
[0078] Exemplarily, the expression image sample may include an RGB three-channel image sequence, and each frame of the RGB three-channel image is superimposed with the heat map of the two-dimensional expression parameter sample to form four-channel data. The three-dimensional expression prediction parameters corresponding to the two-dimensional expression parameter samples are extracted using the first network model. The three-dimensional expression prediction parameters are essentially a set of three-dimensional coordinates of marked points (X0, Z0, Y0). The value of the three-dimensional coordinate of the marked point on the X-axis is denoted as X0, the value on the Y-axis is denoted as Y0, and the value on the Z-axis is denoted as Z0.
[0079] In practical applications, when the first network model outputs three-dimensional expression prediction parameters, the back propagation algorithm can be used to update the model parameters of the first network model until the first network model meets the training termination condition. The training termination condition can be the loss value of the first network model, the number of iterations of the first network model, or the convergence of the loss value.
[0080] After the first network model outputs the three-dimensional expression prediction parameters, the actual loss of the three-dimensional expression prediction parameters and the three-dimensional expression parameter samples can be calculated using the loss function of the first network model. When the actual loss is greater than or equal to the preset loss, the model parameters of the first network model can be updated using the gradient descent method, thereby completing one iteration of training. The training termination conditions of the exemplary embodiments of the present disclosure are various, and examples are given below.
[0081] Exemplarily, the training termination condition can be set as the actual loss amount is less than the preset loss amount. For example, it can be determined whether the actual loss amount is less than the preset loss amount. If the actual loss amount is less than the preset loss amount, it can be considered that the training of the first network model is completed. Otherwise, the back propagation algorithm can be used to update the model parameters in the first network model.
[0082] For example, after a large number of iterative training, the actual loss amount cannot be less than the preset loss amount. It can be set that when the actual loss amount remains constant after multiple iterations, or its change is within a controllable range, it is considered that the training termination condition is met.
[0083] Exemplarily, the actual number of iterations can be set to be greater than or equal to the maximum number of iterations as a training termination condition. When the number of iterative training is equal to the maximum number of iterations, it is considered that the first network model training is completed and the next iterative training will not be started.
[0084] Exemplarily, the loss function of the first network model of the exemplary embodiment of the present disclosure satisfies the regression constraint and the classification constraint, and the weight of the classification constraint is less than the weight of the regression constraint, such as the weight of the classification constraint is greater than 0 and less than 0.5. When the weight of the classification constraint is less than the weight of the regression constraint, the first network model uses this loss function as a supervisory signal to ensure that the trained first network model not only has good coordinate conversion accuracy, but also has good stability and robustness.
[0085] In practical applications, the loss function of the first network model includes a regression loss function and a classification loss function, and the weight of the classification loss function is less than the weight of the regression loss function, and the weight value of the classification constraint condition can be greater than 0 and less than 0.5, for example, 0.1. The regression loss function can be a mean square error loss function, and of course other regression loss functions can also be selected, such as mean absolute error, logarithmic hyperbolic pre-loss function, quantile loss loss function, etc. The classification loss function can be a KL divergence loss function (Kullback-Leibler Divergence, abbreviated as KL divergence), exponential loss function (exponential loss), cross-entropy loss function (Cross-entropy loss function), etc.
[0086] Taking the mean square error loss function and the KL divergence loss function as examples, the loss function of the first network model of the exemplary embodiment of the present disclosure can satisfy: TotalLoss = MSELoss + lambda*KLDivLoss, TotalLoss is expressed as the loss function of the first network model, MSELoss is expressed as the mean square error loss, which can measure the accuracy of the three-dimensional expression prediction parameters relative to the three-dimensional expression parameter samples, and KLDivLoss is expressed as the KL divergence, which can constrain the distance between the distributions of the three-dimensional expression prediction parameters and the three-dimensional expression parameter samples. Lambda is the weight of the classification constraint, which can be a value greater than 0 and less than 0.5, such as 0.1.
[0087] For example, for MSELoss, MSELoss(Q pi , Q gi )=(Q pi -Q gi ) 2 , Q pi It is represented as the three-dimensional expression prediction coordinates of the i-th expression marker point, Q gi It is represented as the three-dimensional expression coordinate sample of the i-th first expression marker point, MSELoss(Q pi , Q gi) is represented by the mean square error loss of the three-dimensional expression prediction coordinates of the i-th first expression marker point and the three-dimensional expression coordinate samples of the i-th first expression marker point; for the KLDivLoss divergence, it can be expressed as:
[0088]
[0089] Q p Expressed as three-dimensional expression prediction coordinates, Q g Represented as a three-dimensional expression coordinate sample, KLDivLoss(Q p , Q g ) represents the distance between the three-dimensional expression prediction coordinates and the three-dimensional expression coordinate sample distribution, also known as the KL divergence, and n represents the total number of the first expression landmarks.
[0090] Step 204: Based on the trained first network model, the target expression image of the target character having the second expression marker point and the two-dimensional expression parameters of the second expression marker point in the target expression image are processed to obtain the three-dimensional expression parameters of the second expression marker point.
[0091] In terms of the state of the target expression image, it can be a target expression picture, i.e., a static image, or a target expression video, i.e., a dynamic image; in terms of the source of the target expression image, it can be a target expression image of a virtual character, or a target expression image of a physical character; in terms of the acquisition of the target expression image, it can be an expression image acquired by various image acquisition devices for the face of the target character. These target characters can be not limited to humans, but also various organisms, such as cats, dogs, pigs, etc., but are not limited to these.
[0092] The second expression mark point on the face of the target character is used to mark the facial expression of the target character, which can be referred to Figure 3 The position of the second expression mark point on the target character's face may be the same as the position of the first expression mark point on the virtual character's face during the training phase, or there may be a difference.
[0093] For example, for a first network model with relatively low generalization ability, the position of the second expression marker on the target character's face can be the same as the position of the first expression marker on the virtual character's face during the training phase. For a first network model with relatively high generalization ability, the position of the second expression marker on the target character's face can be different from the position of the first expression marker on the virtual character's face during the training phase.
[0094] In practical applications, when the image acquisition device of the expression image sample used in the first network model training phase is constant, the image acquisition parameters of the target expression image match the image acquisition parameters of the expression image sample. At this time, the image content ratio and clarity of the target expression image input to the first network model match the image content ratio and clarity of the expression image sample of the virtual character, thereby ensuring that the first network model can output accurate three-dimensional expression parameters of the second expression marker point.
[0095] Exemplarily, the difference between the image acquisition parameters of the target expression image and the image acquisition parameters of the expression image sample is less than or equal to the preset difference. The difference here can be the absolute value of the difference. For example: when the image acquisition parameters include the position coordinates and focal length of the image acquisition device, the absolute value of the difference between the position coordinates of the expression image sample and the position coordinates of the target expression image can be less than the preset coordinate difference, and the absolute value of the difference between the focal length of the expression image sample and the focal length of the expression image sample can be less than the preset focal length. It should be understood that the preset coordinate difference and the preset focal length can be set according to actual conditions, and the smaller these differences are, the more accurate the three-dimensional expression parameters output by the first network model are. It should be understood that the image acquisition parameters of the target expression image can refer to the parameters of the image acquisition device that acquires the target expression image. The image acquisition device can include a physical camera such as a monocular camera, a binocular camera, or a digital virtual camera in a virtual world. In practical applications, the two-dimensional expression parameters of the second expression marker point are used to represent the two-dimensional expression parameters of the expression marker point of the target character, and the two-dimensional expression parameters can be obtained from the target expression image of the target character based on the second expression marker point.
[0096] Exemplarily, a second expression mark point can be marked on the actor's face. The second expression mark point can be a mark point drawn at a preset reference position on the actor's face, such as lips, eyebrows, cheeks, etc., by using a water-based pen, or a reflective ball set at a preset reference position on the actor's face to mark the muscle details during facial movement. A monocular infrared camera is then used to capture the actor's facial expressions to obtain a target expression image.
[0097] When the target expression image is captured by a virtual camera, the virtual character captured by the camera is a virtual character. For example, the user device can retrieve a digital human in the digital human database of the data storage system as a virtual character, bind facial expressions to the virtual character, draw a second expression marker on the face of the virtual character according to a preset reference, and configure the virtual camera to capture the target expression image of the virtual character.
[0098] When a monocular infrared camera or a virtual camera captures an expression of a certain state on the actor's face, the target expression image obtained is a static image, that is, a target expression picture; when a monocular infrared camera or a virtual camera captures the actor's facial expressions at different times, the target expression image obtained is a target expression video or a target expression video stream.
[0099] When the target expression image is a target expression video, the two-dimensional expression parameters may include the two-dimensional coordinates of the second expression punctuation point in multiple frames of expression images contained in the target expression video. Based on this, some embodiments of the present disclosure may filter out the initial frame of the expression image from the target expression video, and track the two-dimensional expression parameters of the second expression marker point in the target expression video based on the initial frame of the expression image to obtain the two-dimensional expression parameters.
[0100] In practical applications, the initial frame of the expression image may not be the first frame of the target expression image. Therefore, the image frame where the facial expression appears for the first time can be identified from the target expression video. The identification method can be a key point detection algorithm or other target detection algorithm. Then, the image frame is used as the initial frame of the expression image, and the image processing method, key point detection algorithm, optical flow tracking algorithm, etc. are used to track the two-dimensional expression parameters of the second expression marker point in multiple frames of expression images to obtain the two-dimensional expression parameters.
[0101] Exemplarily, the key point detection algorithm may include an ASM (Active Shape Model) key point detection algorithm, an AAM (Active Appearance Models) key point detection algorithm, and a DCNN (Deep Convolutional Network) key point detection algorithm, but is not limited thereto. The optical flow tracking algorithm may be a Lucas-Kanade optical flow algorithm (also known as an LK optical flow algorithm), a pyramid Lucas-Kanade optical flow method, and the like.
[0102] Figure 4A to Figure 4E FIG. 2 shows a schematic diagram of a marker tracking effect of an exemplary embodiment of the present disclosure. Figure 4A to Figure 4E As shown, the exemplary embodiment of the present disclosure can filter out five frames of expression images that can reflect the dynamic changes of expression from the target image video, and the five frames of expression images are respectively the first frame expression image 401, the second frame expression image 402, the third frame expression image 403, the fourth frame expression image 404 and the fifth frame expression image 405.
[0103] like Figure 4A to Figure 4EAs shown, the first frame of the expression image 401 is the initial frame of the expression image, the coordinates of the marking points are identified from the first frame of the expression image 401, and the coordinates of the marking points in the second frame of the expression image 402, the third frame of the expression image 403, the fourth frame of the expression image 404 and the fifth frame of the expression image 405 are tracked, thereby obtaining the coordinates of the marking points of the second frame of the expression image 402, the third frame of the expression image 403, the fourth frame of the expression image 404 and the fifth frame of the expression image 405. For example: Figure 4A to Figure 4E It can be seen from the five frames of expression images that the positions of the various marking points marking the lips change with the different degrees of opening of the lips. Based on this, the coordinates of the marking points on the remaining four frames of expression images can be directly tracked based on the coordinates of the marking points on the first frame of expression image 401, without extracting the corresponding coordinates of the marking points from each frame of expression image.
[0104] Exemplarily, when the two-dimensional expression parameters of the exemplary embodiment of the present disclosure include two-dimensional expression parameters of a multi-frame target expression video indicating expression motion records, the multi-frame target expression video can be a continuous frame or a discontinuous multi-frame target expression video, and can be adjusted according to the actual application scenario, without limitation here.
[0105] The first network model of the exemplary embodiment of the present disclosure includes a backbone network and a fully connected layer, wherein the backbone network is used to determine the deep expression features based on the target expression image and the two-dimensional expression parameters, and the fully connected layer is used to determine the three-dimensional expression parameters based on the deep expression features. The first network model can be used to determine the deep expression features based on the expression and two-dimensional expression parameters of the target expression image, and determine the three-dimensional expression parameters based on the deep expression features. It can be understood that the backbone network can understand various deep semantic information from the target expression image, such as depth of field, marker point coordinates, etc., and fuse them with the features of the two-dimensional expression parameters, so that the deep expression features can fuse the features of the second expression marker points in the two-dimensional coordinate system and the three-dimensional coordinate system. On this basis, the fully connected network can determine the three-dimensional expression parameters based on the deep expression features.
[0106] Figure 5 A schematic diagram of the architecture of a first network model of some embodiments of the present disclosure is shown, such as Figure 5 As shown, the first network model 500 in the exemplary embodiment of the present disclosure includes a backbone network 501 and a fully connected layer 502. It should be understood that the backbone network can be various possible end-to-end models, such as a stacked hourglass network model, a fully convolutional neural network model, a U-Net network model, etc. The stacked hourglass network model is used as an example below.
[0107] During the training phase, the backbone model 501 is used to determine deep expression prediction features based on expression image samples and two-dimensional expression parameter samples, and the fully connected layer 502 is used to determine three-dimensional expression prediction parameters based on the deep expression prediction features, and to update the model parameters based on the back propagation algorithm using the deep expression prediction features and three-dimensional expression parameter samples.
[0108] In the inference stage, the backbone network 501 can be used to determine the deep expression features based on the target expression image and the two-dimensional expression parameters, and the fully connected layer 502 is used to determine the three-dimensional expression parameters based on the deep expression features.
[0109] Taking the stacked hourglass network model as an example, the stacked hourglass network model can include at least two cascaded sub-hourglass networks. For a sub-hourglass network, the sub-hourglass network includes an encoder and a decoder. The encoder is used to generate multi-level expression coding features based on the target expression image and two-dimensional expression parameters, and the decoder is used to extract features from the multi-level expression coding features to obtain deep expression features. Since the decoder can fuse multi-level expression coding features, the obtained deep expression features will not have the problem of shallow feature loss, ensuring data integrity.
[0110] When at least two sub-hourglass networks are cascaded, the deep expression features output by the current sub-hourglass network are the input features of the next sub-hourglass network. As the network depth increases, different sub-hourglass networks respond to markers on different parts of the face to different degrees. Therefore, when the deep expression features output by the current sub-hourglass network are input to the next sub-hourglass network, the next sub-hourglass network can use the deep expression features output by the current sub-hourglass network as a reference to generate the deep expression features more accurately. It can be seen that when the backbone network of the first network model of the exemplary embodiment of the present disclosure adopts a stacked hourglass network model, the generation accuracy and robustness of three-dimensional expression parameters can be effectively improved.
[0111] Fig. 6A FIG. 2 shows a schematic diagram of the architecture of a stacked hourglass network model in some embodiments of the present disclosure. Fig. 6A As shown, the stacked hourglass network model 600 in the exemplary embodiment of the present disclosure includes a backbone network 601 and a fully connected layer 602, and the backbone network 601 includes N sub-hourglass networks 6011, where N is an integer greater than or equal to 2, for example, N=4. Figure 6B FIG. 1 shows a schematic diagram of the architecture of a sub-hourglass network according to an exemplary embodiment of the present disclosure. Figure 6B As shown, the sub-hourglass network model 6011 includes an encoder and a decoder. The encoder may contain multiple convolutional layers 6011a, and the decoder may contain multiple deconvolutional layers 6011b.
[0112] like Fig. 6A and Figure 6BAs shown, in the sub-hourglass network 6011, the encoder can generate four levels of expression coding features of different scales based on the three-channel RGB target expression image 603 and the one-channel two-dimensional expression marker point heat map 604. The decoder can not only generate deep expression features with the minimum scale expression coding features output by the encoder as input, but also integrate the four levels of expression coding features of different scales into the generation process of deep expression features during the decoding process, thereby ensuring the accuracy and completeness of the acquired deep expression features. On this basis, the fully connected layer 602 can extract features from the deep expression features to generate three-dimensional expression parameters 605 expressed in the form of a heat map.
[0113] Step 205: Convert the three-dimensional expression parameters into expression driving parameters based on the trained second network model. The second network model of the exemplary embodiment of the present disclosure is essentially a transformation network model, which is used to downsample the three-dimensional expression parameters to obtain expression driving parameters. It is understandable that the expression driving parameters corresponding to different digital human modeling software may be different.
[0114] Exemplarily, in MetaHumanCreator, its expression driving parameter can be RigValue, also known as the expression coefficient, which meets the requirements of the expression coefficient in MetaHuman Creator, that is, the expression driving parameter includes a set of 168-dimensional vectors, which can represent the parameters of bone movement, which is a set of 168×1 vectors. The value of each dimension represents the movement state (pulling and stretching) of each muscle of the face. At the same time, the bone parameters including the 3D coordinate system position, rotation and scale of the bone can be calculated. Based on this, when the transformation network downsamples the three-dimensional expression parameters, it ultimately needs to convert the three-dimensional expression parameters into a set of 168-dimensional vectors, and define the set of 168-dimensional vectors as the expression driving parameters.
[0115] Figure 7 FIG. 2 shows a schematic diagram of the architecture of a second network model of an exemplary embodiment of the present disclosure. Figure 7 As shown, the second network model 700 of the exemplary embodiment of the present disclosure may include K cascaded fully connected layers 701, and the number K of the fully connected layers 701 can be designed according to actual needs, which is an integer greater than or equal to 2. The three-dimensional expression parameters can be reduced in dimension using multiple fully connected layers until the dimension meets the requirements of the corresponding expression driving strategy, thereby obtaining the expression driving parameters.
[0116] Taking metahumancreator as an example, the second network model of the exemplary embodiment of the present disclosure may input a 3D coordinate vector of 3×n, where n is the total number of second expression markers, and output a 168×1 vector as an expression coefficient, where n is an integer greater than or equal to 1. Assuming that the number of second expression markers is 60, there are a total of 180 three-dimensional expression coordinates, and the transformation network model may be used to reduce the dimensionality of the three-dimensional expression coordinates to generate a set of 168×1 vectors as expression driving parameters, thereby using the expression driving parameters to drive the face to show corresponding expressions.
[0117] As can be seen from the above, in one or more technical solutions provided in the exemplary embodiments of the present disclosure, when the first network model constructs a training set in the training stage, after collecting the expression image samples of the virtual character with the first expression mark point in the engine, the two-dimensional expression parameter samples and the three-dimensional expression parameter samples of the first expression mark point of the virtual character can be automatically exported directly from the engine, thereby completing the construction of the training set of the first network model. It can be seen that when the first network model constructs the training set in the training stage, there is no need to manually mark the two-dimensional expression parameter samples and the three-dimensional expression parameter samples, so that the first network model can achieve semi-supervised training, thereby saving training costs. On this basis, the target expression image and the two-dimensional expression parameters of the second expression mark point in the target expression image are input to the trained first network model, and the three-dimensional expression parameters can be automatically obtained, and the trained second network model is used to convert the three-dimensional expression parameters into expression driving parameters, so as to quickly and accurately capture the facial expressions of the actors and output the face capture effect of film and television. Compared with the solution with higher accuracy than the solution without marker points, the factor of offset jitter is avoided.
[0118] In an optional manner, the training set of the second network model of the exemplary embodiment of the present disclosure in the training phase may include three-dimensional expression parameter samples and expression drive samples. The second network model can use the mean square error loss function as a supervision signal to supervise whether the second network model is trained. The training end condition can refer to the first network model, which is not described here. It should be understood that the first network model and the second network model of the exemplary embodiment of the present disclosure can be trained independently or jointly.
[0119] When the three-dimensional expression parameter sample is the three-dimensional coordinates of the first annotation point in the virtual character expression picture sample, the expression driving parameter sample can drive the virtual character to present the facial expression described by the virtual character expression picture sample. When the three-dimensional expression parameter sample is the three-dimensional coordinates of the first annotation point in the virtual character expression video sample, the expression driving parameter sample can drive the virtual character to present the facial expression changes described by the virtual character expression video sample. At this time, the three-dimensional expression parameter sample matches the expression driving real parameter.
[0120] In practical applications, the expression driving sample of the first expression mark point in the expression image sample is obtained from the engine, and then the second network model is trained based on the three-dimensional expression parameter sample and the expression driving parameter sample. For example, the vertex coordinates of the first annotation point in the Mesh and the corresponding real expression coefficient (such as RigValue) can be exported in metahuman.
[0121] Exemplarily, the expression driving parameter samples derived from the engine are substantially corrected by the image acquisition parameters to ensure that the obtained expression driving parameter samples and the three-dimensional expression parameter samples belong to the same three-dimensional space system.
[0122] For the training set of the second network model, the three-dimensional expression parameter samples can be the three-dimensional expression parameters output by the trained first network model, or can be the three-dimensional expression parameter samples of the first expression marker points in the expression image samples obtained from the engine. When the two-dimensional expression parameter samples and the three-dimensional expression parameter samples are exported from the engine, the expression drive samples can be exported together. It can be seen that the training set of the exemplary embodiment of the present disclosure can be automatically constructed with the help of the engine and image acquisition parameters, without the need for manual annotation, so that the second network model can achieve semi-supervised training, thereby saving training costs.
[0123] For example, an exemplary embodiment of the present disclosure inputs 6 facial expression images to the trained first network model, all of which have first marker points marked on the face, and the first marker points are set in different ways. Figure 8A to Figure 8F FIG. 1 is a schematic diagram of a visualization space of three-dimensional expression parameter samples of six characters according to an exemplary embodiment of the present disclosure. Figure 8A to Figure 8F As shown, the six three-dimensional expression parameter sample visualization images 800 are compared with the six facial expression images with marked points, and the position accuracy of the marked points is extremely high.
[0124] The above is an introduction to the solution provided by the embodiment of the present disclosure. It is understandable that, in order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.
[0125] The disclosed embodiment can divide the terminal into functional units according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the disclosed embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0126] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a device for determining expression driving parameters. Fig. 9 FIG. 1 is a schematic block diagram of a module of a device for determining expression drive parameters according to an exemplary embodiment of the present disclosure. Fig. 9 As shown, the expression driving parameter determination device 900 includes:
[0127] A construction module 901 is used to collect expression image samples of a virtual character having a first expression mark point in the engine, and obtain a two-dimensional expression parameter sample and a three-dimensional expression parameter sample of the first expression mark point in the expression image sample from the engine;
[0128] A training module 902, configured to train a first network model based on the expression image samples, the two-dimensional expression parameter samples, and the three-dimensional expression parameter samples;
[0129] The inference module 903 is used to process the target expression image of the target character with the second expression marker point and the two-dimensional expression parameters of the second expression marker point in the target expression image based on the trained first network model, obtain the three-dimensional expression parameters of the second expression marker point, and convert the three-dimensional expression parameters into expression driving parameters based on the trained second network model.
[0130] As a possible implementation manner, the target expression image is a target expression image of a virtual character or a target expression image of a real character.
[0131] As a possible implementation method, the target expression image is a target expression picture or a target expression video. When the target expression image is a target expression video, the construction module 901 is also used to filter out the initial frame of the expression image from the target expression video, and track the two-dimensional expression parameters of the second expression marker point in the target expression video based on the initial frame of the expression image to obtain the two-dimensional expression parameters.
[0132] As a possible implementation method, the first network model includes a backbone network and a fully connected layer. The backbone network is used to determine deep expression features based on the expression and two-dimensional expression parameters of the target expression image, and the fully connected layer is used to determine three-dimensional expression parameters based on the deep expression features.
[0133] As a possible implementation manner, image acquisition parameters of the target expression image are matched with image acquisition parameters of the expression image sample.
[0134] As a possible implementation method, the loss function of the first network model satisfies the regression constraint and the classification constraint, the weight of the classification constraint is less than the weight of the regression constraint, and the weight of the classification constraint is greater than 0 and less than 0.5.
[0135] As a possible implementation manner, the second network model is a transformation network model, which is used to downsample the three-dimensional expression parameters to obtain expression driving parameters.
[0136] As a possible implementation method, the construction module 901 is also used to obtain expression driving parameter samples of the first expression marker point in the expression image sample from the engine, and the training module 902 is also used to train the second network model based on the three-dimensional expression parameter samples and the expression driving samples.
[0137] Fig.10 Schematic block diagram of a chip of an exemplary embodiment of the present disclosure is shown. Fig.10 As shown, the chip 1000 includes one or more (including two) processors 1001 and a communication interface 1002. The communication interface 1002 can support the server to perform the data sending and receiving steps in the above method, and the processor 1001 can support the server to perform the data processing steps in the above method.
[0138] Optional, such as Fig.10 As shown, the chip 1000 also includes a memory 1003, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).
[0139] In some embodiments, Fig.10 As shown, the processor 1001 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 1001 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 1003 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1001. A portion of the memory 1003 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Fig.10 Various buses are labeled as bus system 1004 .
[0140] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.
[0141] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.
[0142] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.
[0143] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.
[0144] refer to Fig.11, a block diagram of an electronic device 1100 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0145] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.
[0146] A plurality of components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 may be any type of device capable of inputting information to the electronic device 1100, and the input unit 1106 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1107 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1104 may include, but is not limited to, a disk, an optical disk. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0147] like Fig.11As shown, the computing unit 1101 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the method of the exemplary embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. In some embodiments, the computing unit 1101 can be configured to execute the method in any other appropriate manner (e.g., by means of firmware).
[0148] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0149] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0150] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0152] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0153] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0154] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).
[0155] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A method for determining expression driving parameters, characterized in that: The method comprises: Collecting expression image samples of the virtual character having the first expression mark point in the engine; Acquire a two-dimensional expression parameter sample and a three-dimensional expression parameter sample of the first expression marker in the expression image sample from the engine; Training a first network model based on the expression image samples, the two-dimensional expression parameter samples, and the three-dimensional expression parameter samples; Processing a target expression image of a target character having a second expression marker point and two-dimensional expression parameters of the second expression marker point in the target expression image based on the trained first network model to obtain three-dimensional expression parameters of the second expression marker point; Converting the three-dimensional expression parameters into expression driving parameters based on the trained second network model; The first network model includes a backbone network and a fully connected layer, wherein the backbone network is used to determine the deep expression features based on the target expression image and the two-dimensional expression parameters, and the fully connected layer is used to determine the three-dimensional expression parameters based on the deep expression features, wherein the backbone network is used to determine the deep expression features based on the target expression image and the two-dimensional expression parameters, including: Determining the deep semantic information corresponding to the target expression image by using the backbone network, wherein the deep semantic information includes depth of field and coordinates of marker points; The depth of field, the coordinates of the marker point and the features of the two-dimensional expression parameter are integrated to obtain the deep expression feature, so that the deep expression feature integrates the features of the second expression marker point in the two-dimensional coordinate system and the three-dimensional coordinate system; The second network model is a transformation network model, and the transformation network model is used to downsample the three-dimensional expression parameters to obtain expression driving parameters; The loss function of the first network model satisfies the regression constraint and the classification constraint, the weight of the classification constraint is less than the weight of the regression constraint, and the weight of the classification constraint is greater than 0 and less than 0.
5.
2. The method according to claim 1, characterized in that The target expression image is a target expression image of a virtual character or a target expression image of a real character.
3. The method according to claim 1, characterized in that: The target expression image is a target expression picture or a target expression video. When the target expression image is a target expression video, the method further includes: Filtering out an initial frame of an expression image from the target expression video; Based on the initial frame of the expression image, two-dimensional expression parameters of the second expression marker point in the target expression video are tracked to obtain two-dimensional expression parameters.
4. The method according to claim 1, characterized in that: The image acquisition parameters of the target expression image match the image acquisition parameters of the expression image sample.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire, from the engine, an expression driving parameter sample of the first expression marker point in the expression image sample; A second network model is trained based on the three-dimensional expression parameter samples and the expression drive samples.
6. A device for determining expression drive parameters, characterized in that: The device comprises: A construction module is used to collect expression image samples of a virtual character having a first expression mark point in the engine; Acquire a two-dimensional expression parameter sample and a three-dimensional expression parameter sample of the first expression marker in the expression image sample from the engine; A training module, used for training a first network model based on the expression image samples, the two-dimensional expression parameter samples and the three-dimensional expression parameter samples; An inference module is used to process a target expression image of a target character having a second expression mark point and two-dimensional expression parameters of the second expression mark point in the target expression image based on the trained first network model to obtain three-dimensional expression parameters of the second expression mark point; and convert the three-dimensional expression parameters into expression driving parameters based on the trained second network model; The first network model includes a backbone network and a fully connected layer, wherein the backbone network is used to determine the deep expression features based on the target expression image and the two-dimensional expression parameters, and the fully connected layer is used to determine the three-dimensional expression parameters based on the deep expression features, wherein the backbone network is used to determine the deep expression features based on the target expression image and the two-dimensional expression parameters, including: Determining the deep semantic information corresponding to the target expression image by using the backbone network, wherein the deep semantic information includes depth of field and coordinates of marker points; The depth of field, the coordinates of the marker point and the features of the two-dimensional expression parameter are integrated to obtain the deep expression feature, so that the deep expression feature integrates the features of the second expression marker point in the two-dimensional coordinate system and the three-dimensional coordinate system; The second network model is a transformation network model, and the transformation network model is used to downsample the three-dimensional expression parameters to obtain expression driving parameters; The loss function of the first network model satisfies the regression constraint and the classification constraint, the weight of the classification constraint is less than the weight of the regression constraint, and the weight of the classification constraint is greater than 0 and less than 0.
5.
7. An electronic device, characterized in that: include: processor; as well as, A memory for storing programs; The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.
