Virtual object driving and model training method and device, equipment and medium

Abstract

The invention provides a virtual object driving and model training method and device, equipment and a medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC, digital humans, intelligent e-commerce and the like. The virtual object driving method comprises the following steps: performing feature extraction on target audio to obtain audio features; the target audio is used for sounding of a virtual object; decoding the audio features and the initial expression coefficient to obtain a target expression coefficient; and based on the target expression coefficient, driving the lip action of the virtual object.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, deep learning, large models and other technical fields, which can be applied to scenarios such as AIGC, digital humans, and smart e-commerce, and in particular to a virtual object driving and model training method, device, equipment and storage medium. Background Art

[0002] Digital human refers to a three-dimensional animation model that can be driven by animators. It can be applied to games, metaverse, virtual reality (VR), augmented reality (AR), virtual customer service, virtual anchors and other scenarios. Summary of the invention

[0003] The present disclosure provides a virtual object driving and model training method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, a virtual object driving method is provided, comprising: extracting features of target audio to obtain audio features; the target audio is used for the virtual object to make sounds; decoding the audio features and initial expression coefficients to obtain target expression coefficients; and driving the lip movements of the virtual object based on the target expression coefficients.

[0005] According to another aspect of the present disclosure, a virtual object driving model training method is provided, the model includes a decoder, and the method includes: extracting features from audio samples to obtain audio features; using the decoder to decode the audio features and initial expression coefficients to obtain predicted expression coefficients; constructing a loss function based on the predicted expression coefficients and real expression coefficients corresponding to the audio samples; and adjusting the parameters of the decoder based on the loss function.

[0006] According to another aspect of the present disclosure, a virtual object driving device is provided, including: an extraction module for performing feature extraction on a target audio to obtain an audio feature; the target audio is used for the virtual object to make a sound; a decoding module for decoding the audio feature and an initial expression coefficient to obtain a target expression coefficient; and a driving module for driving the lip movement of the virtual object based on the target expression coefficient.

[0007] According to another aspect of the present disclosure, a virtual object driving model training device is provided, wherein the model includes a decoder, and the device includes: an extraction module for extracting features from audio samples to obtain audio features; a decoding module for decoding the audio features and initial expression coefficients using the decoder to obtain predicted expression coefficients; a construction module for constructing a loss function based on the predicted expression coefficients and real expression coefficients corresponding to the audio samples; and an adjustment module for adjusting parameters of the decoder based on the loss function.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method as described in any of the above aspects.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the methods described in any one of the above aspects.

[0011] The present disclosure can improve the accuracy of virtual object lip driving.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram of an application scenario for implementing an embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram of a virtual object driving model provided according to an embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram of a decoding process of a single iteration round provided according to an embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a second embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a third embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0021] Figure 8 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0022] Fig. 9 It is a schematic diagram of an electronic device used to implement the virtual object driving method or the virtual object driving model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Taking the virtual object as a digital human as an example, the lip movements of the digital human can be driven by the expression coefficient.

[0025] The expression coefficient can specifically be a blendshape coefficient. Taking the ARKIT standard (an AR development framework) as an example, the Blendshape coefficient provided by the ARKIT standard is a 51-dimensional vector. The value of each dimension of the vector is used to represent the degree of change of an expression, and the corresponding expression is gradually enhanced from 0 to 1. Expressions include, for example, eye expressions, lip expressions, etc., and each expression can be further refined, such as there are 27 lip expressions. When in use, all or part of the expression coefficients can be used for driving, such as using lip expressions for lip driving.

[0026] In the related art, the Blendshape coefficient can be obtained through an image or audio. For example, an image is input into a pre-trained neural network model, and the output is the Blendshape coefficient corresponding to the image. Alternatively, an audio is input into a pre-trained neural network model, and the output is the Blendshape coefficient corresponding to the audio.

[0027] The above method directly converts the image or audio into the Blendshape coefficient, which results in that the accuracy of the Blendshape coefficient needs to be improved, and thus the lip driving effect based on the Blendshape coefficient also needs to be improved.

[0028] In order to improve the accuracy of virtual object lip driving, the present disclosure provides the following embodiments.

[0029] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure. This embodiment provides a virtual object driving method, the method comprising:

[0030] 101. Perform feature extraction on target audio to obtain audio features; the target audio is used for the virtual object to make sounds.

[0031] 102. Decode the audio features and the initial expression coefficients to obtain target expression coefficients.

[0032] 103. Drive the lip movement of the virtual object based on the target expression coefficient.

[0033] Among them, virtual objects refer to objects to be driven, such as digital humans.

[0034] Target audio refers to the audio that the virtual object is going to make.

[0035] The target audio is the given information. Taking the virtual object as a digital human as an example, the driving goal is to drive the digital human to speak the voice content of the target audio.

[0036] The expression coefficient is used to drive the lip movements of the virtual object to obtain a lip shape that matches the target audio. For example, the expression coefficient is a Blendshape coefficient. Taking the ARKIT standard as an example, the Blendshape coefficient is a 51-dimensional vector.

[0037] The initial expression coefficient is a preset value, for example, a 51-dimensional vector of all zeros is used as the initial expression coefficient.

[0038] The target expression coefficient is the final expression coefficient.

[0039] The target expression coefficient is also a 51-dimensional vector. The elements of each dimension are between 0 and 1, and the corresponding expression is gradually enhanced from 0 to 1. For example, if a certain dimension corresponds to the action of curling the upper lip outward, 0 means there is no such action, and 1 means the degree of this action is the deepest.

[0040] After obtaining the target expression coefficient, the target expression coefficient is used to drive the lip movement of the virtual object.

[0041] Since the elements of each dimension of the target expression coefficient represent a degree of expression, the virtual object can be driven to show the corresponding lip shape by combining elements of different dimensions.

[0042] In this embodiment, the target expression coefficient is obtained by decoding the audio features and the initial expression coefficient, and the lip movements of the virtual object are driven based on the target expression coefficient. Compared with the method of directly converting the image or audio into the expression coefficient, the accuracy of the expression coefficient can be improved, thereby improving the accuracy of the virtual object lip driving.

[0043] In order to better understand the embodiments of the present disclosure, application scenarios to which the embodiments of the present disclosure can be applied are described.

[0044] Figure 2 It is a schematic diagram of an application scenario for implementing the embodiment of the present disclosure. The scenario includes: a user terminal 201 and a server 202, and the user terminal 201 may include: a personal computer (PC), a mobile device (such as a mobile phone), a tablet computer, a laptop computer, a smart wearable device, etc. The server 202 may be a cloud server or a local server, and the user terminal 201 and the server 202 may communicate using a communication network, and the communication network may include, for example, a wired network and / or a wireless network.

[0045] Taking the digital human scenario as an example, an application (Application, APP) can be installed in the user terminal, and the APP can provide digital human services. For example, the digital human provides virtual anchor services to users through the APP. The server end of the virtual anchor service is deployed on the server.

[0046] Among them, a segment of audio can be given in advance as the target audio; in addition, an initial expression coefficient can be set in advance, such as a vector of all 0s. Then, the target expression coefficient is obtained according to the audio features of the target audio and the initial expression coefficient, and the lip movement of the virtual object is driven by the target expression coefficient.

[0047] The virtual object can simulate different target objects, and different target objects have different voice styles. For example, for the pronunciation of the same word, different speakers can have different lip shapes. For example, the target object is a speaker.

[0048] In order to meet personalized needs, in this embodiment, Figure 2 As shown, the style characteristics of the target object simulated by the virtual object can also be obtained; the target expression characteristics are obtained according to the audio characteristics, the initial expression coefficients and the style characteristics; and the lip movements of the virtual object are driven according to the target expression characteristics, so that the virtual object can be driven to speak the speech content in the target audio, and the lip shape of the virtual object conforms to the speaking style of the target object.

[0049] Lip driving can be implemented based on a virtual object driving model.

[0050] Figure 3 is a schematic diagram of a virtual object driving model provided according to an embodiment of the present disclosure.

[0051] like Figure 3 As shown, the virtual object driving model includes: a decoder 301 . In order to meet personalized needs, the model may also include: a style extractor 302 .

[0052] The input of the decoder 301 includes: audio features, initial expression coefficients, and in the scenario of meeting personalized needs, also includes: style features; the decoder decodes the above input information, and the output is the target expression feature.

[0053] The audio feature is obtained by extracting the audio feature of the target audio. For example, the target audio is input into an audio feature extraction network, and the output is the audio feature. The audio feature extraction network is, for example, a pre-trained wav2vec model, which is a neural network model that converts audio into features.

[0054] The initial expression coefficients are preset, such as being set to a 51-dimensional vector of all zeros.

[0055] The decoder may specifically adopt a Transformer Decoder structure.

[0056] The style extractor 302 is used to obtain the style features of the target object simulated by the virtual object.

[0057] In this embodiment, the style extractor may specifically obtain the style features according to the facial expression coefficients of the target object.

[0058] Based on this, the input of the style extractor is the facial expression coefficient of the target object, and the output is the style feature.

[0059] The facial expression coefficient can be obtained by extracting the facial expression of the target image of the target object.

[0060] Specifically, a speaking image of the speaker may be captured as a target image, and there may be one or more target images.

[0061] Taking multiple as an example, for each target image in the target video, each target image can be input into the facial capture network, and the output is the facial expression coefficient of the target image.

[0062] The facial capture network is a network that can obtain facial expression coefficients of a person in an image based on an image, and the network can be an existing pre-trained network.

[0063] Assuming that there are M (positive integer) target images, M groups of facial expression coefficients can be obtained. Then, the M groups of facial expression coefficients are input into the style extractor, and the output is the style features of the target object.

[0064] After obtaining the audio features, initial expression coefficients and style features, a decoder is used to decode this information to obtain the target expression coefficients.

[0065] The decoding can be performed in an iterative manner until the target expression coefficient is obtained.

[0066] Specifically, based on the target duration of the target audio and the frame rate of the target video corresponding to the virtual object, multiple iteration rounds are determined; for each of the multiple iteration rounds: the audio features and the historical expression coefficients before each iteration round are decoded to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient; the above process is repeated until the expression coefficient corresponding to the last iteration round of the multiple iteration rounds is obtained as the target expression coefficient.

[0067] For example, assuming that the target duration of the target audio is 10 seconds and the frame rate of the target video is 30 frames per second, that is, 30 images of virtual objects are contained per second, then the target audio will correspond to 10*30 images, and this number (10*30=300) will be used as the number of iteration rounds.

[0068] For the i-th (i=1,2...,300) iteration round, based on the audio features, style features and the historical expression coefficients obtained in the (i-1)-th iteration round, the expression coefficient of the i-th iteration number is obtained, and used as the historical expression coefficient of the next (i+1) iteration number. The process is repeated until the expression coefficient corresponding to the 300th iteration round is obtained as the target expression coefficient.

[0069] Specifically, taking the expression coefficient as the Blendshape coefficient as an example, when predicting the first frame of the image, the historical expression coefficient (Past Face Motions) is first used as the input of the decoder with all 51-dimensional Blendshape coefficients set to 0, and then combined with the audio features and style features to predict the Blendshape coefficient 1*51 of the first frame; then when predicting the second frame of the image, the 51-dimensional coefficients of all 0 and the 1*51 predicted in the previous frame are concatenated together to obtain 2*51 and input into the decoder. The decoder will then predict the coefficient 2*51 of the first two frames; iterate in sequence until the 300*51 corresponding to the target audio is predicted, and the final 300*51 is used as the target expression coefficient to drive the lip movements of the virtual object in each frame of the 300 frames of images.

[0070] Figure 4 It is a schematic diagram of a decoding process of a single iteration round provided according to an embodiment of the present disclosure.

[0071] like Figure 4 As shown, for the Tth (T=1, 2, ..., N)th iteration round, the Tth expression coefficient can be obtained through the historical expression coefficient of the (T-1)th iteration round. Wherein, N is determined according to the target duration and frame rate, for example, based on the above example, N=300.

[0072] In combination with the above application scenarios, the present disclosure also provides the following embodiments.

[0073] Figure 5 is a schematic diagram according to a second embodiment of the present disclosure. This embodiment provides a virtual object driving method, the method comprising:

[0074] 501. Perform feature extraction on target audio to obtain audio features; the target audio is used for the virtual object to make sounds.

[0075] 502. Acquire style features of a target object simulated by the virtual object; the style features are used to characterize the vocal style of the target object.

[0076] 503. Use a decoder to decode the input audio features, the initial expression coefficients, and the style features to output the target expression coefficients.

[0077] The decoder is pre-trained, and the training process of the decoder can refer to the relevant description of the subsequent embodiments.

[0078] 504. Drive lip movements of the virtual object based on the target expression coefficient.

[0079] In this embodiment, by acquiring the style features of the target object and obtaining the target expression coefficient according to the style features, personalized needs can be met.

[0080] Furthermore, facial expression extraction may be performed on the target image of the target object to obtain the facial expression coefficient of the target object; and a style extractor may be used to extract style features from the input facial expression coefficient to output the style features.

[0081] The style extractor is pre-trained, and the training process of the style extractor may refer to the relevant description of the subsequent embodiments.

[0082] In this embodiment, the style features are obtained based on the facial expression coefficients, so that the style features can be obtained efficiently and accurately, thereby improving the accuracy of lip driving.

[0083] Furthermore, multiple iteration rounds can be determined based on the target duration of the target audio and the frame rate of the target video of the virtual object; for each of the multiple iteration rounds: the audio features and the historical expression coefficients before each iteration round are decoded to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient; the above process is repeated until the expression coefficient corresponding to the last iteration round of the multiple iteration rounds is obtained as the target expression coefficient.

[0084] In this embodiment, the target expression coefficient is obtained through an iterative process, which can improve the accuracy of the target expression coefficient and further improve the accuracy of lip driving.

[0085] Figure 6 is a schematic diagram according to a third embodiment of the present disclosure. This embodiment provides a method for training a virtual object driving model, the model including a decoder, and the method including:

[0086] 601. Perform feature extraction on the audio sample to obtain audio features.

[0087] 602. Use the decoder to decode the audio features and initial expression coefficients to obtain predicted expression coefficients.

[0088] 603. Construct a loss function based on the predicted expression coefficient and the real expression coefficient corresponding to the audio sample.

[0089] 604. Adjust parameters of the decoder based on the loss function.

[0090] The audio in the training phase is called audio samples, which can be obtained from samples in existing public datasets.

[0091] Similar to the inference stage, an existing audio feature extraction network, such as a wav2vec model, can be used to perform feature extraction processing on the audio sample to obtain audio features. The initial expression coefficient is a preset value, such as a 51-dimensional vector of all zeros.

[0092] The audio features and initial expression coefficients are input into the decoder, and the output is the predicted expression coefficients.

[0093] After obtaining the predicted expression coefficient, a loss function is constructed based on the predicted expression coefficient and the true expression coefficient, such as a loss function indicating regression loss. The true expression coefficient may be manually annotated, or in combination with the subsequent description, may also be obtained based on a facial capture network.

[0094] Afterwards, the loss function is used to adjust the parameters of the decoder until a preset end condition is reached, and the end condition is, for example, reaching a preset number of times or model convergence.

[0095] The final virtual object driving model is used for the virtual object driving process in the reasoning stage.

[0096] In this embodiment, by decoding the audio features and the initial expression coefficients to obtain the predicted expression coefficients, constructing a loss function based on the predicted expression coefficients and the true expression coefficients, and using the loss function to adjust the model parameters of the decoder, the model accuracy can be improved, thereby improving the model driving effect.

[0097] In some embodiments, it also includes:

[0098] Obtain style features of target objects in image samples;

[0099] The method of using the decoder to decode the audio feature and the initial expression coefficient to obtain the predicted expression coefficient includes:

[0100] A decoder is used to decode the input audio features, the initial expression coefficients and the style features to output the predicted expression coefficients.

[0101] In this embodiment, by acquiring the style features of the target object and obtaining the predicted expression coefficient according to the style features, the personalized effect of the model can be improved to meet personalized needs.

[0102] In some embodiments, the model also includes: a style extractor; obtaining the style features of the target object in the image sample includes: performing facial expression extraction on the image sample to obtain the facial expression coefficient of the target object in the image sample; using the style extractor to extract style features from the input facial expression coefficient to output the style features.

[0103] In this embodiment, the style features are obtained based on the facial expression coefficients, so that the style features can be obtained efficiently and accurately, thereby improving the accuracy of the lip driving model.

[0104] In addition, the parameters of the style extractor can be adjusted according to the loss function to achieve synchronous training of the style extractor and decoder.

[0105] Specifically, for the data collection process: a video (video sample) and corresponding audio samples of multiple different actors reading some fixed texts at normal speaking speed are recorded by camera; the age, gender, etc. of these collected people are required to be diverse. After obtaining the video sample, a trained facial capture network is used to calculate the facial expression coefficient corresponding to the face in each frame of the above video sample, such as the Blendshape coefficient of the face of the ARKIT standard. In this way, many sets of training data pairs of audio and corresponding facial expression coefficients can be constructed. Afterwards, several sets of facial expression coefficients are extracted from the facial expression coefficients corresponding to each audio according to the principle of uniform distribution, and the extracted facial expression coefficients are used to calculate the style features. For example, there are N frames of image samples for a video of a certain actor, and N sets of facial expression coefficients are obtained through facial capture; the average value of the N sets of facial expression coefficients of different actors is calculated to obtain the N sets of facial expression coefficients after the mean calculation, and the N sets of facial expression coefficients are sorted; then, M sets of facial expression coefficients are uniformly sampled from the sorted N sets of facial expression coefficients for calculating the style features.

[0106] For the training process: multiple iteration rounds can be determined based on the target duration of the audio sample and the frame rate of the video sample corresponding to the audio sample; for each of the multiple iteration rounds: the audio features and the historical expression coefficients before each iteration round are decoded to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient; the above process is repeated until the expression coefficient corresponding to the last iteration round in the multiple iteration rounds is obtained as the predicted expression coefficient.

[0107] Specifically, taking the Transformer decoder as an example, the input of the Transformer decoder includes: audio features. The audio features use the wav2vec pre-trained model. This model receives a segment of speech wav (audio sample) input and outputs a corresponding audio feature. In addition, in order to introduce personalized information, the decoder input also includes style features. The style extractor is used to encode the M groups of facial expression coefficients extracted above to obtain style features. In addition, the input of the decoder also includes a preset initial expression coefficient, and the predicted expression coefficient is obtained through a continuous iterative process. For example, if the audio sample is 10 seconds and the frame rate of the video sample is 30 frames / second, a predicted expression coefficient with a dimension of 300*51 can be obtained.

[0108] In this embodiment, the predicted expression coefficient is obtained through an iterative process, which can improve the accuracy of the predicted expression coefficient and further improve the accuracy of the model.

[0109] After obtaining the predicted expression coefficient, a loss function is constructed based on the predicted expression coefficient and the true expression coefficient corresponding to the above audio sample.

[0110] The real expression coefficient can be obtained according to the facial capture network, that is, the real expression coefficient can be obtained according to the facial expression coefficient obtained by the facial capture network.

[0111] Specifically, if the video sample includes video samples of multiple target objects, the facial expression coefficients corresponding to the video samples of each target object are averaged to obtain the average of the facial expression coefficients as the true expression coefficients; or, if the video sample includes video samples of a single target object, the facial expression coefficients corresponding to the video samples of the single target object are used as the true expression coefficients.

[0112] For example, for the same audio sample, different actors can have different video samples. After processing the video samples through the facial capture network, N sets of facial expression coefficients corresponding to each actor are obtained. After calculating the mean of the N sets of facial expression coefficients of different actors, the averaged N sets of facial expression coefficients are used as the real expression coefficients. Based on the above example, N = 300, that is, the dimension of the real expression coefficient is also 300*51.

[0113] In this way, the real expression coefficient is obtained through the facial capture network without manual annotation, which can reduce the cost and improve the accuracy. In addition, the real expression coefficient is obtained based on the average of the facial expression coefficients corresponding to multiple video samples, which can improve the generalization of the model.

[0114] Figure 7 is a schematic diagram according to a fourth embodiment of the present disclosure. This embodiment provides a virtual object driving device, such as Figure 7 As shown, the device 700 includes: an extraction module 701, a decoding module 702 and a driving module 703.

[0115] The extraction module 701 is used to extract features from the target audio to obtain audio features; the target audio is used for the virtual object to make sounds; the decoding module 702 is used to decode the audio features and the initial expression coefficients to obtain the target expression coefficients; the driving module 703 is used to drive the lip movements of the virtual object based on the target expression coefficients.

[0116] In this embodiment, the target expression coefficient is obtained by decoding the audio features and the initial expression coefficient, and the lip movements of the virtual object are driven based on the target expression coefficient. Compared with the method of directly converting the image or audio into the expression coefficient, the accuracy of the expression coefficient can be improved, thereby improving the accuracy of the virtual object lip driving.

[0117] In some embodiments, the device 700 further includes: an acquisition module.

[0118] An acquisition module, used to acquire style features of a target object simulated by the virtual object; the style features are used to characterize the vocalization style of the target object;

[0119] The decoding module 702 is further used for:

[0120] A decoder is used to decode the input audio features, the initial expression coefficients and the style features to output the target expression coefficients.

[0121] In this embodiment, by acquiring the style features of the target object and obtaining the target expression coefficient according to the style features, personalized needs can be met.

[0122] In some embodiments, the acquisition module is further used to:

[0123] Performing facial expression extraction on the target image of the target object to obtain a facial expression coefficient of the target object;

[0124] A style extractor is used to extract style features from the input facial expression coefficients to output the style features.

[0125] In this embodiment, the style features are obtained based on the facial expression coefficients, so that the style features can be obtained efficiently and accurately, thereby improving the accuracy of lip driving.

[0126] In some embodiments, the decoding module 702 is further configured to:

[0127] Determining a plurality of iteration rounds based on a target duration of the target audio and a frame rate of a target video of the virtual object;

[0128] For each iteration round of the multiple iteration rounds: decoding the audio feature and the historical expression coefficients before each iteration round to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient;

[0129] The above process is repeated until the expression coefficient corresponding to the last iteration round in the multiple iteration rounds is obtained as the target expression coefficient.

[0130] In this embodiment, the target expression coefficient is obtained through an iterative process, which can improve the accuracy of the target expression coefficient and further improve the accuracy of lip driving.

[0131] Figure 8 is a schematic diagram according to the fifth embodiment of the present disclosure. This embodiment provides a virtual object driving model training device, the model includes a decoder, such as Figure 8 As shown, the device 800 includes: an extraction module 801, a decoding module 802, a construction module 803 and an adjustment module 804.

[0132] The extraction module 801 is used to extract features from audio samples to obtain audio features; the decoding module 802 is used to use the decoder to decode the audio features and initial expression coefficients to obtain predicted expression coefficients; the construction module 803 is used to construct a loss function based on the predicted expression coefficients and the real expression coefficients corresponding to the audio samples; the adjustment module 804 is used to adjust the parameters of the decoder based on the loss function.

[0133] In this embodiment, by decoding the audio features and the initial expression coefficients to obtain the predicted expression coefficients, constructing a loss function based on the predicted expression coefficients and the true expression coefficients, and using the loss function to adjust the model parameters of the decoder, the model accuracy can be improved, thereby improving the model driving effect.

[0134] In some embodiments, the apparatus 800 further includes:

[0135] An acquisition module, used to acquire style features of a target object in a video sample corresponding to the audio sample; the style features are used to characterize the vocal style of the target object;

[0136] The decoding module 802 is further used for:

[0137] A decoder is used to decode the input audio features, the initial expression coefficients and the style features to output the predicted expression coefficients.

[0138] In this embodiment, by acquiring the style features of the target object and obtaining the predicted expression coefficient according to the style features, the personalized effect of the model can be improved to meet personalized needs.

[0139] In some embodiments, the model further comprises: a style extractor;

[0140] The acquisition module is further used for:

[0141] Extracting facial expressions from the video sample to obtain facial expression coefficients of a target object in the video sample;

[0142] The style extractor is used to extract style features from the input facial expression coefficients to output the style features.

[0143] In this embodiment, the style features are obtained based on the facial expression coefficients, so that the style features can be obtained efficiently and accurately, thereby improving the accuracy of the lip driving model.

[0144] In some embodiments, the apparatus 800 further includes:

[0145] A determination module is used to, if the video sample includes video samples of multiple target objects, calculate the mean of the facial expression coefficients corresponding to the video samples of each target object to obtain the mean of the facial expression coefficients as the true expression coefficient; or, if the video sample includes video samples of a single target object, use the facial expression coefficients corresponding to the video samples of the single target object as the true expression coefficient.

[0146] In this way, the real expression coefficient is obtained through the facial capture network without manual annotation, which can reduce the cost and improve the accuracy. In addition, the real expression coefficient is obtained based on the average of the facial expression coefficients corresponding to multiple video samples, which can improve the generalization of the model.

[0147] In some embodiments, the decoding module 802 is further configured to:

[0148] Determining a plurality of iteration rounds based on a target duration of the audio sample and a frame rate of a video sample corresponding to the audio sample;

[0149] For each iteration round of the multiple iteration rounds: decoding the audio feature and the historical expression coefficients before each iteration round to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient;

[0150] The above process is repeated until the expression coefficient corresponding to the last iteration round in the multiple iteration rounds is obtained as the predicted expression coefficient.

[0151] In this embodiment, the predicted expression coefficient is obtained through an iterative process, which can improve the accuracy of the predicted expression coefficient and further improve the accuracy of the model.

[0152] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0153] It can be understood that the “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0154] It is understandable that, unless otherwise specified, the sequence of steps in the process indicates that the timing relationship between these steps is not limited.

[0155] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0156] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0157] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0158] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0159] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0160] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a virtual object driving method or a virtual object driving model training method. For example, in some embodiments, the virtual object driving method or the virtual object driving model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the virtual object driving method or the virtual object driving model training method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the virtual object driving method or the virtual object driving model training method in any other appropriate manner (for example, by means of firmware).

[0161] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0163] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0165] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0166] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0167] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A virtual object driving method, comprising: Perform feature extraction on the target audio to obtain audio features; The target audio is used for the virtual object to make sounds; Decoding the audio features and the initial expression coefficients to obtain target expression coefficients; Based on the target expression coefficient, the lip movement of the virtual object is driven.

2. The method according to claim 1, further comprising: Acquire style features of a target object simulated by the virtual object; The style feature is used to characterize the vocal style of the target object; The decoding of the audio feature and the initial expression coefficient to obtain the target expression coefficient includes: A decoder is used to decode the input audio features, the initial expression coefficients and the style features to output the target expression coefficients.

3. The method according to claim 2, wherein: The step of obtaining the style feature of the target object simulated by the virtual object includes: Performing facial expression extraction on the target image of the target object to obtain a facial expression coefficient of the target object; A style extractor is used to extract style features from the input facial expression coefficients to output the style features.

4. The method according to claim 1, wherein: The decoding of the audio feature and the initial expression coefficient to obtain the target expression coefficient includes: Determining a plurality of iteration rounds based on a target duration of the target audio and a frame rate of a target video of the virtual object; For each iteration round of the multiple iteration rounds: decoding the audio feature and the historical expression coefficients before each iteration round to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient; The above process is repeated until the expression coefficient corresponding to the last iteration round in the multiple iteration rounds is obtained as the target expression coefficient.

5. A method for training a virtual object driving model, the model comprising a decoder, the method comprising: Perform feature extraction on the audio sample to obtain audio features; Using the decoder, decoding the audio features and initial expression coefficients to obtain predicted expression coefficients; Constructing a loss function based on the predicted expression coefficient and the true expression coefficient corresponding to the audio sample; Based on the loss function, parameters of the decoder are adjusted.

6. The method according to claim 5, further comprising: Acquire style features of a target object in a video sample corresponding to the audio sample; The style feature is used to characterize the vocal style of the target object; The method of using the decoder to decode the audio feature and the initial expression coefficient to obtain the predicted expression coefficient includes: A decoder is used to decode the input audio features, the initial expression coefficients and the style features to output the predicted expression coefficients.

7. The method according to claim 6, wherein: The model also includes: a style extractor; The step of obtaining the style feature of the target object in the video sample corresponding to the audio sample includes: Extracting facial expressions from the video sample to obtain facial expression coefficients of a target object in the video sample; The style extractor is used to extract style features from the input facial expression coefficients to output the style features.

8. The method according to claim 7, further comprising: If the video sample includes video samples of multiple target objects, the facial expression coefficients corresponding to the video samples of each target object are averaged to obtain the average of the facial expression coefficients as the true expression coefficients; or, If the video sample includes a video sample of a single target object, the facial expression coefficient corresponding to the video sample of the single target object is used as the true expression coefficient.

9. The method according to claim 5, wherein: The decoding of the audio feature and the initial expression coefficient to obtain the target expression coefficient includes: Determining a plurality of iteration rounds based on a target duration of the audio sample and a frame rate of a video sample corresponding to the audio sample; For each iteration round of the multiple iteration rounds: decoding the audio feature and the historical expression coefficients before each iteration round to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient; The above process is repeated until the expression coefficient corresponding to the last iteration round in the multiple iteration rounds is obtained as the predicted expression coefficient.

10. A virtual object driving device, comprising: An extraction module, used for extracting features from the target audio to obtain audio features; The target audio is used for the virtual object to make sounds; A decoding module, used for decoding the audio features and the initial expression coefficients to obtain target expression coefficients; A driving module is used to drive the lip movement of the virtual object based on the target expression coefficient.

11. The apparatus according to claim 10, further comprising: An acquisition module, used to acquire style features of a target object simulated by the virtual object; the style features are used to characterize the vocalization style of the target object; The decoding module is further used for: A decoder is used to decode the input audio features, the initial expression coefficients and the style features to output the target expression coefficients.

12. The device according to claim 11, wherein The acquisition module is further used for: Performing facial expression extraction on the target image of the target object to obtain a facial expression coefficient of the target object; A style extractor is used to extract style features from the input facial expression coefficients to output the style features.

13. The device according to claim 10, wherein: The decoding module is further used for: Determining a plurality of iteration rounds based on a target duration of the target audio and a frame rate of a target video of the virtual object; For each iteration round of the multiple iteration rounds: decoding the audio feature and the historical expression coefficients before each iteration round to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient; The above process is repeated until the expression coefficient corresponding to the last iteration round in the multiple iteration rounds is obtained as the target expression coefficient.

14. A virtual object driving model training device, the model comprising a decoder, the device comprising: An extraction module, used for extracting features from audio samples to obtain audio features; A decoding module, used to decode the audio features and initial expression coefficients using the decoder to obtain predicted expression coefficients; A construction module, used to construct a loss function based on the predicted expression coefficient and the real expression coefficient corresponding to the audio sample; An adjustment module is used to adjust the parameters of the decoder based on the loss function.

15. The apparatus according to claim 14, further comprising: An acquisition module, used to acquire style features of a target object in a video sample corresponding to the audio sample; the style features are used to characterize the vocal style of the target object; The decoding module is further used for: A decoder is used to decode the input audio features, the initial expression coefficients and the style features to output the predicted expression coefficients.

16. The device according to claim 15, wherein: The model also includes: a style extractor; The acquisition module is further used for: Extracting facial expressions from the video sample to obtain facial expression coefficients of a target object in the video sample; The style extractor is used to extract style features from the input facial expression coefficients to output the style features.

17. The apparatus according to claim 16, further comprising: A determination module, configured to, if the video sample includes video samples of multiple target objects, calculate the mean of the facial expression coefficients corresponding to the video samples of each target object to obtain the mean of the facial expression coefficients as the true expression coefficient; Alternatively, if the video sample includes a video sample of a single target object, the facial expression coefficient corresponding to the video sample of the single target object is used as the true expression coefficient.

18. The device according to claim 14, wherein: The decoding module is further used for: Determining a plurality of iteration rounds based on a target duration of the audio sample and a frame rate of a video sample corresponding to the audio sample; For each iteration round of the multiple iteration rounds: decoding the audio feature and the historical expression coefficients before each iteration round to obtain the expression coefficient corresponding to each iteration round; the initial value of the historical expression coefficient is the initial expression coefficient; The above process is repeated until the expression coefficient corresponding to the last iteration round in the multiple iteration rounds is obtained as the predicted expression coefficient.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images