Mouth shape driving method and apparatus, and training method and apparatus for mouth shape driving model
By using the lip-type driving model of multi-layer extraction layer and output layer in the process of generating lip-type animations in the audio driver, the inter-frame lip-type jitter problem is solved, and a smoother and more stable lip-type animation is generated, avoiding sliding windows Delay caused by enlargement.
Patent Information
- Application Number
- PCT/CN2024/094189
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-05-20
- Publication Date
- 2024-11-28
AI Technical Summary
In the prior art, when using audio drivers to generate lip animations, inter-frame lip jitter problems are prone to occur, and when increasing the sliding window to alleviate this problem, the effect is limited and the lip delay will be introduced.
The lip-shaped driving model is adopted, including a first extraction layer, a second extraction layer and a first output layer. By performing multi-layer feature extraction and processing of audio features, smooth lip-shaped driving parameters are generated to reduce lip-shaped jitter. The specific steps include: extracting the target feature subsequence from the audio feature sequence, performing multiple feature extraction and sliding window processing, and generating lip drive parameters for generating lip animation.
Without adding sliding windows, more adjacent audio feature information is extracted to generate smoother lip animations, reducing the jitter problem in lip animations, and improving the real-time and stability of lip animations.
Smart Images

Figure CN2024094189_28112024_PF_FP_ABST
Abstract
Description
A lip-activated and lip-activated model training method and device
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on and claims priority to an application filed in China with application number 2023105725834, filed on May 19, 2023. The disclosed content of the application is hereby incorporated into this application as a whole. Technical Field
[0003] The present disclosure relates to the field of lip-activated technology, and in particular to a lip-activated method and device, and a lip-activated model training method and device. Background Art
[0004] Lip-sync animation plays a crucial role in new forms of human-computer interaction, including film, gaming, and virtual reality. When watching animated models, viewers are often sensitive to subtle changes in the model's lip movements. Therefore, generating realistic and natural lip-sync animation is a challenging task in computer graphics today.
[0005] Summary of the Invention
[0006] One or more embodiments of the present disclosure provide a lip-activated method and apparatus, and a lip-activated model training method and apparatus, so as to extract lip-activated parameters that help smooth the lip shape and reduce the problem of lip jitter.
[0007] According to a first aspect, a lip-activated method is provided, comprising:
[0008] Sequentially taking each audio feature in an audio feature sequence as a target audio feature, and determining a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed;
[0009] Performing a first feature extraction on the target feature subsequence to obtain an audio intermediate feature;
[0010] The audio intermediate features corresponding to the respective audio features are sequentially used as target intermediate features, and an intermediate feature subsequence corresponding to the target intermediate features is determined from the respective audio intermediate features;
[0011] Performing a second feature extraction based on the intermediate feature subsequence to determine a corresponding audio smoothing feature;
[0012] The audio smoothing features corresponding to each audio intermediate feature are processed to determine a target lip-sync driving parameter sequence corresponding to the audio feature sequence for generating lip-sync animation.
[0013] According to a second aspect, a method for training a lip-activated model is provided, wherein the lip-activated model includes a first extraction layer, a second extraction layer, and a first output layer; the method includes:
[0014] determining current parameters of the first abstraction layer;
[0015] Sequentially taking each sample audio feature in a sample audio feature sequence as a first feature, and determining a first feature subsequence corresponding to the first feature from the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio;
[0016] Performing first feature extraction on the first feature subsequence using current parameters of the first extraction layer to obtain sample intermediate features;
[0017] Sequentially taking the sample intermediate features corresponding to the audio features of each sample as the second feature, and determining a second feature subsequence corresponding to the second feature from the intermediate features of each sample;
[0018] Using the second extraction layer, based on the second feature subsequence, a second feature extraction is performed to determine a corresponding sample smoothing feature;
[0019] Using the first output layer, processing the sample smoothing features corresponding to the intermediate features of each sample to determine a lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence;
[0020] Parameters of the second extraction layer and the first output layer are adjusted based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.
[0021] According to a third aspect, there is provided a lip-actuating device comprising:
[0022] A first determination module is configured to sequentially use each audio feature in an audio feature sequence as a target audio feature and determine a target feature subsequence corresponding to the target audio feature from the audio feature sequence, where the audio feature sequence is a feature sequence obtained by filtering the audio to be processed;
[0023] A first feature extraction module is configured to perform first feature extraction on the target feature subsequence to obtain an audio intermediate feature;
[0024] A second determination module is configured to sequentially use the audio intermediate features corresponding to the respective audio features as target intermediate features, and determine an intermediate feature subsequence corresponding to the target intermediate features from the respective audio intermediate features;
[0025] A second feature extraction module is configured to perform a second feature extraction based on the intermediate feature subsequence to determine a corresponding audio smoothness feature;
[0026] The first processing module is configured to process the audio smoothness features corresponding to the respective audio intermediate features, and determine a lip-sync driving parameter sequence corresponding to the audio feature sequence for generating lip-sync animation.
[0027] According to a fourth aspect, a training device for a lip-activated model is provided, wherein the lip-activated model includes a first extraction layer, a second extraction layer, and a first output layer; the device includes:
[0028] a third determining module, configured to determine current parameters of the first extraction layer;
[0029] The fourth determining module is configured to sequentially use each sample audio feature in the sample audio feature sequence as a first feature, and determine a first feature subsequence corresponding to the first feature from the sample audio feature sequence, where the sample audio feature sequence is a feature sequence obtained by filtering the sample audio;
[0030] a third feature extraction module configured to perform first feature extraction on the first feature subsequence using current parameters of the first extraction layer to obtain a sample intermediate feature;
[0031] a fifth determining module configured to sequentially use the sample intermediate features corresponding to the respective sample audio features as second features, and determine, from the respective sample intermediate features, a second feature subsequence corresponding to the second feature;
[0032] a fourth feature extraction module configured to utilize the second extraction layer and perform second feature extraction based on the second feature subsequence to determine a corresponding sample smoothing feature;
[0033] a second processing module configured to process, using the first output layer, sample smoothing features corresponding to the intermediate features of each sample, and determine a lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence;
[0034] The first adjustment module is configured to adjust parameters of the second extraction layer and the first output layer based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.
[0035] According to a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect or the second aspect.
[0036] According to a sixth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.
[0037] According to the lip-sync driving and lip-sync driving model training method and device provided by the embodiments of the present disclosure, in the lip-sync driving method, each audio feature in the audio feature sequence is sequentially used as the target audio feature, and a target feature subsequence corresponding to the target audio feature is determined from the audio feature sequence, where the audio feature sequence is a feature sequence obtained by filtering the audio to be processed; a first feature extraction is performed on the target feature subsequence to obtain an audio intermediate feature; the audio intermediate feature corresponding to each audio feature is sequentially used as the target intermediate feature, and an intermediate feature subsequence corresponding to the target intermediate feature is determined from each audio intermediate feature; a second feature extraction is performed based on the intermediate feature subsequence to determine the corresponding audio smoothing feature; the audio smoothing feature corresponding to each audio intermediate feature is processed to determine a lip-sync driving parameter sequence corresponding to the audio feature sequence for generating a lip-sync animation. In the above process, after the first feature extraction, the audio intermediate features corresponding to the audio features obtained are subjected to sliding windows to obtain the corresponding intermediate feature subsequence, and then the second feature extraction is performed on the intermediate feature subsequence, thereby achieving the effect of performing secondary feature extraction on the features obtained after feature extraction. Without increasing the sliding window, audio smoothing features containing more information about adjacent audio features can be obtained. By using this type of feature to generate lip-shaped driving parameters, the lip-shaped animation corresponding to the lip-shaped driving parameters can be made smoother, thereby reducing the problem of lip-shaped jitter in the lip-shaped animation. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.
[0039] FIG1A is a schematic flow chart of a method for training a lip-activated model according to an embodiment;
[0040] FIG1B is a schematic diagram of the data processing process of the lip-activated model provided in an embodiment;
[0041] FIG2A is a schematic diagram showing a sample intermediate feature;
[0042] FIG2B is a schematic diagram showing a second characteristic subsequence;
[0043] FIG2C is a schematic diagram of a feature subsequence to be processed obtained after the dimensionality adjustment operation is performed on the intermediate features of each sample in the second feature subsequence shown in FIG2B ;
[0044] FIG3 is a schematic flow chart of a lip-activated method according to an embodiment;
[0045] FIG4 is a schematic block diagram of a lip-actuating device according to an embodiment;
[0046] FIG5 is a schematic block diagram of a training device for a lip-activated model according to an embodiment;
[0047] FIG6 is a schematic block diagram of an electronic device provided by an embodiment. DETAILED DESCRIPTION
[0048] The technical solutions of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0049] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0050] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.
[0051] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.
[0052] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0053] Among the methods for generating lip-sync animation, there are a number of audio-driven approaches. These methods generally take an audio clip as input and generate a lip-sync animation synchronized with the audio. However, when using audio-driven lip-sync animation, inter-frame lip-sync jitter often occurs.
[0054] Therefore, an improved solution needs to be provided to alleviate the problem of inter-frame lip-syncing jitter in the process of audio-driven lip-syncing animation generation.
[0055] One or more embodiments of the present disclosure provide a lip-activated method and apparatus, and a lip-activated model training method and apparatus, so as to extract lip-activated parameters that help smooth the lip shape and reduce the problem of lip jitter.
[0056] As mentioned above, when using audio to drive lip animation, the problem of lip jitter between image frames (also known as animation frames) often occurs. Currently, the process of using audio to drive lip animation generally involves: obtaining the audio to be processed, filtering the audio to be processed, and obtaining the corresponding audio feature sequence to be processed; using a sliding window method, for each audio feature to be processed in the audio feature sequence to be processed, determining the corresponding feature subsequence; then, using the feature subsequence corresponding to each audio feature to be processed, determining the lip drive parameter sequence corresponding to the audio feature sequence to be processed for generating the lip animation; and then determining the corresponding lip image frame sequence (i.e., lip animation, which can also be referred to as a lip animation frame sequence) based on the lip drive parameter sequence.
[0057] In the above process, to reduce the problem of lip-syncing jitter between image frames, the sliding window is generally increased (i.e., the sliding window length is increased). In theory, increasing the sliding window can alleviate the problem of lip-syncing jitter between image frames. However, when the sliding window is increased to a certain extent, the effect of alleviating the problem of lip-syncing jitter between image frames is minimal and it may cause a significant lip-syncing delay.
[0058] In order to better reduce the problem of lip jitter while ensuring real-time lip-forming drive, the embodiments of the present disclosure disclose a lip-forming drive method and device, and a lip-forming drive model training method and device. The lip-forming drive method and device, and the lip-forming drive model training method and device provided by the present disclosure are described in detail below in conjunction with specific embodiments.
[0059] It is understandable that the lip-activated process provided by the embodiments of the present disclosure may be implemented based on a pre-trained lip-activated model. For clarity, the training process of the lip-activated model will be first introduced below.
[0060] During the training process of the lip-sync driven model, as shown in FIG1A , a flow chart illustrating a method for training a lip-sync driven model in some embodiments of the present disclosure is provided. As shown in FIG1B , the lip-sync driven model may include a first extraction layer, a second extraction layer, and a first output layer. This method may be implemented using a target program. The target program may be installed in an electronic device, which may be implemented using any device, equipment, platform, or device cluster with computing and processing capabilities.
[0061] It should be noted that, in the process of audio-driven lip-syncing, the lip-syncing drive parameters for generating lip-syncing drive animation determined based on the audio, that is, the output of the lip-syncing drive model, can be in the form of a mesh, in the form of a blendshape weight value set, or in the form of a viseme. In the embodiment of the present disclosure, corresponding to the different forms of lip-syncing drive parameters, the first output layer of the lip-syncing drive model can be a classification layer or a regression layer. When the first output layer is a classification layer (referred to as the first classification layer), the form of the lip-syncing drive parameters it outputs can be in the form of a viseme; when the first output layer is the first regression layer, the form of the lip-syncing drive parameters it outputs can be in the form of a mesh or a blendshape weight value set.
[0062] When the first output layer of the lip-driven model is the first classification layer or the first regression layer, the training process (and the process of determining the lip-driven parameters) is similar, but the label data required for training are different. When the first output layer of the lip-driven model is the first classification layer, its label data includes lip-driven parameter labels in the form of viseme; when the first output layer of the lip-driven model is the first regression layer, its label data includes lip-driven parameter labels in the form of mesh, or its label data includes lip-driven parameter labels in the form of a blendshape weight value set.
[0063] The following describes the training process of the lip-activated model. As shown in FIG1A , the training method of the lip-activated model may include the following steps S110 to S170:
[0064] In step S110 , current parameters of the first extraction layer are determined.
[0065] In one implementation, the target program can read the current parameters of the first extraction layer from a preset storage space of the electronic device in which it resides, i.e., read the values of the current parameters of the first extraction layer, and then execute the subsequent training process of the lip-activated model based on the current parameters of the first extraction layer. The first extraction layer can be a pre-trained extraction layer.
[0066] In another implementation, the target program can determine the current parameters of the first extraction layer through training. Accordingly, the training process of the lip-activated model provided by the embodiment of the present disclosure can be divided into two stages, namely, the first stage, the training stage of the first extraction layer, and the second stage, based on the trained first extraction layer, the training stage of the second extraction layer and the first output layer, wherein, during the training process of the second extraction layer and the first output layer, the current parameters of the first extraction layer (i.e., the values of the current parameters) can be fixed. It can be understood that the first extraction layer can be implemented based on any neural network layer in the relevant technology that can extract audio features.
[0067] The training phase of the first extraction layer (i.e., the first phase) is introduced below. Specifically, the target program can first obtain the first extraction layer to be trained and its corresponding output layer (which can be a classification layer or a regression layer). For the sake of clarity, the output layer corresponding to the first extraction layer is called the second output layer. Among them, when the first output layer of the lip-driven model is a classification layer (called the first classification layer), when training the first extraction layer, the second output layer corresponding to the first extraction layer is a classification layer, called the second classification layer. When the first output layer of the lip-driven model is a regression layer (called the first regression layer), when training the first extraction layer, the second output layer corresponding to the first extraction layer is a regression layer, called the second regression layer.
[0068] Next, the target program obtains the training data required for training the first extraction layer and the second output layer, wherein the training data includes a training audio feature sequence corresponding to the audio data (referred to as training audio) and its corresponding label data, and the label data includes a lip-type driving parameter sequence corresponding to the training audio feature sequence. For the sake of clarity, the lip-type driving parameter sequence corresponding to the training audio feature sequence is referred to as the first lip-type driving parameter sequence.
[0069] In this case, the first extraction layer corresponds to the second classification layer, that is, when the second output layer is the second classification layer, the first lip-shaping parameter sequence exists in the form of a viseme, comprising a sequence of phonemes. It is understood that each first lip-shaping parameter in the first lip-shaping parameter sequence has a one-to-one correspondence with each training audio feature in the training audio feature sequence.
[0070] Accordingly, each phoneme in the phoneme sequence has a one-to-one correspondence with each training audio feature. Based on each phoneme in the phoneme sequence, the corresponding lip animation frame can be determined (i.e., the lip animation frame corresponding to each training audio feature is determined), and then the corresponding lip animation is generated. Alternatively, based on the phoneme sequence, the weight value set of the blendshape corresponding to each phoneme can be determined, and then based on the weight value set of the blendshape corresponding to each phoneme, the corresponding mouth movement, i.e., the lip shape, of each phoneme can be obtained, and then the corresponding lip animation can be generated.
[0071] The first extraction layer corresponds to the second regression layer, that is, when the second output layer is the second regression layer, in one case, the first lip-shaped driving parameter sequence can exist in the form of a mesh, and each first lip-shaped driving parameter in the first lip-shaped driving parameter sequence is a position information set corresponding to a set of preset lip-shaped key points (or a set of preset facial key points including the preset lip-shaped key points). Based on the position information set, the corresponding mouth movement, i.e., the lip shape (or the facial expression movement including the mouth movement) can be obtained. Accordingly, the corresponding lip-shaped animation can be obtained based on the first lip-shaped driving parameter sequence.
[0072] In another case where the first extraction layer corresponds to the second regression layer, the first lip-shape driving parameter sequence can exist in the form of a blendshape weight value set, that is, the first lip-shape driving parameter sequence is a series of weight value set sequences. The training audio feature sequence includes a plurality of training audio features, wherein each training audio feature can correspond to a set of weight value sets, each of which is a blendshape weight value set. For each training audio feature, by combining the blendshape weight value set corresponding to the training audio feature, at least the mouth movement, i.e., the lip shape, can be obtained, and then the corresponding lip shape animation frame can be generated. Accordingly, by combining the blendshape weight value set corresponding to each training audio feature in the training audio feature sequence, the mouth movement, i.e., the lip shape, corresponding to each training audio feature can be obtained, and then the corresponding lip shape animation can be generated.
[0073] In some embodiments, the training audio feature sequence corresponding to the training audio may be obtained by filtering the training audio, specifically, the MFCC feature sequence corresponding to the training audio obtained after filtering the training audio using a preset Mel-frequency cepstral coefficient MFCC filtering algorithm.
[0074] After obtaining the training audio feature sequence corresponding to the training audio, the target program sequentially uses each training audio feature in the training audio feature sequence as a feature to be processed (referred to as the third feature), and determines a third feature subsequence corresponding to the third feature from the training audio feature sequence. In one implementation, the target program may use a sliding window method to determine a third feature subsequence corresponding to the third feature from the training audio feature sequence. The third feature subsequence may include the third feature, a third number m of training audio features before the third feature in the training audio feature sequence, and a fourth number n of training audio features after the third feature. The third number m and the fourth number n are values pre-set according to actual needs.
[0075] In one implementation, when the number of training audio features preceding the third feature in the training audio feature sequence is less than a third number m, a preset filling rule can be used to fill in m. Specifically, the first training audio feature in the training audio feature sequence can be used for filling in. Similarly, when the number of training audio features following the third feature in the training audio feature sequence is less than a fourth number n, a preset filling rule can be used to fill in n. Specifically, the last training audio feature in the training audio feature sequence can be used for filling in.
[0076] After obtaining the third feature subsequence corresponding to the third feature, the target program inputs the third feature subsequence into the first extraction layer, and uses the first extraction layer to perform first feature extraction on the third feature subsequence to obtain the training intermediate features corresponding to the third feature (i.e., the third feature subsequence). Next, the training intermediate features are input into the second output layer, and the second output layer is used to process the training intermediate features to obtain the current lip-activated parameters corresponding to the third feature (i.e., the third feature subsequence). Accordingly, the second output layer can be used to obtain the current lip-activated parameter sequence corresponding to the training audio feature sequence based on the training intermediate features corresponding to each training audio feature in the training audio feature sequence.
[0077] Afterwards, the target program can adjust the parameters of the first extraction layer and the second output layer based on the current lip-driven parameter sequence and the first lip-driven parameter sequence corresponding to the training audio feature sequence. Specifically, the target program can use a preset loss function to determine the corresponding prediction loss (referred to as the first prediction loss) based on the current lip-driven parameter sequence and the first lip-driven parameter sequence corresponding to the training audio feature sequence, and then adjust the parameters of the first extraction layer and the second output layer with the goal of minimizing the first prediction loss until the first extraction layer and the second output layer reach the first convergence condition, thereby obtaining the current parameters of the first extraction layer. The first convergence condition may include but is not limited to: the number of parameter adjustments is not less than the first adjustment threshold, or the calculated first prediction loss is lower than the first loss threshold.
[0078] In one case, when the first extraction layer corresponds to the second classification layer (i.e., the second output layer is the second classification layer), the preset loss function may be a cross entropy loss function. In another case, when the first extraction layer corresponds to the second regression layer (i.e., the second output layer is the second regression layer), the preset loss function may be an MSE (mean square error) loss function.
[0079] Let's review the aforementioned training process for the first extraction layer and the second output layer. The above embodiment is explained by taking a training audio feature sequence corresponding to a training audio as an example. In another embodiment, the above training process can also be performed on a batch of samples, that is, a training audio feature sequence corresponding to multiple training audios, to obtain the current lip-shaped drive parameter sequence corresponding to the training audio feature sequence corresponding to each training audio, and then based on the current lip-shaped drive parameter sequence corresponding to the training audio feature sequence corresponding to each training audio, and the first lip-shaped drive parameter sequence corresponding to each training audio feature sequence, the first prediction loss is jointly determined, and then the first extraction layer and the second output layer are trained with the goal of minimizing the first prediction loss. In this embodiment, the first prediction loss is determined for a batch of samples, and then the parameters of the first extraction layer and the second output layer are adjusted. This can reduce the number of parameter adjustments for the first extraction layer and the second output layer, making it easier to implement the training process.
[0080] After determining the current parameters of the first extraction layer, in step S120, each sample audio feature in the sample audio feature sequence is taken as the first feature in turn, and a first feature subsequence corresponding to the first feature is determined from the sample audio feature sequence. The sample audio feature sequence is a feature sequence obtained by filtering the sample audio.
[0081] Among them, the target program can first obtain the data of the second extraction layer and the first output layer for training the lip-activated model, which is called sample data. The sample data includes a sample audio feature sequence obtained by filtering the sample audio, and a lip-activated label parameter sequence corresponding to the sample audio feature sequence.
[0082] In one implementation, the sample audio feature sequence may be an audio feature sequence obtained by filtering the sample audio using a preset Mel-Frequency Cepstral Coefficient (MFCC) filtering algorithm. The sample audio feature sequence includes a number of sample audio features sorted in time sequence, and the lip-syncing label parameter sequence includes a number of lip-syncing label parameters corresponding to the sample audio features.
[0083] In one implementation, the sample data used to train the second extraction layer and the first output layer may be the same as or different from the training data used to train the first extraction layer and the second output layer.
[0084] In one case, when the first output layer of the lip-activated model is the first classification layer, the lip-activated label parameter sequence exists in a viseme form, including a string of phoneme sequences.
[0085] In another case, when the first output layer of the lip-shaped drive model is the first regression layer, the lip-shaped drive label parameter sequence can exist in the form of a mesh (for example, a sequence of lip-shaped key point position sets), wherein each sample audio feature included in the sample audio feature sequence can correspond to a set of lip-shaped key point position sets respectively; or the lip-shaped drive label parameter sequence exists in the form of a blendshape weight value set (a sequence of weight value sets), wherein each sample audio feature included in the sample audio feature sequence corresponds to a set of blendshape weight value sets respectively.
[0086] In one implementation, in step S120, a sliding window approach may be employed to sequentially use each sample audio feature in the sample audio feature sequence as a first feature, and determine a first feature subsequence corresponding to the first feature from the sample audio feature sequence. Specifically, a sliding window (referred to as the first sliding window) may be pre-set and slid across the sample audio feature sequence to determine a corresponding feature subsequence from the sample audio feature sequence for each sample audio feature in the sample audio feature sequence.
[0087] Among them, the length of the first sliding window can be set according to actual needs. As shown in Figure 1B, for the first feature A (for example, sample audio feature A), the first feature subsequence corresponding to the first feature A is determined from the sample audio feature sequence, wherein the first feature subsequence corresponding to the first feature A can include the first feature A, the third number m of sample audio features before the first feature A in the sample audio feature sequence, and the fourth number n of sample audio features after the first feature A. In one case, the length of the first sliding window can be set to 23, and accordingly, the third number m can be set to 15, and the fourth number n can be set to 7.
[0088] In one implementation, when the number of sample audio features preceding the first feature in the sample audio feature sequence is less than a third number m, a preset filling rule may be used to fill in the m number. The filling may be performed using the first sample audio feature in the sample audio feature sequence. Similarly, when the number of sample audio features following the first feature in the sample audio feature sequence is less than a fourth number n, a preset filling rule may be used to fill in the n number. The filling may be performed using the last sample audio feature in the sample audio feature sequence.
[0089] After obtaining the first feature subsequence corresponding to the first feature (i.e., a sample audio feature), in step S130, the current parameters of the first extraction layer are used to perform first feature extraction on the first feature subsequence to obtain the sample intermediate feature. In this step, the target program inputs the first feature subsequence into the first extraction layer obtained through the aforementioned training, performs first feature extraction on the first feature subsequence using the current parameters of the first extraction layer, and obtains the sample intermediate feature corresponding to the first feature subsequence (i.e., the first feature).
[0090] The target program uses the current parameters of the first extraction layer to obtain the corresponding sample intermediate features for each sample audio feature, wherein each sample intermediate feature can be sorted according to the sorting order of its corresponding sample audio features, or it can be considered that each sample intermediate feature is sorted according to the output time order of the corresponding first extraction layer. As shown in Figure 1B, the sample intermediate feature corresponding to the first feature A (i.e., sample audio feature A) can be expressed as the feature output at time t of the first extraction layer. Correspondingly, the sample intermediate feature corresponding to the sample audio feature B before the first feature A in the sample audio feature sequence can be expressed as the feature output at time t-1 of the first extraction layer; the sample intermediate feature corresponding to the sample audio feature C after the first feature A in the sample audio feature sequence can be expressed as the feature output at time t+1 of the first extraction layer, and so on.
[0091] After the target program obtains the sample intermediate features corresponding to each sample audio feature (or obtains the sample intermediate features corresponding to at least n sample audio features after the current sample audio feature as the first feature), it then takes the sample intermediate features corresponding to each sample audio feature as the second feature in turn in step S140, and determines the second feature subsequence corresponding to the second feature from each sample intermediate feature.
[0092] In some embodiments, the target program may sequentially use the sample intermediate features corresponding to each sample audio feature as the second feature, and use a sliding window (e.g., a second sliding window) to determine a second feature subsequence corresponding to the second feature from each sample intermediate feature, wherein the second feature subsequence may include the second feature and a first number of sample intermediate features before and a second number of sample intermediate features after the second feature, i.e., the length of the second sliding window (i.e., the number of windows) is equal to the sum of the first number, the second number, and 1. Specifically, the process of determining the second feature subsequence corresponding to the second feature can refer to the aforementioned process of determining the first feature subsequence corresponding to the first feature, and will not be repeated here.
[0093] In one case, to ensure the real-time performance of lip-sync driving, the value of the second number should not be too large, for example, not exceeding a preset value, for example, the second number can be 1 or 2. In one implementation, the first number can be smaller than the aforementioned third number m, and the second number can be smaller than the aforementioned fourth number n, so as to better ensure the real-time performance of driving while improving inter-frame jitter.
[0094] Then, in step S150, a second feature extraction is performed based on the second feature subsequence using the second extraction layer to determine the corresponding sample smoothness feature. In this step, the second feature subsequence is input into the second extraction layer, and the second feature extraction is performed based on the second feature subsequence using the parameters of the second extraction layer to determine the corresponding sample smoothness feature.
[0095] In some embodiments, the second extraction layer can be a fully convolutional network to perform second feature extraction on the second feature subsequence. It can be implemented by performing feature extraction again on the basis of the sample intermediate features extracted by the first extraction layer (fused with the information of the audio features of each sample in the corresponding first feature subsequence), so that the obtained sample smoothing features are fused with the information of each sample intermediate feature in the second feature subsequence (that is, each sample intermediate feature corresponds to each sample audio feature in the first feature subsequence). Compared with the sample intermediate features corresponding to each sample smoothing feature, each sample smoothing feature fuses the information of a larger number of sample audio features. By using this type of feature to generate lip-type drive parameters, the lip-type animation corresponding to the lip-type drive parameters can be made smoother, so as to reduce the problem of lip-type jitter in the lip-type animation. Accordingly, in one case, the second extraction layer can also be called a smoother, which can alleviate the problem of lip-type jitter in lip-type drive.
[0096] In some embodiments, the second feature subsequence can be directly input into the second extraction layer, and the second feature extraction is performed on the second feature subsequence using the parameters of the second extraction layer to determine the sample smoothing feature corresponding to the second feature subsequence (ie, the second feature).
[0097] In some further embodiments, each sample intermediate feature output by the first extraction layer may include multiple dimensional sub-features (as shown in FIG2A , sample intermediate feature 1 may include dimensional sub-feature 1 corresponding to feature dimension 1, dimensional sub-feature 2 corresponding to feature dimension 2, ... dimensional sub-feature N corresponding to feature dimension N).
[0098] The target program uses a sliding window approach to obtain a second feature subsequence corresponding to the sample intermediate feature as the second feature. The second feature subsequence includes the number of sample intermediate features of the sliding window (i.e., the aforementioned second sliding window). A sample intermediate feature can be called a window feature, and a sample intermediate feature includes multiple dimensional sub-features, i.e., a window feature includes multiple dimensional sub-features. Accordingly, the dimensional representation of the second feature subsequence can be (batchsize, number of windows, feature dimensions), where batchsize represents the batch size of the training data for the lip-activated model, the number of windows represents the height of the second feature subsequence, and the number of feature dimensions represents the width of the second feature subsequence.
[0099] As shown in Figure 2B, this is a schematic diagram of a representation of the second feature subsequence. As shown in Figure 2B, the number of windows, that is, the height of the second feature subsequence, is 3, and the number of feature dimensions, that is, the width of the second feature subsequence, is 3. Among them, one square represents one dimension sub-feature, and one row of dimension sub-features represents one sample intermediate feature, that is, one window feature.
[0100] Considering that the second extraction layer can be a fully convolutional network, the dimension of its input data is represented as (batchsize, channel, height, width), and considering that the number of feature dimensions is represented as the width of the second feature subsequence, when using the second extraction layer to directly perform feature fusion extraction (convolution processing) on the second feature subsequence, the sub-features of each dimension of the sample intermediate feature may be split and fused, which will lead to incomplete fusion of the sub-features of each dimension of the sample intermediate feature (window feature).
[0101] Specifically, based on the second feature subsequence as shown in Figure 2B, if the size of the convolution kernel included in the second extraction layer is 2*2, accordingly, when the second feature subsequence is convolved (feature fusion extraction) using the 2*2 convolution kernel, for the sample intermediate feature a shown in the first row, the 2*2 convolution kernel can only simultaneously convolve the two dimensional sub-features a1 and a2 of the sample intermediate feature a (and the two dimensional sub-features b1 and b2 of the sample intermediate feature b), while the other dimensional sub-feature a3 (and b3 of the sample intermediate feature b) is separated and cannot be completely fused.
[0102] In order to better and more completely fuse the multiple dimensional sub-features included in the intermediate features of each sample, that is, to better and more completely fuse the window features, in step S150, the target program can perform a dimensionality adjustment operation (such as a reshape operation) on the intermediate features of each sample in the second feature subsequence to obtain a feature subsequence including the intermediate features of each sample after dimensionality adjustment, which can be called a feature subsequence to be processed. The dimensionality adjustment operation can set the dimensional sub-features contained in the intermediate features of each sample (window feature) as the input of different channels of the second feature extraction layer. The dimensional representation of the intermediate features of each sample after dimensionality adjustment can be (batchsize, feature dimension, number of windows, 1). As shown in Figure 2C, it is a schematic diagram of the feature subsequence to be processed obtained after the dimensionality adjustment operation is performed on the intermediate features of each sample in the second feature subsequence as shown in Figure 2B.
[0103] After the dimensionality adjustment operation is performed on the intermediate features of each sample in the second feature subsequence, all dimensional sub-features corresponding to each sample intermediate feature (i.e., each window feature) serve as inputs to different channels of the second extraction layer. In one implementation, the second extraction layer may include several convolutional layers, and the all dimensional sub-features corresponding to each sample intermediate feature (i.e., each window feature) mentioned here as inputs to different channels of the second extraction layer may refer to the inputs to different channels of the first convolutional layer of the second extraction layer.
[0104] Then in step S150, the target program can then use the second extraction layer to perform a second feature extraction on the feature subsequence to be processed, and obtain the sample smoothing feature corresponding to the second feature. For example, when using the convolution kernel in the first convolution layer of the second extraction layer (for example, a convolution kernel of size 2*1) to perform convolution on any sample intermediate feature (the sample intermediate feature a as shown in Figure 2C) in the feature subsequence to be processed, first use the 2*1 convolution kernel corresponding to each channel to perform convolution on the sample intermediate feature a on each channel to obtain the convolution result of each channel, and then perform weighted summation on the convolution result of each channel to obtain the final convolution result corresponding to the sample intermediate feature a. Among them, the final convolution result obtained includes all the dimensional sub-features contained in the sample intermediate feature a, and includes all the dimensional sub-features contained in its adjacent sample intermediate feature b, which can achieve complete convolution of different sample intermediate features and obtain a final convolution result containing more comprehensive information. Correspondingly, the sample smoothness feature corresponding to the obtained second feature subsequence (i.e., the second feature) contains more comprehensive sample audio feature information. This type of feature is used to generate lip-sync driving parameters, which can make the lip-sync animation corresponding to the lip-sync driving parameters smoother and better reduce the problem of lip jitter in the lip-sync animation.
[0105] Then, in step S160, the first output layer is used to process the sample smoothing features corresponding to the intermediate features of each sample to determine the lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence.
[0106] In this step, the sample smoothing features corresponding to each sample intermediate feature are input to the first output layer. The parameters of the first output layer are used to process the sample smoothing features corresponding to each sample intermediate feature to determine a lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence. In one embodiment, the first output layer can be implemented as a fully connected layer.
[0107] Wherein, the first output layer can be a classification layer or a regression layer. Specifically, when the first output layer is the first classification layer, the lip-shaped drive prediction parameter sequence can exist in the form of viseme, which is a string of phoneme sequences. When the first output layer is the first regression layer, the lip-shaped drive prediction parameter sequence can exist in the form of mesh (for example, a string of lip-shaped key point position set sequences), wherein each sample audio feature included in the sample audio feature sequence corresponds to a set of lip-shaped key point position sets; or the lip-shaped drive prediction parameter sequence can exist in the form of a blendshape weight value set (a string of weight value set sequences), wherein each sample audio feature corresponds to a set of weight value sets, and the set of weight value sets includes several blendshape weight values.
[0108] After determining that the sample audio feature sequence corresponds to the lip-actuated prediction parameter sequence, in step S170 , the parameters of the second extraction layer and the first output layer are adjusted based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.
[0109] Specifically, a preset loss function is used to determine the corresponding prediction loss (called the current prediction loss) based on the lip-driven prediction parameter sequence and the lip-driven label parameter sequence corresponding to the sample audio feature sequence. Then, with the goal of minimizing the current prediction loss, the parameters of the second extraction layer and the first output layer are adjusted until the second extraction layer and the first output layer reach the second convergence condition, thereby obtaining the trained second extraction layer and first output layer.
[0110] The second convergence condition may include, but is not limited to, the number of parameter adjustments being no less than a second adjustment threshold, or the calculated current predicted loss being less than a second loss threshold. The second adjustment threshold may be the same as or different from the aforementioned first adjustment threshold, and the second loss threshold may be the same as or different from the aforementioned first loss threshold.
[0111] In one case, when the first output layer is a first classification layer, the preset loss function may be a cross entropy loss function. In another case, when the first output layer is a first regression layer, the preset loss function may be an MSE (mean square error) loss function.
[0112] Let's review the execution process of the aforementioned steps S110-S170. The above embodiment is explained by taking the sample audio feature sequence corresponding to a sample audio as an example. In another embodiment, the above training process can also be performed on a batch of samples, that is, the sample audio feature sequence corresponding to multiple sample audios, to obtain the lip-shaped drive prediction parameter sequence corresponding to the sample audio feature sequence corresponding to each sample audio, and then based on the lip-shaped drive prediction parameter sequence corresponding to the sample audio feature sequence corresponding to each sample audio, and the lip-shaped drive label parameter sequence corresponding to the sample audio feature sequence, the current prediction loss is determined, and the second extraction layer and the first output layer are trained with the goal of minimizing the current prediction loss. In this embodiment, the current prediction loss is determined for a batch of samples, and then the parameters of the second extraction layer and the first output layer are adjusted. This can reduce the number of parameter adjustments for the second extraction layer and the first output layer, making it easier to implement the training process.
[0113] In this embodiment, a new lip-sync drive model structure is provided. Through its second extraction layer, the second feature subsequence corresponding to the sample intermediate features output by the first extraction layer is subjected to feature extraction and fusion again, so that sample smoothing features that fuse information of a larger number of sample audio features can be extracted without increasing the sliding window. By using such features to generate lip-sync drive parameters, the lip-sync animation corresponding to the lip-sync drive parameters can be made smoother, providing a basis for reducing the problem of lip jitter in lip-sync animation.
[0114] After the lip-activated model is trained, it can be used to perform lip-activated animation (i.e., generate lip animation). FIG3 shows a flow chart of the lip-activated animation method in some embodiments of the present disclosure. The method can be implemented by a target program. The target program can be installed in an electronic device, which can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. As shown in FIG3, the method includes the following steps:
[0115] In step S310, each audio feature in the audio feature sequence is sequentially used as a target audio feature, and a target feature subsequence corresponding to the target audio feature is determined from the audio feature sequence. The audio feature sequence is the feature sequence obtained by filtering the audio to be processed. The audio to be processed can be any audio data used as input for lip-syncing (i.e., for generating lip-sync animation).
[0116] Among them, the process of determining the target feature subsequence corresponding to the target audio feature from the audio feature sequence can refer to the aforementioned process of determining the third feature subsequence corresponding to the third feature from the training audio feature sequence, or the process of determining the first feature subsequence corresponding to the first feature from the sample audio feature sequence, and will not be repeated here.
[0117] Next, in step S320, a first feature extraction is performed on the target feature subsequence to obtain audio intermediate features. Specifically, the first extraction layer obtained through the aforementioned training can be used to perform first feature extraction on the target feature subsequence to obtain audio intermediate features. The process of obtaining audio intermediate features can be referred to the process of obtaining sample intermediate features (or obtaining training intermediate features) described above and will not be repeated here.
[0118] Then, in step S330, the audio intermediate features corresponding to the respective audio features are sequentially used as target intermediate features, and an intermediate feature subsequence corresponding to the target intermediate features is determined from the respective audio intermediate features.
[0119] In one implementation, the target program sequentially uses the audio intermediate features corresponding to each audio feature as the target intermediate feature, and can use a sliding window method to determine an intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature, where the intermediate feature subsequence includes the target intermediate feature, a first number of audio intermediate features preceding it, and a second number of audio intermediate features following it. The process of determining the intermediate feature subsequence corresponding to the target intermediate feature can be referred to the aforementioned process of determining the second feature subsequence corresponding to the second feature, and will not be repeated here.
[0120] Next, in step S340, a second feature extraction is performed based on the intermediate feature subsequence to determine the corresponding audio smoothness feature. Specifically, the second extraction layer obtained by the aforementioned training can be used to perform a second feature extraction based on the intermediate feature subsequence to determine the corresponding audio smoothness feature.
[0121] In some embodiments, in order to ensure that the audio smoothing feature includes more comprehensive and rich information, in step S340, the target program can perform a dimension adjustment operation on each audio intermediate feature in the intermediate feature subsequence to obtain a dimension adjustment feature subsequence including each audio intermediate feature after dimension adjustment, wherein the dimension adjustment operation is used to set each dimensional sub-feature contained in each audio intermediate feature as an input of different channels; then, a second feature extraction is performed on the dimension adjustment feature subsequence to obtain the audio smoothing feature corresponding to the target intermediate feature, so as to realize setting each dimensional sub-feature contained in each audio intermediate feature as an input of different channels of the first convolution layer of the second extraction layer, so as to facilitate the fusion of all dimensional sub-features of different audio intermediate features, so as to better alleviate the problem of lip jitter in the subsequent lip driving process.
[0122] The process of performing the dimension adjustment operation on each audio intermediate feature in the intermediate feature subsequence can refer to the aforementioned process of performing the dimension adjustment operation on each sample intermediate feature in the second feature subsequence, and will not be repeated here.
[0123] Then, in step S350, the audio smoothing features corresponding to each audio intermediate feature are processed to determine a target lip-sync driving parameter sequence corresponding to the audio feature sequence used to generate the lip-sync animation. Specifically, the aforementioned first output layer is used to process the audio smoothing features corresponding to each audio intermediate feature to determine a target lip-sync driving parameter sequence corresponding to the audio feature sequence used to generate the lip-sync animation. Wherein, when the first output layer is a first classification layer, the target lip-sync driving parameter sequence is a predicted phoneme sequence; when the first output layer is a first regression layer, the target lip-sync driving parameter sequence is a predicted weight value set sequence or a predicted lip key point position set sequence.
[0124] The process of determining the target lip-activated parameter sequence corresponding to the audio feature sequence used to generate the lip animation can refer to the aforementioned process of determining the lip-activated prediction parameter sequence corresponding to the sample audio feature sequence, which will not be repeated here.
[0125] After obtaining the target lip-activated parameter sequence, a corresponding lip animation can be generated based thereon, and then the lip animation can be played. Specifically, the target program can generate a corresponding lip animation frame based on each target lip-activated parameter in the target lip-activated parameter sequence, and then play the lip animation frame, thereby achieving real-time playback of the lip animation according to the audio feature sequence.
[0126] In this embodiment, a sliding window is performed on the audio intermediate features corresponding to each audio feature obtained after the first feature extraction to obtain a corresponding intermediate feature subsequence, and then a second feature extraction is performed on the intermediate feature subsequence, thereby achieving the effect of performing secondary feature extraction on the features obtained after the feature extraction, and obtaining an audio smoothing feature containing information of more audio features. Without increasing the sliding window, an audio smoothing feature containing information of more adjacent audio features can be obtained. By using this type of feature to generate lip-shaped drive parameters, the lip-shaped animation corresponding to the lip-shaped drive parameters can be made smoother, thereby reducing the problem of lip-shaped jitter in the lip-shaped animation.
[0127] In addition, a dimensionality adjustment operation is performed on each audio intermediate feature in the intermediate feature sub-sequence, and each dimensional sub-feature contained in each audio intermediate feature is set as the input of different channels of the first convolutional layer of the second extraction layer, which facilitates the fusion of all dimensional sub-features of different audio intermediate features and can better alleviate the problem of lip jitter during the subsequent lip-shaping driving process.
[0128] The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily need to be performed in the specific order shown or in a sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0129] Corresponding to the above method embodiment, the present disclosure provides a lip-actuating device 400, a schematic block diagram of which is shown in FIG4 , including:
[0130] A first determination module 410 is configured to sequentially use each audio feature in an audio feature sequence as a target audio feature and determine a target feature subsequence corresponding to the target audio feature from the audio feature sequence, where the audio feature sequence is a feature sequence obtained by filtering the audio to be processed;
[0131] A first feature extraction module 420 is configured to perform first feature extraction on the target feature subsequence to obtain an audio intermediate feature;
[0132] The second determining module 430 is configured to sequentially use the audio intermediate features corresponding to the respective audio features as target intermediate features, and determine an intermediate feature subsequence corresponding to the target intermediate features from the respective audio intermediate features;
[0133] A second feature extraction module 440 is configured to perform a second feature extraction based on the intermediate feature subsequence to determine a corresponding audio smoothness feature;
[0134] The first processing module 450 is configured to process the audio smoothness features corresponding to the respective audio intermediate features, and determine a lip-sync driving parameter sequence corresponding to the audio feature sequence for generating lip-sync animation.
[0135] In an optional implementation, the second feature extraction module 440 includes:
[0136] an adjustment unit (not shown in the figure), configured to perform a dimensionality adjustment operation on each audio intermediate feature in the intermediate feature subsequence to obtain a dimensionality-adjusted feature subsequence including each audio intermediate feature after dimensionality adjustment, wherein the dimensionality adjustment operation is used to set each dimensional sub-feature included in each audio intermediate feature as an input of a different channel;
[0137] A feature extraction unit (not shown in the figure) is configured to perform the second feature extraction on the dimension adjustment feature subsequence to obtain an audio smoothing feature corresponding to the target intermediate feature.
[0138] In an optional implementation, the feature extraction unit is specifically configured to perform the second feature extraction on the dimensionally adjusted feature subsequence using a fully convolutional network.
[0139] In an optional embodiment, the second determination module 430 is specifically configured to use a sliding window method to determine an intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature, wherein the intermediate feature subsequence includes the target intermediate feature and a first number of audio intermediate features before and after the target intermediate feature.
[0140] In an optional embodiment, the invention further comprises:
[0141] A first acquisition module (not shown in the figure), configured to acquire audio to be processed;
[0142] The filtering module (not shown in the figure) is configured to filter the audio to be processed using a preset Mel-Frequency Cepstral Coefficient (MFCC) filtering algorithm to obtain the audio feature sequence.
[0143] In an optional implementation, the target lip-actuating parameter sequence is a predicted phoneme sequence, a predicted weight value set sequence, or a predicted lip-actuating key point position set sequence.
[0144] Corresponding to the above method embodiment, the present disclosure provides a training device 500 for a lip-activated model. The lip-activated model includes a first extraction layer, a second extraction layer, and a first output layer. The schematic block diagram thereof is shown in FIG5 , and includes:
[0145] A third determining module 510 is configured to determine current parameters of the first extraction layer;
[0146] a fourth determining module 520 configured to sequentially use each sample audio feature in the sample audio feature sequence as a first feature, and determine a first feature subsequence corresponding to the first feature from the sample audio feature sequence, where the sample audio feature sequence is a feature sequence obtained by filtering the sample audio;
[0147] A third feature extraction module 530 is configured to perform first feature extraction on the first feature subsequence using the current parameters of the first extraction layer to obtain a sample intermediate feature;
[0148] A fifth determining module 540 is configured to sequentially use the sample intermediate feature corresponding to each sample audio feature as the second feature, and determine a second feature subsequence corresponding to the second feature from each sample intermediate feature;
[0149] a fourth feature extraction module 550 configured to perform second feature extraction based on the second feature subsequence using the second extraction layer to determine a corresponding sample smoothing feature;
[0150] The second processing module 560 is configured to process the sample smoothing features corresponding to the intermediate features of each sample using the first classification layer or the first regression layer to determine a lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence;
[0151] The first adjustment module 570 is configured to adjust parameters of the second extraction layer and the first output layer based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.
[0152] In an optional embodiment, the first adjustment module 570 is specifically configured to determine the prediction loss based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence;
[0153] With the goal of minimizing the prediction loss, parameters of the second extraction layer and the first output layer are adjusted.
[0154] In an optional implementation, the third determining module 510 is specifically configured to obtain a training audio feature sequence, where the training audio feature sequence is a feature sequence obtained by filtering the training audio;
[0155] sequentially taking each training audio feature in the training audio feature sequence as a third feature, and determining a third feature subsequence corresponding to the third feature from the training audio feature sequence;
[0156] Using the first extraction layer, performing first feature extraction on the third feature subsequence to obtain a training intermediate feature;
[0157] Using the second output layer, processing the training intermediate features corresponding to each training audio feature to determine a current lip-actuation parameter sequence corresponding to the training audio feature sequence;
[0158] Based on the current lip-actuating parameter sequence and the first lip-actuating parameter sequence corresponding to the training audio feature sequence, parameters of the first extraction layer and the second output layer are adjusted.
[0159] The above-mentioned device embodiments correspond to the method embodiments. For detailed descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For detailed descriptions, please refer to the corresponding method embodiments.
[0160] An embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method shown in FIG1A or FIG3 is implemented. Referring now to FIG6 , a schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure is shown. The electronic device shown in FIG6 is merely an example and should not limit the functionality or scope of use of the embodiments of the present disclosure.
[0161] As shown in Figure 6, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0162] Typically, the following devices can be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although FIG6 shows an electronic device with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively. Each box shown in FIG6 may represent one device, or may represent multiple devices as needed.
[0163] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
[0164] The embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the lip-activated method shown in FIG. 3 or the lip-activated model training method shown in FIG. 1A provided by the present disclosure.
[0165] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer-readable medium may be conveyed using any suitable medium, including but not limited to wires, optical cables, RF (Radio Frequency), or any suitable combination thereof.
[0166] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device sequentially uses each audio feature in the audio feature sequence as a target audio feature, and determines a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed; performs a first feature extraction on the target feature subsequence to obtain an audio intermediate feature; sequentially uses the audio intermediate feature corresponding to each audio feature as a target intermediate feature, and determines an intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature; performs a second feature extraction based on the intermediate feature subsequence to determine a corresponding audio smoothing feature; processes the audio smoothing feature corresponding to each audio intermediate feature, and determines a target lip-sync driving parameter sequence corresponding to the audio feature sequence for generating a lip-sync animation. Or, the electronic device determines the current parameters of the first extraction layer; sequentially uses each sample audio feature in the sample audio feature sequence as a first feature, and determines a first feature subsequence corresponding to the first feature from the sample audio feature sequence, where the sample audio feature sequence is a feature sequence obtained by filtering the sample audio; uses the current parameters of the first extraction layer to perform first feature extraction on the first feature subsequence to obtain a sample intermediate feature; sequentially uses the sample intermediate feature corresponding to each sample audio feature as a second feature, and determines a second feature subsequence corresponding to the second feature from each sample intermediate feature; uses the second extraction layer to perform second feature extraction based on the second feature subsequence to determine the corresponding sample smoothing feature; uses the first output layer to process the sample smoothing feature corresponding to each sample intermediate feature, and determines a lip-type drive prediction parameter sequence corresponding to the sample audio feature sequence; adjusts the parameters of the second extraction layer and the first output layer based on the lip-type drive prediction parameter sequence and the lip-type drive label parameter sequence corresponding to the sample audio feature sequence.
[0167] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0168] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences between the other embodiments. In particular, the storage medium and computing device embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, reference can be made to the description of the method embodiments.
[0169] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described in the embodiments of the present disclosure may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0170] The specific implementation methods described above further illustrate the purpose, technical solutions, and beneficial effects of the embodiments of the present disclosure. It should be understood that the above description is only a specific implementation method of the embodiments of the present disclosure and is not intended to limit the scope of protection of the present disclosure. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present disclosure shall be included in the scope of protection of the present disclosure.
Claims
1. A lip-activated method, comprising: Taking each audio feature in the audio feature sequence as a target audio feature in turn, and determining a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed; Performing a first feature extraction on the target feature subsequence to obtain an audio intermediate feature; The audio intermediate features corresponding to the audio features are sequentially used as target intermediate features, and an intermediate feature subsequence corresponding to the target intermediate features is determined from the audio intermediate features; Based on the intermediate feature subsequence, performing a second feature extraction to determine a corresponding audio smoothing feature; The audio smoothing features corresponding to each audio intermediate feature are processed to determine a target lip-sync driving parameter sequence corresponding to the audio feature sequence for generating lip-sync animation.
2. The method of claim 1, wherein: The step of extracting a second feature based on the intermediate feature subsequence to determine a corresponding audio smoothing feature includes: Performing a dimension adjustment operation on each audio intermediate feature in the intermediate feature subsequence to obtain a dimension adjustment feature subsequence including each audio intermediate feature after dimension adjustment, wherein the dimension adjustment operation is used to set each dimensional sub-feature included in each audio intermediate feature as an input of different channels; The second feature extraction is performed on the dimension adjustment feature subsequence to obtain an audio smoothing feature corresponding to the target intermediate feature.
3. The method according to any one of claims 1 to 2, wherein: The extracting the second feature from the dimension adjustment feature subsequence includes: The second feature extraction is performed on the dimensionally adjusted feature subsequence using a fully convolutional network.
4. The method according to any one of claims 1 to 3, wherein: The step of determining, from each of the audio intermediate features, an intermediate feature subsequence corresponding to the target intermediate feature comprises: A sliding window method is used to determine an intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature, wherein the intermediate feature subsequence includes the target intermediate feature and a first number of audio intermediate features before and a second number of audio intermediate features after the target intermediate feature.
5. The method according to any one of claims 1 to 4, further comprising: Get the audio to be processed; The audio to be processed is filtered using a preset Mel-frequency cepstral coefficient (MFCC) filtering algorithm to obtain the audio feature sequence.
6. The method according to any one of claims 1 to 4, wherein the target lip-shape driving parameter sequence is a predicted phoneme sequence, a predicted weight value set sequence, or a predicted lip-shape key point position set sequence.
7. A method for training a lip-activated model, wherein: The lip-driven model includes a first extraction layer, a second extraction layer and a first output layer; the method includes: determining current parameters of the first abstraction layer; Taking each sample audio feature in a sample audio feature sequence as a first feature in turn, and determining a first feature subsequence corresponding to the first feature from the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio; Using the current parameters of the first extraction layer, performing first feature extraction on the first feature subsequence to obtain sample intermediate features; Sequentially taking the sample intermediate features corresponding to the sample audio features as the second features, and determining the second feature subsequence corresponding to the second features from the sample intermediate features; Using the second extraction layer, based on the second feature subsequence, performing second feature extraction to determine a corresponding sample smoothing feature; Using the first output layer, processing the sample smoothing features corresponding to the intermediate features of each sample, and determining the lip-driven prediction parameter sequence corresponding to the sample audio feature sequence; Based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence, the parameters of the second extraction layer and the first output layer are adjusted.
8. The method of claim 7, wherein: The adjusting the parameters of the second extraction layer and the first output layer comprises: Determining a prediction loss based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence; The parameters of the second extraction layer and the first output layer are adjusted with the goal of minimizing the prediction loss.
9. The method according to any one of claims 7 to 8, wherein: The determining the current parameter of the first extraction layer comprises: Acquire a training audio feature sequence, where the training audio feature sequence is a feature sequence obtained by filtering the training audio; sequentially taking each training audio feature in the training audio feature sequence as a third feature, and determining a third feature subsequence corresponding to the third feature from the training audio feature sequence; Using the first extraction layer, performing first feature extraction on the third feature subsequence to obtain a training intermediate feature; Using the second output layer, processing the training intermediate features corresponding to each training audio feature, and determining a current lip-actuating parameter sequence corresponding to the training audio feature sequence; Based on the current lip-driven parameter sequence and the first lip-driven parameter sequence corresponding to the training audio feature sequence, the parameters of the first extraction layer and the second output layer are adjusted until the first extraction layer and the second output layer reach a first convergence condition, thereby obtaining the current parameters of the first extraction layer.
10. A lip-actuating device, comprising: A first determination module is configured to sequentially use each audio feature in an audio feature sequence as a target audio feature, and determine a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed; A first feature extraction module is configured to perform a first feature extraction on the target feature subsequence to obtain an audio intermediate feature; A second determination module is configured to sequentially use the audio intermediate features corresponding to each audio feature as the target intermediate feature, and determine the intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature; A second feature extraction module is configured to perform a second feature extraction based on the intermediate feature subsequence to determine a corresponding audio smoothness feature; The first processing module is configured to process the audio smoothness features corresponding to each audio intermediate feature, and determine a lip-sync driving parameter sequence corresponding to the audio feature sequence for generating a lip-sync animation.
11. A training device for a lip-activated model, wherein: The lip-driven model includes a first extraction layer, a second extraction layer and a first output layer; the device includes: A third determination module, configured to determine current parameters of the first extraction layer; The fourth determining module is configured to sequentially use each sample audio feature in the sample audio feature sequence as the first feature, from Determine a first feature subsequence corresponding to the first feature in the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio; A third feature extraction module is configured to perform first feature extraction on the first feature subsequence using current parameters of the first extraction layer to obtain a sample intermediate feature; A fifth determination module is configured to sequentially use the sample intermediate features corresponding to each sample audio feature as the second feature, and determine a second feature subsequence corresponding to the second feature from each sample intermediate feature; a fourth feature extraction module, configured to use the second extraction layer and based on the second feature subsequence, perform second feature extraction to determine a corresponding sample smoothing feature; A second processing module is configured to use the first output layer to process the sample smoothing features corresponding to the intermediate features of each sample, and determine the lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence; The first adjustment module is configured to adjust parameters of the second extraction layer and the first output layer based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.
12. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 9 is implemented.
13. A computer-readable storage medium storing computer-executable instructions, wherein when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.
14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Real-time audio-driven virtual character mouth shape synchronous control method
CN111081270A
Method and system for driving mouth shape by voice based on time sequence convolution
CN114495908A
Method and device for driving mouth shape of virtual image by sound and medium
CN114581567A
Audio-driven figure mouth shape method, model and training method thereof
CN115035604A
Prediction model training method and device, mouth shape animation generation method and device, equipment and medium
CN115602157A
