Mouth shape driving method and device and mouth shape driving model training method and device

By extracting audio features multiple times in the lip-drive model to generate smooth features, the problem of inter-frame jitter in the lip-driven animation is solved, and smoother lip-driven animation and reduced delay effect is achieved.

CN119007725APending Publication Date: 2024-11-22BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310572583.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the process of using audio driver to generate lip animations, inter-frame lip jitter often occurs. The existing methods can alleviate the sliding window by increasing the sliding window, but the effect is limited and will introduce lip delay.

Method used

Using the lip-driven model, multiple feature extractions of audio features are performed through the first extraction layer and the second extraction layer to generate smooth features, reduce jitter in the lip-shaped animation, and do not need to increase the sliding window to obtain more adjacent ones. Audio feature information.

Benefits of technology

It realizes that while ensuring real-time driving, it significantly reduces the jitter problem in lip animations, and the generated lip animations are smoother and reduces lip delay.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007725A_ABST
    Figure CN119007725A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a mouth shape driving method and device and a mouth shape driving model training method and device, and the method comprises the steps: sequentially taking each audio feature in an audio feature sequence as a target audio feature, and determining a target feature subsequence corresponding to the target audio feature from the audio feature sequence, the audio feature sequence is a feature sequence obtained by filtering the to-be-processed audio; performing first feature extraction on the target feature sub-sequence to obtain audio intermediate features; sequentially taking the audio intermediate features corresponding to the audio features as target intermediate features, and determining an intermediate feature subsequence corresponding to the target intermediate features from the audio intermediate features; performing second feature extraction based on the intermediate feature sub-sequence, and determining a corresponding audio smooth feature; and processing the audio smooth features corresponding to the audio intermediate features, and determining a mouth shape driving parameter sequence corresponding to the audio feature sequence used for generating the mouth shape animation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of lip-activated technology, and in particular to a lip-activated method and device, and a lip-activated model training method and device. Background Art

[0002] Lip-sync animation plays an important role in new human-computer interaction modes such as movies, games, and virtual reality. When watching model animations, viewers are often sensitive to some subtle changes in the model's lip movements. Therefore, how to generate truly natural lip-sync animations is the first challenging topic in the field of computer graphics today.

[0003] Among the methods for generating lip animation, there are a series of methods driven by audio. These methods generally take a piece of audio as input and generate a lip animation synchronized with the audio. In the process of generating lip animation by audio drive, the problem of lip jitter between frames often occurs.

[0004] Therefore, an improved solution needs to be provided to alleviate the problem of inter-frame lip-syncing jitter in the process of audio-driven lip-syncing animation generation. Summary of the invention

[0005] One or more embodiments of the present disclosure provide a lip-shaped driving method and device, and a lip-shaped driving model training method and device, so as to extract lip-shaped driving parameters that are helpful for smoothing the lip shape and reduce the lip-shaped jitter problem.

[0006] According to a first aspect, a lip-activated driving method is provided, comprising:

[0007] Taking each audio feature in the audio feature sequence as a target audio feature in turn, and determining a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed;

[0008] Performing a first feature extraction on the target feature subsequence to obtain an audio intermediate feature;

[0009] The audio intermediate features corresponding to the audio features are sequentially used as target intermediate features, and an intermediate feature subsequence corresponding to the target intermediate features is determined from the audio intermediate features;

[0010] Based on the intermediate feature subsequence, performing a second feature extraction to determine a corresponding audio smoothing feature;

[0011] The audio smoothing features corresponding to each audio intermediate feature are processed to determine a target lip-sync driving parameter sequence corresponding to the audio feature sequence for generating lip animation.

[0012] According to a second aspect, a method for training a lip-activated model is provided, wherein the lip-activated model comprises a first extraction layer, a second extraction layer and a first output layer; the method comprises:

[0013] determining current parameters of the first abstraction layer;

[0014] Taking each sample audio feature in a sample audio feature sequence as a first feature in turn, and determining a first feature subsequence corresponding to the first feature from the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio;

[0015] Using the current parameters of the first extraction layer, performing first feature extraction on the first feature subsequence to obtain sample intermediate features;

[0016] Sequentially taking the sample intermediate features corresponding to the sample audio features as the second features, and determining the second feature subsequence corresponding to the second features from the sample intermediate features;

[0017] Using the second extraction layer, based on the second feature subsequence, performing second feature extraction to determine a corresponding sample smoothing feature;

[0018] Using the first output layer, processing the sample smoothing features corresponding to the intermediate features of each sample, and determining the lip-driven prediction parameter sequence corresponding to the sample audio feature sequence;

[0019] Based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence, the parameters of the second extraction layer and the first output layer are adjusted.

[0020] According to a third aspect, there is provided a lip drive device, comprising:

[0021] A first determination module is configured to sequentially use each audio feature in an audio feature sequence as a target audio feature, and determine a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed;

[0022] The first feature extraction module is configured to perform a first feature extraction on the target feature subsequence to obtain an audio intermediate feature:

[0023] A second determination module is configured to sequentially use the audio intermediate features corresponding to each audio feature as the target intermediate feature, and determine the intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature;

[0024] A second feature extraction module is configured to perform a second feature extraction based on the intermediate feature subsequence to determine a corresponding audio smoothness feature;

[0025] The first processing module is configured to process the audio smoothness features corresponding to each audio intermediate feature, and determine a lip-sync driving parameter sequence corresponding to the audio feature sequence for generating a lip-sync animation.

[0026] According to a fourth aspect, a training device for a lip-activated model is provided, wherein the lip-activated model comprises a first extraction layer, a second extraction layer and a first output layer; the device comprises:

[0027] A third determination module, configured to determine current parameters of the first extraction layer;

[0028] The fourth determination module is configured to sequentially use each sample audio feature in the sample audio feature sequence as a first feature, and determine a first feature subsequence corresponding to the first feature from the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio;

[0029] A third feature extraction module is configured to perform first feature extraction on the first feature subsequence using current parameters of the first extraction layer to obtain a sample intermediate feature;

[0030] A fifth determination module is configured to sequentially use the sample intermediate features corresponding to each sample audio feature as the second feature, and determine a second feature subsequence corresponding to the second feature from each sample intermediate feature;

[0031] a fourth feature extraction module, configured to use the second extraction layer and based on the second feature subsequence, perform second feature extraction to determine a corresponding sample smoothing feature;

[0032] A second processing module is configured to use the first output layer to process the sample smoothing features corresponding to the intermediate features of each sample, and determine the lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence;

[0033] The first adjustment module is configured to adjust parameters of the second extraction layer and the first output layer based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.

[0034] According to a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect or the second aspect.

[0035] According to a sixth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0036] According to the lip-sync driving, lip-sync driving model training method and device provided by the embodiments of the present disclosure, in the lip-sync driving method, each audio feature in the audio feature sequence is sequentially used as the target audio feature, and a target feature subsequence corresponding to the target audio feature is determined from the audio feature sequence, where the audio feature sequence is a feature sequence obtained by filtering the audio to be processed; a first feature extraction is performed on the target feature subsequence to obtain an audio intermediate feature; the audio intermediate features corresponding to each audio feature are sequentially used as the target intermediate feature, and an intermediate feature subsequence corresponding to the target intermediate feature is determined from each audio intermediate feature; based on the intermediate feature subsequence, a second feature extraction is performed to determine the corresponding audio smoothing feature; the audio smoothing features corresponding to each audio intermediate feature are processed to determine a lip-sync driving parameter sequence corresponding to the audio feature sequence for generating a lip animation. In the above process, after the first feature extraction, the audio intermediate features corresponding to the audio features obtained are subjected to sliding windows to obtain the corresponding intermediate feature subsequences, and then the second feature extraction is performed on the intermediate feature subsequences, thereby achieving the effect of performing secondary feature extraction on the features obtained after the feature extraction. Without increasing the sliding window, audio smoothing features containing more information about adjacent audio features can be obtained. By using such features to generate lip-drive parameters, the lip animation corresponding to the lip-drive parameters can be made smoother, thereby reducing the problem of lip jitter in the lip animation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1A A schematic flow chart of a method for training a lip-driven model provided in an embodiment;

[0039] Figure 1B A schematic diagram of the data processing process of the lip-driven model provided in the embodiment;

[0040] Figure 2A A schematic diagram showing the intermediate features of a sample;

[0041] Figure 2B is a schematic diagram showing a second characteristic subsequence;

[0042] Figure 2C for Figure 2B A schematic diagram of a feature subsequence to be processed obtained after the dimension adjustment operation is performed on the intermediate features of each sample in the second feature subsequence shown;

[0043] Figure 3A schematic diagram of a flow chart of a lip-actuating method provided in an embodiment;

[0044] Figure 4 A schematic block diagram of a lip-actuating device provided in an embodiment;

[0045] Figure 5 A schematic block diagram of a training device for a lip-driven model provided in an embodiment;

[0046] Figure 6 A schematic block diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION

[0047] The technical solution of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0048] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0049] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0050] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0051] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0052] As mentioned above, in the process of using audio driver to generate lip animation, the problem of lip jitter between image frames (also called animation frames) often occurs. At present, the process of using audio driver to generate lip animation is generally as follows: obtain the audio to be processed, filter the audio to be processed, and obtain the corresponding audio feature sequence to be processed; use a sliding window method to determine the corresponding feature subsequence for each audio feature to be processed in the audio feature sequence to be processed; then use the feature subsequence corresponding to each audio feature to be processed to determine the lip drive parameter sequence corresponding to the audio feature sequence to be processed for generating lip animation, and then determine the corresponding lip image frame sequence (i.e., lip animation, which can also be called lip animation frame sequence) based on the lip drive parameter sequence.

[0053] In the above process, in order to reduce the jitter problem of lip shape between image frames, the sliding window is generally increased (i.e., the sliding window length is increased). In theory, increasing the sliding window can alleviate the problem of lip shape jitter between image frames. However, when the sliding window is increased to a certain extent, the effect of alleviating the problem of lip shape jitter between image frames is very limited, and it will cause a large lip shape delay problem.

[0054] In order to better reduce the problem of lip jitter while ensuring real-time lip driving, the embodiments of the present invention disclose a lip driving method and device, and a lip driving model training method and device. The lip driving method and device, and the lip driving model training method and device provided by the present invention are described in detail below in conjunction with specific embodiments.

[0055] It is understandable that the lip-driven process provided by the embodiment of the present disclosure may be implemented based on a pre-trained lip-driven model. For the sake of clarity, the training process of the lip-driven model is first introduced below.

[0056] In the training process of the lip-driven model, such as Figure 1A As shown, a flow chart of a method for training a lip-driven model in one embodiment of the present disclosure is shown. Figure 1B As shown, the lip-driven model may include a first extraction layer, a second extraction layer and a first output layer. The method may be implemented by a target program. The target program may be installed in an electronic device, and the electronic device may be implemented by any device, equipment, platform, device cluster, etc. having computing and processing capabilities.

[0057] It should be noted that, in the process of audio-driven lip shape, the lip-driven parameters used to generate lip-driven animation determined based on the audio, that is, the output of the lip-driven model, can be in the form of a mesh, in the form of a blendshape weight value set, or in the form of a viseme. In the disclosed embodiment, corresponding to the different forms of lip-driven parameters, the first output layer of the lip-driven model can be a classification layer or a regression layer. When the first output layer is a classification layer (referred to as the first classification layer), the output lip-driven parameters can be in the form of a viseme; when the first output layer is the first regression layer, the output lip-driven parameters can be in the form of a mesh or a blendshape weight value set.

[0058] When the first output layer of the lip-driven model is the first classification layer or the first regression layer, the training process (and the process of determining the lip-driven parameters) is similar, but the label data required for training are different. When the first output layer of the lip-driven model is the first classification layer, its label data includes lip-driven parameter labels in the form of viseme; when the first output layer of the lip-driven model is the first regression layer, its label data includes lip-driven parameter labels in the form of mesh, or its label data includes lip-driven parameter labels in the form of blendshape weight value sets.

[0059] The following is an introduction to the training process of the lip-driven model. Figure 1A As shown, the training method of the lip-driven model may include the following steps S110-S170:

[0060] In step S110 , current parameters of the first abstraction layer are determined.

[0061] In one implementation, the target program can read the current parameters of the first extraction layer from the preset storage space of the electronic device where the target program is located, that is, read the values ​​of the current parameters of the first extraction layer, and then execute the subsequent training process of the lip-driven model based on the current parameters of the first extraction layer. The first extraction layer can be a pre-trained extraction layer.

[0062] In another implementation, the target program can determine the current parameters of the first extraction layer by training. Accordingly, the training process of the lip-driven model provided by the embodiment of the present disclosure can be divided into two stages, namely, the first stage, the training stage of the first extraction layer, and the second stage, based on the trained first extraction layer, the training stage of the second extraction layer and the first output layer, wherein, during the training process of the second extraction layer and the first output layer, the current parameters of the first extraction layer (i.e., the values ​​of the current parameters) can be fixed. It can be understood that the first extraction layer can be implemented based on any neural network layer in the relevant technology that can extract audio features.

[0063] The training phase (i.e., the first phase) of the first extraction layer is introduced below. Specifically, the target program can first obtain the first extraction layer to be trained and its corresponding output layer (which can be a classification layer or a regression layer). For the sake of clarity, the output layer corresponding to the first extraction layer is called the second output layer. Among them, when the first output layer of the lip-driven model is a classification layer (called the first classification layer), when training the first extraction layer, the second output layer corresponding to the first extraction layer is a classification layer, called the second classification layer. When the first output layer of the lip-driven model is a regression layer (called the first regression layer), when training the first extraction layer, the second output layer corresponding to the first extraction layer is a regression layer, called the second regression layer.

[0064] Next, the target program obtains the training data required for training the first extraction layer and the second output layer, wherein the training data includes a training audio feature sequence corresponding to the audio data (referred to as training audio) and its corresponding label data, and the label data includes a lip-type driving parameter sequence corresponding to the training audio feature sequence. For the sake of clear description, the lip-type driving parameter sequence corresponding to the training audio feature sequence is referred to as the first lip-type driving parameter sequence.

[0065] Wherein, when the first extraction layer corresponds to the second classification layer, that is, when the second output layer is the second classification layer, the first lip shape driving parameter sequence exists in the form of viseme, including a string of phoneme sequences. It can be understood that each first lip shape driving parameter in the first lip shape driving parameter sequence has a one-to-one correspondence with each training audio feature in the training audio feature sequence.

[0066] Accordingly, each phoneme in the phoneme sequence has a one-to-one correspondence with each training audio feature. Based on each phoneme in the phoneme sequence, the corresponding lip animation frame can be determined (i.e., the lip animation frame corresponding to each training audio feature is determined), and then the corresponding lip animation is generated. Alternatively, based on the phoneme sequence, the weight value set of the blendshape corresponding to each phoneme can be determined, and then based on the weight value set of the blendshape corresponding to each phoneme, the corresponding mouth movement, i.e., lip shape, of each phoneme can be obtained, and then the corresponding lip animation can be generated.

[0067] When the first extraction layer corresponds to the second regression layer, that is, when the second output layer is the second regression layer, in one case, the first lip shape driving parameter sequence can exist in the form of a mesh, and each first lip shape driving parameter in the first lip shape driving parameter sequence is a position information set corresponding to a set of preset lip shape key points (or a set of preset face key points including the preset lip shape key points), and the corresponding mouth movement, that is, the lip shape (or the facial expression movement including the mouth movement) can be obtained based on the position information set. Accordingly, the corresponding lip shape animation can be obtained based on the first lip shape driving parameter sequence.

[0068] In another case when the first extraction layer corresponds to the second regression layer, the first lip shape driving parameter sequence can exist in the form of a blendshape weight value set, that is, the first lip shape driving parameter sequence is a series of weight value set sequences. The training audio feature sequence includes a plurality of training audio features, wherein each training audio feature can correspond to a set of weight value sets, and the set of weight value sets is a blendshape weight value set. For each training audio feature, the blendshape weight value set corresponding to the training audio feature is combined to at least obtain the mouth action, i.e., the lip shape, and then generate the corresponding lip shape animation frame. Accordingly, by combining the blendshape weight value set corresponding to each training audio feature in the training audio feature sequence, the mouth action, i.e., the lip shape, corresponding to each training audio feature can be obtained, and then generate the corresponding lip shape animation.

[0069] In one embodiment, the training audio feature sequence corresponding to the training audio may be obtained by filtering the training audio, and specifically may be the MFCC feature sequence corresponding to the training audio obtained after filtering the training audio using a preset Mel-frequency cepstral coefficient MFCC filtering algorithm.

[0070] After obtaining the training audio feature sequence corresponding to the training audio, the target program sequentially uses each training audio feature in the training audio feature sequence as a feature to be processed (referred to as the third feature), and determines a third feature subsequence corresponding to the third feature from the training audio feature sequence. In one implementation, the target program may use a sliding window method to determine a third feature subsequence corresponding to the third feature from the training audio feature sequence, and the third feature subsequence may include the third feature, a third number m of training audio features before the third feature in the training audio feature sequence, and a fourth number n of training audio features after the third feature. Among them, the third number m and the fourth number n are values ​​pre-set according to actual needs.

[0071] In one implementation, when the number of training audio features before the third feature in the training audio feature sequence is less than the third number m, the preset filling rule can be used to make up to m. Specifically, the first training audio feature in the training audio feature sequence can be used for filling. Similarly, when the number of training audio features after the third feature in the training audio feature sequence is less than the fourth number n, the preset filling rule can be used to make up to n. Specifically, the last training audio feature in the training audio feature sequence can be used for filling.

[0072] After obtaining the third feature subsequence corresponding to the third feature, the target program inputs the third feature subsequence into the first extraction layer, uses the first extraction layer to perform first feature extraction on the third feature subsequence, and obtains the training intermediate feature corresponding to the third feature (i.e., the third feature subsequence). Next, the training intermediate feature is input into the second output layer, and the training intermediate feature is processed by the second output layer to obtain the current lip-type drive parameter corresponding to the third feature (i.e., the third feature subsequence). Accordingly, the second output layer can be used to obtain the current lip-type drive parameter sequence corresponding to the training audio feature sequence based on the training intermediate features corresponding to each training audio feature in the training audio feature sequence.

[0073] Afterwards, the target program can adjust the parameters of the first extraction layer and the second output layer based on the current lip-driven parameter sequence and the first lip-driven parameter sequence corresponding to the training audio feature sequence. Specifically, the target program can use a preset loss function to determine the corresponding prediction loss (referred to as the first prediction loss) based on the current lip-driven parameter sequence and the first lip-driven parameter sequence corresponding to the training audio feature sequence, and then adjust the parameters of the first extraction layer and the second output layer with the goal of minimizing the first prediction loss until the first extraction layer and the second output layer reach the first convergence condition, thereby obtaining the current parameters of the first extraction layer. The first convergence condition may include, but is not limited to: the number of parameter adjustments is not less than the first adjustment threshold, or the calculated first prediction loss is less than the first loss threshold.

[0074] In one case, when the first extraction layer corresponds to the second classification layer (i.e., the second output layer is the second classification layer), the preset loss function may be a cross entropy loss function. In another case, when the first extraction layer corresponds to the second regression layer (i.e., the second output layer is the second regression layer), the preset loss function may be an MSE (mean square error) loss function.

[0075] Let's review the aforementioned training process for the first extraction layer and the second output layer. The above embodiment is explained by taking a training audio feature sequence corresponding to a training audio as an example. In another embodiment, the above training process can also be performed on a batch of samples, i.e., a training audio feature sequence corresponding to a plurality of training audios, to obtain the current lip-type drive parameter sequence corresponding to the training audio feature sequence corresponding to each training audio, and then based on the current lip-type drive parameter sequence corresponding to the training audio feature sequence corresponding to each training audio, and the first lip-type drive parameter sequence corresponding to each training audio feature sequence, jointly determine the first prediction loss, and then train the first extraction layer and the second output layer with the goal of minimizing the first prediction loss. In this embodiment, the first prediction loss is determined for a batch of samples, and then the parameters of the first extraction layer and the second output layer are adjusted. This can reduce the number of parameter adjustments for the first extraction layer and the second output layer, making it easier to implement the training process.

[0076] After determining the current parameters of the first extraction layer, in step S120, each sample audio feature in the sample audio feature sequence is taken as the first feature in turn, and a first feature subsequence corresponding to the first feature is determined from the sample audio feature sequence, and the sample audio feature sequence is a feature sequence obtained by filtering the sample audio.

[0077] Among them, the target program can first obtain data of the second extraction layer and the first output layer for training the lip-driven model, which is called sample data. The sample data includes a sample audio feature sequence obtained by filtering the sample audio, and a lip-driven label parameter sequence corresponding to the sample audio feature sequence.

[0078] In one implementation, the sample audio feature sequence may be an audio feature sequence obtained by filtering the sample audio using a preset Mel-frequency cepstral coefficient (MFCC) filtering algorithm, wherein the sample audio feature sequence includes a number of sample audio features sorted in time sequence, and the lip-driven label parameter sequence includes a number of lip-driven label parameters corresponding to the sample audio features.

[0079] In one implementation, the sample data for training the second extraction layer and the first output layer may be the same as or different from the training data for training the first extraction layer and the second output layer.

[0080] In one case, when the first output layer of the lip-driven model is the first classification layer, the lip-driven label parameter sequence exists in a viseme form, including a string of phoneme sequences.

[0081] In another case, when the first output layer of the lip-shaped drive model is the first regression layer, the lip-shaped drive label parameter sequence can exist in the form of a mesh (for example, a sequence of lip-shaped key point position sets), wherein each sample audio feature included in the sample audio feature sequence can correspond to a set of lip-shaped key point position sets respectively; or the lip-shaped drive label parameter sequence exists in the form of a blendshape weight value set (a sequence of weight value sets), wherein each sample audio feature included in the sample audio feature sequence corresponds to a set of blendshape weight value sets respectively.

[0082] In one implementation, in step S120, a sliding window method may be used to sequentially use each sample audio feature in the sample audio feature sequence as a first feature, and determine a first feature subsequence corresponding to the first feature from the sample audio feature sequence. Specifically, a sliding window (referred to as a first sliding window) may be pre-set, and the first sliding window may be slid on the sample audio feature sequence to determine a corresponding feature subsequence from the sample audio feature sequence for each sample audio feature in the sample audio feature sequence.

[0083] The length of the first sliding window can be set according to actual needs. Figure 1B As shown, for a first feature A (for example, sample audio feature A), a first feature subsequence corresponding to the first feature A is determined from a sample audio feature sequence, wherein the first feature subsequence corresponding to the first feature A may include the first feature A, a third number m of sample audio features before the first feature A in the sample audio feature sequence, and a fourth number n of sample audio features after the first feature A. In one case, the length of the first sliding window may be set to 23, and correspondingly, the third number m may be set to 15, and the fourth number n may be set to 7.

[0084] In one implementation, when the number of sample audio features before the first feature in the sample audio feature sequence is less than the third number m, a preset filling rule can be used to make up to m. Among them, the first sample audio feature in the sample audio feature sequence can be used for filling. Similarly, when the number of sample audio features after the first feature in the sample audio feature sequence is less than the fourth number n, a preset filling rule can be used to make up to n. Among them, the last sample audio feature in the sample audio feature sequence can be used for filling.

[0085] After obtaining the first feature subsequence corresponding to the first feature (i.e., a certain sample audio feature), in step S130, the first feature subsequence is subjected to first feature extraction using the current parameters of the first extraction layer to obtain the sample intermediate feature. In this step, the target program inputs the first feature subsequence into the first extraction layer obtained by the aforementioned training, and uses the current parameters of the first extraction layer to perform first feature extraction on the first feature subsequence to obtain the sample intermediate feature corresponding to the first feature subsequence (i.e., the first feature).

[0086] The target program uses the current parameters of the first extraction layer to obtain the corresponding sample intermediate features for each sample audio feature, wherein each sample intermediate feature can be sorted according to the sorting order of its corresponding sample audio features, or it can be considered that each sample intermediate feature is sorted according to the output time order of the corresponding first extraction layer. Figure 1B As shown, the sample intermediate feature corresponding to the first feature A (i.e., sample audio feature A) can be represented as the feature output at time t of the first extraction layer. Correspondingly, the sample intermediate feature corresponding to the sample audio feature B before the first feature A in the sample audio feature sequence can be represented as the feature output at time t-1 of the first extraction layer; the sample intermediate feature corresponding to the sample audio feature C after the first feature A in the sample audio feature sequence can be represented as the feature output at time t+1 of the first extraction layer, and so on.

[0087] After the target program obtains the sample intermediate features corresponding to each sample audio feature (or obtains the sample intermediate features corresponding to at least n sample audio features after the sample audio feature currently serving as the first feature), then in step S140, the sample intermediate features corresponding to each sample audio feature are sequentially used as second features, and a second feature subsequence corresponding to the second feature is determined from each sample intermediate feature.

[0088] In one embodiment, the target program may sequentially use the sample intermediate features corresponding to each sample audio feature as the second feature, and use a sliding window (e.g., a second sliding window) method to determine the second feature subsequence corresponding to the second feature from each sample intermediate feature, wherein the second feature subsequence may include the second feature and the first number of sample intermediate features before it and the second number of sample intermediate features after it, i.e., the length of the second sliding window (i.e., the number of windows) is equal to the sum of the first number, the second number, and 1. Specifically, the process of determining the second feature subsequence corresponding to the second feature can refer to the aforementioned process of determining the first feature subsequence corresponding to the first feature, and will not be repeated here.

[0089] In one case, in order to ensure the real-time performance of lip-sync driving, the value of the second number is not too large, for example, not exceeding a preset value, for example, the second number can be 1 or 2. In one implementation, the first number can be less than the aforementioned third number m, and the second number can be less than the aforementioned fourth number n, so as to better ensure the real-time performance of driving while improving inter-frame jitter.

[0090] Then, in step S150, the second feature extraction is performed based on the second feature subsequence using the second extraction layer to determine the corresponding sample smoothness feature. In this step, the second feature subsequence is input into the second extraction layer, and the second feature extraction is performed based on the second feature subsequence using the parameters of the second extraction layer to determine the corresponding sample smoothness feature.

[0091] In one embodiment, the second extraction layer can be a fully convolutional network to perform second feature extraction on the second feature subsequence. It can be achieved that based on the sample intermediate features extracted by the first extraction layer (information of the audio features of each sample in the corresponding first feature subsequence is integrated), feature extraction is performed again, so that the obtained sample smooth features are integrated with information of each sample intermediate feature in the second feature subsequence (that is, each sample audio feature in the first feature subsequence to which each sample intermediate feature corresponds). Relative to the sample intermediate features corresponding to each sample smooth feature, each sample smooth feature integrates information of a larger number of sample audio features. By using this type of feature to generate lip drive parameters, the lip animation corresponding to the lip drive parameters can be made smoother, so as to reduce the problem of lip jitter in the lip animation. Accordingly, in one case, the second extraction layer can also be called a smoother, which can alleviate the problem of lip jitter in lip drive.

[0092] In one embodiment, the second feature subsequence may be directly input into the second extraction layer, and the second feature extraction may be performed on the second feature subsequence using the parameters of the second extraction layer to determine the sample smoothing feature corresponding to the second feature subsequence (ie, the second feature).

[0093] In another embodiment, each sample intermediate feature output by the first extraction layer may include multiple dimensional sub-features (such as Figure 2A As shown, the sample intermediate feature 1 may include dimension sub-feature 1 corresponding to feature dimension 1, dimension sub-feature 2 corresponding to feature dimension 2, ..., dimension sub-feature N corresponding to feature dimension N).

[0094] The target program uses a sliding window method to obtain a second feature subsequence corresponding to the sample intermediate feature as the second feature. The second feature subsequence includes the window number of sample intermediate features of the sliding window (i.e., the aforementioned second sliding window), wherein a sample intermediate feature can be called a window feature, and a sample intermediate feature includes multiple dimensional subfeatures, i.e., a window feature includes multiple dimensional subfeatures. Correspondingly, the dimensional representation of the second feature subsequence can be (batchsize, number of windows, feature dimension), batchsize represents the batch size of the training data of the lip-driven model, the number of windows represents the height of the second feature subsequence, and the number of feature dimensions represents the width of the second feature subsequence.

[0095] like Figure 2B As shown in FIG. 1 , a schematic diagram of the second characteristic subsequence is shown in FIG. Figure 2B As shown, the number of windows, i.e., the height of the second feature subsequence, is 3, and the number of feature dimensions, i.e., the width of the second feature subsequence, is 3, wherein one square represents one dimensional sub-feature, and one row of dimensional sub-features represents one sample intermediate feature, i.e., one window feature.

[0096] Considering that the second extraction layer can be a fully convolutional network, the dimension of its input data is represented by (batchsize, channel, height, width), and considering that the number of feature dimensions is represented by the width of the second feature subsequence, when using the second extraction layer to directly perform feature fusion extraction (convolution processing) on ​​the second feature subsequence, the sub-features of each dimension of the sample intermediate feature may be split and fused, which will result in incomplete fusion of the sub-features of each dimension of the sample intermediate feature (window feature).

[0097] Specifically, in Figure 2B On the basis of the second feature subsequence shown, if the size of the convolution kernel included in the second extraction layer is 2*2, accordingly, when the second feature subsequence is convolved (feature fusion extraction) using the 2*2 convolution kernel, for the sample intermediate feature a shown in the first row, the 2*2 convolution kernel can only simultaneously convolve the two dimensional sub-features a1 and a2 of the sample intermediate feature a (as well as the two dimensional sub-features b1 and b2 of the sample intermediate feature b), while the other dimensional sub-feature a3 (as well as b3 of the sample intermediate feature b) is separated and cannot be completely fused.

[0098] In order to better and more completely fuse the multiple dimensional sub-features included in the intermediate features of each sample, that is, to better and more completely fuse the window features, in step S150, the target program can perform a dimensionality adjustment operation (such as a reshape operation) on the intermediate features of each sample in the second feature subsequence to obtain a feature subsequence including the intermediate features of each sample after dimensionality adjustment, which can be called a feature subsequence to be processed. The dimensionality adjustment operation can set the dimensional sub-features contained in the intermediate features of each sample (window feature) as the input of different channels of the second feature extraction layer. The dimensional representation of the intermediate features of each sample after dimensionality adjustment can be (batchsize, feature dimension, number of windows, 1). Figure 2C As shown, Figure 2B A schematic diagram of a feature subsequence to be processed obtained after the dimension adjustment operation is performed on the intermediate features of each sample in the second feature subsequence shown.

[0099] After the dimensionality adjustment operation is performed on the intermediate features of each sample in the second feature subsequence, all dimensional sub-features corresponding to each sample intermediate feature (i.e., each window feature) are used as inputs of different channels of the second extraction layer. In one implementation, the second extraction layer may include several layers of convolutional layers, and all dimensional sub-features corresponding to each sample intermediate feature (i.e., each window feature) mentioned here as inputs of different channels of the second extraction layer may refer to inputs of different channels of the first convolutional layer of the second extraction layer.

[0100] Then, in step S150, the target program can then use the second extraction layer to perform a second feature extraction on the feature subsequence to be processed, and obtain a sample smoothing feature corresponding to the second feature. For example, using the convolution kernel (e.g., a convolution kernel of size 2*1) in the first convolution layer of the second extraction layer, any sample intermediate feature (e.g., Figure 2C When the sample intermediate feature a) shown in the figure is convolved, the 2*1 convolution kernel corresponding to each channel is first used to convolve the sample intermediate feature a on each channel to obtain the convolution result of each channel. After that, the convolution result of each channel is weighted summed to obtain the final convolution result corresponding to the sample intermediate feature a. Among them, the final convolution result obtained includes all the dimensional sub-features contained in the sample intermediate feature a, and includes all the dimensional sub-features contained in its adjacent sample intermediate feature b, which can achieve complete convolution of different sample intermediate features and obtain a final convolution result containing more comprehensive information. Correspondingly, the sample smoothness feature corresponding to the obtained second feature subsequence (i.e., the second feature) contains more comprehensive sample audio feature information. This type of feature is used to generate lip-type drive parameters, which can make the lip animation corresponding to the lip-type drive parameters smoother and better reduce the problem of lip jitter in the lip animation.

[0101] Then, in step S160, the first output layer is used to process the sample smoothing features corresponding to the intermediate features of each sample, and determine the lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence.

[0102] In this step, the sample smoothing features corresponding to the intermediate features of each sample are input into the first output layer, and the sample smoothing features corresponding to the intermediate features of each sample are processed using the parameters of the first output layer to determine the lip-driven prediction parameter sequence corresponding to the sample audio feature sequence. In one case, the first output layer can be implemented by a fully connected layer.

[0103] Wherein, the first output layer can be a classification layer or a regression layer. Specifically, when the first output layer is the first classification layer, the lip-type drive prediction parameter sequence can exist in the form of viseme, which is a string of phoneme sequences. When the first output layer is the first regression layer, the lip-type drive prediction parameter sequence can exist in the form of mesh (for example, a string of lip-type key point position set sequences), wherein each sample audio feature included in the sample audio feature sequence corresponds to a set of lip-type key point position sets; or the lip-type drive prediction parameter sequence can exist in the form of blendshape weight value set (a string of weight value set sequences), wherein each sample audio feature corresponds to a set of weight value sets, and the set of weight value sets includes several blendshape weight values.

[0104] After determining that the sample audio feature sequence corresponds to the lip-actuated prediction parameter sequence, in step S170, the parameters of the second extraction layer and the first output layer are adjusted based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.

[0105] Specifically, a preset loss function is used to determine the corresponding prediction loss (called the current prediction loss) based on the lip-driven prediction parameter sequence and the lip-driven label parameter sequence corresponding to the sample audio feature sequence, and then the parameters of the second extraction layer and the first output layer are adjusted with the goal of minimizing the current prediction loss until the second extraction layer and the first output layer reach the second convergence condition, thereby obtaining the trained second extraction layer and first output layer.

[0106] The second convergence condition may include, but is not limited to, the number of parameter adjustments is not less than a second adjustment threshold, or the calculated current predicted loss is less than a second loss threshold. The second adjustment threshold may be the same as or different from the aforementioned first adjustment threshold, and the second loss threshold may be the same as or different from the aforementioned first loss threshold.

[0107] In one case, when the first output layer is a first classification layer, the preset loss function may be a cross entropy loss function. In another case, when the first output layer is a first regression layer, the preset loss function may be an MSE (mean square error) loss function.

[0108] Let's review the execution process of the aforementioned steps S110-S170. The above embodiment is explained by taking a sample audio feature sequence corresponding to a sample audio as an example. In another embodiment, the above training process can also be performed on a batch of samples, i.e., a sample audio feature sequence corresponding to a plurality of sample audios, to obtain the lip-type driven prediction parameter sequence corresponding to the sample audio feature sequence corresponding to each sample audio, and then determine the current prediction loss based on the lip-type driven prediction parameter sequence corresponding to the sample audio feature sequence corresponding to each sample audio, and the lip-type driven label parameter sequence corresponding to the sample audio feature sequence, and train the second extraction layer and the first output layer with the goal of minimizing the current prediction loss. In this embodiment, the current prediction loss is determined for a batch of samples, and then the parameters of the second extraction layer and the first output layer are adjusted. This can reduce the number of parameter adjustments for the second extraction layer and the first output layer, making it easier to implement the training process.

[0109] In this embodiment, a new structure of a lip-sync driving model is provided. Through its second extraction layer, the second feature subsequence corresponding to the sample intermediate feature output by the first extraction layer is subjected to feature extraction and fusion again, so that sample smoothing features that fuse information of a larger number of sample audio features can be extracted without increasing the sliding window. By using this type of feature to generate lip-sync driving parameters, the lip animation corresponding to the lip-sync driving parameters can be made smoother, which provides a basis for reducing the problem of lip jitter in lip animation.

[0110] After the lip-driven model is trained, the lip-driven model can be used to perform lip-driven operation (ie, generate lip animation). Figure 3 The flowchart of the lip-activated method in one embodiment of the present disclosure is shown. The method can be implemented by a target program. The target program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 3 As shown, the method comprises the following steps:

[0111] In step S310, each audio feature in the audio feature sequence is used as a target audio feature in turn, and a target feature subsequence corresponding to the target audio feature is determined from the audio feature sequence, and the audio feature sequence is a feature sequence obtained by filtering the audio to be processed. The audio to be processed can be any audio data used as input for lip-sync driving (i.e., for generating lip-sync animation).

[0112] Among them, the process of determining the target feature subsequence corresponding to the target audio feature from the audio feature sequence can refer to the aforementioned process of determining the third feature subsequence corresponding to the third feature from the training audio feature sequence, or the process of determining the first feature subsequence corresponding to the first feature from the sample audio feature sequence, and will not be repeated here.

[0113] Then, in step S320, the first feature extraction is performed on the target feature subsequence to obtain the audio intermediate feature. Specifically, the first extraction layer obtained by the aforementioned training can be used to perform the first feature extraction on the target feature subsequence to obtain the audio intermediate feature. The process of obtaining the audio intermediate feature can refer to the process of obtaining the sample intermediate feature (or obtaining the training intermediate feature) mentioned above, which will not be repeated here.

[0114] Then, in step S330, the audio intermediate features corresponding to the respective audio features are sequentially used as target intermediate features, and an intermediate feature subsequence corresponding to the target intermediate feature is determined from the respective audio intermediate features.

[0115] In one implementation, the target program sequentially uses the audio intermediate features corresponding to each audio feature as the target intermediate feature, and can use a sliding window method to determine the intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature, wherein the intermediate feature subsequence includes the target intermediate feature and the first number of audio intermediate features before it and the second number of audio intermediate features after it. The process of determining the intermediate feature subsequence corresponding to the target intermediate feature can refer to the aforementioned process of determining the second feature subsequence corresponding to the second feature, and will not be repeated here.

[0116] Next, in step S340, based on the intermediate feature subsequence, a second feature extraction is performed to determine the corresponding audio smoothness feature. Specifically, the second extraction layer obtained by the aforementioned training can be used to perform a second feature extraction based on the intermediate feature subsequence to determine the corresponding audio smoothness feature.

[0117] In one embodiment, in order to ensure that the audio smoothness feature includes more comprehensive and rich information, in step S340, the target program can perform a dimension adjustment operation on each audio intermediate feature in the intermediate feature subsequence to obtain a dimension adjustment feature subsequence including each audio intermediate feature after dimension adjustment, wherein the dimension adjustment operation is used to set each dimensional sub-feature contained in each audio intermediate feature as an input of different channels; then, a second feature extraction is performed on the dimension adjustment feature subsequence to obtain the audio smoothness feature corresponding to the target intermediate feature, so as to set each dimensional sub-feature contained in each audio intermediate feature as an input of different channels of the first convolutional layer of the second extraction layer, so as to facilitate the fusion of all dimensional sub-features of different audio intermediate features, so as to better alleviate the problem of lip jitter in the subsequent lip driving process.

[0118] The process of performing the dimension adjustment operation on each audio intermediate feature in the intermediate feature subsequence can refer to the aforementioned process of performing the dimension adjustment operation on each sample intermediate feature in the second feature subsequence, which will not be repeated here.

[0119] Then, in step S350, the audio smoothing features corresponding to each audio intermediate feature are processed to determine the target lip-type driving parameter sequence corresponding to the audio feature sequence used to generate the lip animation. Specifically, the aforementioned first output layer is used to process the audio smoothing features corresponding to each audio intermediate feature to determine the target lip-type driving parameter sequence corresponding to the audio feature sequence used to generate the lip animation. Wherein, when the first output layer is the first classification layer, the target lip-type driving parameter sequence is a predicted phoneme sequence, and when the first output layer is the first regression layer, the target lip-type driving parameter sequence is a predicted weight value set sequence or a predicted lip key point position set sequence.

[0120] The process of determining the target lip-actuating parameter sequence corresponding to the audio feature sequence used to generate the lip animation can refer to the aforementioned process of determining the lip-actuating prediction parameter sequence corresponding to the sample audio feature sequence, which will not be repeated here.

[0121] After obtaining the target lip-shaped driving parameter sequence, a corresponding lip-shaped animation can be generated based on it, and then the lip-shaped animation is played. Specifically, the target program can obtain a target lip-shaped driving parameter in the target lip-shaped driving parameter sequence, that is, generate a corresponding lip-shaped animation frame based on the target lip-shaped driving parameter, and then play the lip-shaped animation frame to achieve real-time playback of the lip-shaped animation according to the audio feature sequence.

[0122] In this embodiment, a sliding window is performed on the audio intermediate features corresponding to each audio feature obtained after the first feature extraction to obtain a corresponding intermediate feature subsequence, and then a second feature extraction is performed on the intermediate feature subsequence, thereby achieving the effect of performing a secondary feature extraction on the features obtained after the feature extraction, and obtaining an audio smoothing feature containing more audio feature information. Without increasing the sliding window, an audio smoothing feature containing more adjacent audio feature information can be obtained. By using this type of feature to generate lip drive parameters, the lip animation corresponding to the lip drive parameters can be made smoother, thereby reducing the problem of lip jitter in the lip animation.

[0123] In addition, a dimensionality adjustment operation is performed on each audio intermediate feature in the intermediate feature sub-sequence, and each dimensional sub-feature contained in each audio intermediate feature is set as the input of different channels of the first convolutional layer of the second extraction layer, so as to facilitate the fusion of all dimensional sub-features of different audio intermediate features, which can better alleviate the problem of lip jitter in the subsequent lip-shaking process.

[0124] The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible, or may be advantageous.

[0125] Corresponding to the above method embodiment, the present disclosure provides a lip-actuating device 400, a schematic block diagram of which is shown in FIG. Figure 4 As shown, including:

[0126] A first determination module 410 is configured to sequentially use each audio feature in an audio feature sequence as a target audio feature, and determine a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed;

[0127] A first feature extraction module 420 is configured to perform a first feature extraction on the target feature subsequence to obtain an audio intermediate feature;

[0128] The second determination module 430 is configured to sequentially use the audio intermediate features corresponding to the respective audio features as target intermediate features, and determine the intermediate feature subsequence corresponding to the target intermediate features from the respective audio intermediate features;

[0129] A second feature extraction module 440 is configured to perform a second feature extraction based on the intermediate feature subsequence to determine a corresponding audio smoothness feature;

[0130] The first processing module 450 is configured to process the audio smoothness features corresponding to the audio intermediate features, and determine a lip-sync driving parameter sequence corresponding to the audio feature sequence for generating lip animation.

[0131] In an optional implementation, the second feature extraction module 440 includes:

[0132] an adjustment unit (not shown in the figure), configured to perform a dimension adjustment operation on each audio intermediate feature in the intermediate feature subsequence to obtain a dimension-adjusted feature subsequence including each audio intermediate feature after dimension adjustment, wherein the dimension adjustment operation is used to set each dimensional sub-feature included in each audio intermediate feature as an input of different channels;

[0133] A feature extraction unit (not shown in the figure) is configured to perform the second feature extraction on the dimension adjustment feature subsequence to obtain an audio smoothing feature corresponding to the target intermediate feature.

[0134] In an optional implementation, the feature extraction unit is specifically configured to perform the second feature extraction on the dimensionally adjusted feature subsequence using a fully convolutional network.

[0135] In an optional embodiment, the second determination module 430 is specifically configured to determine, from each audio intermediate feature, an intermediate feature subsequence corresponding to the target intermediate feature by using a sliding window method, wherein the intermediate feature subsequence includes the target intermediate feature and a first number of audio intermediate features before and a second number of audio intermediate features after the target intermediate feature.

[0136] In an optional implementation, it also includes:

[0137] A first acquisition module (not shown in the figure), configured to acquire audio to be processed;

[0138] The filtering module (not shown in the figure) is configured to filter the audio to be processed using a preset Mel-frequency cepstral coefficient MFCC filtering algorithm to obtain the audio feature sequence.

[0139] In an optional implementation, the target lip-shape driving parameter sequence is a predicted phoneme sequence, a predicted weight value set sequence, or a predicted lip-shape key point position set sequence.

[0140] Corresponding to the above method embodiment, the present disclosure embodiment provides a training device 500 for a lip-activated model, wherein the lip-activated model includes a first extraction layer, a second extraction layer and a first output layer; a schematic block diagram thereof is shown in FIG. Figure 5 As shown, including:

[0141] A third determination module 510 is configured to determine current parameters of the first extraction layer;

[0142] The fourth determination module 520 is configured to sequentially use each sample audio feature in the sample audio feature sequence as a first feature, and determine a first feature subsequence corresponding to the first feature from the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio;

[0143] A third feature extraction module 530 is configured to perform first feature extraction on the first feature subsequence using the current parameters of the first extraction layer to obtain a sample intermediate feature;

[0144] A fifth determination module 540 is configured to sequentially use the sample intermediate features corresponding to each sample audio feature as the second feature, and determine a second feature subsequence corresponding to the second feature from each sample intermediate feature;

[0145] A fourth feature extraction module 550 is configured to use the second extraction layer to perform second feature extraction based on the second feature subsequence to determine a corresponding sample smoothing feature;

[0146] The second processing module 560 is configured to process the sample smoothing features corresponding to the intermediate features of each sample by using the first classification layer or the first regression layer to determine the lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence;

[0147] The first adjustment module 570 is configured to adjust the parameters of the second extraction layer and the first output layer based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.

[0148] In an optional implementation, the first adjustment module 570 is specifically configured to determine the prediction loss based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence;

[0149] The parameters of the second extraction layer and the first output layer are adjusted with the goal of minimizing the prediction loss.

[0150] In an optional implementation, the third determination module 510 is specifically configured to obtain a training audio feature sequence, where the training audio feature sequence is a feature sequence obtained by filtering the training audio;

[0151] sequentially taking each training audio feature in the training audio feature sequence as a third feature, and determining a third feature subsequence corresponding to the third feature from the training audio feature sequence;

[0152] Using the first extraction layer, performing first feature extraction on the third feature subsequence to obtain a training intermediate feature;

[0153] Using the second output layer, processing the training intermediate features corresponding to each training audio feature, and determining a current lip-actuating parameter sequence corresponding to the training audio feature sequence;

[0154] Based on the current lip-driven parameter sequence and the first lip-driven parameter sequence corresponding to the training audio feature sequence, the parameters of the first extraction layer and the second output layer are adjusted.

[0155] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0156] The present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the following is implemented: Figure 1A or Figure 3 The method shown below. Figure 6 , which shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0157] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0158] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0159] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0160] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer executes the Figure 3 The lip-actuated method shown or Figure 1A The training method of the lip-driven model shown.

[0161] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0162] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device sequentially uses each audio feature in the audio feature sequence as a target audio feature, and determines a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed; performs a first feature extraction on the target feature subsequence to obtain an audio intermediate feature; sequentially uses the audio intermediate feature corresponding to each audio feature as a target intermediate feature, and determines an intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature; performs a second feature extraction based on the intermediate feature subsequence to determine the corresponding audio smoothing feature; processes the audio smoothing feature corresponding to each audio intermediate feature, and determines a target lip-sync driving parameter sequence corresponding to the audio feature sequence for generating lip animation. Or the electronic device determines the current parameters of the first extraction layer; takes each sample audio feature in the sample audio feature sequence as the first feature in turn, and determines a first feature subsequence corresponding to the first feature from the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio; uses the current parameters of the first extraction layer to extract the first feature from the first feature subsequence to obtain a sample intermediate feature; takes the sample intermediate features corresponding to each sample audio feature in turn as the second feature, and determines a second feature subsequence corresponding to the second feature from each sample intermediate feature; uses the second extraction layer to extract the second feature based on the second feature subsequence to determine the corresponding sample smoothing feature; uses the first output layer to process the sample smoothing features corresponding to each sample intermediate feature to determine a lip-driven prediction parameter sequence corresponding to the sample audio feature sequence; adjusts the parameters of the second extraction layer and the first output layer based on the lip-driven prediction parameter sequence and the lip-driven label parameter sequence corresponding to the sample audio feature sequence.

[0163] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0165] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0166] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A lip-activated method, comprising: Taking each audio feature in the audio feature sequence as a target audio feature in turn, and determining a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed; Performing a first feature extraction on the target feature subsequence to obtain an audio intermediate feature; The audio intermediate features corresponding to the audio features are sequentially used as target intermediate features, and an intermediate feature subsequence corresponding to the target intermediate features is determined from the audio intermediate features; Based on the intermediate feature subsequence, performing a second feature extraction to determine a corresponding audio smoothing feature; The audio smoothing features corresponding to each audio intermediate feature are processed to determine a target lip-sync driving parameter sequence corresponding to the audio feature sequence for generating lip animation.

2. The method of claim 1, wherein: The step of extracting a second feature based on the intermediate feature subsequence to determine a corresponding audio smoothing feature includes: Performing a dimension adjustment operation on each audio intermediate feature in the intermediate feature subsequence to obtain a dimension adjustment feature subsequence including each audio intermediate feature after dimension adjustment, wherein the dimension adjustment operation is used to set each dimensional sub-feature included in each audio intermediate feature as an input of different channels; The second feature extraction is performed on the dimension adjustment feature subsequence to obtain an audio smoothing feature corresponding to the target intermediate feature.

3. The method of claim 1, wherein: The extracting the second feature from the dimension adjustment feature subsequence includes: The second feature extraction is performed on the dimensionally adjusted feature subsequence using a fully convolutional network.

4. The method of claim 1, wherein: The step of determining, from each of the audio intermediate features, an intermediate feature subsequence corresponding to the target intermediate feature comprises: A sliding window method is used to determine an intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature, wherein the intermediate feature subsequence includes the target intermediate feature and a first number of audio intermediate features before and a second number of audio intermediate features after the target intermediate feature.

5. The method according to any one of claims 1 to 4, further comprising: Get the audio to be processed; The audio to be processed is filtered using a preset Mel-frequency cepstral coefficient (MFCC) filtering algorithm to obtain the audio feature sequence.

6. The method according to any one of claims 1 to 4, wherein the target lip-shape driving parameter sequence is a predicted phoneme sequence, a predicted weight value set sequence, or a predicted lip-shape key point position set sequence.

7. A method for training a lip-activated model, wherein: The lip-driven model includes a first extraction layer, a second extraction layer and a first output layer; the method includes: determining current parameters of the first abstraction layer; Taking each sample audio feature in a sample audio feature sequence as a first feature in turn, and determining a first feature subsequence corresponding to the first feature from the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio; Using the current parameters of the first extraction layer, performing first feature extraction on the first feature subsequence to obtain sample intermediate features; Sequentially taking the sample intermediate features corresponding to the sample audio features as the second features, and determining the second feature subsequence corresponding to the second features from the sample intermediate features; Using the second extraction layer, based on the second feature subsequence, performing second feature extraction to determine a corresponding sample smoothing feature; Using the first output layer, processing the sample smoothing features corresponding to the intermediate features of each sample, and determining the lip-driven prediction parameter sequence corresponding to the sample audio feature sequence; Based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence, the parameters of the second extraction layer and the first output layer are adjusted.

8. The method of claim 7, wherein: The adjusting the parameters of the second extraction layer and the first output layer comprises: Determining a prediction loss based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence; The parameters of the second extraction layer and the first output layer are adjusted with the goal of minimizing the prediction loss.

9. The method of claim 7, wherein: The determining the current parameter of the first extraction layer comprises: Acquire a training audio feature sequence, where the training audio feature sequence is a feature sequence obtained by filtering the training audio; sequentially taking each training audio feature in the training audio feature sequence as a third feature, and determining a third feature subsequence corresponding to the third feature from the training audio feature sequence; Using the first extraction layer, performing first feature extraction on the third feature subsequence to obtain a training intermediate feature; Using the second output layer, processing the training intermediate features corresponding to each training audio feature, and determining a current lip-actuating parameter sequence corresponding to the training audio feature sequence; Based on the current lip-driven parameter sequence and the first lip-driven parameter sequence corresponding to the training audio feature sequence, the parameters of the first extraction layer and the second output layer are adjusted until the first extraction layer and the second output layer reach a first convergence condition, thereby obtaining the current parameters of the first extraction layer.

10. A lip-actuating device, comprising: A first determination module is configured to sequentially use each audio feature in an audio feature sequence as a target audio feature, and determine a target feature subsequence corresponding to the target audio feature from the audio feature sequence, wherein the audio feature sequence is a feature sequence obtained by filtering the audio to be processed; A first feature extraction module is configured to perform a first feature extraction on the target feature subsequence to obtain an audio intermediate feature; A second determination module is configured to sequentially use the audio intermediate features corresponding to each audio feature as the target intermediate feature, and determine the intermediate feature subsequence corresponding to the target intermediate feature from each audio intermediate feature; A second feature extraction module is configured to perform a second feature extraction based on the intermediate feature subsequence to determine a corresponding audio smoothness feature; The first processing module is configured to process the audio smoothness features corresponding to each audio intermediate feature, and determine a lip-sync driving parameter sequence corresponding to the audio feature sequence for generating a lip-sync animation.

11. A training device for a lip-activated model, wherein: The lip-driven model includes a first extraction layer, a second extraction layer and a first output layer; the device includes: A third determination module is configured to determine current parameters of the first extraction layer; a fourth determination module configured to sequentially use each sample audio feature in a sample audio feature sequence as a first feature, and determine a first feature subsequence corresponding to the first feature from the sample audio feature sequence, wherein the sample audio feature sequence is a feature sequence obtained by filtering the sample audio; A third feature extraction module is configured to perform first feature extraction on the first feature subsequence using current parameters of the first extraction layer to obtain a sample intermediate feature; A fifth determination module is configured to sequentially use the sample intermediate features corresponding to each sample audio feature as the second feature, and determine a second feature subsequence corresponding to the second feature from each sample intermediate feature; a fourth feature extraction module, configured to use the second extraction layer and based on the second feature subsequence, perform second feature extraction to determine a corresponding sample smoothing feature; A second processing module is configured to use the first output layer to process the sample smoothing features corresponding to the intermediate features of each sample, and determine the lip-actuation prediction parameter sequence corresponding to the sample audio feature sequence; The first adjustment module is configured to adjust parameters of the second extraction layer and the first output layer based on the lip-actuated prediction parameter sequence and the lip-actuated label parameter sequence corresponding to the sample audio feature sequence.

12. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 9 is implemented.