Training of a prediction model, lip sync animation generation method and apparatus, device and medium

Abstract

The application provides a training method of a prediction model, a lip animation generation method and device, equipment and a medium, and relates to the technical field of data processing. The method comprises: sampling a sample voice signal and sample lip data corresponding to the sample voice signal from dynamic capture data; extracting a sample audio feature of the sample voice signal by using a feature extractor; generating predicted lip data according to the sample audio feature and sample initial lip animation data by using a lip generator; and adjusting model parameters of the lip prediction model according to the predicted lip data and the sample lip data to obtain a target lip prediction model. Compared with the prior art, the method avoids the problem of insufficient accuracy and the problem of poor generalization ability.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method, apparatus, device, and medium for training a prediction model and generating lip-sync animation. Background Technology

[0002] Virtual humans currently have many applications, such as virtual characters in games, virtual idols, and virtual narrators. As a medium for interaction between computers and humans, the accuracy of lip-syncing when a virtual human speaks greatly affects the user experience of virtual interaction.

[0003] Generally, a parameter-controlled approach is used to control the generation of lip movements in virtual humans. These parameters are a set of facial expression and lip-movement control systems defined by artists, with n dimensions, each clearly defined. The goal of lip-movement synthesis is to automatically synthesize the parameters related to lip movements. Current technologies typically determine these parameters by using artificial intelligence (AI) to generate the corresponding lip movements based on the input speech.

[0004] However, this generation method suffers from insufficient accuracy in lip shape prediction and poor generalization ability. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a method, apparatus, device, and medium for training a prediction model and generating lip-sync animation, thereby solving the problems of insufficient accuracy and poor generalization ability in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, one embodiment of this application provides a method for training a lip shape prediction model, the lip shape prediction model comprising: a feature extractor and a lip shape generator, the method comprising:

[0008] Sample speech signals and corresponding lip-sync data are collected from the dynamic capture data.

[0009] The feature extractor is used to extract sample audio features from the sample speech signal;

[0010] Using the lip shape generator, predicted lip shape data is generated based on the sample audio features and the initial lip animation data of the sample;

[0011] Based on the predicted lip shape data and the sample lip shape data, the model parameters of the lip shape prediction model are adjusted to obtain the target lip shape prediction model.

[0012] Secondly, another embodiment of this application provides a method for generating lip-sync animation, the method comprising:

[0013] Based on the target speech signal and initial mouth animation data, a mouth shape prediction model is used to generate target mouth shape data; wherein, the mouth shape prediction model is a model trained using any of the methods described in the first aspect above.

[0014] Based on the target speech and the target lip-sync data, control the mouth of the preset virtual object to perform lip-sync animation.

[0015] Thirdly, another embodiment of this application provides a training device for a lip shape prediction model, the device comprising: a sampling module, an extraction module, a generation module, and an adjustment module, wherein:

[0016] The sampling module is used to sample sample speech signals and sample lip shape data corresponding to the sample speech signals from the dynamically captured data;

[0017] The extraction module is used to extract sample audio features of the sample speech signal using the feature extractor;

[0018] The generation module is used to generate predicted lip shape data using the lip shape generator based on the sample audio features and the initial lip animation data of the sample.

[0019] The adjustment module is used to adjust the model parameters of the lip shape prediction model according to the predicted lip shape data and the sample lip shape data, so as to obtain the target lip shape prediction model.

[0020] Fourthly, another embodiment of this application provides a lip-sync animation generation device, the device comprising: a generation module and a control module, wherein:

[0021] The generation module is used to generate target lip shape data based on the target speech signal and the initial mouth animation data, using a lip shape prediction model.

[0022] The control module is used to control the mouth of the preset virtual object to perform lip-sync animation based on the target speech and the target lip-sync data.

[0023] Fifthly, another embodiment of this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in either the first or second aspect above.

[0024] In a sixth aspect, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of the method as described in either the first or second aspect above.

[0025] The beneficial effects of this application are as follows: By using the training method of the lip shape prediction model provided in this application, sample speech signals and corresponding sample lip shape data are sampled from the dynamic capture data. Then, the sample audio features of the sample speech signals are extracted using a feature extractor. After the lip shape generator generates predicted lip shape data based on the sample audio features and the initial lip animation data, the lip shape prediction model is adjusted based on the predicted lip shape data and the sample lip shape data. The adjusted model is then determined as the target lip shape prediction model. In this way, the target lip shape prediction model trained in this way can make the generated lip shape animation more stable and accurate when generating lip shape animation based on the speech signal. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating the training method of a lip shape prediction model provided in an embodiment of this application;

[0028] Figure 2 A flowchart illustrating a training method for a lip shape prediction model provided in another embodiment of this application;

[0029] Figure 3 A flowchart illustrating a training method for a lip shape prediction model provided in another embodiment of this application;

[0030] Figure 4 This is a flowchart illustrating a lip-sync animation generation method provided in an embodiment of this application.

[0031] Figure 5 A schematic diagram of the structure of a training device for a lip shape prediction model provided in an embodiment of this application;

[0032] Figure 6 A schematic diagram of the structure of a training device for a lip shape prediction model provided in another embodiment of this application;

[0033] Figure 7 This is a schematic diagram of the structure of a lip-sync animation generation method and lip-sync animation generation device provided in an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0036] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0037] Furthermore, the flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed in order or performed simultaneously. Moreover, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0038] The following explanation, using several specific application examples, illustrates the training method for a lip shape prediction model provided in this application. Figure 1 This application provides a flowchart illustrating a training method for a lip shape prediction model according to an embodiment of the present application. The lip shape prediction model includes a feature extractor and a lip shape generator, such as... Figure 1 As shown, the method includes:

[0039] S101: Sample speech signals from dynamic capture data, as well as sample lip-sync data corresponding to the speech signals.

[0040] In the embodiments of this application, multiple sample lip-sync animations are sampled sequentially to obtain sample speech signals and sample lip-sync data for each sample lip-sync animation; wherein, the multiple sample lip-sync animations are all acquired by dynamic capture.

[0041] S102: Use a feature extractor to extract sample audio features from the sample speech signal.

[0042] The collected sample audio features are multi-dimensional sample audio features. This extraction method can avoid the problem of inaccurate subsequent lip shape prediction due to errors in spectrogram extraction.

[0043] Before using a feature extractor to extract audio features from samples, the feature extractor needs to be adjusted based on a large amount of dynamically captured sample data to obtain an adjusted feature extractor. This method allows the feature extractor to be trained on multilingual speech signals when the dynamically captured sample data includes a large number of multilingual sample speech signals, thereby enhancing the generalization of lip-sync synthesis.

[0044] S103: Employs a lip shape generator to generate predicted lip shape data based on the audio features of the samples and the initial lip animation data of the samples.

[0045] In some possible embodiments, the initial mouth animation data of the sample can be, for example, the initial mouth animation data generated based on the identity and voice emotion information of the person corresponding to the dynamic capture data; its mathematical expression can be, for example, bs init =f initial (x id ,audio emo ), where bs init This is the initial mouth animation data for the sample; that is, the initial mouth animation data for the sample can be combined with the motion capture data to correspond to the character's identity x. id And the emotion of the voice audio emo As can be defined, the identity can be, for example, the age, gender or other identity of the person corresponding to the dynamic capture data, and the emotion can be determined based on the tone of voice of the person corresponding to the dynamic capture data, such as happiness, anger, sadness, etc. It should be understood that the above embodiments are only illustrative examples, and can be flexibly adjusted according to the user's needs, and are not limited to the embodiments given above.

[0046] In this method, the sample audio features and initial mouth animation data are used as inputs to the lip-shape generator. This setup considers not only the sample audio features but also the identity and emotional information of the person corresponding to the data when generating predicted lip-shape data. Therefore, even with the same sample audio features, the predicted lip-shape data obtained by the lip-shape generator may differ due to differences in the emotional information of the initial mouth animation data. For example, with the same sample audio features, the predicted lip-shape data will be different when the emotional information of the initial mouth animation data is happy or angry. For instance, the mouth opening amplitude may be greater when the initial mouth animation data is angry than when it is happy. It should be understood that the above embodiments are merely illustrative, and the relationship between the emotional information of the initial mouth animation data and the predicted lip-shape data can be flexibly adjusted according to user needs and is not limited to the embodiments described above.

[0047] In the embodiments of this application, a prediction model with a Transformer structure is used as the lip shape generator for the predicted lip shape data of the sample audio features. The Transformer model can establish long-distance dependencies. That is, when generating predicted lip shape data based on the Transformer model, the Transformer model will comprehensively consider the contextual information of the sample audio features before generating the predicted lip shape data, thereby making the generated predicted lip shape data more accurate, enhancing the model's ability to model long distances, and enabling it to cope with situations where lip shape changes rapidly when generating predicted lip shape data.

[0048] The predicted lip shape data can include, for example, lip shape parameter data in n dimensions. Each dimension of lip shape parameter data has a clear definition. Taking parameters related to lip shape synthesis as an example: the JawOpen lip shape parameter data only controls the opening of the mouth; the left lip smile (MouthSmile_L) lip shape parameter data controls the smile of the left lip. When both dimensions are set to 50, the left lip can be opened while smiling; when both are set to 100, the amplitude of the above action is larger. The right lip smile (MouthSmile_R) lip shape parameter data controls the smile of the right lip. When both MouthSmile_L and MouthSmile_R are 50, it means that the current smile amplitude of the left and right sides is the same. The specific content of the lip shape parameter data included in the preset lip shape data, as well as the presentation of the content, can be flexibly adjusted according to the user's needs and is not limited to the examples given above.

[0049] S104: Based on the predicted lip shape data and the sample lip shape data, adjust the model parameters of the lip shape prediction model to obtain the target lip shape prediction model.

[0050] The target lip shape prediction model obtained using the above method can make the generated lip shape animation more stable and accurate when it is generated based on the target prediction model.

[0051] The training method of the lip-shape prediction model provided in this application is adopted. After sampling sample speech signals and corresponding sample lip-shape data from dynamic capture data, the sample audio features of the sample speech signals are extracted using a feature extractor. After the lip-shape generator generates predicted lip-shape data based on the sample audio features and the initial lip animation data, the lip-shape prediction model is adjusted based on the predicted lip-shape data and the sample lip-shape data. The adjusted model is determined as the target lip-shape prediction model. In this way, the target lip-shape prediction model trained in this way can make the generated lip-shape animation more stable and accurate when generating lip-shape animation based on speech signals.

[0052] Optionally, based on the above embodiments, this application embodiment may also provide a method for training a lip shape prediction model. The implementation process of obtaining the target lip shape prediction model in the above method is illustrated below with reference to the accompanying drawings. Figure 2 A flowchart illustrating a training method for a lip shape prediction model, as provided in another embodiment of this application, is shown below. Figure 2 As shown, S104 may include:

[0053] S111: Calculate the lip animation position loss function value based on the predicted lip shape data and the sample lip shape data.

[0054] In the embodiments of this application, in order to make the predicted lip shape data output by the lip shape prediction model as close as possible to the sample lip shape data, while ensuring the sparsity of the predicted lip shape data, this application introduces a lip shape animation position loss function to describe the difference between them, and ensures the sparsity of the predicted lip shape data obtained in subsequent use by reducing the difference between them. The specific mathematical form is as follows:

[0055]

[0056] Among them, L pos This refers to the animation position loss function value; bs i To predict lip shape data, The method for determining the animation position loss function value, which is the sample lip-sync data, is to compare the predicted lip-sync data and the sample lip-sync data frame by frame. Based on the loss value of each frame, the loss value of each frame is accumulated, and the accumulated result is determined as the animation position loss value. In this way, by continuously converging the animation position loss function value, it is ensured that the predicted lip-sync data and the sample lip-sync data of each frame are as close as possible, thereby ensuring the accuracy and sparsity of the predicted lip-sync data.

[0057] S112: Using a preset discriminator, the predicted lip shape data and the sample lip shape data are discriminated to obtain the first discrimination result.

[0058] The preset discriminator is used to distinguish between true and false lip shape data in sample lip shape data and predicted lip shape data.

[0059] S113: Based on the first identification result, generate the first overall evaluation loss function value for the lip-sync animation.

[0060] In the embodiments of this application, the overall evaluation loss function for the first lip-sync animation is defined as follows:

[0061] SGAN-G =E x～audio (D(G(x))-c) 2

[0062] Among them, L LSGAN-G Let be the overall evaluation loss function for the first lip-sync animation, where D represents the discriminator, G represents the lip-sync prediction model, and c is a constant 0.

[0063] S114: Calculate the loss function value of the lip-sync prediction model based on the lip-sync animation position loss function value and the overall evaluation loss function value of the first lip-sync animation.

[0064] In some possible embodiments, for example, the lip-sync animation position loss function value and the first lip-sync animation overall evaluation loss function can be directly added together as the lip-sync prediction model loss function value; or the lip-sync animation position loss function value and the first lip-sync animation overall evaluation loss function can be added together according to preset weights as the lip-sync prediction model loss function value. It should be understood that the above embodiments are only illustrative examples, and the specific method for determining the joint loss function value can be flexibly adjusted according to user needs, and is not limited to the methods given in the above embodiments.

[0065] S115: Adjust the model parameters of the lip shape prediction model based on the loss function value of the lip shape prediction model to obtain the target lip shape prediction model.

[0066] In the embodiments of this application, the lip shape prediction model and the discriminator can be converged based on the lip shape prediction model loss function value until the lip shape prediction model loss function value is less than a preset value.

[0067] Optionally, based on the above embodiments, this application embodiment may also provide a training method for a lip shape prediction model, as further illustrated below with reference to the accompanying drawings. Figure 3 A flowchart illustrating a training method for a lip shape prediction model, as provided in another embodiment of this application, is shown below. Figure 3 As shown, the method may further include:

[0068] S121: Based on the first identification result, generate the second overall evaluation loss function value for the lip-sync animation;

[0069] The overall evaluation loss function for the second lip-sync animation is represented by the discriminator loss function, which is expressed as follows:

[0070]

[0071] Among them, L LSGAN-D Let b be the discriminator loss function, and a be a constant of 1.

[0072] S122: Based on the overall evaluation loss function value of the second lip-sync animation, adjust the model parameters of the discriminator to obtain the target discriminator.

[0073] Subsequently, before S115, the method may further include: adjusting the model parameters of the lip shape prediction model based on the loss function value of the lip shape prediction model to obtain the target lip shape prediction model; before that, the method may further include: identifying the predicted lip shape data and the sample lip shape data based on the target discriminator to obtain a second discrimination result; regenerating the first overall evaluation loss function value of the lip shape animation based on the second discrimination result; recalculating the loss function value of the lip shape prediction model based on the lip shape animation position loss function value and the regenerated first overall evaluation loss function value of the lip shape animation; correspondingly, S115 is to adjust the model parameters of the lip shape prediction model based on the recalculated loss function value of the lip shape prediction model to obtain the target lip shape prediction model.

[0074] That is, in one embodiment of this application, during the convergence stage of the lip-shape prediction model and the discriminator, the model parameters of the lip-shape prediction model can be adjusted first based on the first overall evaluation loss function value of the lip-shape animation and the lip-shape animation position loss function. After the model parameters of the lip-shape prediction model converge to near the optimal solution, the second overall evaluation loss function value of the lip-shape animation is generated based on the first discrimination result. Then, the model parameters of the discriminator are adjusted based on the second overall evaluation loss function value of the lip-shape animation to obtain the target discriminator.

[0075] Subsequently, the lip shape prediction model is converged based on the adjusted target discriminator. That is, the adjusted target discriminator is used to re-discriminate the predicted lip shape data and the sample lip shape data to obtain a second discrimination result. Then, the first overall evaluation function value of the lip shape animation is regenerated based on the second discrimination result. Subsequently, the model parameters of the lip shape prediction model are adjusted based on the regenerated first overall evaluation loss function value of the lip shape animation and the lip shape animation position loss function, so that the model parameters of the lip shape prediction model converge to the optimal solution. At this time, after the target discriminator converges again, the resulting lip shape prediction model is the trained target lip shape prediction model.

[0076] This convergence method involves first pre-training the lip shape prediction model, that is, first converging the lip shape prediction model to the optimal solution, and then re-training the lip shape prediction model and the discriminator together.

[0077] In the embodiments of this application, during the joint training process, the discriminator can be optimized and updated first using the gradient descent method. After the discriminator is optimized, the lip shape prediction model is then optimized and updated using the gradient descent method based on the optimized discriminator. This ensures that the optimized discriminator can provide the correct inverse gradient flow when optimizing the lip shape prediction model, further enhancing the prediction accuracy of the lip shape prediction model provided in this application when predicting lip shape data in the future.

[0078] For example, in some other possible embodiments, before S114, the lip-sync animation speed loss function value can be calculated based on the predicted lip-sync data and the sample lip-sync data; then, the joint loss function value can be calculated based on the lip-sync animation position loss function value, the lip-sync animation speed loss function, and the overall lip-sync animation evaluation loss function value.

[0079] In the embodiments of this application, in order to eliminate the jitter in the predicted lip-sync animation generated based on the predicted lip-sync data, a first-order lip-sync animation velocity loss function is introduced, the specific mathematical form of which is as follows:

[0080]

[0081] Among them, L vel This represents the lip-sync animation speed loss function value. The predicted lip-sync data and the sample lip-sync data are used to calculate the lip-sync animation speed loss function value for each frame based on the predicted lip-sync data of the frame preceding each frame of predicted lip-sync data and the sample lip-sync data of the frame preceding each frame of sample lip-sync data. Then, the lip-sync animation speed loss function values ​​of each frame are accumulated to obtain the final lip-sync animation speed loss function value.

[0082] For example, in some other possible embodiments, before S114, the lip-sync animation smoothness loss function value can be calculated based on the predicted lip-sync data and the sample lip-sync data; then, the joint loss function value can be calculated based on the lip-sync animation position loss function value, the lip-sync animation speed loss function, the lip-sync animation smoothness loss function, and the lip-sync animation overall evaluation loss function value.

[0083] In the embodiments of this application, in order to prevent unnatural and sudden changes in the predicted lip-sync animation generated based on the predicted lip-sync data, a second-order lip-sync animation smoothness loss function is also introduced for the predicted lip-sync data, the specific mathematical form of which is as follows:

[0084]

[0085] Among them, L lap bs represents the loss function value for lip-sync smoothness. i To predict lip-sync data, the lip-sync smoothness loss function value for each frame is calculated based on the preceding and following frames of predicted lip-sync data. Then, the lip-sync smoothness loss function values ​​for each frame are accumulated to obtain the final lip-sync smoothness loss function value.

[0086] In the embodiments of this application, when pre-training the lip-shape prediction model, for example, the loss function value of the lip-shape prediction model and the preset weights of each loss function can be calculated based on the lip-shape animation position loss function value, lip-shape animation speed loss function, lip-shape animation smoothness loss function, and first lip-shape animation overall evaluation loss function value.

[0087] In the embodiments of this application, the loss function value of the lip shape prediction model takes the following form:

[0088]

[0089] Among them, L all-G μ represents the loss function value of the lip shape prediction model. pos μ vel μ lap and μ g These are preset coefficients, and are constant values. In one embodiment of this application, μ pos μ vel μ lap and μ g The values ​​are set to 1.0, 0.5, 0.5 and 0.1 respectively. It should be understood that the above embodiments are only illustrative examples. The specific values ​​of the preset coefficients can be flexibly adjusted according to user needs and are not limited to those given in the above embodiments. They can be set to any constant value. This application does not impose any restrictions on them.

[0090] The training method of the lip-shape prediction model provided in this application improves the expressive power of audio features by extracting sample audio features of sample speech signals through a feature extractor. It has a certain generalization ability for different languages. In addition, the lip-shape generator in this application can establish long-distance dependencies, thereby improving the modeling ability of the model. Furthermore, by introducing a discriminator, this application can improve the overall continuity of the lip-shape animation generated based on the predicted lip-shape data by analyzing the generated predicted lip-shape data.

[0091] The following explanation, using several specific application examples, illustrates a lip-syncing animation generation method provided in this application. Figure 4 This is a flowchart illustrating a lip-sync animation generation method according to an embodiment of this application, as shown below. Figure 4 As shown, the method includes:

[0092] S201: Based on the target speech signal and the initial mouth animation data, a mouth shape prediction model is used to generate target mouth shape data.

[0093] The lip shape prediction model uses the above-mentioned... Figures 1-3 The model obtained through training.

[0094] In the embodiments of this application, the initial mouth animation data is generated based on the identity and voice emotion information of the task corresponding to the target voice signal.

[0095] S202: Based on the target speech and target lip-sync data, control the mouth of the preset virtual object to perform lip-sync animation.

[0096] The above method is to use the above Figures 1-3 The trained model performs the same beneficial effects as those given in the above embodiments, and will not be repeated here.

[0097] The training apparatus for the lip shape prediction model provided in this application will be explained below with reference to the accompanying drawings. This training apparatus for the lip shape prediction model can perform the above-described... Figures 1-4 The training method for any lip shape prediction model, its specific implementation, and its beneficial effects are described above and will not be repeated below.

[0098] Figure 5 This is a schematic diagram of the structure of a training device for a lip shape prediction model provided in an embodiment of this application, as shown below. Figure 5 As shown, the device includes: a sampling module 301, an extraction module 302, a generation module 303, and an adjustment module 304, wherein:

[0099] The sampling module 301 is used to sample sample speech signals and corresponding sample lip shape data from the dynamic capture data.

[0100] Extraction module 302 is used to extract sample audio features of sample speech signals using a feature extractor;

[0101] The generation module 303 is used to generate predicted lip shape data based on the audio features of the sample and the initial mouth animation data of the sample using a lip shape generator.

[0102] The adjustment module 304 is used to adjust the model parameters of the lip shape prediction model based on the predicted lip shape data and the sample lip shape data to obtain the target lip shape prediction model.

[0103] Optionally, based on the above embodiments, this application embodiment may also provide a training device for a lip shape prediction model, as described below with reference to the accompanying drawings. Figure 5 The implementation process of the given device is illustrated with examples. Figure 6 This is a schematic diagram of the structure of a training device for a lip shape prediction model provided in another embodiment of this application, as shown below. Figure 6 As shown, the device also includes: a calculation module 305 and an identification module 306, wherein:

[0104] The calculation module 305 is used to calculate the lip animation position loss function value based on the predicted lip shape data and the sample lip shape data;

[0105] The identification module 306 is used to identify the predicted lip shape data and the sample lip shape data using a preset discriminator to obtain a first identification result;

[0106] The generation module 303 is used to generate the first overall evaluation loss function value of the lip-sync animation based on the first identification result;

[0107] The calculation module 305 is specifically used to calculate the loss function value of the lip-shape prediction model based on the positional loss function value of the lip-shape animation and the overall evaluation loss function value of the first lip-shape animation.

[0108] The adjustment module 304 is specifically used to adjust the model parameters of the lip shape prediction model based on the loss function value of the lip shape prediction model, so as to obtain the target lip shape prediction model.

[0109] Optionally, the generation module 303 is specifically used to generate a second overall evaluation loss function value for lip-sync animation based on the first identification result;

[0110] The adjustment module 304 is specifically used to adjust the model parameters of the discriminator based on the overall evaluation loss function value of the second lip-sync animation, so as to obtain the target discriminator.

[0111] Optionally, the discrimination module 306 is specifically used to discriminate the predicted lip shape data and the sample lip shape data according to the target discriminator to obtain a second discrimination result;

[0112] The generation module 303 is specifically used to regenerate the overall evaluation loss function value of the first lip-sync animation based on the second identification result;

[0113] The calculation module 305 is specifically used to recalculate the loss function value of the lip-shape prediction model based on the positional loss function value of the lip-shape animation and the overall evaluation loss function value of the regenerated first lip-shape animation.

[0114] The adjustment module 304 is specifically used to adjust the model parameters of the lip shape prediction model based on the recalculated loss function value of the lip shape prediction model, so as to obtain the target lip shape prediction model.

[0115] Optionally, the calculation module 305 is specifically used to calculate the lip-shape animation speed loss function value based on the predicted lip-shape data and the sample lip-shape data; and to calculate the lip-shape prediction model loss function value based on the lip-shape animation position loss function value, the lip-shape animation speed loss function, and the first lip-shape animation overall evaluation loss function value.

[0116] Optionally, the calculation module 305 is specifically used to calculate the lip animation smoothness loss function value based on the predicted lip shape data and the sample lip shape data; and to calculate the lip shape prediction model loss function value based on the lip shape animation position loss function value, the lip shape animation speed loss function, the lip shape animation smoothness loss function, and the first lip shape animation overall evaluation loss function value.

[0117] Optionally, the generation module 303 is specifically used to generate initial mouth animation data for the sample based on the identity and voice emotion information of the person corresponding to the dynamic capture data.

[0118] The lip-sync animation generation apparatus provided in this application will be explained below with reference to the accompanying drawings. This lip-sync animation generation apparatus can perform the above-described... Figure 5 The specific implementation and beneficial effects of the lip-sync animation generation method are described above and will not be repeated below.

[0119] Figure 7 This is a schematic diagram of the structure of a lip-sync animation generation device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes: a generation module 401 and a control module 402, wherein:

[0120] The generation module 401 is used to generate target lip shape data based on the target speech signal and the initial mouth animation data, using a lip shape prediction model;

[0121] The control module 402 is used to control the mouth of a preset virtual object to perform lip-sync animation based on the target speech and target lip-sync data.

[0122] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0123] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0124] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be integrated into a terminal device or a chip of a terminal device.

[0125] like Figure 8 As shown, the electronic device includes: a processor 501, a storage medium 502, and a bus 503.

[0126] Processor 501 is used to store programs, and processor 501 calls the programs stored in storage medium 502 to execute the above-mentioned programs. Figures 1-4 The corresponding method implementation is similar in both implementation and technical effect, and will not be described in detail here.

[0127] Optionally, this application also provides a program product, such as a storage medium storing a computer program, including a program that executes the embodiments corresponding to the above-described methods when run by a processor.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0131] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A training method for a lip shape prediction model, characterized in that, The lip shape prediction model includes a feature extractor and a lip shape generator, and the method includes: Sample speech signals and corresponding lip-sync data are collected from the dynamic capture data. The feature extractor is used to extract sample audio features from the sample speech signal; Using the lip-shape generator, predicted lip-shape data is generated based on the sample audio features and the initial lip animation data of the sample; wherein, the initial lip animation data of the sample is generated based on the identity and voice emotion information of the person corresponding to the dynamic capture data, and the voice emotion information is determined based on the tone of voice of the person corresponding to the dynamic capture data; Based on the predicted lip shape data and the sample lip shape data, the model parameters of the lip shape prediction model are adjusted to obtain the target lip shape prediction model; The step of adjusting the model parameters of the lip shape prediction model based on the predicted lip shape data and the sample lip shape data to obtain the target lip shape prediction model includes: Calculate the lip-shape animation position loss function value based on the predicted lip-shape data and the sample lip-shape data; A preset discriminator is used to distinguish between the predicted lip shape data and the sample lip shape data to obtain a first discrimination result; Based on the first identification result, generate the overall evaluation loss function value of the first lip-sync animation; Based on the lip-shape animation position loss function value and the first lip-shape animation overall evaluation loss function value, calculate the lip-shape prediction model loss function value; Based on the loss function value of the lip shape prediction model, the model parameters of the lip shape prediction model are adjusted to obtain the target lip shape prediction model.

2. The method as described in claim 1, characterized in that, The method further includes: Based on the first identification result, a second overall evaluation loss function value for the lip-sync animation is generated; Based on the overall evaluation loss function value of the second lip-sync animation, the model parameters of the discriminator are adjusted to obtain the target discriminator.

3. The method as described in claim 2, characterized in that, Before adjusting the model parameters of the lip shape prediction model based on the loss function value of the lip shape prediction model to obtain the target lip shape prediction model, the method further includes: The predicted lip shape data and the sample lip shape data are identified by the target discriminator to obtain a second identification result; Based on the second identification result, regenerate the overall evaluation loss function value of the first lip-sync animation; Based on the lip-shape animation position loss function value and the overall evaluation loss function value of the regenerated first lip-shape animation, the lip-shape prediction model loss function value is recalculated. The step of adjusting the model parameters of the lip shape prediction model based on the loss function value of the lip shape prediction model to obtain the target lip shape prediction model includes: Based on the recalculated loss function value of the lip shape prediction model, the model parameters of the lip shape prediction model are adjusted to obtain the target lip shape prediction model.

4. The method as described in claim 1, characterized in that, Before calculating the loss function value of the lip-shape prediction model based on the lip-shape animation position loss function value and the overall evaluation loss function value of the first lip-shape animation, the method further includes: Calculate the lip-sync animation speed loss function value based on the predicted lip-sync data and the sample lip-sync data; The step of calculating the loss function value of the lip-shape prediction model based on the lip-shape animation position loss function value and the overall evaluation loss function value of the first lip-shape animation includes: The loss function value of the lip-shape prediction model is calculated based on the lip-shape animation position loss function value, the lip-shape animation velocity loss function value, and the first lip-shape animation overall evaluation loss function value.

5. The method as described in claim 4, characterized in that, Before calculating the loss function value of the lip-shape prediction model based on the lip-shape animation position loss function value and the overall evaluation loss function value of the first lip-shape animation, the method further includes: Calculate the lip animation smoothness loss function value based on the predicted lip shape data and the sample lip shape data; The step of calculating the loss function value of the lip-shape prediction model based on the lip-shape animation position loss function value and the overall evaluation loss function value of the first lip-shape animation includes: The loss function value of the lip-sync prediction model is calculated based on the lip-sync animation position loss function value, the lip-sync animation speed loss function, the lip-sync animation smoothness loss function, and the first lip-sync animation overall evaluation loss function value.

6. A method for generating lip-sync animation, characterized in that, The method includes: The lip shape prediction model is trained using the method described in any one of claims 1-5; Based on the target speech signal and the initial mouth animation data, the lip shape prediction model is used to generate target lip shape data; Based on the target speech and the target lip-sync data, control the mouth of the preset virtual object to perform lip-sync animation.

7. A training device for a lip shape prediction model, characterized in that, The device includes: a sampling module, an extraction module, a generation module, and an adjustment module, wherein: The sampling module is used to sample sample speech signals and sample lip shape data corresponding to the sample speech signals from the dynamically captured data; The extraction module is used to extract sample audio features of the sample speech signal using a feature extractor; The generation module is used to generate predicted lip shape data by employing a lip shape generator based on the sample audio features and the initial lip animation data of the sample; wherein, the initial lip animation data of the sample is generated based on the identity and voice emotion information of the person corresponding to the dynamic capture data, and the emotion information is determined based on the tone of voice of the person corresponding to the dynamic capture data. The adjustment module is used to adjust the model parameters of the mouth shape prediction model according to the predicted mouth shape data and the sample mouth shape data to obtain the target mouth shape prediction model. The adjustment module is specifically used for: Calculate the lip-shape animation position loss function value based on the predicted lip-shape data and the sample lip-shape data; A preset discriminator is used to distinguish between the predicted lip shape data and the sample lip shape data to obtain a first discrimination result; Based on the first identification result, generate the overall evaluation loss function value of the first lip-sync animation; Based on the lip-shape animation position loss function value and the first lip-shape animation overall evaluation loss function value, calculate the lip-shape prediction model loss function value; Based on the loss function value of the lip shape prediction model, the model parameters of the lip shape prediction model are adjusted to obtain the target lip shape prediction model.

8. A lip-sync animation generation device, characterized in that, The apparatus includes: a generation module, a control module, and a training device for the lip shape prediction model as described in claim 7, wherein: The generation module is used to generate target lip shape data based on the target speech signal and initial mouth animation data, using a lip shape prediction model; wherein, the lip shape prediction model is a model trained using the training device described above. The control module is used to control the mouth of a preset virtual object to perform lip-sync animation based on the target speech and the target lip-sync data.

9. An electronic device, characterized in that, The device includes a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the method described in any one of claims 1-6.

Images