Mouth shape animation generation method and device and storage medium

Abstract

The invention provides a mouth shape animation generation method and device and a storage medium, and relates to the technical field of videos, and the method comprises the steps: obtaining the audio features of the speaking audio of a target object; image coding features of a target object image are obtained, and the target object image comprises the face of a target object; acquiring target object emotion information; the audio features, the image coding features and the target object emotion information are input into a mouth shape animation generation model to obtain a mouth shape animation, and the mouth shape of the target object in the mouth shape animation is matched with the target object speaking audio and the target object emotion information. By adopting the method and the device, the generated mouth shape animation can be more expressive.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to a method, device and storage medium for generating lip animation. Background Art

[0002] With the rapid development of artificial intelligence, neural network models, as an important means of artificial intelligence, have been widely used in various fields. In the field of audio and video, neural network models are usually used to generate a video that matches the target object's lip shape and audio based on the target object's image and human voice audio. This video is also called lip animation.

[0003] In the related art, after feature extraction of the target object image and human voice audio, the features are input into a neural network model, and the neural network model outputs the corresponding lip animation. The lip animation obtained in this way is not realistic and lacks expressiveness. Summary of the invention

[0004] The embodiment of the present application provides a method, device and storage medium for generating lip animation, which can make the generated lip animation more realistic and expressive. The technical solution is as follows:

[0005] In a first aspect, a method for generating a lip animation is provided, the method comprising:

[0006] Obtaining audio features of the target object's speaking audio;

[0007] Acquire image coding features of a target object image, wherein the target object image includes a face of the target object;

[0008] Obtain emotional information of the target object;

[0009] The audio features, the image coding features and the target object emotional information are input into a lip animation generation model to obtain a lip animation, in which the lip shape of the target object matches the target object speaking audio and the target object emotional information.

[0010] In a possible implementation, obtaining the target object's emotion information includes:

[0011] Get the target object's emotional text entered by the user account;

[0012] The target object emotion text is input into a text encoder to obtain text features of the target object emotion text, and the text features are used as the target object emotion information.

[0013] In a possible implementation, obtaining the target object's emotion information includes:

[0014] Obtaining a pleasure value corresponding to a pleasure dimension and an activation value corresponding to an activation dimension input by a user account, wherein the pleasure dimension and the activation dimension are dimensions used to quantify emotions;

[0015] Generate a target object emotion text based on the pleasure value and the activation value;

[0016] The target object emotion text is input into a text encoder to obtain text features of the target object emotion text, and the text features are used as the target object emotion information.

[0017] In a possible implementation, obtaining audio features of the target object's speech audio includes:

[0018] The target object speech audio is input into an audio coding model to obtain audio features of the target object speech audio, wherein the audio coding model is trained based on sample human voice audio of at least one language.

[0019] In a possible implementation, the method further includes:

[0020] Generate a target object picture sequence based on the number of audio frames of the target object's speaking audio and the target object picture;

[0021] The step of acquiring the image coding features of the target object image comprises:

[0022] Obtain image coding features of each target object picture in the target object picture sequence.

[0023] In a possible implementation, the method further includes:

[0024] Obtaining target mouth area information of the target object face in the target object picture;

[0025] The step of inputting the audio features, the image coding features and the target object emotion information into a lip animation generation model to obtain a lip animation comprises:

[0026] The audio features, the image coding features, the target mouth area information and the target object emotion information are input into a lip animation generation model to obtain a lip animation.

[0027] In a possible implementation, the acquiring target mouth region information of the target object face in the target object picture includes:

[0028] Extracting facial key points of the target object in the target object image to obtain a plurality of target facial key points;

[0029] Calculating the mouth length of the target object based on the target mouth corner point among the multiple facial key points;

[0030] Determine a standard mouth corner point and a standard nose bottom point of the target object face in the target object picture in a standard posture based on the mouth length, the target mouth corner point and the target nose bottom point among the multiple target facial key points;

[0031] Determine an affine transformation matrix based on the target mouth corner point, the target nose bottom point, the standard mouth corner point, and the standard nose bottom point;

[0032] Based on the affine transformation matrix, the plurality of target facial key points are mapped to obtain standard facial key points corresponding to the plurality of target facial key points;

[0033] Determine standard mouth region information of the target object face in the target object image in a standard posture based on a standard cheek corner point, a standard eye outer point, a standard ear middle point, and a standard chin center point among the standard facial key points;

[0034] Based on the affine transformation matrix, the standard mouth region information is mapped to obtain the target mouth region information of the target object face in the target object picture.

[0035] In a possible implementation, the method further includes:

[0036] Get the subject's speaking video;

[0037] intercepting a speaking video sample of the subject in the speaking video of the subject, and obtaining an image sample of the subject, wherein the image of the subject includes the face of the subject;

[0038] Obtaining a sample of the subject's emotion information in the subject's speaking video sample;

[0039] Obtain an audio sample of the subject speaking corresponding to the video sample of the subject speaking;

[0040] Acquire audio features corresponding to the audio samples of the subject's speech and image coding features of the image samples of the subject;

[0041] Based on the object emotion information sample, the audio features corresponding to the object speech audio sample, the image coding features of the object image sample and the object speech video sample, the initial lip animation generation model is trained to obtain the lip animation generation model.

[0042] In a second aspect, a device for generating lip animation is provided, the device comprising:

[0043] An acquisition module is used to acquire audio features of the target object's speech audio; acquire image coding features of the target object's image, wherein the target object's image includes the target object's face; and acquire emotional information of the target object;

[0044] The generation module is used to input the audio features, the image coding features and the target object emotional information into a lip animation generation model to obtain a lip animation, in which the lip shape of the target object matches the target object speaking audio and the target object emotional information.

[0045] In a possible implementation, the acquisition module is used to:

[0046] Get the target object's emotional text entered by the user account;

[0047] The target object emotion text is input into a text encoder to obtain text features of the target object emotion text, and the text features are used as the target object emotion information.

[0048] In a possible implementation, the acquisition module is used to:

[0049] Obtaining a pleasure value corresponding to a pleasure dimension and an activation value corresponding to an activation dimension input by a user account, wherein the pleasure dimension and the activation dimension are dimensions used to quantify emotions;

[0050] Generate a target object emotion text based on the pleasure value and the activation value;

[0051] The target object emotion text is input into a text encoder to obtain text features of the target object emotion text, and the text features are used as the target object emotion information.

[0052] In a possible implementation, the acquisition module is used to:

[0053] The target object speech audio is input into an audio coding model to obtain audio features of the target object speech audio, wherein the audio coding model is trained based on sample human voice audio of at least one language.

[0054] In a possible implementation, the acquisition module is used to:

[0055] Generate a target object picture sequence based on the number of audio frames of the target object's speaking audio and the target object picture;

[0056] Obtain image coding features of each target object picture in the target object picture sequence.

[0057] In a possible implementation, the acquisition module is further used to:

[0058] Obtaining target mouth area information of the target object face in the target object picture;

[0059] The generating module is used for:

[0060] The audio features, the image coding features, the target mouth area information and the target object emotion information are input into a lip animation generation model to obtain a lip animation.

[0061] In a possible implementation, the acquisition module is used to:

[0062] Extracting facial key points of the target object in the target object image to obtain a plurality of target facial key points;

[0063] Calculating the mouth length of the target object based on the target mouth corner point among the multiple facial key points;

[0064] Determine a standard mouth corner point and a standard nose bottom point of the target object face in the target object picture in a standard posture based on the mouth length, the target mouth corner point and the target nose bottom point among the multiple target facial key points;

[0065] Determine an affine transformation matrix based on the target mouth corner point, the target nose bottom point, the standard mouth corner point, and the standard nose bottom point;

[0066] Based on the affine transformation matrix, the plurality of target facial key points are mapped to obtain standard facial key points corresponding to the plurality of target facial key points;

[0067] Determine standard mouth region information of the target object face in the target object image in a standard posture based on a standard cheek corner point, a standard eye outer point, a standard ear middle point, and a standard chin center point among the standard facial key points;

[0068] Based on the affine transformation matrix, the standard mouth region information is mapped to obtain the target mouth region information of the target object face in the target object picture.

[0069] In a possible implementation, the apparatus further includes a training module, configured to:

[0070] Get the subject's speaking video;

[0071] intercepting a speaking video sample of the subject in the speaking video of the subject, and obtaining an image sample of the subject, wherein the image of the subject includes the face of the subject;

[0072] Obtaining a sample of the subject's emotion information in the subject's speaking video sample;

[0073] Obtain an audio sample of the subject speaking corresponding to the video sample of the subject speaking;

[0074] Acquire audio features corresponding to the audio samples of the subject's speech and image coding features of the image samples of the subject;

[0075] Based on the object emotion information sample, the audio features corresponding to the object speech audio sample, the image coding features of the object image sample and the object speech video sample, the initial lip animation generation model is trained to obtain the lip animation generation model.

[0076] According to a third aspect, a terminal is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the method for generating lip animation according to the first aspect and any possible implementation of the first aspect.

[0077] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to implement the operations performed by the method for generating lip animation as described in the first aspect and any possible implementation of the first aspect.

[0078] In a fifth aspect, a computer program product is provided, wherein at least one instruction is stored in the computer program product, and the instruction is loaded and executed by a processor to implement the operations performed by the method for generating lip animation as described in the first aspect and any possible implementation of the first aspect.

[0079] The beneficial effects of the technical solution provided by this application are:

[0080] In the technical solution provided in the present application, the emotional information of the target object is added to the input of the lip-sync animation generation model. When the object says the same words under different emotions, the lip shapes are also different. Therefore, in the present application, by introducing the emotional information of the target object, the target object in the lip-sync animation generated by the lip-sync animation generation model has emotional representation, which is more realistic and expressive. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 is a flow chart of a method for generating lip animation provided in an embodiment of the present application;

[0083] Figure 2 This is a schematic diagram of the corresponding relationship between activation, pleasure and emotion provided in an embodiment of the present application;

[0084] Figure 3 is a schematic diagram of key points of a face provided in an embodiment of the present application;

[0085] Figure 4 is a schematic diagram of a target object picture provided in an embodiment of the present application;

[0086] Figure 5 is a schematic diagram of a mouth area provided in an embodiment of the present application;

[0087] Figure 6 It is a schematic diagram of the structure of a device for generating lip animation provided in an embodiment of the present application;

[0088] Figure 7 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0089] Figure 8 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0091] An embodiment of the present application provides a method for generating lip animation, which can be implemented by a computing device. The computing device can be a terminal or a server. The terminal can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, etc., and the server can be a single server, a server cluster, etc.

[0092] The implementation scenario of the lip animation generation method provided by the embodiment of the present application is described below. In a possible implementation scenario, a user wants to generate a video (lip animation) in which a specified target object is speaking a specified speech. In this case, the user can input a target object picture and a human voice audio containing the specified target object to the terminal, wherein the target object picture contains the face of the specified target object, the specified target object can be a real person, a virtual person, etc., and the target object speech audio is an audio containing the above-mentioned specified speech, and the specified speech can be Chinese, Japanese, English, French, German, etc. Then, the terminal generates a video indicating that the specified target object is speaking the specified speech through a lip animation generation model, or the terminal sends the target object picture and the target object speech audio of the specified target object to a server, and the server generates a video indicating that the specified target object is speaking the specified speech through a lip animation generation model. However, in the related art, the target object has no emotional representation in the generated lip animation, and such lip animation has poor expressiveness. Based on this, in the lip-sync animation generation method provided in the embodiment of the present application, the target object's emotional information is added to the input of the lip-sync animation generation model. By introducing the target object's emotional information, the target object in the lip-sync animation generated by the lip-sync animation generation model has emotional representation, which is more realistic and expressive.

[0093] The following is a description of the method for generating lip animation provided in an embodiment of the present application. The method can be implemented by a computing device. Figure 1 , the processing flow of the method may include the following steps:

[0094] Step 101: Acquire audio features of the target object's speech audio.

[0095] In implementation, the user can input the target object's speech audio into the computing device according to actual needs. Then, the computing device extracts audio features from the target object's speech audio to obtain audio features. There are many algorithms for audio feature extraction, such as MFCC (Mel-scale Frequency Cepstral Coefficients) algorithm, audio coding model, etc. The embodiment of the present application does not limit the specific audio feature extraction method. The audio coding model is obtained by training the speech audio of a sample object in at least one language.

[0096] In a possible implementation, in order to better identify various minority languages, a Whisper model (universal language model) can be used to extract audio features. Specifically, the target object's speaking audio can be input into the Whisper model, and the Whisper model outputs the audio features of the target object's speaking audio. Among them, audio features may include features such as pitch, rhythm, and intensity, which are compressed representations of the target object's speaking audio. The dimension of the audio feature can be configured according to actual needs. For example, it can be configured as [50*t, 384], where 50 means that every 1s of audio is converted into 50 audio feature points, t represents the duration of the target object's speaking audio, and 384 is the feature dimension.

[0097] Step 102: Obtain image coding features of the target object image.

[0098] In implementation, the user can input a target object picture to the computing device through a user account according to actual needs, wherein the target object picture may include the face of the target object. Then, the computing device generates a target object picture sequence based on the number of audio frames of the target object's speaking audio and the target object picture. Specifically, considering that the target object's lip shape may be affected by the front and back parts of the audio, the N frames of the target object picture are matched with the 2N-M, ... 2N, ... 2M+M frames of audio, where N and M are positive integers. Based on this correspondence, a sliding window operation is performed on the target object picture input by the user to generate a target object picture sequence, and each picture in the target object picture sequence is the above-mentioned target object picture.

[0099] After obtaining the target object image sequence, image coding processing is performed on the target object image sequence in the target object image sequence to obtain image coding features of each target object image. Specifically, the target object image can be input into a VAE (Variational Autoencoder) model, and the VAE model outputs the corresponding image coding features.

[0100] Step 103: Obtain emotional information of the target object.

[0101] In implementation, the target object emotion information is used to indicate the emotion of the target object, and there are many ways to obtain it, and several of them are listed below for illustration:

[0102] Method 1:

[0103] A text input box for the target object's emotion is provided to the user through a computing device, and the user can enter the target object's emotion text in the text input box. Then, the computing device obtains the target object's emotion text input by the user, inputs the obtained target object's emotion text into a text encoder, and obtains the text features of the target object's emotion text as the target object's emotion information. The text encoder can be a text encoder in a CLIP (Contrastive Language-Image Pre-training, multimodal pre-training) model.

[0104] Method 2:

[0105] An input box for a pleasure value corresponding to the pleasure dimension and an activation value corresponding to the activation dimension is provided to the user through a computing device. The pleasure dimension and the activation dimension are dimensions used to quantify emotions. The user can enter the pleasure value and the activation value in the input box through the user account, wherein the value range of the pleasure value and the activation value can both be -1 to 1, wherein the pleasure value is used to indicate the positive or negative attribute of the emotion, and the activation value is used to indicate the intensity of the emotion. Then, the computing device obtains the pleasure value and the activation value input by the user account, and then generates the target object emotion text based on the correspondence between the pleasure value, the activation value and the emotion. The correspondence between the pleasure value, the activation value and the emotion can also be called a VA (Valence-Arousal) model, a VA mapping diagram, etc. See Figure 2 , showing a correspondence between a pleasure value, an activation value and an emotion. Then, the computing device inputs the acquired target object emotion text into a text encoder to obtain text features of the target object emotion text as the target object emotion information. The text encoder may be a text encoder in the CLIP model.

[0106] Step 104: input the audio features, image coding features and target object emotion information into a lip animation generation model to obtain a lip animation.

[0107] In implementation, after obtaining the audio features of the target object's speech audio, the image coding features of the target object's image, and the target object's emotional information, the audio features, the image coding features of the target object's image, and the target object's emotional information are input into a pre-trained lip animation generation model, which performs feature fusion on the human voice audio features, the image coding features of the target object's image, and the target object's emotional information, and outputs a lip animation. In the lip animation, the target object's lip shape matches the emotion indicated by the target object's speech audio and the target object's emotional information.

[0108] The above-mentioned lip animation generation model can be trained by the following method: obtain a video of an object speaking, intercept a video sample of the object speaking in the video of the object speaking, and obtain an image sample of the object. Among them, the image sample of the object may not be included in the video sample of the object speaking, and the face of the object is included in the image of the object. Then, obtain the object emotion information sample in the video sample of the object speaking. Obtain the object speaking audio sample corresponding to the object speaking video sample. Obtain the audio features corresponding to the object speaking audio sample and the image coding features of the object image sample. Then, input the object emotion information sample, the audio features corresponding to the object speaking audio sample, and the image coding features of the object image sample into the lip animation generation model to be trained to obtain the predicted lip animation. Finally, calculate the loss value of the predicted lip animation and the object speaking video sample, and adjust the parameters of the initial lip animation generation model according to the loss value until the conditions are met, stop training, and obtain the above-mentioned lip animation generation model.

[0109] In a possible implementation, the above-mentioned lip animation generation model may include Backbone Net and Reference Net. The image coding features of the above-mentioned target object image are input into Reference Net, and the audio features and target object emotional information are input into Backbone Net. After Reference Net processes the image coding features of the target object image, they are input into Backbone Net for feature fusion, and finally the lip animation is output.

[0110] In a possible implementation, in order to more accurately adjust the target object's lip shape, the target object's mouth area can be located to obtain the target mouth area information of the target object's face in the target object image. Accordingly, the above step 104 is processed as follows: input the audio features, image coding features, target mouth area information and target object emotion information into the lip shape animation generation model to obtain the lip shape animation. The target mouth area information is also input into Backbone Net.

[0111] The following is an explanation of how to obtain the target mouth area information of the target object face in the target object image:

[0112] First, extract the facial key points of the target object face in the target object image to obtain multiple target facial key points. Specifically, the DW pose (Effective Whole-body Pose Estimation with Two-stages Distillation) algorithm can be used to extract 68 target facial key points of the target object face in the target object image. Figure 3, showing a schematic diagram of the locations of 68 facial key points on a face.

[0113] Then, the coordinates of the left corner of the target mouth, the right corner of the target mouth, and the bottom of the target nose are obtained. Figure 3 For example, the left corner of the target mouth is the key point of the face numbered 49, and its coordinates can be recorded as: [x_49, y_49], [x_55, y_55], the bottom of the target nose is the key point of the face numbered 34, and its coordinates can be recorded as: [x_34, y_34], and the right corner of the target mouth is the key point of the face numbered 55, and its coordinates can be recorded as: [x_55, y_55]. It should be noted that the coordinates here are the coordinates of the coordinate system established with the upper left corner of the image as the origin, the horizontal side as the x-axis, and the vertical side as the y-axis of the two adjacent sides of the upper left corner.

[0114] Then, based on the coordinates of the left corner point of the target mouth and the coordinates of the right corner point of the target mouth, the mouth length of the target object is calculated. Specifically, the calculation formula can be as follows:

[0115] L=sqrt((x_49-x_55)^2+(y_49-x_55)^2)

[0116] Wherein, L is the mouth length of the target object, and sqrt() means the square root of ().

[0117] Next, based on the mouth length, the coordinates of the target mouth left corner, the coordinates of the target mouth right corner and the target nose bottom point, determine the coordinates of the standard mouth left corner, the coordinates of the standard mouth right corner and the standard nose bottom point of the target object face in the target object image in the standard pose.

[0118] Among them, the coordinates of the left corner of the standard mouth can be: [(x_49+x_55) / 2+L / 2, y_49];

[0119] The coordinates of the right corner of the standard mouth can be: [(x_49+x_55) / 2+L / 2, y_55];

[0120] The coordinates of the standard nose bottom point can be: [x_34, y_34].

[0121] See also Figure 4 , showing the original target object image and the target object face in the standard pose.

[0122] Then, based on the coordinates of the left corner of the target mouth, the right corner of the target mouth, the bottom of the target nose, and the left corner of the standard mouth, the right corner of the standard mouth, and the bottom of the standard nose, the affine transformation matrix is ​​determined. The affine transformation matrix includes a rotation matrix and a translation matrix. The specific calculation method belongs to the means in the related art and will not be repeated in this application.

[0123] Then, based on the affine transformation matrix, multiple target facial key points are mapped (rotated, translated) to obtain standard facial key points corresponding to the multiple target facial key points. Specifically, the coordinates of each target facial key point are multiplied by the affine transformation matrix to obtain the coordinates of the corresponding standard facial key point.

[0124] Next, based on the coordinates of the upper left corner of the standard cheek, the upper right corner of the standard cheek, the lower outer point of the standard left eye, the lower outer point of the standard right eye, the middle point of the standard left ear, the middle point of the standard right ear and the center point of the standard chin among the standard facial key points, the standard mouth area information of the target object face in the target object image in the standard posture is determined.

[0125] Specific, combined Figure 3 The 68 facial key points shown in the figure are numbered, and the coordinates of the upper left corner of the standard cheek are marked as: [x_1', y_1'], the coordinates of the upper right corner of the standard cheek are marked as: [x_17', y_17'], the coordinates of the lower outer point of the standard left eye are marked as: [x_42', y_42'], the coordinates of the lower outer point of the standard right eye are marked as: [x_48', y_48'], the coordinates of the middle point of the standard left ear are marked as: [x_2', y_2'], the coordinates of the middle point of the standard right ear are marked as: [x_16', y_16'], and the coordinates of the center point of the standard chin are marked as: [x_9', y_9'].

[0126] The standard facial key point corresponding to the minimum value of y_1', y_42', y_48', and y_17' is determined as the upper boundary point of the mouth area, [x_9', y_9'], i.e. the standard chin center point, is determined as the lower boundary point of the mouth area, [x_2', y_2'], i.e. the standard left ear middle point, is determined as the left boundary point of the mouth area, and [x_16', y_16'], i.e. the standard right ear middle point, is determined as the right boundary point of the mouth area. The rectangle passing through the above four boundary points is used as the standard mouth area of ​​the target object face in the target object image in the standard posture, such as Figure 5 shown.

[0127] Then, based on the affine transformation matrix, the standard mouth area is mapped (rotated, translated) to obtain the target mouth area of ​​the target object face in the target object image, such as Figure 5The position information of the target mouth area, i.e., the target mouth area information, can be represented by the positions of four corner points. The target mouth area can be called a mask.

[0128] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0129] In the technical solution provided in the embodiment of the present application, the target object's emotional information is added to the input of the lip animation generation model. When a person says the same words under different emotions, the lip shape is also different. Therefore, in the present application, by introducing the target object's emotional information, the target object in the lip animation generated by the lip animation generation model has emotional representation, which is more realistic and expressive. In addition, in the technical solution provided in the embodiment of the present application, the target mouth area information is introduced, so that the lip animation generation model uses the mouth as the area of ​​interest, and the lip shape can be adjusted more accurately, and the lip shape and audio in the generated lip animation are more matched.

[0130] Based on the same technical concept, the embodiment of the present application also provides a device for generating lip animation, which can be a computing device, such as Figure 6 As shown, the device includes an acquisition module 510 and a generation module 520:

[0131] The acquisition module 510 is used to acquire audio features of the target object's speech audio; acquire image coding features of the target object's image, wherein the target object's image includes the target object's face; and acquire emotional information of the target object;

[0132] The generation module 520 is used to input the audio features, the image coding features and the target object emotional information into a lip animation generation model to obtain a lip animation, in which the lip shape of the target object matches the target object speaking audio and the target object emotional information.

[0133] In a possible implementation, the acquisition module 520 is used to:

[0134] Get the target object's emotional text entered by the user account;

[0135] The target object emotion text is input into a text encoder to obtain text features of the target object emotion text, and the text features are used as the target object emotion information.

[0136] In a possible implementation, the acquisition module 520 is used to:

[0137] Obtaining a pleasure value corresponding to a pleasure dimension and an activation value corresponding to an activation dimension input by a user account, wherein the pleasure dimension and the activation dimension are dimensions used to quantify emotions;

[0138] Generate a target object emotion text based on the pleasure value and the activation value;

[0139] The target object emotion text is input into a text encoder to obtain text features of the target object emotion text, and the text features are used as the target object emotion information.

[0140] In a possible implementation, the acquisition module 520 is used to:

[0141] The target object speech audio is input into an audio coding model to obtain audio features of the target object speech audio, wherein the audio coding model is trained based on sample human voice audio of at least one language.

[0142] In a possible implementation, the acquisition module 520 is used to:

[0143] Generate a target object picture sequence based on the number of audio frames of the target object's speaking audio and the target object picture;

[0144] Obtain image coding features of each target object picture in the target object picture sequence.

[0145] In a possible implementation, the acquisition module 520 is further configured to:

[0146] Obtaining target mouth area information of the target object face in the target object picture;

[0147] The generating module 520 is used for:

[0148] The audio features, the image coding features, the target mouth area information and the target object emotion information are input into a lip animation generation model to obtain a lip animation.

[0149] In a possible implementation, the acquisition module 510 is used to:

[0150] Extracting facial key points of the target object in the target object image to obtain a plurality of target facial key points;

[0151] Calculating the mouth length of the target object based on the target mouth corner point among the multiple facial key points;

[0152] Determine a standard mouth corner point and a standard nose bottom point of the target object face in the target object picture in a standard posture based on the mouth length, the target mouth corner point and the target nose bottom point among the multiple target facial key points;

[0153] Determine an affine transformation matrix based on the target mouth corner point, the target nose bottom point, the standard mouth corner point, and the standard nose bottom point;

[0154] Based on the affine transformation matrix, the plurality of target facial key points are mapped to obtain standard facial key points corresponding to the plurality of target facial key points;

[0155] Determine standard mouth region information of the target object face in the target object image in a standard posture based on a standard cheek corner point, a standard eye outer point, a standard ear middle point, and a standard chin center point among the standard facial key points;

[0156] Based on the affine transformation matrix, the standard mouth region information is mapped to obtain the target mouth region information of the target object face in the target object picture.

[0157] In a possible implementation, the apparatus further includes a training module, configured to:

[0158] Get the subject's speaking video;

[0159] intercepting a speaking video sample of the subject in the speaking video of the subject, and obtaining an image sample of the subject, wherein the image of the subject includes the face of the subject;

[0160] Obtaining a sample of the subject's emotion information in the subject's speaking video sample;

[0161] Obtain an audio sample of the subject speaking corresponding to the video sample of the subject speaking;

[0162] Acquire audio features corresponding to the audio samples of the subject's speech and image coding features of the image samples of the subject;

[0163] Based on the object emotion information sample, the audio features corresponding to the object speech audio sample, the image coding features of the object image sample and the object speech video sample, the initial lip animation generation model is trained to obtain the lip animation generation model.

[0164] In the technical solution provided in the embodiment of the present application, the emotional information of the target object is added to the input of the lip-sync animation generation model. When a person says the same words under different emotions, the lip shape is also different. Therefore, in the present application, by introducing the emotional information of the target object, the target object in the lip-sync animation generated by the lip-sync animation generation model has emotional representation, which is more realistic and expressive.

[0165] It should be noted that: the device for generating lip animation provided in the above embodiment only uses the division of the above functional modules as an example when generating lip animation. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computing device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device for generating lip animation provided in the above embodiment and the method embodiment for generating lip animation belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0166] Figure 7 The structure block diagram of a terminal 600 provided by an exemplary embodiment of the present application is shown. The terminal 600 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 600 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0167] Typically, the terminal 600 includes a processor 601 and a memory 602 .

[0168] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0169] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices.

[0170] In some embodiments, the terminal 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608 and a power supply 609.

[0171] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0172] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0173] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, set on the front panel of the terminal 600; in other embodiments, the display screen 605 can be at least two, respectively set on different surfaces of the terminal 600 or in a folding design; in other embodiments, the display screen 605 can be a flexible display screen, set on the curved surface or folding surface of the terminal 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0174] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0175] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0176] The positioning component 608 is used to locate the current geographical location of the terminal 600 to implement navigation or LBS (Location Based Service). The positioning component 608 can be a positioning component based on the GPS (Global Positioning System), Beidou system or Galileo system.

[0177] The power supply 609 is used to power various components in the terminal 600. The power supply 609 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0178] In some embodiments, the terminal 600 further includes one or more sensors 610 , including but not limited to: an acceleration sensor 611 , a gyroscope sensor 612 , a pressure sensor 613 , a fingerprint sensor 614 , an optical sensor 615 , and a proximity sensor 616 .

[0179] The acceleration sensor 611 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 600. For example, the acceleration sensor 611 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 601 can control the display screen 605 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for collecting game or user motion data.

[0180] The gyro sensor 612 can detect the body direction and rotation angle of the terminal 600, and the gyro sensor 612 can cooperate with the acceleration sensor 611 to collect the user's 3D actions on the terminal 600. The processor 601 can implement the following functions based on the data collected by the gyro sensor 612: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0181] The pressure sensor 613 can be set on the side frame of the terminal 600 and / or the lower layer of the display screen 605. When the pressure sensor 613 is set on the side frame of the terminal 600, it can detect the user's holding signal of the terminal 600, and the processor 601 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 613. When the pressure sensor 613 is set on the lower layer of the display screen 605, the processor 601 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 605. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0182] The fingerprint sensor 614 is used to collect the user's fingerprint, and the processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 614 can be set on the front, back, or side of the terminal 600. When a physical button or a manufacturer logo is set on the terminal 600, the fingerprint sensor 614 can be integrated with the physical button or the manufacturer logo.

[0183] The optical sensor 615 is used to collect the ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 according to the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is reduced. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 according to the ambient light intensity collected by the optical sensor 615.

[0184] The proximity sensor 616, also called a distance sensor, is usually arranged on the front panel of the terminal 600. The proximity sensor 616 is used to collect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from the screen-on state to the screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from the screen-off state to the screen-on state.

[0185] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the terminal 600, and the terminal 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0186] Figure 8 1 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1001 and one or more memories 1002, wherein the memory 1002 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1001 to implement the methods provided in the above-mentioned various method embodiments. Of course, the computing device may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the computing device may also include other components for implementing device functions, which will not be described in detail here.

[0187] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in a terminal to complete the method for generating lip animation in the above embodiment. The computer-readable storage medium can be non-transient. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0188] An embodiment of the present application provides a computer program product, in which at least one instruction is stored. The instruction is loaded and executed by a processor to implement the operations performed by the above-mentioned method for generating lip animation.

[0189] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the target object images, target object pictures, target object speech audio, emotional information, etc. involved in this application are all obtained with full authorization.

[0190] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0191] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating lip animation, characterized in that: The method comprises: Obtaining audio features of the target object's speaking audio; Acquire image coding features of a target object image, wherein the target object image includes a face of the target object; Obtain emotional information of the target object; The audio features, the image coding features and the target object emotional information are input into a lip animation generation model to obtain a lip animation, in which the lip shape of the target object matches the target object speaking audio and the target object emotional information.

2. The method according to claim 1, characterized in that The step of obtaining the target object's emotional information includes: Get the target object's emotional text entered by the user account; The target object emotion text is input into a text encoder to obtain text features of the target object emotion text, and the text features are used as the target object emotion information.

3. The method according to claim 1, characterized in that The step of obtaining the target object's emotional information includes: Obtaining a pleasure value corresponding to a pleasure dimension and an activation value corresponding to an activation dimension input by a user account, wherein the pleasure dimension and the activation dimension are dimensions used to quantify emotions; Generate a target object emotion text based on the pleasure value and the activation value; The target object emotion text is input into a text encoder to obtain text features of the target object emotion text, and the text features are used as the target object emotion information.

4. The method according to claim 1, characterized in that The step of obtaining audio features of the target object's speech audio includes: The target object speech audio is input into an audio coding model to obtain audio features of the target object speech audio, wherein the audio coding model is trained based on sample object speech audio of at least one language.

5. The method according to claim 1, characterized in that The method further comprises: Generate a target object picture sequence based on the number of audio frames of the target object's speaking audio and the target object picture; The step of acquiring the image coding features of the target object image comprises: Obtain image coding features of each target object picture in the target object picture sequence.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining target mouth area information of the target object face in the target object picture; The step of inputting the audio features, the image coding features and the target object emotion information into a lip animation generation model to obtain a lip animation comprises: The audio features, the image coding features, the target mouth area information and the target object emotion information are input into a lip animation generation model to obtain a lip animation.

7. The method according to claim 6, characterized in that The step of obtaining target mouth region information of the target object face in the target object picture includes: Extracting facial key points of the target object in the target object image to obtain a plurality of target facial key points; Calculating the mouth length of the target object based on the target mouth corner point among the multiple facial key points; Determine a standard mouth corner point and a standard nose bottom point of the target object face in the target object picture in a standard posture based on the mouth length, the target mouth corner point and the target nose bottom point among the multiple target facial key points; Determine an affine transformation matrix based on the target mouth corner point, the target nose bottom point, the standard mouth corner point, and the standard nose bottom point; Based on the affine transformation matrix, the plurality of target facial key points are mapped to obtain standard facial key points corresponding to the plurality of target facial key points; Determine standard mouth region information of the target object face in the target object image in a standard posture based on a standard cheek corner point, a standard eye outer point, a standard ear middle point, and a standard chin center point among the standard facial key points; Based on the affine transformation matrix, the standard mouth region information is mapped to obtain the target mouth region information of the target object face in the target object picture.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Get the subject's speaking video; intercepting a speaking video sample of the subject in the speaking video of the subject, and obtaining an image sample of the subject, wherein the image of the subject includes the face of the subject; Obtaining a sample of the subject's emotion information from the subject's speaking video sample; Obtain an audio sample of the subject speaking corresponding to the video sample of the subject speaking; Acquire audio features corresponding to the audio samples of the subject's speech and image coding features of the image samples of the subject; Based on the object emotion information sample, the audio features corresponding to the object speech audio sample, the image coding features of the object image sample and the object speech video sample, the initial lip animation generation model is trained to obtain the lip animation generation model.

9. A computing device, characterized in that The computing device includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operation performed by the method for generating lip animation according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operation performed by the method for generating lip animation according to any one of claims 1 to 8.

11. A computer program product, characterized in that The computer program product stores at least one instruction, which is loaded and executed by a processor to implement the operations performed by the method for generating lip animation according to any one of claims 1 to 8.

Images