Face video generation and face feature prediction model training method, device and equipment

Abstract

The invention provides a face video generation and face feature prediction model training method, device and equipment, and relates to the technical field of video processing. In some embodiments of the present disclosure, original speech is obtained; inputting the original voice into the trained facial feature prediction model to obtain facial feature parameters corresponding to the original voice; generating a target video corresponding to the facial feature parameters, and predicting the facial feature parameters through the original voice by using the model; the facial feature prediction model comprises an encoder and a decoder, the encoder is used for encoding original speech into target audio feature vectors, the decoder is used for decoding the target audio feature vectors and historical facial feature parameters into facial feature parameters, and the encoder predicts the facial feature parameters by combining the historical facial feature parameters; the accuracy of a prediction result is improved, and the precision of a facial feature prediction model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a method, device and equipment for facial video generation and facial feature prediction model training. Background Art

[0002] With the rapid development of Internet technology, virtual digital people have emerged. The matching requirements between the voice information of virtual digital people and the corresponding video information are gradually increasing.

[0003] At present, by extracting features from the sample speech data and sample video data of the sample object when speaking, we obtain the speech features and facial features of the sample data, and use the obtained facial features as training labels for the speech features, so that the model can learn the method from speech features to facial features.

[0004] The facial feature prediction model trained by the above method has low accuracy. Summary of the invention

[0005] The present disclosure provides a method, device and equipment for facial video generation and facial feature prediction model training, so as to at least solve the problem of low accuracy of existing facial feature prediction models.

[0006] The present disclosure provides a facial feature prediction model training method, including:

[0007] Collecting sample multimedia data, the sample multimedia data including: sample speech and sample video corresponding to the sample speech;

[0008] Labeling the image data in the sample video to generate a training sample set, wherein the training sample set includes a plurality of samples, each sample includes: audio data corresponding to the image data and a label corresponding to the image data, wherein the label is used to represent a facial expression in the image data corresponding to the audio data;

[0009] The initial model is trained based on the training sample set to obtain a facial feature prediction model.

[0010] Optionally, labeling the image data in the sample video to generate a training sample set includes:

[0011] Separating the sample voice and the sample video from the sample multimedia data;

[0012] Performing face detection on the image frames contained in the sample video to select a target image containing a face image from the image frames;

[0013] The target image is input into a pre-trained parameter estimation model to obtain the label.

[0014] Optionally, the training of the initial model based on the training sample set to obtain the facial feature prediction model includes:

[0015] Inputting the audio data into the initial model to obtain facial feature prediction parameters;

[0016] Determining a loss function according to a difference between the facial feature prediction parameter and the label;

[0017] The initial model is trained according to the loss function, the audio data and the label to obtain the facial feature prediction model.

[0018] Optionally, determining a loss function according to a difference between the facial feature prediction parameter and the label includes:

[0019] Determining a first loss value according to an error between the facial feature prediction parameter and the label;

[0020] Determining a second loss value according to an error between a first-order difference of the facial feature prediction parameter and the label;

[0021] A target loss function is determined according to the first loss value and the second loss value.

[0022] The present disclosure also provides a facial video generation method, including:

[0023] Get the original voice;

[0024] The original speech is input into the facial feature prediction model trained by the facial feature prediction model training method, so as to obtain facial feature parameters corresponding to the original speech; wherein the facial feature prediction model comprises an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, and the decoder is used to decode the target audio feature vector and pre-stored historical facial feature parameters into the facial feature parameters;

[0025] Based on the facial feature parameters corresponding to the original speech, a corresponding target facial video is generated.

[0026] Optionally, the historical facial feature parameters are facial feature parameters at a previous moment, and the step of inputting the original speech into a pre-trained facial feature prediction model to obtain the facial feature parameters corresponding to the original speech includes:

[0027] Within the facial feature prediction model, inputting the original speech into the encoder to obtain the target audio feature vector;

[0028] The target audio feature vector and the facial feature parameters at the previous moment are input into the decoder to obtain the facial feature parameters corresponding to the original speech.

[0029] Optionally, the encoder includes: an audio feature extraction layer, a linear interpolation network layer, a multi-head attention network layer and a first feedforward network layer; the inputting the original speech into the encoder to obtain the target audio feature vector includes:

[0030] Inside the encoder, inputting the original speech into the audio feature extraction layer to perform feature extraction to obtain a first audio feature vector, wherein the first audio feature vector is a time domain feature vector and / or a frequency domain feature vector;

[0031] Inputting the first audio feature vector into the linear interpolation network layer to obtain a second audio feature vector, wherein the second audio feature vector is a sampling rate feature vector;

[0032] Inputting the second audio feature vector into the multi-head attention network to obtain a third audio feature vector, wherein the third audio feature vector is an audio sequence relationship feature vector;

[0033] The third audio feature vector is input into the first feedforward network to obtain the target audio feature vector.

[0034] Optionally, the decoder includes: a first motion encoder, a periodic position encoding layer, a biased multi-head attention network layer, a biased cross-modal multi-head attention network layer, a second feedforward network and a second motion encoder, and the target audio feature vector and the facial feature parameters at the previous moment are input into the decoder to obtain the facial feature parameters corresponding to the original speech, including:

[0035] Inputting the facial feature parameters at the previous moment into the first motion encoder to obtain a first feature vector, wherein the first feature vector is a vector containing audio dynamic features;

[0036] Inputting the first feature vector into the periodic position coding layer to obtain a second feature vector, wherein the second feature vector is a vector of periodic or position information features in the audio signal;

[0037] Inputting the second feature vector into the biased multi-head attention network layer to obtain a third feature vector, wherein the third feature vector is a vector of features at different positions in the audio sequence;

[0038] Inputting the third feature vector and the target audio feature vector into the biased cross-modal multi-head attention network layer to obtain a fourth feature vector, wherein the fourth feature vector is a vector of associated features in the cross-modal data;

[0039] Inputting the fourth eigenvector into the second feedforward network to obtain a fifth eigenvector, wherein the fifth eigenvector is a vector obtained by performing a nonlinear transformation on the audio feature;

[0040] The fifth feature vector is input into a second motion encoder to obtain facial feature parameters corresponding to the original speech.

[0041] The present disclosure also provides a facial video generation device, including:

[0042] Acquisition module, to obtain the original speech;

[0043] A prediction model module, inputting the original speech into a pre-trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; wherein the facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, and the decoder is used to decode the target audio feature vector and pre-stored historical facial feature parameters into the facial feature parameters;

[0044] A generation module is used to generate a corresponding target face video based on the facial feature parameters corresponding to the original speech.

[0045] The present disclosure also provides a facial feature prediction model training device, comprising:

[0046] A collection module, used for collecting sample multimedia data, wherein the sample multimedia data includes: sample speech and sample video corresponding to the sample speech;

[0047] A labeling module, used to label the image data in the sample video to generate a training sample set, wherein the training sample set includes a plurality of samples, each sample includes: audio data corresponding to the image data and a label corresponding to the image data, wherein the label is used to represent a facial expression in the image data corresponding to the audio data;

[0048] The training module is used to train the initial model based on the training sample set to obtain a facial feature prediction model.

[0049] The present disclosure also provides an electronic device, including:

[0050] processor;

[0051] a memory for storing instructions executable by the processor;

[0052] The processor is configured to execute the instructions to implement each step in the above method.

[0053] The embodiment of the present disclosure also provides a computer-readable storage medium, wherein the computer instructions are used to enable the computer to execute each step in the above method.

[0054] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, characterized in that the computer program / instruction implements each step of the above method when executed by a processor.

[0055] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0056] In some embodiments of the present disclosure, original speech is obtained; the original speech is input into a pre-trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; a target video corresponding to the facial feature parameters is generated, and the facial feature parameters are predicted using the model through the original speech; the facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, the decoder is used to decode the target audio feature vector and historical facial feature parameters into facial feature parameters, the encoder predicts the facial feature parameters in combination with the historical facial feature parameters, thereby improving the accuracy of the prediction results and enhancing the accuracy of the facial feature prediction model.

[0057] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0059] Figure 1 A flowchart of a facial video generation method provided by an exemplary embodiment of the present disclosure;

[0060] Figure 2 A flowchart of a facial feature prediction model training method provided for an exemplary embodiment of the present disclosure;

[0061] Figure 3 A schematic diagram of the structure of an encoder provided for an exemplary embodiment of the present disclosure;

[0062] Figure 4 A schematic diagram of a decoder provided in an embodiment of the present disclosure;

[0063] Figure 5A schematic structural diagram of a facial video generation device provided by an exemplary embodiment of the present disclosure;

[0064] Figure 6 A schematic diagram of the structure of a facial feature prediction model training device provided by an exemplary embodiment of the present disclosure;

[0065] Figure 7 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of the present disclosure;

[0066] Figure 8 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0068] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0069] It should be noted that the user information involved in this disclosure includes but is not limited to: user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure comply with the relevant laws and regulations and do not violate public order and good morals.

[0070] At present, virtual digital humans can be divided into two categories. The first category relies on manual driving, that is, the interaction of body and expression is carried out through dynamic capture technology; the second category relies on machines, that is, the content presented by the virtual human is calculated through the computer artificial intelligence model in terms of language, voice, expression and body interaction. This disclosure scheme is explained for the second category of virtual digital humans.

[0071] At present, the voice features and facial features of the sample data are obtained by extracting features from the sample speech data and sample video data when the sample object speaks, and the facial features obtained are used as training labels for the voice features, so that the model can learn the method from voice features to facial features. The current training method does not combine the audio features of historical voice data to participate in the training of the model, resulting in low accuracy of the prediction results and low model precision.

[0072] In response to the above technical problems, in some embodiments of the present disclosure, original speech is obtained; the original speech is input into a trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; a target video corresponding to the facial feature parameters is generated, and the facial feature parameters are predicted using the model through the original speech; the facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, the decoder is used to decode the target audio feature vector and historical facial feature parameters into facial feature parameters, the encoder predicts the facial feature parameters in combination with the historical facial feature parameters, thereby improving the accuracy of the prediction results and improving the accuracy of the facial feature prediction model.

[0073] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0074] Figure 1 FIG. 1 is a flow chart of a facial video generation method provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0075] S101: obtaining original speech;

[0076] S102: inputting the original speech into a pre-trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; wherein the facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, and the decoder is used to decode the target audio feature vector and pre-stored historical facial feature parameters into facial feature parameters;

[0077] S103: generating a corresponding target face video based on the facial feature parameters corresponding to the original speech;

[0078] The facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, and the decoder is used to decode the target audio feature vector and historical facial feature parameters into facial feature parameters.

[0079] In this embodiment, the execution subject of the above method may be a server or a terminal device.

[0080] In this embodiment, when the execution subject of the above method is a server, the type of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, and other server devices. Among them, the composition of the server mainly includes a processor, a hard disk, a memory, a system bus, etc., and a general computer architecture type.

[0081] In this embodiment, original speech is obtained; the original speech is input into a trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; a target video corresponding to the facial feature parameters is generated, and the facial feature parameters are predicted using the model through the original speech; the facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, the decoder is used to decode the target audio feature vector and historical facial feature parameters into facial feature parameters, the encoder predicts the facial feature parameters in combination with the historical facial feature parameters, thereby improving the accuracy of the prediction results and improving the accuracy of the facial feature prediction model.

[0082] Before using the facial feature prediction model, it is necessary to first train the facial feature prediction model. Figure 2 The following is a flow chart of a facial feature prediction model training method provided by an exemplary embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0083] S201: Collecting sample multimedia data, where the sample multimedia data includes: sample speech and sample video corresponding to the sample speech;

[0084] S202: labeling the image data in the sample video to generate a training sample set, where the training sample set includes multiple samples, each sample includes: audio data corresponding to the image data and a label corresponding to the image data, where the label is used to represent the facial expression in the image data corresponding to the audio data;

[0085] S203: Training the initial model based on the training sample set to obtain a facial feature prediction model.

[0086] In the prediction model training method, the execution subject may not be limited to the type of server. For example, the server may be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly includes a processor, a hard disk, a memory, a system bus, etc., and a general computer architecture type.

[0087] In some embodiments of the present disclosure, sample multimedia data is collected. It should be noted that the multimedia data includes at least audio data and video data, and may also include text, etc. Generally speaking, different people may have different facial features when speaking, so the multimedia data in the embodiments of the present disclosure may be multimedia data corresponding to a single person.

[0088] It should be noted that the label is used to represent the facial expression of the person in the image data, and the label includes any one of: smile, laugh, calm, anger, disdain, eye roll, and grimace.

[0089] In some embodiments of the present disclosure, labels are annotated on the image data in the sample video to generate a training sample set. One achievable method is to separate sample speech and sample video from the sample multimedia data; perform face detection on the image frames contained in the sample video to select a target image containing a face image from the image frames; and input the target image into a pre-trained parameter estimation model to obtain a label.

[0090] In some embodiments of the present disclosure, sample speech and sample video are separated from sample multimedia data. One achievable method is to use an existing separation tool to separate sample speech and sample video corresponding to the sample speech from sample multimedia data. For example, the separated sample speech is saved as a 16khz, 16bit single-channel wav file. Each frame of the separated sample video is saved as a jpg file.

[0091] In some embodiments of the present disclosure, face detection is performed on image frames included in the sample video to select a target image containing a face image from the image frames. Specifically, a face detection model is used to perform face detection on all image frames in the sample video frame by frame to determine whether there is a face in the image frame, and a target image containing a face image is selected from all image frames.

[0092] In some embodiments of the present disclosure, the target image is input into a trained parameter estimation model to obtain sample facial feature parameters, wherein the parameter estimation model is a PE module (parameter estimation) of a CPEM model (Consistent Parameter Estimation Model).

[0093] In some embodiments of the present disclosure, the initial model is trained according to the sample speech and the sample facial feature parameters to obtain the facial feature prediction model. According to the set sampling rate and frame rate, the sample speech and the sample facial feature parameters are time-aligned and length-segmented to obtain a training data set, and the initial model is trained using the training data set to obtain a trained facial feature prediction model.

[0094] In some embodiments of the present disclosure, the initial model is trained based on sample speech and sample facial feature parameters to obtain a facial feature prediction model. One achievable method is to input the sample speech into the initial model to obtain facial feature prediction parameters; determine the loss function based on the difference between the facial feature prediction parameters and the sample facial feature parameters; train the initial model based on the loss function, sample speech and sample facial feature parameters to obtain the facial feature prediction model. Among them, the first loss value is determined based on the error between the facial feature prediction parameters and the sample facial feature parameters; the second loss value is determined based on the error of the first-order difference between the facial feature prediction parameters and the sample facial feature parameters; the target loss function is determined based on the first loss value and the second loss value. Among them, the error loss of the first-order difference is added to the loss function to improve the accuracy of the facial feature prediction model.

[0095] After the facial feature prediction model is trained, facial feature prediction is performed using the trained facial feature prediction model. The following describes the application scenario of the facial feature prediction model.

[0096] In some embodiments of the present disclosure, the original speech is input into the trained facial feature prediction model to obtain the facial feature parameters corresponding to the original speech. One achievable method is to input the original speech into the encoder inside the facial feature prediction model to obtain the target audio feature vector; and input the target audio feature vector and the facial feature parameters at the previous moment into the decoder to obtain the facial feature parameters corresponding to the original speech.

[0097] Figure 3 Schematic diagram of the structure of an encoder provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the encoder includes: an audio feature extraction layer, a linear interpolation network layer, a multi-head attention network layer and a first feedforward network layer. The original speech is input into the encoder to obtain a target audio feature vector. One achievable method is that, inside the encoder, the original speech is input into the audio feature extraction layer for feature extraction to obtain a first audio feature vector, wherein the first audio feature vector is a time domain feature vector and / or a frequency domain feature vector; the first audio feature vector is input into the linear interpolation network layer to obtain a second audio feature vector, wherein the second audio feature vector is a sampling rate feature vector; the second audio feature vector is input into the multi-head attention network to obtain a third audio feature vector, wherein the third audio feature vector is an audio sequence relationship feature vector; the third audio feature vector is input into the first feedforward network to obtain a target audio feature vector.

[0098] Figure 4 Schematic diagram of a decoder provided by an embodiment of the present disclosure. Figure 4As shown, the decoder includes: a first motion encoder, a periodic position encoding layer, a biased multi-head attention network layer, a biased cross-modal multi-head attention network layer, a second feedforward network and a second motion encoder. The target audio feature vector and the facial feature parameters of the previous moment are input into the decoder to obtain the facial feature parameters corresponding to the original speech. One feasible method is to input the facial feature parameters of the previous moment into the first motion encoder to obtain a first feature vector, wherein the first feature vector is a vector containing audio dynamic features; input the first feature vector into the periodic position encoding layer to obtain a second feature vector, wherein the second feature vector is a vector of periodic or position information features in the audio signal; input the second feature vector into the biased multi-head attention network layer to obtain a third feature vector, wherein the third feature vector is a vector of features at different positions in the audio sequence; input the third feature vector and the target audio feature vector into the biased cross-modal multi-head attention network layer to obtain a fourth feature vector, wherein the fourth feature vector is a vector of associated features in cross-modal data; input the fourth feature vector into the second feedforward network to obtain a fifth feature vector, wherein the fifth feature vector is a vector after nonlinear transformation of the audio features; input the fifth feature vector into the second motion encoder to obtain facial feature parameters corresponding to the original speech.

[0099] It should be noted that the facial feature parameters output by the facial feature prediction model are the facial feature parameters corresponding to each frame of the image. After obtaining the facial feature parameters corresponding to the original speech, the facial feature parameters corresponding to each frame of the image can be inserted into the video frame of the video to be synthesized, and video synthesis is performed to obtain the target video. Among them, the embodiment of the present disclosure does not limit the type of target video. The target video includes but is not limited to: real person videos and virtual person videos. Virtual person videos include but are not limited to: virtual anchor videos, virtual assistant videos, virtual experts and virtual idols.

[0100] In the above method embodiment, original speech is obtained; the original speech is input into a trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; a target video corresponding to the facial feature parameters is generated, and the facial feature parameters are predicted by the model through the original speech; the facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, the decoder is used to decode the target audio feature vector and historical facial feature parameters into facial feature parameters, the encoder predicts the facial feature parameters in combination with the historical facial feature parameters, thereby improving the accuracy of the prediction results and improving the accuracy of the facial feature prediction model.

[0101] Figure 5 FIG. 5 is a schematic diagram of a facial video generating device 50 provided by an exemplary embodiment of the present disclosure. Figure 5As shown, the facial video generating device 50 includes: an acquisition module 51, a prediction model module 52 and a generation module 53.

[0102] Wherein, the acquisition module 51 is used to acquire the original speech;

[0103] The prediction model module 52 inputs the original speech into the pre-trained facial feature prediction model to obtain the facial feature parameters corresponding to the original speech;

[0104] The generating module 53 is used to generate the corresponding target face video based on the facial feature parameters corresponding to the original speech.

[0105] The facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, and the decoder is used to decode the target audio feature vector and pre-stored historical facial feature parameters into facial feature parameters.

[0106] Optionally, the historical facial feature parameters are facial feature parameters at a previous moment, and when the prediction model module 52 inputs the original speech into the pre-trained facial feature prediction model to obtain the facial feature parameters corresponding to the original speech, it is used to:

[0107] Inside the facial feature prediction model, the original speech is input into the encoder to obtain the target audio feature vector;

[0108] The target audio feature vector and the facial feature parameters at the previous moment are input into the decoder to obtain the facial feature parameters corresponding to the original speech.

[0109] Optionally, the encoder includes: an audio feature extraction layer, a linear interpolation network layer, a multi-head attention network layer and a first feedforward network layer; when the original speech is input into the encoder and the target audio feature vector is obtained, the prediction model module 52 is used to:

[0110] Inside the encoder, the original speech is input into the audio feature extraction layer for feature extraction to obtain a first audio feature vector, wherein the first audio feature vector is a time domain feature vector and / or a frequency domain feature vector;

[0111] Inputting the first audio feature vector into a linear interpolation network layer to obtain a second audio feature vector, wherein the second audio feature vector is a sampling rate feature vector;

[0112] Input the second audio feature vector into the multi-head attention network to obtain a third audio feature vector;

[0113] The third audio feature vector is input into the first feedforward network to obtain a target audio feature vector, wherein the third audio feature vector is an audio sequence relationship feature vector.

[0114] Optionally, the decoder includes: a first motion encoder, a periodic position encoding layer, a biased multi-head attention network layer, a biased cross-modal multi-head attention network layer, a second feedforward network and a second motion encoder, and the prediction model module 52 inputs the target audio feature vector and the facial feature parameters of the previous moment into the decoder to obtain the facial feature parameters corresponding to the original speech, and is used to:

[0115] Inputting the facial feature parameters at the previous moment into a first motion encoder to obtain a first feature vector, wherein the first feature vector is a vector containing audio dynamic features;

[0116] Inputting the first feature vector into a periodic position coding layer to obtain a second feature vector, wherein the second feature vector is a vector of periodic or position information features in the audio signal;

[0117] Input the second feature vector into the biased multi-head attention network layer to obtain a third feature vector, wherein the third feature vector is a vector of features at different positions in the audio sequence;

[0118] Inputting the third feature vector and the target audio feature vector into a biased cross-modal multi-head attention network layer to obtain a fourth feature vector, wherein the fourth feature vector is a vector of associated features in the cross-modal data;

[0119] Inputting the fourth eigenvector into the second feedforward network to obtain a fifth eigenvector, wherein the fifth eigenvector is a vector obtained by performing a nonlinear transformation on the audio feature;

[0120] The fifth feature vector is input into the second motion encoder to obtain facial feature parameters corresponding to the original speech.

[0121] Figure 6 FIG. 6 is a schematic diagram of a facial feature prediction model training device 60 provided by an exemplary embodiment of the present disclosure. Figure 6 As shown, the facial feature prediction model training device 60 includes: a collection module 61, a labeling module 62 and a training module 63.

[0122] The acquisition module 61 is used to acquire sample multimedia data, and the sample multimedia data includes: sample speech and sample video corresponding to the sample speech;

[0123] The labeling module 62 is used to label the image data in the sample video to generate a training sample set, where the training sample set includes multiple samples, each sample includes: audio data corresponding to the image data and a label corresponding to the image data, where the label is used to represent the facial expression in the image data corresponding to the audio data;

[0124] The training module 63 is used to train the initial model based on the training sample set to obtain a facial feature prediction model.

[0125] Optionally, when labeling the image data in the sample video to generate a training sample set, the labeling module 62 is used to:

[0126] Separating sample voice and sample video from sample multimedia data;

[0127] Performing face detection on the image frames contained in the sample video to select a target image containing a face image from the image frames;

[0128] Input the target image into the pre-trained parameter estimation model to obtain the label.

[0129] Optionally, when the training module 63 trains the initial model based on the training sample set to obtain the facial feature prediction model, it is used to:

[0130] Input the audio data into the initial model to obtain facial feature prediction parameters;

[0131] Determine the loss function based on the difference between the predicted parameters of facial features and the labels;

[0132] The initial model is trained according to the loss function, audio data and labels to obtain a facial feature prediction model.

[0133] Optionally, when determining the loss function according to the difference between the facial feature prediction parameter and the label, the training module 63 is used to:

[0134] Determine a first loss value according to an error between a facial feature prediction parameter and a label;

[0135] Determine a second loss value according to an error between a first-order difference of a facial feature prediction parameter and a label;

[0136] A target loss function is determined according to the first loss value and the second loss value.

[0137] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0138] Figure 7 FIG. 1 is a schematic diagram of a structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 7 As shown, the electronic device includes: a memory 71 and a processor 72. In addition, the electronic device also includes a power supply component 73 and a communication component 74.

[0139] The memory 71 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device.

[0140] The memory 71 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0141] The communication component 74 is used for data transmission with other devices.

[0142] The processor 72 can execute computer instructions stored in the memory 71 to: obtain original speech; input the original speech into a pre-trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; wherein the facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, and the decoder is used to decode the target audio feature vector and pre-stored historical facial feature parameters into facial feature parameters; based on the facial feature parameters corresponding to the original speech, generate a corresponding target facial video.

[0143] Accordingly, the present disclosure also provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 1 Each step in the method embodiment.

[0144] Accordingly, the present disclosure also provides a computer program product, the computer program product includes a computer program / instruction, the computer program / instruction is executed by a processor Figure 1 Each step in the method embodiment.

[0145] Figure 8 FIG. 1 is a schematic diagram of a structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, the electronic device includes: a memory 81 and a processor 82. In addition, the electronic device also includes a power supply component 83 and a communication component 84.

[0146] The memory 81 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.

[0147] The memory 81 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0148] The communication component 84 is used for data transmission with other devices.

[0149] The processor 82 can execute computer instructions stored in the memory 81 to: collect sample multimedia data, the sample multimedia data includes: sample speech and sample video corresponding to the sample speech; label the image data in the sample video to generate a training sample set, the training sample set includes multiple samples, each sample includes: audio data corresponding to the image data and a label corresponding to the image data, the label is used to represent the facial expression in the image data corresponding to the audio data; train the initial model based on the training sample set to obtain a facial feature prediction model.

[0150] Accordingly, the present disclosure also provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 2 Each step in the method embodiment.

[0151] Accordingly, the present disclosure also provides a computer program product, the computer program product includes a computer program / instruction, the computer program / instruction is executed by a processor Figure 2 Each step in the method embodiment.

[0152] Above Figure 7 and Figure 8 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0153] Above Figure 7 and Figure 8The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0154] The electronic device may further include a display screen.

[0155] The display screen includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0156] The electronic device mentioned above also includes an audio component.

[0157] The audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0158] In the embodiments of the above-mentioned apparatus, device, storage medium and computer program product disclosed in the present invention, original speech is obtained; the original speech is input into a trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; a target video corresponding to the facial feature parameters is generated, and the facial feature parameters are predicted by the original speech using the model; the facial feature prediction model includes an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, the decoder is used to decode the target audio feature vector and historical facial feature parameters into facial feature parameters, the encoder predicts the facial feature parameters in combination with the historical facial feature parameters, thereby improving the accuracy of the prediction results and the accuracy of the facial feature prediction model.

[0159] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0164] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0165] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0166] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0167] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A facial feature prediction model training method, characterized in that: include: Collecting sample multimedia data, the sample multimedia data including: sample speech and sample video corresponding to the sample speech; Labeling the image data in the sample video to generate a training sample set, wherein the training sample set includes a plurality of samples, each sample includes: audio data corresponding to the image data and a label corresponding to the image data, wherein the label is used to represent a facial expression in the image data corresponding to the audio data; The initial model is trained based on the training sample set to obtain a facial feature prediction model.

2. The method according to claim 1, characterized in that Labeling the image data in the sample video to generate a training sample set includes: Separating the sample voice and the sample video from the sample multimedia data; Performing face detection on the image frames contained in the sample video to select a target image containing a face image from the image frames; The target image is input into a pre-trained parameter estimation model to obtain the label.

3. The method according to claim 1, characterized in that The initial model is trained based on the training sample set to obtain a facial feature prediction model, including: Inputting the audio data into the initial model to obtain facial feature prediction parameters; Determining a loss function according to a difference between the facial feature prediction parameter and the label; The initial model is trained according to the loss function, the audio data and the label to obtain the facial feature prediction model.

4. The method according to claim 3, characterized in that Determining a loss function according to a difference between the facial feature prediction parameter and the label includes: Determining a first loss value according to an error between the facial feature prediction parameter and the label; Determining a second loss value according to an error between a first-order difference of the facial feature prediction parameter and the label; A target loss function is determined according to the first loss value and the second loss value.

5. A facial video generation method, characterized in that: include: Get the original voice; The original speech is input into a facial feature prediction model trained by the facial feature prediction model training method according to any one of claims 1 to 4, to obtain facial feature parameters corresponding to the original speech; wherein the facial feature prediction model comprises an encoder and a decoder, the encoder is used to encode the original speech into a target audio feature vector, and the decoder is used to decode the target audio feature vector and pre-stored historical facial feature parameters into the facial feature parameters; Based on the facial feature parameters corresponding to the original speech, a corresponding target facial video is generated.

6. The method according to claim 5, characterized in that The historical facial feature parameters are facial feature parameters at a previous moment, and the inputting of the original speech into a pre-trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech includes: Within the facial feature prediction model, inputting the original speech into the encoder to obtain the target audio feature vector; The target audio feature vector and the facial feature parameters at the previous moment are input into the decoder to obtain the facial feature parameters corresponding to the original speech.

7. The method according to claim 6, characterized in that The encoder comprises: an audio feature extraction layer, a linear interpolation network layer, a multi-head attention network layer and a first feedforward network layer; the original speech is input into the encoder to obtain the target audio feature vector, comprising: Inside the encoder, inputting the original speech into the audio feature extraction layer to perform feature extraction to obtain a first audio feature vector, wherein the first audio feature vector is a time domain feature vector and / or a frequency domain feature vector; Inputting the first audio feature vector into the linear interpolation network layer to obtain a second audio feature vector, wherein the second audio feature vector is a sampling rate feature vector; Inputting the second audio feature vector into the multi-head attention network to obtain a third audio feature vector, wherein the third audio feature vector is an audio sequence relationship feature vector; The third audio feature vector is input into the first feedforward network to obtain the target audio feature vector.

8. The method according to claim 6, characterized in that The decoder includes: a first motion encoder, a periodic position encoding layer, a biased multi-head attention network layer, a biased cross-modal multi-head attention network layer, a second feedforward network and a second motion encoder. The target audio feature vector and the facial feature parameters at the previous moment are input into the decoder to obtain the facial feature parameters corresponding to the original speech, including: Inputting the facial feature parameters at the previous moment into the first motion encoder to obtain a first feature vector, wherein the first feature vector is a vector containing audio dynamic features; Inputting the first feature vector into the periodic position coding layer to obtain a second feature vector, wherein the second feature vector is a vector of periodic or position information features in the audio signal; Inputting the second feature vector into the biased multi-head attention network layer to obtain a third feature vector, wherein the third feature vector is a vector of features at different positions in the audio sequence; Inputting the third feature vector and the target audio feature vector into the biased cross-modal multi-head attention network layer to obtain a fourth feature vector, wherein the fourth feature vector is a vector of associated features in the cross-modal data; Inputting the fourth eigenvector into the second feedforward network to obtain a fifth eigenvector, wherein the fifth eigenvector is a vector obtained by performing a nonlinear transformation on the audio feature; The fifth feature vector is input into a second motion encoder to obtain facial feature parameters corresponding to the original speech.

9. A facial video generation device, characterized in that: include: An acquisition module, used to acquire original speech; A prediction model module, used for inputting the original speech into a pre-trained facial feature prediction model to obtain facial feature parameters corresponding to the original speech; wherein the facial feature prediction model includes an encoder and a decoder, the encoder is used for encoding the original speech into a target audio feature vector, and the decoder is used for decoding the target audio feature vector and pre-stored historical facial feature parameters into the facial feature parameters; A generation module is used to generate a corresponding target face video based on the facial feature parameters corresponding to the original speech.

10. A facial feature prediction model training device, characterized in that: include: A collection module, used for collecting sample multimedia data, wherein the sample multimedia data includes: sample speech and sample video corresponding to the sample speech; A labeling module, used to label the image data in the sample video to generate a training sample set, wherein the training sample set includes a plurality of samples, each sample includes: audio data corresponding to the image data and a label corresponding to the image data, wherein the label is used to represent a facial expression in the image data corresponding to the audio data; The training module is used to train the initial model based on the training sample set to obtain a facial feature prediction model.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement each step in the method according to any one of claims 1 to 4 or claims 5 to 8.

12. A computer-readable storage medium, characterized in that: The computer instructions are used to enable the computer to execute each step of the method according to any one of claims 1 to 4 or claims 5 to 8.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 or claims 5 to 8 are implemented.

Images