Method for acquiring human face key point information, method and device for generating human face animation

Abstract

The present disclosure relates to a method for acquiring facial key point information, a method for generating facial animation, and a device. The method for acquiring facial key point information includes: acquiring a sample video and a sample audio synchronously recorded for a sample face; determining the facial key point information in each video frame of the sample video to obtain a key point information sequence; determining the audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence; using the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and using the key point annotation information sequence as the target output of the neural network model, training the neural network model to obtain a facial key point acquisition model; and using the facial key point acquisition model to acquire facial key point information. The facial key point acquisition model trained by the present disclosure has high accuracy and stability in outputting facial key points.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for acquiring facial key point information, and a method and device for generating facial animation. Background Art

[0002] Speech recognition and text-to-speech technology is often used in audio-driven virtual image facial animation generation. The input audio generates a facial animation of a target face that matches the audio stream, thereby completing the audio drive of the target face.

[0003] Currently, in order to generate a facial animation of a target face that matches an audio stream based on audio, it is necessary to first obtain a facial key point sequence of the target face corresponding to the target audio through a facial key point acquisition model based on the target audio, and then generate an animation of the target face based on the facial key point sequence and a reference image of the target face.

[0004] Furthermore, how to train a facial key point model becomes the key to generating facial animation of a target face that matches the audio stream based on audio. Summary of the invention

[0005] This section is provided to introduce the concepts in a brief form, which will be described in detail in the detailed implementation section below. This section is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to be used to limit the scope of the technical solution claimed for protection.

[0006] In a first aspect, the present disclosure provides a method for acquiring facial key point information, the method comprising:

[0007] Obtain sample video and sample audio synchronously recorded for the sample face;

[0008] Determine the facial key point information in each video frame of the sample video to obtain a key point information sequence, wherein the facial key point information includes information of a first facial key point that is not linked to a voice;

[0009] Determine an audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence;

[0010] The key point information sequence and the audio spectrum sequence are used as inputs of a neural network model, and the key point annotation information sequence is used as a target output of the neural network model, and the neural network model is trained to obtain a face key point acquisition model, wherein the key point annotation information sequence includes face key point annotation information of the sample face corresponding to each of the video frames, and the face key point annotation information includes annotation information of the first face key point and annotation information of the second face key point linked to the utterance;

[0011] The facial key point acquisition model is used to acquire facial key point information.

[0012] In a second aspect, the present disclosure provides a method for generating a facial animation, the method comprising:

[0013] Performing spectrum conversion on the acquired target audio to obtain a target audio spectrum sequence;

[0014] Inputting the target audio spectrum sequence into a facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio, wherein the facial key point acquisition model is obtained according to the method described in any one of the first aspects;

[0015] An animation of the target face is generated according to the target face key point information sequence and a reference face image of the target face.

[0016] In a third aspect, the present disclosure provides a device for acquiring facial key point information, the device comprising:

[0017] The first acquisition module is used to acquire sample video and sample audio synchronously recorded for the sample face;

[0018] A first determination module is used to determine the facial key point information in each video frame of the sample video to obtain a key point information sequence, wherein the facial key point information includes information of a first facial key point that is not linked to a voice;

[0019] A second determining module, configured to determine an audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence;

[0020] A training module, used to use the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and use the key point annotation information sequence as a target output of the neural network model, to train the neural network model to obtain a face key point acquisition model, wherein the key point annotation information sequence includes face key point annotation information of the sample face corresponding to each of the video frames, and the face key point annotation information includes annotation information of the first face key point and annotation information of the second face key point linked to the utterance;

[0021] The second acquisition module is used to acquire facial key point information using the facial key point acquisition model.

[0022] In a fourth aspect, the present disclosure provides a device for generating a facial animation, the device comprising:

[0023] A conversion module, used for performing spectrum conversion on the acquired target audio to obtain a target audio spectrum sequence;

[0024] A third determination module is used to input the target audio spectrum sequence into a facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio, wherein the facial key point acquisition model is obtained according to the method described in any one of the first aspects;

[0025] The driving module is used to generate an animation of the target face according to the target face key point information sequence and the reference face image of the target face.

[0026] In a fifth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspect or the second aspect.

[0027] In a sixth aspect, the present disclosure provides an electronic device, including:

[0028] a storage device having at least one computer program stored thereon;

[0029] At least one processing device is used to execute the at least one computer program in the memory to implement the steps of the method in any one of the first aspect or the second aspect.

[0030] Through the above technical solution, since the facial key point information in each video frame of the authorized sample video includes the information of the first facial key point that is not linked to the sound, when extracting the authorized sample video and the authorized sample audio recorded synchronously with the authorized sample face, it is not necessary for the authorized sample face in the authorized sample video to always be a frontal video. Even if the authorized sample face in the authorized sample video has a large angle of facial twist, the neural network model can be trained by jointly driving the first facial key point sequence and the audio spectrum sequence. And because the difficulty of collecting the authorized sample video and the authorized sample audio in the present disclosure is lower, the number of authorized sample videos and authorized sample audio obtained is larger, and the data is richer. The facial key point acquisition model obtained by training the authorized sample video and the authorized sample audio of the present disclosure has high accuracy and stability in outputting facial key points.

[0031] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0033] Figure 1 The figure is a flow chart of a method for acquiring facial key point information according to an exemplary embodiment.

[0034] Figure 2 The present invention is a flowchart of a method for generating facial animation according to an exemplary embodiment.

[0035] Figure 3 The present invention is a block diagram of a device for acquiring facial key point information according to an exemplary embodiment.

[0036] Figure 4 The invention is a block diagram of a device for generating facial animation according to an exemplary embodiment.

[0037] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0039] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0040] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0041] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0042] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0043] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0044] Figure 1 is a flow chart of a method for obtaining facial key point information according to an exemplary embodiment. Figure 1 As shown, the method for obtaining facial key point information includes the following steps.

[0045] In step S11, a sample video and a sample audio synchronously recorded for a sample face are obtained.

[0046] The sample faces involved in the present disclosure are sample faces that have been authorized for use. The sample videos synchronously recorded for the sample faces that have been authorized for use may include frontal videos of the sample faces that have been authorized for use and may also include side-face videos of the sample faces that have been authorized for use.

[0047] In step S12, the facial key point information in each video frame of the sample video is determined to obtain a key point information sequence.

[0048] For example, the sample video that has been authorized for use can be input into a pre-trained neural network model, and the neural network model outputs the facial key point information in each video frame of the sample video that has been authorized for use. Then, based on the facial key point information in each video frame, the information of the first facial key point that is not linked to the sound is screened.

[0049] Among them, the first facial key point can be pre-set and can include at least one of the nose bridge key point, the cheek edge key point, and the ear key point.

[0050] In step S13, an audio spectrum of each audio frame of the sample audio is determined to obtain an audio spectrum sequence.

[0051] For example, each audio frame in the sample audio that has been authorized for use may be converted to a spectrum to obtain an audio spectrum corresponding to each audio frame and then obtain an audio spectrum sequence. For example, each audio frame in the sample audio that has been authorized for use may be converted to a Mel spectrum to obtain a Mel audio spectrum sequence.

[0052] In step S14, the key point information sequence and the audio spectrum sequence are used as inputs of the neural network model, and the key point annotation information sequence is used as the target output of the neural network model. The neural network model is trained to obtain a face key point acquisition model.

[0053] In step S15, the facial key point acquisition model is used to acquire facial key point information.

[0054] Typically, when training a facial key point acquisition model, it is necessary to collect authorized sample videos and authorized sample audios of authorized sample faces always in the front view, so as to ensure the accuracy of the facial key points output by the trained facial key point acquisition model.

[0055] However, when people actually speak, as the tone of their voice changes, their heads always rotate over a large range or more freely. If the collected sample video contains twisting or rotation of the sample face (i.e., including the sample face in profile), it is necessary to discard the collected sample video and audio of the sample face, and re-collect the sample video and audio of the sample face, which increases the difficulty of collecting the sample video and audio when the sample face is always facing forward.

[0056] Therefore, in the present disclosure, the sample video that has been authorized for use is input into a pre-trained neural network model, and after the neural network model outputs the facial key point information in each video frame of the sample video that has been authorized for use, the information of the preset first facial key point that is not linked to voice is screened out based on the facial key point information in each video frame.

[0057] After obtaining the facial key point information including the first facial key point that is not linked to the vocalization, since the first facial key point is not linked to the vocalization, that is, the first facial key point will not change according to the voice, in the present disclosure, the key point information sequence including the first facial key point (not linked to the vocalization) and the audio spectrum sequence can be used as the input for training the facial key point acquisition model, and the second facial key point sequence linked to the vocalization can be obtained by jointly driving the first facial key point sequence and the audio spectrum sequence.

[0058] Furthermore, for the sample video and sample audio that are synchronously recorded for the authorized sample face in the present disclosure, it is not necessary for the sample face in the sample video to always be a frontal face video. Even if the authorized sample face in the sample video has a large angle of facial twist, the neural network can be trained based on the data including the frontal face and the side face to obtain a facial key point acquisition model.

[0059] Furthermore, since it is less difficult to collect authorized sample videos and authorized sample audios in the present disclosure, the number of authorized sample videos and authorized sample audios obtained is larger and the data is richer. The facial key point acquisition model trained based on the authorized sample videos and authorized sample audios of the present disclosure outputs facial key points with high accuracy and high stability.

[0060] For example, the key point information sequence and the audio spectrum sequence can be used as the input of the neural network model, and the key point annotation information sequence can be used as the target output of the neural network model to train the neural network model to obtain the face key point acquisition model:

[0061] The audio spectrum sequence is encoded to obtain an audio spectrum feature vector sequence, and the key point information sequence is encoded to obtain a key point information feature vector sequence. According to the audio spectrum feature vector sequence and the key point information feature vector sequence, a key point prediction information sequence is obtained.

[0062] Among them, the key point prediction information sequence includes the facial key point prediction information of the authorized sample face corresponding to each video frame, the facial key point prediction information includes the prediction information of the first facial key point and the prediction information of the second facial key point. According to the difference between the key point prediction information sequence and the key point annotation information sequence, the neural network model is trained to obtain the facial key point acquisition model.

[0063] After obtaining the facial key point acquisition model, the facial key point acquisition model can be used to acquire facial key point information. That is, the audio spectrum sequence of the audio is input into the facial key point acquisition model, and the facial key point information sequence corresponding to the audio is acquired through the facial key point acquisition model, that is, the facial key point information corresponding to the audio is acquired.

[0064] In an exemplary embodiment of the present disclosure, since the facial key point information in each video frame of the authorized sample video includes the information of the first facial key point that is not linked to the sound, when extracting the authorized sample video and the authorized sample audio recorded synchronously with the authorized sample face, it is not necessary for the authorized sample face in the authorized sample video to always be a frontal video. Even if the authorized sample face in the authorized sample video has a large angle of facial twist, the neural network model can be trained by jointly driving the first facial key point sequence and the audio spectrum sequence. In addition, since the difficulty of collecting the authorized sample video and the authorized sample audio in the present disclosure is lower, the number of authorized sample videos and authorized sample audio obtained is larger, and the data is richer. The facial key point acquisition model obtained by training the authorized sample video and the authorized sample audio of the present disclosure has high accuracy and stability in outputting facial key points.

[0065] In the present disclosure, the structure of the neural network model trained by the key point information sequence and the audio spectrum sequence may include an audio spectrum encoder, a face key point encoder and a decoder.

[0066] In the present disclosure, the audio spectrum encoder may be used to encode an audio spectrum sequence to obtain an audio spectrum feature vector sequence.

[0067] For example, a Mel spectrum conversion is performed on each audio frame of the sample audio that has been authorized for use to obtain a Mel audio spectrum (for example, a Mel audio spectrum of 80 dimensions). After further obtaining a Mel audio spectrum sequence, the Mel audio spectrum sequence is input into an audio spectrum encoder, and the Mel audio spectrum sequence is modeled to obtain a Mel audio spectrum feature vector sequence. The Mel audio spectrum sequence can be in the form of an image of [batch size × 1 × 80 × number of frames]. The batch size indicates the number of samples authorized for use that participate in training at one time, 1 indicates the current Mel audio spectrum, 80 indicates the 80 dimensions of the Mel audio spectrum, and the number of frames indicates the number of frames corresponding to the current Mel audio spectrum.

[0068] The process of modeling the Mel audio spectrum sequence may include: the audio spectrum encoder may process each input Mel audio spectrum through two-dimensional convolution, batch normalization, residual structure and ReLU activation function to obtain a Mel audio spectrum feature vector sequence.

[0069] In the present disclosure, a face key point encoder may be used to encode a key point information sequence to obtain a key point information feature vector sequence.

[0070] Since the key points of the face can be represented by the coordinates (x, y), the coordinates of the key point information sequence including the first key point of the face can be directly used as the input of the neural network model, and the key point information sequence including the first key point of the face can be encoded by the key point encoder of the face. For example, the first key points of the face selected include 4 key points of the nose bridge and two key points of the cheek edges on the left and right sides, totaling 8.

[0071] The coordinates of the first facial key point sequence may be, for example, in the form of [batch size × 16 × number of frames]. The batch size indicates the number of samples authorized for use in training at one time, 16 indicates that the number of first facial key points is 8, each first facial key point is 2 coordinates represented by (x, y) (i.e., 2×8), and the number of frames indicates the number of frames corresponding to the current first facial key point.

[0072] Inputting the coordinates of [batch size × 16 × number of frames] into the face key point encoder, the process of modeling the first face key point sequence may include: the face key point encoder may process the first face key point of each input frame through one-dimensional convolution, batch normalization, residual structure and ReLU activation function to obtain a first face key point information feature vector sequence.

[0073] In the present disclosure, the decoder can be used to obtain prediction information including prediction information of the first facial key points not linked to the utterance and annotation information of the second facial key points linked to the utterance based on the audio spectrum feature vector sequence and the key point information feature vector sequence.

[0074] The decoder structure consists of multiple layers of transposed convolutions. The upsampling magnification factor of the transposed convolution can be flexibly adjusted according to the coordinate latitude of the facial key points to be predicted. The first facial key point input can be used as a reference to obtain the prediction information of the facial key points adjacent to the first facial key point and the set multiple number. Then, based on the previous facial key point prediction information, the adjacent facial key points of other multiple numbers are predicted again until all the facial key point prediction information is obtained. For example, if there are 75 facial key points to be predicted, the magnification factors of 3, 3, and 5 can be used. The final prediction result of the model is [batch size × number of frames × 2 × 75], which is [batch size × number of frames × 150] after transformation. Then, through the upsampling of the transposed convolution, the spatial relationship between the coordinates of the facial key points can be maximized.

[0075] Figure 2 is a flowchart of a method for generating a facial animation according to an exemplary embodiment. Figure 2 As shown, the method for generating facial animation includes the following steps.

[0076] In step S21, spectrum conversion is performed on the acquired target audio to obtain a target audio spectrum sequence.

[0077] In step S22, the target audio spectrum sequence is input into the facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio.

[0078] Among them, the facial key point acquisition model is obtained according to the facial key point information acquisition method of the first embodiment or the second embodiment of the present disclosure.

[0079] In step S23, an animation of the target face is generated according to the target face key point information sequence and the reference face image of the target face.

[0080] In an exemplary embodiment of the present disclosure, after spectrally converting the acquired target audio to obtain a target audio spectrum sequence, the target audio spectrum sequence is input into a facial key point acquisition model obtained by the facial key point information acquisition method according to the first embodiment or the second embodiment. Since the number of authorized training samples is larger and the data is richer during the training process of the facial key point acquisition model, the facial key point acquisition model trained based on the present disclosure has high accuracy and stability in outputting facial key points, thereby generating a more realistic and accurate animation of the target face.

[0081] Figure 3 FIG3 is a block diagram 300 of a device for acquiring facial key point information according to an exemplary embodiment. Figure 3 The device for acquiring facial key point information includes a first acquisition module 301, a first determination module 302, a second determination module 303, a training module 304 and a second acquisition module 305.

[0082] The first acquisition module 301 is used to acquire sample video and sample audio synchronously recorded for the sample face;

[0083] A first determination module 302 is used to determine facial key point information in each video frame of the sample video to obtain a key point information sequence, wherein the facial key point information includes information of a first facial key point that is not linked to a voice;

[0084] A second determination module 303 is used to determine the audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence;

[0085] A training module 304 is used to use the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and use the key point annotation information sequence as a target output of the neural network model, to train the neural network model to obtain a face key point acquisition model, wherein the key point annotation information sequence includes face key point annotation information of the sample face corresponding to each of the video frames, and the face key point annotation information includes annotation information of the first face key point and annotation information of the second face key point linked to the utterance;

[0086] The second acquisition module 305 is used to acquire facial key point information using the facial key point acquisition model.

[0087] Optionally, the training module 304 uses the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and uses the key point annotation information sequence as a target output of the neural network model, and trains the neural network model to obtain a face key point acquisition model in the following manner:

[0088] Encoding the audio spectrum sequence to obtain an audio spectrum feature vector sequence;

[0089] Encoding the key point information sequence to obtain a key point information feature vector sequence;

[0090] According to the audio spectrum feature vector sequence and the key point information feature vector sequence, a key point prediction information sequence is obtained, wherein the key point prediction information sequence includes the face key point prediction information of the sample face corresponding to each of the video frames, and the face key point prediction information includes the prediction information of the first face key point and the prediction information of the second face key point;

[0091] The neural network model is trained according to the difference between the key point prediction information sequence and the key point annotation information sequence to obtain the face key point acquisition model.

[0092] Optionally, the neural network model includes an audio spectrum encoder, a facial key point encoder and a decoder;

[0093] The audio spectrum encoder is used to encode the audio spectrum sequence to obtain the audio spectrum feature vector sequence;

[0094] The facial key point encoder is used to encode the key point information sequence to obtain the key point information feature vector sequence;

[0095] The decoder is used to obtain the key point prediction information sequence according to the audio spectrum feature vector sequence and the key point information feature vector sequence.

[0096] Optionally, the decoder is used to obtain the key point prediction information sequence based on the audio spectrum feature vector sequence and the key point information feature vector sequence by an up-sampling method based on transposed convolution.

[0097] Optionally, the first facial key point includes at least one of a nose bridge key point, a cheek edge key point, and an ear key point.

[0098] Figure 4 FIG4 is a block diagram 400 showing a device for generating facial animation according to an exemplary embodiment. Figure 4 The device for generating facial animation includes a conversion module 401, a third determination module 402 and a driving module 403.

[0099] The conversion module 401 is used to perform spectrum conversion on the acquired target audio to obtain a target audio spectrum sequence;

[0100] A third determination module 402 is used to input the target audio spectrum sequence into a facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio, wherein the facial key point acquisition model is obtained according to any one of the methods in the first aspect;

[0101] The driving module 403 is used to generate an animation of the target face according to the target face key point information sequence and the reference face image of the target face.

[0102] Reference below Figure 5 , which shows an electronic device (eg, Figure 1 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0103] like Figure 5As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0104] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0105] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0106] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0107] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0108] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains sample video and sample audio recorded synchronously for a sample face; determines the facial key point information in each video frame of the sample video to obtain a key point information sequence, wherein the facial key point information includes information of a first facial key point that is not linked to vocalization; determines the audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence; uses the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and uses the key point annotation information sequence as a target output of the neural network model, trains the neural network model to obtain a facial key point acquisition model, wherein the key point annotation information sequence includes facial key point annotation information of the sample face corresponding to each of the video frames, and the facial key point annotation information includes annotation information of the first facial key point and annotation information of the second facial key point linked to vocalization; and uses the facial key point acquisition model to acquire facial key point information.

[0110] Alternatively, the computer-readable medium carries one or more programs, which, when executed by the electronic device, causes the electronic device to: perform spectrum conversion on the acquired target audio to obtain a target audio spectrum sequence; input the target audio spectrum sequence into a facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio, wherein the facial key point acquisition model is obtained according to the method described in any one of the first aspects; and generate an animation of the target face based on the target facial key point information sequence and a reference facial image of the target face.

[0111] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first acquisition module may also be described as a "module for acquiring at least two Internet Protocol addresses".

[0114] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0115] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] According to one or more embodiments of the present disclosure, Example 1 provides a method for acquiring facial key point information, including:

[0117] Obtain sample video and sample audio synchronously recorded for the sample face;

[0118] Determine the facial key point information in each video frame of the sample video to obtain a key point information sequence, wherein the facial key point information includes information of a first facial key point that is not linked to a voice;

[0119] Determine an audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence;

[0120] The key point information sequence and the audio spectrum sequence are used as inputs of a neural network model, and the key point annotation information sequence is used as a target output of the neural network model, and the neural network model is trained to obtain a face key point acquisition model, wherein the key point annotation information sequence includes face key point annotation information of the sample face corresponding to each of the video frames, and the face key point annotation information includes annotation information of the first face key point and annotation information of the second face key point linked to the utterance;

[0121] The facial key point acquisition model is used to acquire facial key point information.

[0122] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, further comprising: taking the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and taking the key point annotation information sequence as a target output of the neural network model, and training the neural network model to obtain a face key point acquisition model, including:

[0123] Encoding the audio spectrum sequence to obtain an audio spectrum feature vector sequence;

[0124] Encoding the key point information sequence to obtain a key point information feature vector sequence;

[0125] According to the audio spectrum feature vector sequence and the key point information feature vector sequence, a key point prediction information sequence is obtained, wherein the key point prediction information sequence includes the face key point prediction information of the sample face corresponding to each of the video frames, and the face key point prediction information includes the prediction information of the first face key point and the prediction information of the second face key point;

[0126] The neural network model is trained according to the difference between the key point prediction information sequence and the key point annotation information sequence to obtain the face key point acquisition model.

[0127] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, further comprising:

[0128] The neural network model includes an audio spectrum encoder, a facial key point encoder and a decoder;

[0129] The audio spectrum encoder is used to encode the audio spectrum sequence to obtain the audio spectrum feature vector sequence;

[0130] The facial key point encoder is used to encode the key point information sequence to obtain the key point information feature vector sequence;

[0131] The decoder is used to obtain the key point prediction information sequence according to the audio spectrum feature vector sequence and the key point information feature vector sequence.

[0132] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 3, further comprising:

[0133] The decoder is used to obtain the key point prediction information sequence based on the up-sampling method of transposed convolution according to the audio spectrum feature vector sequence and the key point information feature vector sequence.

[0134] According to one or more embodiments of the present disclosure, Example 5 provides the method of any one of Examples 1-4, further comprising:

[0135] The first facial key points include at least one of nose bridge key points, cheek edge key points, and ear key points.

[0136] According to one or more embodiments of the present disclosure, Example 6 provides a method for generating a facial animation, including:

[0137] Performing spectrum conversion on the acquired target audio to obtain a target audio spectrum sequence;

[0138] Inputting the target audio spectrum sequence into a facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio, wherein the facial key point acquisition model is obtained according to the method described in any one of Examples 1-5;

[0139] An animation of the target face is generated according to the target face key point information sequence and a reference face image of the target face.

[0140] According to one or more embodiments of the present disclosure, Example 7 provides a device for acquiring facial key point information, including:

[0141] The first acquisition module is used to acquire sample video and sample audio synchronously recorded for the sample face;

[0142] A first determination module is used to determine the facial key point information in each video frame of the sample video to obtain a key point information sequence, wherein the facial key point information includes information of a first facial key point that is not linked to a voice;

[0143] A second determining module, configured to determine an audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence;

[0144] A training module, used to use the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and use the key point annotation information sequence as a target output of the neural network model, to train the neural network model to obtain a face key point acquisition model, wherein the key point annotation information sequence includes face key point annotation information of the sample face corresponding to each of the video frames, and the face key point annotation information includes annotation information of the first face key point and annotation information of the second face key point linked to the utterance;

[0145] The second acquisition module is used to acquire facial key point information using the facial key point acquisition model.

[0146] According to one or more embodiments of the present disclosure, Example 8 provides the apparatus of Example 7, further comprising: the training module uses the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and uses the key point annotation information sequence as a target output of the neural network model, and trains the neural network model to obtain a face key point acquisition model in the following manner:

[0147] Encoding the audio spectrum sequence to obtain an audio spectrum feature vector sequence;

[0148] Encoding the key point information sequence to obtain a key point information feature vector sequence;

[0149] According to the audio spectrum feature vector sequence and the key point information feature vector sequence, a key point prediction information sequence is obtained, wherein the key point prediction information sequence includes the face key point prediction information of the sample face corresponding to each of the video frames, and the face key point prediction information includes the prediction information of the first face key point and the prediction information of the second face key point;

[0150] The neural network model is trained according to the difference between the key point prediction information sequence and the key point annotation information sequence to obtain the face key point acquisition model.

[0151] According to one or more embodiments of the present disclosure, Example 9 provides the apparatus of Example 8, further comprising:

[0152] The neural network model includes an audio spectrum encoder, a facial key point encoder and a decoder;

[0153] The audio spectrum encoder is used to encode the audio spectrum sequence to obtain the audio spectrum feature vector sequence;

[0154] The facial key point encoder is used to encode the key point information sequence to obtain the key point information feature vector sequence;

[0155] The decoder is used to obtain the key point prediction information sequence according to the audio spectrum feature vector sequence and the key point information feature vector sequence.

[0156] According to one or more embodiments of the present disclosure, Example 10 provides the apparatus of Example 9, further comprising:

[0157] The decoder is used to obtain the key point prediction information sequence based on the up-sampling method of transposed convolution according to the audio spectrum feature vector sequence and the key point information feature vector sequence.

[0158] According to one or more embodiments of the present disclosure, Example 11 provides the apparatus of any one of Examples 7-10, further comprising:

[0159] The first facial key points include at least one of nose bridge key points, cheek edge key points, and ear key points.

[0160] According to one or more embodiments of the present disclosure, Example 12 provides a device for generating a facial animation, including:

[0161] A conversion module, used for performing spectrum conversion on the acquired target audio to obtain a target audio spectrum sequence;

[0162] A third determination module is used to input the target audio spectrum sequence into a facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio, wherein the facial key point acquisition model is obtained according to the method described in any one of Examples 1-5;

[0163] The driving module is used to generate an animation of the target face according to the target face key point information sequence and the reference face image of the target face.

[0164] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0165] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0166] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A method for obtaining facial key point information, characterized in that: The method comprises: Obtain sample video and sample audio synchronously recorded for the sample face; Determine the facial key point information in each video frame of the sample video to obtain a key point information sequence, wherein the facial key point information includes information of a first facial key point that is not linked to a voice; Determine an audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence; The key point information sequence and the audio spectrum sequence are used as inputs of a neural network model, and the key point annotation information sequence is used as a target output of the neural network model, and the neural network model is trained to obtain a face key point acquisition model, wherein the key point annotation information sequence includes face key point annotation information of the sample face corresponding to each of the video frames, and the face key point annotation information includes annotation information of the first face key point and annotation information of the second face key point linked to the utterance; The facial key point acquisition model is used to acquire facial key point information.

2. The method according to claim 1, characterized in that The method of using the key point information sequence and the audio spectrum sequence as inputs of a neural network model, using the key point annotation information sequence as a target output of the neural network model, and training the neural network model to obtain a face key point acquisition model includes: Encoding the audio spectrum sequence to obtain an audio spectrum feature vector sequence; Encoding the key point information sequence to obtain a key point information feature vector sequence; According to the audio spectrum feature vector sequence and the key point information feature vector sequence, a key point prediction information sequence is obtained, wherein the key point prediction information sequence includes the face key point prediction information of the sample face corresponding to each of the video frames, and the face key point prediction information includes the prediction information of the first face key point and the prediction information of the second face key point; The neural network model is trained according to the difference between the key point prediction information sequence and the key point annotation information sequence to obtain the face key point acquisition model.

3. The method according to claim 2, characterized in that The neural network model includes an audio spectrum encoder, a facial key point encoder and a decoder; The audio spectrum encoder is used to encode the audio spectrum sequence to obtain the audio spectrum feature vector sequence; The facial key point encoder is used to encode the key point information sequence to obtain the key point information feature vector sequence; The decoder is used to obtain the key point prediction information sequence according to the audio spectrum feature vector sequence and the key point information feature vector sequence.

4. The method according to claim 3, characterized in that: The decoder is used to obtain the key point prediction information sequence based on the up-sampling method of transposed convolution according to the audio spectrum feature vector sequence and the key point information feature vector sequence.

5. The method according to any one of claims 1 to 4, characterized in that The first facial key points include at least one of nose bridge key points, cheek edge key points, and ear key points.

6. A method for generating facial animation, characterized in that: The method comprises: Performing spectrum conversion on the acquired target audio to obtain a target audio spectrum sequence; Inputting the target audio spectrum sequence into a facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio, wherein the facial key point acquisition model is obtained according to any one of the methods of claims 1-5; An animation of the target face is generated according to the target face key point information sequence and a reference face image of the target face.

7. A device for acquiring facial key point information, characterized in that: The device comprises: The first acquisition module is used to acquire sample video and sample audio synchronously recorded for the sample face; A first determination module is used to determine the facial key point information in each video frame of the sample video to obtain a key point information sequence, wherein the facial key point information includes information of a first facial key point that is not linked to a voice; A second determining module, configured to determine an audio spectrum of each audio frame of the sample audio to obtain an audio spectrum sequence; A training module, used to use the key point information sequence and the audio spectrum sequence as inputs of a neural network model, and use the key point annotation information sequence as a target output of the neural network model, to train the neural network model to obtain a face key point acquisition model, wherein the key point annotation information sequence includes face key point annotation information of the sample face corresponding to each of the video frames, and the face key point annotation information includes annotation information of the first face key point and annotation information of the second face key point linked to the utterance; The second acquisition module is used to acquire facial key point information using the facial key point acquisition model.

8. A device for generating facial animation, characterized in that: The device comprises: A conversion module, used to perform spectrum conversion on the acquired target audio to obtain a target audio spectrum sequence; A third determination module is used to input the target audio spectrum sequence into a facial key point acquisition model to obtain a target facial key point information sequence corresponding to the target audio, wherein the facial key point acquisition model is obtained according to any one of the methods of claims 1-5; The driving module is used to generate an animation of the target face according to the target face key point information sequence and the reference face image of the target face.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 6 are implemented.

10. An electronic device, characterized in that: include: a storage device having at least one computer program stored thereon; At least one processing device, configured to execute the at least one computer program in the storage device to implement the steps of the method according to any one of claims 1 to 6.

Images