Method, device, storage medium and electronic device for generating key points of human face

By extracting the audio data and processing the facial features, a natural change of virtual images face key point sequence is generated, which solves the problem that the existing virtual images are not realistic enough, and improves the authenticity and fluency of the mouth movements of the virtual image. , enhances the interactive experience.

CN112668407BActive Publication Date: 2024-11-26BEIJING DA MI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011462207.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-11
Publication Date
2024-11-26
Estimated Expiration
2040-12-11

AI Technical Summary

Technical Problem

The existing virtual image synthesis technology is not realistic enough, especially when talking to different people in a noisy environment, the performance of sound turning to key points to generate lip changes in images is unstable, resulting in a reduced interactive experience.

Method used

By extracting the audio data features, generating sound domain features and face features, and superimposing them to generate input features, using multi-layer fully connected networks and long and short-term memory neural networks to generate face key points sequences of naturally changing virtual images , improve the authenticity and fluency of mouth movements.

Benefits of technology

It realizes the virtual image that is directly generated based on audio data, improves the authenticity and fluency of the mouth movement of the virtual image, and enhances the interactive experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112668407B_ABST
    Figure CN112668407B_ABST
Patent Text Reader

Abstract

The present application discloses a method, device, storage medium and electronic device for generating facial key points, and belongs to the field of computer technology. The facial key point generation method comprises: extracting features from audio data to obtain sound domain features, extracting features from template faces to obtain facial features, processing facial sequences to obtain sequence features, superimposing the sound domain features, facial features and sequence features to generate input features, and generating facial key point sequences according to the input features. Therefore, in the present application, by directly generating relevant features mainly including phoneme features based on audio data, and then processing to obtain relevant information about facial key points of a naturally changing virtual image, the problem that the prior art completely relies on audio, and the performance of generating lip change images for sounds in noisy environments and the sounds of different people's voices is not stable enough, is solved, and the realism and fluency of the virtual image's mouth movements are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and electronic device for generating key points of a human face. Background Art

[0002] At present, virtual image synthesis can be applied in different occasions. For example, in the process of online education, virtual teachers provide teaching services, which can not only greatly reduce the burden of teachers, but also reduce teaching costs, and provide a better teaching experience than simple recorded classes. In addition, virtual images can also be used in a wider range of occasions, such as artificial intelligence (AI) news anchors, games, animations and applications, which have great commercial value in actual business scenarios. In the prior art, the synthesis of virtual images can generate corresponding lip change images based on the input sound data to simulate the mouth movements when speaking, but the existing synthesized virtual images are not real enough, which reduces the interactive experience. The existing virtual image technology of voice-to-key point conversion relies entirely on audio, and the performance of generating lip change images for sounds in noisy environments and the sounds of different people's voices is not stable enough. In response to this problem, there is an urgent need for a method that can directly process the input audio data and generate high-quality virtual images with corresponding mouth movements and natural changes in facial expressions. Summary of the invention

[0003] The embodiments of the present application provide a method, device, storage medium and electronic device for generating key points of a human face, which can directly generate a naturally changing virtual image based on audio data and improve the realism of the mouth. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for generating facial key points, comprising:

[0005] Extracting features from the audio data to obtain sound domain features; wherein the sound domain features include phoneme mouth shape features and sound coding features;

[0006] Extract features from the template face to obtain facial features;

[0007] Processing a face sequence to obtain a sequence feature; wherein the face sequence includes an angle sequence constraint feature and a boundary key point constraint feature;

[0008] Superimposing the voice domain feature, the face feature and the sequence feature to generate an input feature;

[0009] Generate a facial key point sequence according to the input features.

[0010] In a second aspect, an embodiment of the present application provides a device for generating facial key points, the device comprising:

[0011] A first extraction module is used to extract features from audio data to obtain sound domain features; wherein the sound domain features include phoneme mouth shape features and sound coding features;

[0012] The second extraction module is used to extract features from the template face to obtain facial features;

[0013] A processing module, used to process a face sequence to obtain a sequence feature; wherein the face sequence includes an angle sequence constraint feature and a boundary key point constraint feature;

[0014] A superposition module, used for superimposing the voice domain feature, the face feature and the sequence feature to generate an input feature;

[0015] A generation module is used to generate a sequence of facial key points according to the input features.

[0016] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0017] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0018] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0019] When the above-mentioned facial key point generation method, device, storage medium and electronic device are working, feature extraction is performed on audio data to obtain sound domain features, feature extraction is performed on template faces to obtain facial features, and sequence features are processed to obtain sequence features. The sound domain features, facial features and sequence features are superimposed to generate input features, and a facial key point sequence is generated based on the input features. The embodiment of the present application can directly generate relevant features mainly including phoneme features based on audio data, and then process to obtain information related to facial key points of a naturally changing virtual image, thereby improving the realism and fluency of the virtual image's mouth movements. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a schematic diagram of a communication system architecture provided by an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a method for generating key points of a face provided in an embodiment of the present application;

[0023] Figure 3 It is another flowchart of a method for generating key points of a face provided in an embodiment of the present application;

[0024] Figure 4 It is a structural schematic diagram of a facial key point generation device provided in an embodiment of the present application;

[0025] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0027] When the following description is designed with drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0028] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.

[0029] In order to solve the problem that the existing synthesized virtual images mentioned above are not realistic enough, which reduces the interactive experience, and the existing sound-to-key point virtual image technology completely relies on audio, and the performance of generating lip change images for sounds in noisy environments and different people's voices is not stable enough, a method for generating facial key points is proposed. The method of the embodiment of the present application can be implemented by a computer program and can be run on a computer system based on the von Neumann system and equipped with a camera or a rotatable camera. The computer system can be a computer system of a smart phone, a laptop computer, a tablet computer and other devices.

[0030] Figure 1 A schematic diagram of a communication system architecture provided for this application.

[0031] See also Figure 1 , the communication system 01 includes a terminal device 101, a network device 102, and a server 103; when the communication system 01 includes a core network, the network device 102 can also be connected to the core network. The network device 102 can also communicate with an Internet Protocol (IP) network, such as the Internet, a private IP network, or other data networks. The network device 102 provides services for the terminal devices 101 and the server 103 within the coverage area. The user can use the terminal device 101 to interact with the server 103 through the network device 102 to receive or send messages, etc. The terminal device 101 can be installed with various communication client applications, such as voice interaction applications, animation production applications, etc. The server 103 can be a server that stores the face key point generation method provided in the embodiment of the present application and provides various services, and is used to detect, store and process the audio data, template face and other files uploaded by the terminal device 101, and send the processing results to the terminal device 101.

[0032] In the following method embodiments, for the sake of convenience, only the execution subject of each step is introduced and described as a computer.

[0033] The following will be combined with the attached Figures 2 to 3 , the facial key point generation method provided in the embodiment of the present application is introduced in detail.

[0034] See also Figure 2 , a flow chart of a method for generating facial key points is provided for an embodiment of the present application. The method may include the following steps:

[0035] S201, extracting features from audio data to obtain sound domain features.

[0036] Generally, the sound domain features include phoneme mouth shape features and sound coding features. The computer first marks the phonemes and mouth shapes in the audio data, wherein the phonemes include Chinese phonemes and English phonemes, and the mouth shape is represented by the size of the mouth opening. The phonemes and mouth shapes are classified by a clustering algorithm to obtain a classification result, and the mapping relationship between the phonemes and the mouth shapes is determined according to the classification result. Then, the computer calculates the center position of the time interval of the audio data based on a preset frame rate, traverses the time interval to extract the Mel cepstral coefficient MFCC sound features in the sub-time interval of a preset length before and after the center position, and processes the MFCC sound features through a convolutional neural CNN sound coding network and a fully connected network FC to obtain sound coding features. The computer then processes the audio data to obtain a phoneme sequence, and determines the mouth shape features corresponding to the phoneme sequence according to the mapping relationship between phonemes and mouth shapes, wherein the mouth shape features are aligned with the MFCC sound features, and the mouth shape features are processed by dimensional conversion to obtain phoneme mouth shape features, and finally the sound coding features and the mouth shape features are spliced ​​to generate sound domain features.

[0037] S202: Extract features from the template face to obtain facial features.

[0038] Generally, facial features refer to facial key point coordinate information features, such as 81 facial key point coordinates or 68 facial key point coordinates obtained by a computer through a common facial recognition algorithm. The computer identifies the template face in the data set to obtain facial key point coordinate information, counts all the facial key point coordinate information in the data set to obtain average facial key point coordinate information, determines the target facial key point coordinate information, obtains initial input features based on the target facial key point coordinate information and the average facial key point coordinate information, and processes the initial input features to obtain facial features. The data set refers to a set of template faces provided by the user, such as human faces, animated faces, or virtual images.

[0039] S203: Process the face sequence to obtain sequence features.

[0040] Generally, the face sequence includes angle sequence constraint features and boundary key point constraint features. The user can manually set or select an existing stored template in real time. The angle sequence constraint features include parameters in the x and y directions, and the boundary key point constraint features include three boundary point parameters. The computer obtains the angle sequence constraint features and the boundary key point constraint features, processes the angle sequence constraint features to obtain an angle sequence constraint sequence, processes the boundary key point constraint features to obtain a boundary key point constraint sequence, and superimposes the angle sequence constraint sequence and the boundary key point constraint sequence to obtain a sequence feature.

[0041] S204: superimpose the voice domain feature, the face feature and the sequence feature to generate an input feature.

[0042] Generally, superposition refers to the fusion of multiple vectors or arrays into one vector or array, mainly including Cat superposition and Stack superposition. The computer can directly call the corresponding function for processing. The computer performs Cat superposition on the sound domain feature, the face feature and the sequence feature to obtain a first superposition feature, and performs Stack superposition on the first superposition feature to obtain an input feature. Cat superposition can be understood as continuation, which will not increase the dimension, and Stack superposition can be understood as superposition, which will add a new dimension. The added dimension is determined according to the dimension of the input set.

[0043] S205: Generate a facial key point sequence according to the input features.

[0044] Generally, the facial key point sequence includes a sequence size and an audio data length association parameter, the number of facial key points, and corresponding coordinates. The computer processes the input features to obtain facial key point related features, and processes the facial key point related features through a multi-layer fully connected network to obtain a facial key point sequence, wherein the processing of the input features to obtain facial key point related features uses a long short-term memory (LSTM) neural network.

[0045] From the above content, it can be seen that the feature extraction of audio data obtains the sound domain feature, the feature extraction of the template face obtains the face feature, the face sequence is processed to obtain the sequence feature, the sound domain feature, the face feature and the sequence feature are superimposed to generate the input feature, and the face key point sequence is generated according to the input feature. The embodiment of the present application can directly generate relevant features mainly including phoneme features based on audio data, and then process to obtain the relevant information of the face key points of the naturally changing virtual image, thereby improving the realism and fluency of the virtual image's mouth movements.

[0046] See also Figure 3 , another flow chart of a method for generating key points of a human face is provided in an embodiment of the present application. The method for generating key points of a human face may include the following steps:

[0047] S301, marking the phonemes and mouth shapes in the audio data, and classifying the phonemes and mouth shapes by a clustering algorithm to obtain classification results.

[0048] Generally, phonemes are the smallest speech units divided according to the natural properties of speech, and are analyzed based on the pronunciation actions in a syllable. One action constitutes a phoneme. The phonemes include Chinese phonemes and English phonemes. The mouth shape is represented by the size of the mouth opening. For example: there are 32 phonemes in Chinese, such as b, p, m, f..., and 48 phonemes in English, including 20 vowel phonemes and 28 consonant phonemes. The computer marks the phonemes and mouth shapes in the audio data. For example, the phonemes of "Mandarin" in the audio data are marked as p, u, t, o, ng, h, u, a, a total of 8 phonemes, and the corresponding mouth shapes are 2, 28, 9, 24, 22, 21, 28, 23, among which mouth shape 2 corresponds to a mouth opening size of 1 cm, and mouth shape 28 corresponds to a mouth opening size of 2 cm, etc. According to the actual situation, the mouth opening size is divided into 10 groups, namely 0-0.5 cm, 0.5 cm-1 cm, ..., 4.5 cm-5 cm, so mouth shape 2 is in the second group, mouth shape 28 is in the fourth group, etc. The computer uses clustering algorithms such as category mean Kmeans to cluster the phonemes into different groups according to the size of the mouth shapes, and obtains classification results. For example, phonemes p, u, t are in the first group, and phonemes o, ng, h are in the second group, etc.

[0049] S302: Determine a mapping relationship between the phoneme and the mouth shape according to the classification result.

[0050] Generally, after the computer obtains the classification result, it determines the mapping relationship between the phoneme and the mouth shape according to the classification result, for example: phoneme p corresponds to mouth shape category 1, phoneme o corresponds to mouth shape category 2, phoneme u corresponds to mouth shape category 1, etc.

[0051] S303: Calculate a center position of the time interval of the audio data based on a preset frame rate.

[0052] Generally, the frame rate refers to the frequency / rate at which a bitmap image in units of frames appears continuously on a display. The computer needs to generate a sequence of facial key points based on audio data, and the facial key point sequence corresponds to the facial key point coordinates of consecutive frames. Therefore, it is necessary to first determine the frame rate of the generated virtual image video. For example, if the frame rate is determined to be 25 frames per second and the audio data is 3 minutes, the generated video has 4500 frames, the number of MFCC features of the audio data per second is 100, and there are 100 MFCC features in 1 second that are related to the audio duration. There are 25 corresponding generated faces in 1 second, and the 100 MFCC features are divided into 25 parts, and the MFCC features at the center position of each of the 25 parts are determined as the features at the center position of the generated face.

[0053] S304, traversing the time interval to extract Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position, and processing the MFCC sound features through a convolutional neural network (CNN) sound coding network and a fully connected network (FC) to obtain sound coding features.

[0054] Generally, the sub-time interval of the preset length refers to the duration of the audio data corresponding to each frame of the generated face image, for example, the sub-time interval of the preset length is 150ms. After the computer determines the center position of the time interval of the audio data, it traverses the time interval to extract the Mel cepstral coefficient MFCC sound features in the sub-time interval of the preset length before and after the center position, and processes the MFCC sound features to obtain sound coding features, for example, extracting the 13-dimensional MFCC features of the audio data in the sub-time interval, the first-order derivative features (12 dimensions) and the second-order derivative features (11 dimensions) of MFCC, and Cat to form a first sound feature of (1, 36) dimensions of 36 MFCC related dimensions after Cat is combined, traverse the audio data, and obtain the second sound feature of (30, 36) dimensions corresponding to the corresponding time before and after the center position of 300ms, and use the convolutional network and the fully connected network as sound feature encoders to extract the second sound feature to obtain sound coding features, wherein the first sound feature and the second sound feature are represented by arrays, and the sound coding feature is represented by a one-dimensional vector.

[0055] S305: Process the audio data to obtain a phoneme sequence, and determine the mouth shape features corresponding to the phoneme sequence according to a mapping relationship between phonemes and mouth shapes.

[0056] Generally, the computer annotates the audio data as a phoneme sequence according to Automatic Speech Recognition (ASR), or reads and aligns the phoneme annotations obtained in the text answer script generated in the human-computer dialogue or the manually edited text script by Natural Language Processing (NLP). The phoneme sequence contains information such as phoneme category, start time, end time, etc., and the mouth shape feature corresponding to the phoneme sequence is determined according to the mapping relationship between the phoneme and the mouth shape. For example, in order to synchronize with the sound feature, the computer marks the mouth shape category of the phoneme every 0.01 seconds according to the start time T0 and the end time T1, and forms a one-dimensional mouth shape feature array. The size of the mouth shape feature array is (T1-T0) / 0.01, and the mouth shape feature of the same phoneme has a unified mouth shape category number. Finally, the one-dimensional mouth shape feature array is converted into a two-dimensional one-hot mouth shape feature array, and aligned with the MFCC sound feature extracted from the audio data, wherein the phoneme sequence and the mouth shape feature are represented by an array.

[0057] S306, processing the mouth shape feature through dimensional conversion to obtain a phoneme mouth shape feature, and concatenating the sound coding feature and the mouth shape feature to generate a sound domain feature.

[0058] Generally, after the computer determines the mouth shape features corresponding to the phoneme sequence, it directly converts the mouth shape features into one-dimensional features using one-hot features to obtain sequentially arranged one-dimensional mouth shape features without processing through a fully connected neural network. For example, two consecutive one-hot mouth shape features before conversion are [[0,1,0,…0,0], [0,0,…0,1]], and the mouth shape features after dimensional conversion are [0,1,0,…,0,0,0,0,…0,1]. The sound coding features use MFCC sound features to output sound coding feature vectors through a multi-layer fully connected network, and then Cat-connect with the one-hot mouth shape features to generate sound domain features. For example, the sound coding features are [0,1,2,3], and the mouth shape features are [4,5,6,7,8], then the generated sound domain features are [0,1,2,3,4,5,6,7,8]. Wherein, the mouth shape features are represented by a two-dimensional one-hot array, and the phoneme mouth shape features and the sound domain features are represented by a one-dimensional vector.

[0059] S307, identifying the template face in the data set to obtain facial key point coordinate information, and counting all the facial key point coordinate information in the data set to obtain average facial key point coordinate information.

[0060] Generally, a data set is a collection of multiple template faces pre-provided by a user in a computer. Before the computer identifies the template face in the data set and obtains the coordinate information of the key points of the face, it also includes: detecting the face image in the original image data set based on the face detection algorithm, obtaining the detection result file, the face detection algorithm includes the face recognition dlib face image extraction algorithm, the user preset face key point positioning model algorithm or the artificial intelligence public platform call algorithm (such as Baidu, Kuangshi platform), parsing the detection result file to generate the template face, wherein the information in the detection result file includes the coordinates of multiple key points of the face, such as the cheek coordinates, eyebrow coordinates, eye coordinates, mouth coordinates and nose coordinates. After the computer obtains the sound features, it identifies the template face in the data set to obtain the facial key point coordinate information, and counts all the facial key point coordinate information in the data set to obtain the average facial key point coordinate information. For example, using Dlib feature recognition or a neural network such as a deep network to identify the 68 facial key point coordinate information in the first template face is ((73, 25), (85, 30), (90, 34), ...), and identifying the 68 facial key point coordinate information in the second template face is ((65, 20), (87, 32), (92, 30), ...), then the average facial key point coordinate information is ((69, 22.5), (86, 31), (91, 32), ...).

[0061] S308: Determine target facial key point coordinate information, obtain initial input features based on the target facial key point coordinate information and the average facial key point coordinate information, and process the initial input features to obtain facial features.

[0062] Generally, after obtaining the average facial key point coordinate information, the computer determines the target facial key point coordinate information, obtains the initial input features based on the target facial key point coordinate information and the average facial key point coordinate information, and processes the initial input features to obtain facial features. For example, the computer determines that the 68 facial key point coordinate information ((73, 25), (85, 30), (90, 34), ...) in the template face No. 1 is the target facial key point coordinate information, and subtracts the average facial key point coordinate information ((69, 22.5), (86, 31), (91, 32), ...) from the target facial key point coordinate information to obtain the initial input features, which can be expressed as ((4, 2.5), (-1, -1), (-1, 2), ...). The initial input features are processed by a facial key point feature extraction module (composed of a multi-layer fully connected network) to obtain facial features, wherein the initial input features are represented by an array and the facial features are represented by a vector.

[0063] S309: Obtain angle sequence constraint features and boundary key point constraint features, and process the angle sequence constraint features to obtain an angle sequence constraint sequence.

[0064] Generally, the face sequence includes angle sequence constraint features and boundary key point constraint features, which can be manually set or selected by the user. The angle sequence constraint features include parameters in the x and y directions, and the boundary key point constraint features include 3 boundary point parameters, for example: the angle sequence constraint feature is (30, 60), 30 indicates a 30 degree rotation in the x-axis direction, 60 indicates a 60 degree rotation in the y-axis direction, and the boundary key point constraint features are ((35, 70), (55, 120), (75, 70)), which represent the coordinates of the left boundary point, the lower boundary point, and the right boundary point of the generated face. The computer processes the angle sequence constraint feature dimension (N, 2) through a sequence feature extraction module (composed of a single-layer fully connected network) to obtain an angle sequence constraint sequence dimension (N, 12), wherein the angle sequence constraint feature and the boundary key point constraint feature are represented by arrays, and the angle sequence constraint sequence is represented by a vector. For example: for N consecutive faces, the computer obtains the angle sequence constraint feature dimension of (N, 2), 2 represents the angle x, y direction parameters, and the angle sequence constraint sequence dimension after passing through the sequence feature extraction module (composed of a multi-layer fully connected network) is (N, 12).

[0065] S310, processing the boundary key point constraint feature to obtain a boundary key point constraint sequence, and superimposing the angle sequence constraint sequence and the boundary key point constraint sequence to obtain a sequence feature.

[0066] Generally, after the computer obtains the angle sequence constraint sequence, the boundary key point constraint feature is processed by a sequence feature extraction module (composed of a single-layer fully connected network) to obtain a boundary key point constraint sequence, wherein the boundary key point constraint sequence is represented by a vector, and the angle sequence constraint sequence and the boundary key point constraint sequence are superimposed to obtain a sequence feature, for example: for N consecutive human faces, the boundary key point constraint sequence obtained by the computer is (N, 6), and the dimension of the boundary key point constraint sequence after the sequence feature extraction module (composed of a single-layer fully connected network) is (N, 36). The boundary key point constraint sequence in this example is superimposed with the angle sequence constraint sequence in S309 to obtain a sequence feature of (N, 48), and the sequence feature is represented by a vector.

[0067] S311. Cat-superimpose the voice domain feature, the face feature, and the sequence feature to obtain a first superimposed feature, and Stack-superimpose the first superimposed feature to obtain an input feature.

[0068] Generally, the sound domain feature corresponds to the sound domain feature corresponding to a face generated in one frame, the face feature represents the face feature corresponding to a face generated in one frame, and the sequence feature represents the sequence feature corresponding to a face generated in one frame. For example, assuming that the number of mouth shape classification categories C is 32, the dimension of the sound domain feature corresponding to a face generated in one frame is (1, 256+30*C), that is, (1, 1216), the dimension of the face feature is (1, 512), and the dimension of the sequence feature is (1, 48). The computer Cat superimposes the sound domain feature, the face feature and the sequence feature to obtain a first superimposed feature. For example, N is an integer greater than 1, and the sound domain features of N frames are superimposed to obtain a first sound domain superimposed feature with a dimension of (N, 1216), the face features of N frames are superimposed to obtain a first face superimposed feature with a dimension of (N, 512), and the sequence features of N frames are superimposed to obtain a first sequence superimposed feature with a dimension of (N, 8). After the computer obtains the first superimposed feature, it performs Stack superposition on the first superimposed feature to obtain the input feature. For example, the dimension of the first sound domain superimposed feature is (N, 1216), the dimension of the first face superimposed feature is (N, 512), and the dimension of the first sequence superimposed feature is (N, 48). The computer performs Stack superposition to obtain an input feature dimension of (N, 1776).

[0069] S312: Process the input features to obtain features related to facial key points, and process the features related to facial key points through a multi-layer fully connected network to obtain a facial key point sequence.

[0070] Generally, after the computer obtains the input features, it uses a long short-term memory (LSTM) neural network to process the input features to obtain features related to facial key points, wherein the LSTM neural network has 256 hidden nodes and 3 layers, and the features related to facial key points are represented by vectors. After the computer obtains the features related to facial key points, it processes the features related to facial key points through a multi-layer fully connected network to obtain a facial key point sequence, wherein the facial key point sequence includes a parameter S associated with the sequence size and the length of the audio data, the number of facial key points P and the corresponding coordinates N, where N is always equal to 2, and the facial key point sequence is represented by an array (S, P, N). For example, the facial key point sequence obtained by the computer is (1, 50, 2), which means that a frame of face is generated, and the generated face has 50 facial key points and the x and y axis coordinates of each facial key point ((125, 75), (130, 80), (140, 83)...). The facial key point related features are processed through a multi-layer fully connected network to obtain a facial key point sequence, and the facial features are represented by a (S, 512) dimensional vector after encoding. For example, the 50 facial key point features are encoded into (1, 512) after passing through a multi-layer fully connected network to generate a frame of face.

[0071] When the scheme of the embodiment of the present application is executed, the audio data is feature extracted to obtain the sound domain feature, the template face is feature extracted to obtain the face feature, the face sequence is processed to obtain the sequence feature, the sound domain feature, the face feature and the sequence feature are superimposed to generate the input feature, and the face key point sequence is generated according to the input feature. The embodiment of the present application can directly generate relevant features mainly including phoneme features based on the audio data, and then process to obtain the information related to the face key points of the naturally changing virtual image, thereby improving the realism and fluency of the virtual image's mouth movements.

[0072] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0073] See also Figure 4 , which shows a schematic diagram of the structure of a facial key point generating device provided by an exemplary embodiment of the present application, hereinafter referred to as generating device 4. Generating device 4 can be implemented as all or part of the terminal through software, hardware or a combination of both. It includes:

[0074] The first extraction module 401 is used to extract features from the audio data to obtain sound domain features; wherein the sound domain features include phoneme mouth shape features and sound coding features;

[0075] The second extraction module 402 is used to extract features from the template face to obtain facial features;

[0076] The processing module 403 is used to process the face sequence to obtain sequence features; wherein the face sequence includes angle sequence constraint features and boundary key point constraint features;

[0077] A superposition module 404, configured to superimpose the voice domain feature, the face feature and the sequence feature to generate an input feature;

[0078] The generating module 405 is used to generate a facial key point sequence according to the input features.

[0079] Optionally, the first extraction module 401 further includes:

[0080] The splicing unit is used to extract features from audio data to obtain sound coding features; process the audio data to obtain phoneme mouth shape features; and splice the sound coding features and the mouth shape features to generate sound domain features.

[0081] A traversal unit is used to calculate the center position of the time interval of the audio data based on a preset frame rate; traverse the time interval to extract the Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position; and process the MFCC sound features through a convolutional neural network (CNN) sound coding network and a fully connected network (FC) to obtain sound coding features.

[0082] An alignment unit is used to process the audio data to obtain a phoneme sequence; determine the mouth shape feature corresponding to the phoneme sequence according to the mapping relationship between phonemes and mouth shapes; wherein the mouth shape feature is aligned with the MFCC sound feature; and process the mouth shape feature through dimensional conversion to obtain a phoneme mouth shape feature.

[0083] A classification unit is used to mark phonemes and mouth shapes in audio data; wherein the phonemes include Chinese phonemes and English phonemes, and the mouth shape is represented by the size of the mouth opening; the phonemes and the mouth shapes are classified by a clustering algorithm to obtain a classification result; and a mapping relationship between the phonemes and the mouth shapes is determined according to the classification result.

[0084] Optionally, the second extraction module 402 further includes:

[0085] The recognition unit is used to recognize the template face in the data set to obtain the facial key point coordinate information; count the facial key point coordinate information of all the facial key point in the data set to obtain the average facial key point coordinate information; determine the target facial key point coordinate information, and obtain the initial input feature based on the target facial key point coordinate information and the average facial key point coordinate information; and process the initial input feature to obtain the facial feature.

[0086] Optionally, the processing module 403 further includes:

[0087] An acquisition unit is used to acquire angle sequence constraint features and boundary key point constraint features; process the angle sequence constraint features to obtain an angle sequence constraint sequence; process the boundary key point constraint features to obtain a boundary key point constraint sequence; and superimpose the angle sequence constraint sequence and the boundary key point constraint sequence to obtain a sequence feature.

[0088] Optionally, the superposition module 404 further includes:

[0089] The merging unit is used to perform Cat superposition on the voice domain feature, the face feature and the sequence feature to obtain a first superposition feature; and perform Stack superposition on the first superposition feature to obtain an input feature.

[0090] Optionally, the generating module 405 further includes:

[0091] A processing unit is used to process the input features to obtain features related to facial key points; and to process the features related to facial key points through a multi-layer fully connected network to obtain a facial key point sequence; wherein the facial key point sequence includes a parameter associating a sequence size with an audio data length, the number of facial key points, and corresponding coordinates.

[0092] The embodiments of this application and Figures 2 to 3 The method embodiments are based on the same concept, and the technical effects they bring are also the same. The specific process can be referred to Figures 2 to 3 The description of the method embodiment will not be repeated here.

[0093] The device 4 can be a field-programmable gate array (FPGA) for realizing relevant functions, a dedicated integrated chip, a system on chip (SoC), a central processor unit (CPU), a graphics processor (GPU), an embedded neural network processor (NPU), a tensor processor (TPU), and other similar server-side and mobile-side image processors, neural network acceleration processors, network processors (NP), digital signal processing circuits, microcontrollers (MCU), and programmable logic devices (PLD) or other integrated chips can also be used.

[0094] When the scheme of the embodiment of the present application is executed, the audio data is feature extracted to obtain the sound domain feature, the template face is feature extracted to obtain the face feature, the face sequence is processed to obtain the sequence feature, the sound domain feature, the face feature and the sequence feature are superimposed to generate the input feature, and the face key point sequence is generated according to the input feature. The embodiment of the present application can directly generate relevant features mainly including phoneme features based on the audio data, and then process to obtain the information related to the face key points of the naturally changing virtual image, thereby improving the realism and fluency of the virtual image's mouth movements.

[0095] The present application also provides a computer storage medium, which can store multiple instructions, which are suitable for being loaded by a processor and executing the above method steps. The specific execution process can be seen in Figure 2 or Figure 3 The specific description of the illustrated embodiment will not be repeated here.

[0096] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the template control method described in the above embodiments.

[0097] See also Figure 5 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 5 may include: at least one processor 501 , at least one network interface 504 , a user interface 503 , a memory 505 , and at least one communication bus 502 .

[0098] The communication bus 502 is used to realize the connection and communication between these components.

[0099] The user interface 503 may include a display screen (Display) and a microphone (Microphone). The optional user interface 503 may also include a standard wired interface and a wireless interface.

[0100] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0101] Among them, the processor 501 may include one or more processing cores. The processor 501 uses various interfaces and lines to connect various parts in the entire terminal 500, and executes various functions and processes data of the terminal 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 505, and calling data stored in the memory 505. Optionally, the processor 501 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 501 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for all tensor operations in the deep learning network, and is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 501, and it can be implemented separately through a chip.

[0102] Among them, the memory 505 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 505 includes a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 505 may also be optionally at least one storage device located away from the aforementioned processor 501. As Figure 5 As shown, the memory 505 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a facial key point generation application.

[0103] exist Figure 5 In the electronic device 500 shown, the user interface 503 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 501 can be used to call the facial key point generation application stored in the memory 505, and specifically perform the following operations:

[0104] Extracting features from the audio data to obtain sound domain features; wherein the sound domain features include phoneme mouth shape features and sound coding features;

[0105] Extract features from the template face to obtain facial features;

[0106] Processing a face sequence to obtain a sequence feature; wherein the face sequence includes an angle sequence constraint feature and a boundary key point constraint feature;

[0107] Superimposing the voice domain feature, the face feature and the sequence feature to generate an input feature;

[0108] Generate a facial key point sequence according to the input features.

[0109] In one embodiment, the processor 501 performs the feature extraction on the audio data to obtain the sound domain feature, including:

[0110] Extract features from audio data to obtain sound coding features;

[0111] Processing the audio data to obtain phoneme mouth shape features;

[0112] The sound coding feature and the mouth shape feature are concatenated to generate a sound domain feature.

[0113] In one embodiment, the processor 501 performs the feature extraction on the audio data to obtain the sound coding feature, including:

[0114] Calculate the center position of the time interval of the audio data based on a preset frame rate;

[0115] Traversing the time interval, extracting Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position;

[0116] The MFCC sound features are processed by a convolutional neural network (CNN) sound coding network and a fully connected network (FC) to obtain sound coding features.

[0117] In one embodiment, the processor 501 performs the processing of the audio data to obtain the phoneme mouth shape feature, including:

[0118] Processing the audio data to obtain a phoneme sequence;

[0119] Determining the mouth shape feature corresponding to the phoneme sequence according to the mapping relationship between the phoneme and the mouth shape; wherein the mouth shape feature is aligned with the MFCC sound feature;

[0120] The mouth shape features are processed by dimensional conversion to obtain phoneme mouth shape features.

[0121] In one embodiment, before the processor 501 performs the feature extraction of the audio data to obtain the sound domain feature, the process further includes:

[0122] Marking phonemes and mouth shapes in audio data; wherein the phonemes include Chinese phonemes and English phonemes, and the mouth shapes are represented by the size of the mouth opening;

[0123] Classifying the phonemes and the mouth shapes by a clustering algorithm to obtain a classification result;

[0124] A mapping relationship between the phoneme and the mouth shape is determined according to the classification result.

[0125] In one embodiment, the processor 501 performs the feature extraction on the template face to obtain the facial features, including:

[0126] Identify the template face in the data set to obtain the coordinate information of the key points of the face;

[0127] Counting all the facial key point coordinate information in the data set to obtain average facial key point coordinate information;

[0128] Determine the target face key point coordinate information, and obtain the initial input feature based on the target face key point coordinate information and the average face key point coordinate information;

[0129] The initial input features are processed to obtain facial features.

[0130] In one embodiment, the processor 501 performs the processing of the face sequence to obtain the sequence feature, including:

[0131] Obtain angle sequence constraint features and boundary key point constraint features;

[0132] Processing the angle sequence constraint feature to obtain an angle sequence constraint sequence;

[0133] Processing the boundary key point constraint features to obtain a boundary key point constraint sequence;

[0134] The angle sequence constraint sequence and the boundary key point constraint sequence are superimposed to obtain a sequence feature.

[0135] In one embodiment, the processor 501 performs the step of superimposing the voice domain feature, the face feature, and the sequence feature to generate an input feature, including:

[0136] Cat-superimpose the voice domain feature, the face feature and the sequence feature to obtain a first superimposed feature;

[0137] The first superposition features are stacked to obtain input features.

[0138] In one embodiment, the processor 501 executes the step of generating a facial key point sequence according to the input feature, including:

[0139] Processing the input features to obtain features related to key points of the face;

[0140] The facial key point related features are processed through a multi-layer fully connected network to obtain a facial key point sequence; wherein the facial key point sequence includes a parameter associated with the sequence size and the audio data length, the number of facial key points and the corresponding coordinates.

[0141] The technical concept and Figure 2 or Figure 3 The technical concept is the same as that of Figure 2 or Figure 3 The method embodiments are not described in detail here.

[0142] In the embodiment of the present application, feature extraction is performed on audio data to obtain sound domain features, feature extraction is performed on template faces to obtain facial features, and sequence features are processed to obtain sequence features. The sound domain features, facial features, and sequence features are superimposed to generate input features, and a sequence of facial key points is generated based on the input features. The embodiment of the present application can directly generate relevant features mainly including phoneme features based on audio data, and then process to obtain information related to facial key points of a naturally changing virtual image, thereby improving the realism and fluency of the virtual image's mouth movements.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0144] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for generating facial key points, characterized in that: The method comprises: Extracting features from the audio data to obtain sound domain features; wherein the sound domain features include phoneme mouth shape features and sound coding features; Extract features from the template face to obtain facial features; Processing a face sequence to obtain a sequence feature; wherein the face sequence includes an angle sequence constraint feature and a boundary key point constraint feature; Superimposing the voice domain feature, the face feature and the sequence feature to generate an input feature; Generate a facial key point sequence according to the input features; The step of extracting features from the audio data to obtain sound domain features includes: Extract features from audio data to obtain sound coding features; Processing the audio data to obtain phoneme mouth shape features; Splicing the sound coding feature and the mouth shape feature to generate a sound domain feature; The step of extracting features from the template face to obtain facial features includes: Identify the template face in the data set to obtain the coordinate information of the key points of the face; Counting all the facial key point coordinate information in the data set to obtain average facial key point coordinate information; Determine the target face key point coordinate information, and obtain the initial input feature based on the target face key point coordinate information and the average face key point coordinate information; Processing the initial input features to obtain facial features; The step of processing the face sequence to obtain sequence features includes: Obtain angle sequence constraint features and boundary key point constraint features; Processing the angle sequence constraint feature to obtain an angle sequence constraint sequence; Processing the boundary key point constraint features to obtain a boundary key point constraint sequence; Superimposing the angle sequence constraint sequence and the boundary key point constraint sequence to obtain a sequence feature; The step of generating a facial key point sequence according to the input features includes: Processing the input features to obtain features related to key points of the face; The facial key point related features are processed through a multi-layer fully connected network to obtain a facial key point sequence; wherein the facial key point sequence includes a parameter associated with the sequence size and the audio data length, the number of facial key points and the corresponding coordinates.

2. The method according to claim 1, characterized in that: The step of extracting features from the audio data to obtain sound coding features includes: Calculate the center position of the time interval of the audio data based on a preset frame rate; Traversing the time interval, extracting Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position; The MFCC sound features are processed by a convolutional neural network (CNN) sound coding network and a fully connected network (FC) to obtain sound coding features.

3. The method according to claim 2, characterized in that: The step of processing the audio data to obtain phoneme mouth shape features includes: Processing the audio data to obtain a phoneme sequence; Determining the mouth shape feature corresponding to the phoneme sequence according to the mapping relationship between the phoneme and the mouth shape; wherein the mouth shape feature is aligned with the MFCC sound feature; The mouth shape features are processed by dimensional conversion to obtain phoneme mouth shape features.

4. The method according to claim 3, characterized in that: Before extracting the features of the audio data to obtain the sound domain features, the method further includes: Marking phonemes and mouth shapes in audio data; wherein the phonemes include Chinese phonemes and English phonemes, and the mouth shapes are represented by the size of the mouth opening; Classifying the phonemes and the mouth shapes by a clustering algorithm to obtain a classification result; A mapping relationship between the phoneme and the mouth shape is determined according to the classification result.

5. The method according to claim 1, characterized in that: The step of superimposing the voice domain feature, the face feature and the sequence feature to generate the input feature comprises: Cat-superimpose the voice domain feature, the face feature and the sequence feature to obtain a first superimposed feature; The first superposition features are stacked to obtain input features.

6. The method according to claim 1, characterized in that: The processing of the input features to obtain features related to key points of the face is processing the input features using a long short-term memory (LSTM) neural network to obtain features related to key points of the face.

7. A facial key point generation device, characterized in that: include: A first extraction module is used to extract features from audio data to obtain sound domain features; wherein the sound domain features include phoneme mouth shape features and sound coding features; The second extraction module is used to extract features from the template face to obtain facial features; A processing module, used to process a face sequence to obtain a sequence feature; wherein the face sequence includes an angle sequence constraint feature and a boundary key point constraint feature; A superposition module, used for superimposing the voice domain feature, the face feature and the sequence feature to generate an input feature; A generation module, used to generate a sequence of facial key points according to the input features; The first extraction module further includes: A splicing unit is used to extract features from audio data to obtain sound coding features; process the audio data to obtain phoneme mouth shape features; and splice the sound coding features and the mouth shape features to generate sound domain features; The second extraction module further includes: The recognition unit is used to recognize the template face in the data set to obtain the coordinate information of the key points of the face; to count the coordinate information of all the key points of the face in the data set to obtain the average coordinate information of the key points of the face; to determine the coordinate information of the key points of the target face, and to obtain the initial input features based on the coordinate information of the key points of the target face and the average coordinate information of the key points of the face; and to process the initial input features to obtain the face features; The processing module further includes: An acquisition unit is used to acquire angle sequence constraint features and boundary key point constraint features; process the angle sequence constraint features to obtain an angle sequence constraint sequence; process the boundary key point constraint features to obtain a boundary key point constraint sequence; and superimpose the angle sequence constraint sequence and the boundary key point constraint sequence to obtain a sequence feature; The generating module further includes: A processing unit is used to process the input features to obtain features related to facial key points; and to process the features related to facial key points through a multi-layer fully connected network to obtain a facial key point sequence; wherein the facial key point sequence includes a parameter associating a sequence size with an audio data length, the number of facial key points, and corresponding coordinates.

8. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as described in any one of claims 1 to 6.