Method, device, storage medium and electronic device for generating key points of human face
By performing feature extraction and face sequence processing on audio data, combining multi-layer fully connected networks and LSTM neural networks, a sequence of virtual faces with natural changes is solved, and the problem of unnatural virtual face movements in the existing technology is solved. It significantly improves the authenticity and fluency of virtual images.
Patent Information
- Application Number
- CN202011463289.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-11
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2040-12-11
AI Technical Summary
It is difficult for the prior art to directly generate natural-changing virtual face images through audio data. The generated face actions of the faces are mostly fixed, and lack realism and fluency.
By extracting the audio data feature, combining the template face features and face sequence features, a multi-layer fully connected network and long and short-term memory neural network are used to generate a naturally changing virtual face key point sequence, improving the authenticity and smoothness of the virtual image Spend.
It realizes the direct generation of natural changes in the key points of the virtual face based on audio data, which significantly improves the authenticity and fluency of virtual image speech actions.
Smart Images
Figure CN112668408B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and electronic device for generating key points of a human face. Background Art
[0002] Sound and vision play an important role in the process of human transmission / reception of information, and these two methods contain overlapping information to a certain extent. For example, when we hear a familiar person speaking, we can infer what the person in the picture said, which shows that the two types of information can be converted into each other. In some actual business scenarios, it has great commercial application value. With the development of deep learning and the increase in computer computing power, many audio processing and image generation methods based on deep learning have achieved good results. The current method of directly generating facial images through audio, the movements of the generated speaking faces cannot change naturally, or can only change according to a fixed pattern. In response to this phenomenon, there is an urgent need for a method that can directly process the input audio data and generate high-quality virtual images with corresponding mouth movements and natural changes in facial expressions. Summary of the invention
[0003] The embodiments of the present application provide a method, device, storage medium and electronic device for generating key points of a human face, which can directly generate a naturally changing virtual image based on audio data to improve the realism. The technical solution is as follows:
[0004] In a first aspect, an embodiment of the present application provides a method for generating facial key points, comprising:
[0005] Extract features from audio data to obtain sound features;
[0006] Extract features from the template face to obtain facial features;
[0007] Processing a face sequence to obtain a sequence feature; wherein the face sequence includes an angle sequence constraint feature and a boundary key point constraint feature;
[0008] Superimposing the sound feature, the face feature and the sequence feature to generate an input feature;
[0009] Generate a facial key point sequence according to the input features.
[0010] In a second aspect, an embodiment of the present application provides a device for generating facial key points, the device comprising:
[0011] A first extraction module, used for extracting features from audio data to obtain sound features;
[0012] The second extraction module is used to extract features from the template face to obtain facial features;
[0013] A processing module, used to process a face sequence to obtain sequence features; wherein the face sequence includes angle sequence constraint features and boundary key point constraint features;
[0014] A superposition module, used for superimposing the sound feature, the face feature and the sequence feature to generate an input feature;
[0015] A generation module is used to generate a sequence of facial key points according to the input features.
[0016] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.
[0017] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.
[0018] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:
[0019] When the above-mentioned facial key point generation method, device, storage medium and electronic device are working, feature extraction is performed on audio data to obtain sound features, feature extraction is performed on template faces to obtain facial features, and facial sequences are processed to obtain sequence features. The sound features, facial features and sequence features are superimposed to generate input features, and facial key point sequences are generated based on the input features. The embodiments of the present application can directly generate information related to facial key points of a naturally changing virtual image based on audio data, thereby improving the authenticity and fluency of the virtual image's speech movements. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 It is a schematic diagram of a communication system architecture provided by an embodiment of the present application;
[0022] Figure 2 It is a flowchart of a method for generating key points of a face provided in an embodiment of the present application;
[0023] Figure 3It is another flowchart of a method for generating key points of a face provided in an embodiment of the present application;
[0024] Figure 4 It is a structural schematic diagram of a facial key point generation device provided in an embodiment of the present application;
[0025] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
[0027] When the following description is designed with drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0028] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.
[0029] In order to solve the problem mentioned above that the current method of directly generating facial images through audio, the movements of the generated speaking face cannot change naturally or can only change according to a fixed pattern, a method for generating facial key points is proposed. The method of the embodiment of the present application can be implemented by a computer program and can be run on a computer system based on the von Neumann system and equipped with a camera or a rotatable camera. The computer system can be a computer system of a device such as a smart phone, a laptop computer, or a tablet computer.
[0030] Figure 1 A schematic diagram of a communication system architecture provided for this application.
[0031] See also Figure 1, the communication system 01 includes a terminal device 101, a network device 102, and a server 103; when the communication system 01 includes a core network, the network device 102 can also be connected to the core network. The network device 102 can also communicate with an Internet Protocol (IP) network, such as the Internet, a private IP network, or other data networks. The network device 102 provides services for the terminal devices 101 and the server 103 within the coverage area. The user can use the terminal device 101 to interact with the server 103 through the network device 102 to receive or send messages, etc. The terminal device 101 can be installed with various communication client applications, such as voice interaction applications, animation production applications, etc. The server 103 can be a server that stores the face key point generation method provided in the embodiment of the present application and provides various services, and is used to detect, store and process the audio data, template face and other files uploaded by the terminal device 101, and send the processing results to the terminal device 101.
[0032] In the following method embodiments, for the sake of convenience, only the execution subject of each step is introduced and described as a computer.
[0033] The following will be combined with the attached Figures 2 to 3 , the facial key point generation method provided in the embodiment of the present application is introduced in detail.
[0034] See also Figure 2 , a flow chart of a method for generating facial key points is provided for an embodiment of the present application. The method may include the following steps:
[0035] S201, extracting features from audio data to obtain sound features.
[0036] Generally, a feature can be a characteristic or property that distinguishes a certain type of object from other types of objects, or a collection of these characteristics or properties. A feature can be data that can be extracted by measuring or processing. For audio data, a sound feature can be a characteristic of the audio that can be distinguished from other types of audio, such as: Mel-scale Frequency Cepstral Cofficients (MFCC) features, Mel Filter Bank (MFB) features, Spectral Subband Centroid (SSC) features, etc. The computer calculates the center position of the time interval of the audio data based on a preset frame rate, traverses the time interval to extract the Mel-scale Cepstral Coefficient MFCC sound features in the sub-time interval of a preset length before and after the center position, and processes the MFCC sound features to obtain sound features.
[0037] S202: Extract features of the template face to obtain facial features.
[0038] Generally, facial features refer to facial key point coordinate information features, such as 81 facial key point coordinates or 68 facial key point coordinates. The computer recognizes the template face in the data set to obtain facial key point coordinate information, counts all the facial key point coordinate information in the data set to obtain average facial key point coordinate information, determines the target facial key point coordinate information, obtains initial input features based on the target facial key point coordinate information and the average facial key point coordinate information, and processes the initial input features to obtain facial features.
[0039] S203: Process the face sequence to obtain sequence features.
[0040] Generally, the face sequence includes angle sequence constraint features and boundary key point constraint features, and the user can manually set or select a template. The angle sequence constraint features include parameters in the x and y directions, and the boundary key point constraint features include three boundary point parameters. The computer obtains the angle sequence constraint features and the boundary key point constraint features, processes the angle sequence constraint features to obtain an angle sequence constraint sequence, processes the boundary key point constraint features to obtain a boundary key point constraint sequence, and superimposes the angle sequence constraint sequence and the boundary key point constraint sequence to obtain a sequence feature.
[0041] S204: Superimpose the sound feature, the face feature and the sequence feature to generate an input feature.
[0042] Generally, superposition refers to fusing multiple vectors or arrays into one vector or array, including Cat superposition and Stack superposition. The computer performs Cat superposition on the sound feature, the face feature and the sequence feature to obtain a first superposition feature, and performs Stack superposition on the first superposition feature to obtain an input feature. Cat superposition can be understood as continuation, which does not increase the dimension, and Stack superposition can be understood as superposition, which adds a new dimension (the added dimension depends on the input dimension).
[0043] S205: Generate a facial key point sequence according to the input features.
[0044] Generally, the facial key point sequence includes a sequence size and an audio data length association parameter, the number of facial key points, and corresponding coordinates. The computer processes the input features to obtain facial key point related features, and processes the facial key point related features through a multi-layer fully connected network to obtain a facial key point sequence, wherein the processing of the input features to obtain facial key point related features is to use a long short-term memory (LSTM) neural network to process the input features to obtain facial key point related features.
[0045] As can be seen from the above content, the audio data is feature extracted to obtain the sound feature, the template face is feature extracted to obtain the face feature, the face sequence is processed to obtain the sequence feature, the sound feature, the face feature and the sequence feature are superimposed to generate the input feature, and the face key point sequence is generated according to the input feature. The embodiment of the present application can directly generate the face key point related information of the naturally changing virtual image based on the audio data, and improve the realism and fluency of the virtual image's speech action.
[0046] See also Figure 3 , another flow chart of a method for generating key points of a human face is provided in an embodiment of the present application. The method for generating key points of a human face may include the following steps:
[0047] S301. Calculate a center position of a time interval of the audio data based on a preset frame rate.
[0048] Generally, the frame rate refers to the frequency / rate at which a bitmap image in units of frames appears continuously on a display. The computer needs to generate a sequence of facial key points based on audio data, and the facial key point sequence corresponds to the facial key point coordinates of consecutive frames. Therefore, it is necessary to first determine the frame rate of the generated virtual image video. For example, if the frame rate is determined to be 25 frames per second and the audio data is 3 minutes, the generated video has 4500 frames, the number of MFCC features of the audio data per second is 100, and there are 100 MFCC features in 1 second that are related to the audio duration. There are 25 corresponding generated faces in 1 second, and the 100 MFCC features are divided into 25 parts, and the MFCC features at the center position of each of the 25 parts are determined as the features at the center position of the generated face.
[0049] S302, traversing the time interval to extract Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position, and processing the MFCC sound features to obtain sound features.
[0050] Generally, the sub-time interval of the preset length refers to the duration of the audio data corresponding to each frame of the generated face image, for example, the sub-time interval of the preset length is 150ms. After the computer determines the center position of the time interval of the audio data, it traverses the time interval to extract the Mel cepstral coefficient MFCC sound features in the sub-time interval of the preset length before and after the center position, and processes the MFCC sound features to obtain sound features, for example, extracting the 13-dimensional MFCC features of the audio data in the sub-time interval, the first-order derivative features (12 dimensions) and the second-order derivative features (11 dimensions) of MFCC, and Cat are combined to form a first sound feature of (1, 36) dimensions of 36 MFCC related dimensions, traversing 1 second of audio data, obtaining a second sound feature of (30, 36) dimensions corresponding to a total of 300ms before and after the center position, using a convolutional network and a fully connected network as a sound feature encoder to extract the second sound feature to obtain a sound feature, and the dimension of the encoded sound feature vector is (1, 256), wherein the first sound feature and the second sound feature are represented by arrays, and the sound feature is represented by a vector.
[0051] S303, identifying the template face in the data set to obtain the facial key point coordinate information, and counting the facial key point coordinate information of all the facial key point in the data set to obtain the average facial key point coordinate information.
[0052] Generally, a data set is a collection of multiple template faces pre-provided by a user in a computer. Before the computer identifies the template face in the data set and obtains the coordinate information of the key points of the face, it also includes: detecting the face image in the data set based on the face detection algorithm, obtaining the detection result file, the face detection algorithm includes the face recognition dlib face image extraction algorithm, the user preset face key point positioning model algorithm or the artificial intelligence public platform call algorithm (such as Baidu, Kuangshi platform), parsing the detection result file to generate the template face, wherein the information in the detection result file includes the coordinates of multiple key points of the face, such as the cheek coordinates, eyebrow coordinates, eye coordinates, mouth coordinates and nose coordinates. After the computer obtains the sound features, it identifies the template face in the data set to obtain the facial key point coordinate information, and counts all the facial key point coordinate information in the data set to obtain the average facial key point coordinate information. For example, using Dlib feature recognition or a neural network such as a deep network to identify the 68 facial key point coordinate information in the first template face is ((73, 25), (85, 30), (90, 34), ...), and identifying the 68 facial key point coordinate information in the second template face is ((65, 20), (87, 32), (92, 30), ...), then the average facial key point coordinate information is ((69, 22.5), (86, 31), (91, 32), ...).
[0053] S304: Determine target facial key point coordinate information, obtain initial input features based on the target facial key point coordinate information and the average facial key point coordinate information, and process the initial input features to obtain facial features.
[0054] Generally, after obtaining the average facial key point coordinate information, the computer determines the target facial key point coordinate information, obtains the initial input features based on the target facial key point coordinate information and the average facial key point coordinate information, and processes the initial input features to obtain facial features. For example, the computer determines that the 68 facial key point coordinate information ((73, 25), (85, 30), (90, 34), ...) in the template face No. 1 is the target facial key point coordinate information, and subtracts the average facial key point coordinate information ((69, 22.5), (86, 31), (91, 32), ...) from the target facial key point coordinate information to obtain the initial input features, which can be expressed as ((4, 2.5), (-1, -1), (-1, 2), ...). The initial input features are processed by a facial key point feature extraction module (composed of a multi-layer fully connected network) to obtain facial features, wherein the initial input features are represented by an array and the facial features are represented by a vector.
[0055] S305 , obtaining angle sequence constraint features and boundary key point constraint features, and processing the angle sequence constraint features to obtain an angle sequence constraint sequence.
[0056] Generally, the face sequence includes angle sequence constraint features and boundary key point constraint features. The user can manually set or select a template. The angle sequence constraint features include parameters in the x and y directions. For example, the angle sequence constraint feature is (30, 60), where 30 represents a 30-degree rotation in the x-axis direction and 60 represents a 60-degree rotation in the y-axis direction. The computer processes the angle sequence constraint features through a sequence feature extraction module (composed of a single-layer fully connected network) to obtain an angle sequence constraint sequence, wherein the angle sequence constraint features and the boundary key point constraint features are represented by arrays, and the angle sequence constraint sequence is represented by a vector. For example, for N consecutive faces, the computer obtains the angle sequence constraint feature dimension of (N, 2), where 2 represents the angle x and y direction parameters, and the angle sequence constraint sequence dimension after passing through the sequence feature extraction module (composed of a single-layer fully connected network) is (N, 12).
[0057] S306: Process the boundary key point constraint features to obtain a boundary key point constraint sequence, and superimpose the angle sequence constraint sequence and the boundary key point constraint sequence to obtain a sequence feature.
[0058] Generally, after the computer obtains the angle sequence constraint sequence, the boundary key point constraint feature is processed by the sequence feature extraction module (composed of a single-layer fully connected network) to obtain the boundary key point constraint sequence. The boundary key point constraint feature includes 6 parameters in the x and y directions of the three boundary points. For example, the boundary key point constraint feature is ((35,70), (55,120), (75,70)), which represents the coordinates of the three points of the left boundary point, the lower boundary point and the right boundary point of the generated face, a total of 6 parameters. Among them, the boundary key point constraint sequence is represented by a vector, and the angle sequence constraint sequence and the boundary key point constraint sequence are superimposed to obtain a sequence feature. For example, for N consecutive human faces, the computer obtains the boundary key point constraint sequence as (N,6), and the dimension of the boundary key point constraint sequence after the sequence feature extraction module (composed of a single-layer fully connected network) is (N,36). The boundary key point constraint sequence in this embodiment is superimposed with the angle sequence constraint sequence in S305 to obtain a sequence feature of (N,48), and the sequence feature is represented by a vector.
[0059] S307: Cat-superimpose the sound feature, the face feature and the sequence feature to obtain a first superimposed feature.
[0060] Generally, the sound feature corresponds to the sound feature corresponding to a face generated in one frame, the face feature represents the face feature corresponding to a face generated in one frame, and the sequence feature represents the sequence feature corresponding to a face generated in one frame. For example, the dimension of the sound feature corresponding to a face generated in one frame is (1,256), the dimension of the face feature is (1,512), and the dimension of the sequence feature is (1,48). The computer Cat-superimposes the sound feature, the face feature and the sequence feature to obtain a first superimposed feature. For example, N is an integer greater than 1, and the sound features of N frames are superimposed to obtain a first sound superimposed feature with a dimension of (N,256), the face features of N frames are superimposed to obtain a first face superimposed feature with a dimension of (N,512), and the sequence features of N frames are superimposed to obtain a first sequence superimposed feature with a dimension of (N,48).
[0061] S308: Stack the first superposition features to obtain input features.
[0062] Generally, after the computer obtains the first superimposed feature, it performs Stack superposition on the first superimposed feature to obtain the input feature. For example, the dimension of the first sound superimposed feature is (N, 256), the dimension of the first face superimposed feature is (N, 512), and the dimension of the first sequence superimposed feature is (N, 48). The computer performs Stack superposition to obtain an input feature dimension of (N, 816).
[0063] S309: Process the input features to obtain features related to facial key points.
[0064] Generally, after the computer obtains the input features, it uses a long short-term memory (LSTM) neural network to process the input features to obtain features related to facial key points, wherein the LSTM neural network has 256 hidden nodes and 3 layers, and the features related to facial key points are represented by vectors.
[0065] S310, processing the facial key point related features through a multi-layer fully connected network to obtain a facial key point sequence.
[0066] Generally, after the computer obtains the features related to the key points of the face, it processes the features related to the key points of the face through a multi-layer fully connected network to obtain a sequence of key points of the face, wherein the features related to the key points of the face include a parameter S associated with the sequence size and the length of the audio data, the number of key points of the face P and the corresponding coordinates N, where N is always equal to 2, and the sequence of key points of the face is represented by an array (S, P, N). For example, the sequence of key points of the face obtained by the computer is (1, 50, 2), which indicates that a frame of face is generated, and the generated face has 50 key points of the face and the x-axis and y-axis coordinates of each key point of the face ((125, 75), (130, 80), (140, 83)...). The key points of the face are processed through a multi-layer fully connected network to obtain a sequence of key points of the face, and after encoding, the face features are represented by a (S, 512) dimensional vector. For example, the 50 key points of the face are encoded as (1, 512) through a multi-layer fully connected network to indicate that a frame of face is generated.
[0067] When the scheme of the embodiment of the present application is executed, the center position of the time interval of the audio data is calculated based on a preset frame rate, the time interval is traversed to extract the Mel-frequency cepstral coefficient MFCC sound features in the sub-time interval of a preset length before and after the center position, the MFCC sound features are processed to obtain sound features, the template face in the data set is identified to obtain the facial key point coordinate information, all the facial key point coordinate information in the data set are counted to obtain the average facial key point coordinate information, the target facial key point coordinate information is determined, the initial input features are obtained based on the target facial key point coordinate information and the average facial key point coordinate information, and the initial input features are processed to obtain Facial features, obtaining angle sequence constraint features and boundary key point constraint features, processing the angle sequence constraint features to obtain angle sequence constraint sequences, processing the boundary key point constraint features to obtain boundary key point constraint sequences, superimposing the angle sequence constraint sequences and the boundary key point constraint sequences to obtain sequence features, Cat superimposing the sound features, the facial features and the sequence features to obtain a first superimposed feature, Stack superimposing the first superimposed feature to obtain an input feature, processing the input feature to obtain facial key point related features, and processing the facial key point related features through a multi-layer fully connected network to obtain a facial key point sequence. The embodiment of the present application can directly generate facial key point related information of a naturally changing virtual image based on audio data, thereby improving the authenticity and fluency of the virtual image's speech movements.
[0068] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0069] See also Figure 4 , which shows a schematic diagram of the structure of a facial key point generating device provided by an exemplary embodiment of the present application, hereinafter referred to as generating device 4. Generating device 4 can be implemented as all or part of the terminal through software, hardware or a combination of both. It includes:
[0070] The first extraction module 401 is used to extract features from audio data to obtain sound features;
[0071] The second extraction module 402 is used to extract features from the template face to obtain facial features;
[0072] The processing module 403 is used to process the face sequence to obtain sequence features; wherein the face sequence includes angle sequence constraint features and boundary key point constraint features;
[0073] A superposition module 404, configured to superimpose the sound feature, the face feature and the sequence feature to generate an input feature;
[0074] The generating module 405 is used to generate a facial key point sequence according to the input features.
[0075] Optionally, the first extraction module 401 further includes:
[0076] A calculation unit is used to calculate the center position of the time interval of the audio data based on a preset frame rate; traverse the time interval to extract the Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position; and process the MFCC sound features to obtain sound features.
[0077] Optionally, the second extraction module 402 further includes:
[0078] The recognition unit is used to recognize the template face in the data set to obtain the facial key point coordinate information; count the facial key point coordinate information of all the facial key point in the data set to obtain the average facial key point coordinate information; determine the target facial key point coordinate information, and obtain the initial input feature based on the target facial key point coordinate information and the average facial key point coordinate information; and process the initial input feature to obtain the facial feature.
[0079] The parsing unit is used to detect the face image in the data set based on the face detection algorithm and obtain the detection result file; parse the detection result file to generate a template face; wherein the information in the detection result file includes the coordinates of multiple face key points such as cheek coordinates, eyebrow coordinates, eye coordinates, mouth coordinates and nose coordinates.
[0080] Optionally, the processing module 403 further includes:
[0081] An acquisition unit is used to acquire angle sequence constraint features and boundary key point constraint features; process the angle sequence constraint features to obtain an angle sequence constraint sequence; process the boundary key point constraint features to obtain a boundary key point constraint sequence; and superimpose the angle sequence constraint sequence and the boundary key point constraint sequence to obtain a sequence feature.
[0082] Optionally, the superposition module 404 further includes:
[0083] The merging unit is used to perform Cat superposition on the sound feature, the face feature and the sequence feature to obtain a first superposition feature; and perform Stack superposition on the first superposition feature to obtain an input feature.
[0084] Optionally, the generating module 405 further includes:
[0085] A processing unit is used to process the input features to obtain features related to facial key points; and to process the features related to facial key points through a multi-layer fully connected network to obtain a facial key point sequence; wherein the facial key point sequence includes a parameter associating a sequence size with an audio data length, the number of facial key points, and corresponding coordinates.
[0086] The embodiments of this application and Figures 2 to 3 The method embodiments are based on the same concept, and the technical effects they bring are also the same. The specific process can be referred to Figures 2 to 3 The description of the method embodiment will not be repeated here.
[0087] The device 4 can be a field-programmable gate array (FPGA) for realizing relevant functions, a dedicated integrated chip, a system on chip (SoC), a central processor unit (CPU), a graphics processor (GPU), an embedded neural network processor (NPU), a tensor processor (TPU), and other similar server-side and mobile-side image processors, neural network acceleration processors, network processors (NP), digital signal processing circuits, microcontrollers (MCU), and programmable logic devices (PLD) or other integrated chips can also be used.
[0088] When the scheme of the embodiment of the present application is executed, the center position of the time interval of the audio data is calculated based on a preset frame rate, the time interval is traversed to extract the Mel-frequency cepstral coefficient MFCC sound features in the sub-time interval of a preset length before and after the center position, the MFCC sound features are processed to obtain sound features, the template face in the data set is identified to obtain the facial key point coordinate information, all the facial key point coordinate information in the data set are counted to obtain the average facial key point coordinate information, the target facial key point coordinate information is determined, the initial input features are obtained based on the target facial key point coordinate information and the average facial key point coordinate information, and the initial input features are processed to obtain Facial features, obtaining angle sequence constraint features and boundary key point constraint features, processing the angle sequence constraint features to obtain angle sequence constraint sequences, processing the boundary key point constraint features to obtain boundary key point constraint sequences, superimposing the angle sequence constraint sequences and the boundary key point constraint sequences to obtain sequence features, Cat superimposing the sound features, the facial features and the sequence features to obtain a first superimposed feature, Stack superimposing the first superimposed feature to obtain an input feature, processing the input feature to obtain facial key point related features, and processing the facial key point related features through a multi-layer fully connected network to obtain a facial key point sequence. The embodiment of the present application can directly generate facial key point related information of a naturally changing virtual image based on audio data, thereby improving the authenticity and fluency of the virtual image's speech movements.
[0089] The present application also provides a computer storage medium, which can store multiple instructions, which are suitable for being loaded by a processor and executing the above method steps. The specific execution process can be seen in Figure 2 or Figure 3 The specific description of the illustrated embodiment will not be repeated here.
[0090] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the template control method described in the above embodiments.
[0091] See also Figure 5 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 5 may include: at least one processor 501 , at least one network interface 504 , a user interface 503 , a memory 505 , and at least one communication bus 502 .
[0092] The communication bus 502 is used to realize the connection and communication between these components.
[0093] The user interface 503 may include a display screen (Display) and a microphone (Microphone). The optional user interface 503 may also include a standard wired interface and a wireless interface.
[0094] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0095] Among them, the processor 501 may include one or more processing cores. The processor 501 uses various interfaces and lines to connect various parts in the entire terminal 500, and executes various functions and processes data of the terminal 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 505, and calling data stored in the memory 505. Optionally, the processor 501 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 501 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is used to perform all tensor operations in the deep learning network, and is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 501, and it can be implemented separately through a chip.
[0096] Among them, the memory 505 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 505 includes a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 505 may also be optionally at least one storage device located away from the aforementioned processor 501. As Figure 5 As shown, the memory 505 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a facial key point generation application.
[0097] exist Figure 5 In the electronic device 500 shown, the user interface 503 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 501 can be used to call the facial key point generation application stored in the memory 505, and specifically perform the following operations:
[0098] Extract features from audio data to obtain sound features;
[0099] Extract features from the template face to obtain facial features;
[0100] Processing a face sequence to obtain a sequence feature; wherein the face sequence includes an angle sequence constraint feature and a boundary key point constraint feature;
[0101] Superimposing the sound feature, the face feature and the sequence feature to generate an input feature;
[0102] Generate a facial key point sequence according to the input features.
[0103] In one embodiment, the processor 501 performs feature extraction on the audio data to obtain the sound feature, including:
[0104] Calculate the center position of the time interval of the audio data based on a preset frame rate;
[0105] Traversing the time interval, extracting Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position;
[0106] The MFCC sound features are processed to obtain sound features.
[0107] In one embodiment, the processor 501 performs the feature extraction on the template face to obtain the facial features, including:
[0108] Identify the template face in the data set to obtain the coordinate information of the key points of the face;
[0109] Counting all the facial key point coordinate information in the data set to obtain average facial key point coordinate information;
[0110] Determine the target face key point coordinate information, and obtain the initial input feature based on the target face key point coordinate information and the average face key point coordinate information;
[0111] The initial input features are processed to obtain facial features.
[0112] In one embodiment, before the processor 501 executes the recognition of the template face in the data set to obtain the coordinate information of the key points of the face, the process further includes:
[0113] Detect face images in the dataset based on the face detection algorithm and obtain the detection result file;
[0114] Parsing the detection result file to generate a template face;
[0115] The information in the detection result file includes the coordinates of multiple facial key points, including cheek coordinates, eyebrow coordinates, eye coordinates, mouth coordinates and nose coordinates.
[0116] In one embodiment, the processor 501 performs the processing of the face sequence to obtain the sequence feature, including:
[0117] Obtain angle sequence constraint features and boundary key point constraint features;
[0118] Processing the angle sequence constraint feature to obtain an angle sequence constraint sequence;
[0119] Processing the boundary key point constraint features to obtain a boundary key point constraint sequence;
[0120] The angle sequence constraint sequence and the boundary key point constraint sequence are superimposed to obtain a sequence feature.
[0121] In one embodiment, the processor 501 performs the step of superimposing the sound feature, the face feature, and the sequence feature to generate the input feature, including:
[0122] Cat superimposes one of the voice feature, the face feature and the sequence feature to obtain a first superimposed feature;
[0123] The first superposition features are stacked to obtain input features.
[0124] In one embodiment, the processor 501 executes the step of generating a facial key point sequence according to the input feature, including:
[0125] Processing the input features to obtain features related to key points of the face;
[0126] The facial key point related features are processed through a multi-layer fully connected network to obtain a facial key point sequence; wherein the facial key point sequence includes a parameter associated with the sequence size and the audio data length, the number of facial key points and the corresponding coordinates.
[0127] The technical concept and Figure 2 or Figure 3 The technical concept is the same as that of Figure 2 or Figure 3 The method embodiments are not described in detail here.
[0128] In an embodiment of the present application, a center position on a time interval of the audio data is calculated based on a preset frame rate, the time interval is traversed to extract Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position, the MFCC sound features are processed to obtain sound features, a template face in a data set is identified to obtain facial key point coordinate information, all facial key point coordinate information in the data set is counted to obtain average facial key point coordinate information, target facial key point coordinate information is determined, initial input features are obtained based on the target facial key point coordinate information and the average facial key point coordinate information, and the initial input features are processed to obtain facial features. Characteristic, obtain angle sequence constraint features and boundary key point constraint features, process the angle sequence constraint features to obtain angle sequence constraint sequences, process the boundary key point constraint features to obtain boundary key point constraint sequences, superimpose the angle sequence constraint sequences and the boundary key point constraint sequences to obtain sequence features, Cat superimpose the sound features, the face features and the sequence features to obtain a first superimposed feature, Stack superimpose the first superimposed feature to obtain an input feature, process the input feature to obtain face key point related features, and process the face key point related features through a multi-layer fully connected network to obtain a face key point sequence. The embodiment of the present application can directly generate face key point related information of a naturally changing virtual image based on audio data, thereby improving the authenticity and fluency of the virtual image's speech movements.
[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0130] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A method for generating facial key points, characterized in that: The method comprises: Extracting features from the audio data to obtain sound features, wherein the feature extraction of the sound features includes measuring or processing; Extracting features from the template face to obtain facial features, wherein the facial features are obtained by processing the coordinates of facial key points of the template face; Processing the face sequence to obtain sequence features; wherein the face sequence includes angle sequence constraint features and boundary key point constraint features, and the processing of the face sequence includes processing and superimposing the angle sequence constraint features and the boundary key point constraint features respectively; Superimposing the sound feature, the face feature and the sequence feature to generate an input feature; Generate a facial key point sequence according to the input features, wherein the facial key point sequence is obtained by processing through a multi-layer fully connected network; The step of extracting features from the audio data to obtain sound features includes: Calculate the center position of the time interval of the audio data based on a preset frame rate; Traversing the time interval, extracting Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position; Processing the MFCC sound features to obtain sound features; The step of extracting features from the template face to obtain facial features includes: Identify the template face in the data set to obtain the coordinate information of the key points of the face; Counting all the facial key point coordinate information in the data set to obtain average facial key point coordinate information; Determine the target face key point coordinate information, and obtain the initial input feature based on the target face key point coordinate information and the average face key point coordinate information; The initial input features are processed to obtain facial features.
2. The method according to claim 1, characterized in that: Before obtaining the facial key point coordinate information of the template face in the recognition data set, the method further includes: Detect face images in the dataset based on the face detection algorithm and obtain the detection result file; Parsing the detection result file to generate a template face; The information in the detection result file includes the coordinates of multiple facial key points, including cheek coordinates, eyebrow coordinates, eye coordinates, mouth coordinates and nose coordinates.
3. The method according to claim 1, characterized in that: The step of processing the face sequence to obtain sequence features includes: Obtain angle sequence constraint features and boundary key point constraint features; Processing the angle sequence constraint feature to obtain an angle sequence constraint sequence; Processing the boundary key point constraint features to obtain a boundary key point constraint sequence; The angle sequence constraint sequence and the boundary key point constraint sequence are superimposed to obtain a sequence feature.
4. The method according to claim 1, characterized in that: The step of superimposing the sound feature, the face feature and the sequence feature to generate the input feature comprises: Cat-superimpose the sound feature, the face feature and the sequence feature to obtain a first superimposed feature; The first superposition features are stacked to obtain input features.
5. The method according to claim 1, characterized in that: The step of generating a facial key point sequence according to the input features comprises: Processing the input features to obtain features related to key points of the face; The facial key point related features are processed through a multi-layer fully connected network to obtain a facial key point sequence; wherein the facial key point sequence includes a parameter associated with the sequence size and the audio data length, the number of facial key points and the corresponding coordinates.
6. The method according to claim 5, characterized in that: The processing of the input features to obtain features related to key points of the face is processing the input features using a long short-term memory (LSTM) neural network to obtain features related to key points of the face.
7. A facial key point generation device, characterized in that: include: A first extraction module, configured to extract features from the audio data to obtain sound features, wherein the feature extraction of the sound features includes measurement or processing; A second extraction module is used to extract features from the template face to obtain facial features, wherein the facial features are obtained by processing the coordinates of facial key points of the template face; A processing module, used for processing a face sequence to obtain sequence features; wherein the face sequence includes angle sequence constraint features and boundary key point constraint features; wherein the face sequence includes angle sequence constraint features and boundary key point constraint features, and the processing of the face sequence includes processing and superimposing the angle sequence constraint features and the boundary key point constraint features respectively; A superposition module, used for superimposing the sound feature, the face feature and the sequence feature to generate an input feature; A generation module, used to generate a facial key point sequence according to the input features, wherein the facial key point sequence is obtained by processing through a multi-layer fully connected network; Wherein, the first extraction module further includes: A calculation unit is used to calculate the center position of the time interval of the audio data based on a preset frame rate; traverse the time interval to extract the Mel-frequency cepstral coefficient (MFCC) sound features in a sub-time interval of a preset length before and after the center position; and process the MFCC sound features to obtain sound features; Wherein, the second extraction module further includes: The recognition unit is used to recognize the template face in the data set to obtain the facial key point coordinate information; count the facial key point coordinate information of all the facial key point in the data set to obtain the average facial key point coordinate information; determine the target facial key point coordinate information, and obtain the initial input feature based on the target facial key point coordinate information and the average facial key point coordinate information; and process the initial input feature to obtain the facial feature.
8. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps as claimed in any one of claims 1 to 6.
9. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as described in any one of claims 1 to 6.