Virtual avatar generation method, virtual avatar generation system, and computing device

Abstract

The application discloses a virtual image generation method, a virtual image generation system and a computing device, and relates to the technical field of virtual image generation, and specifically discloses a method comprising: obtaining speech features of answer speech data corresponding to input speech data based on the input speech data obtained from a user; obtaining a face mask image containing face feature points based on the speech features of each frame of the answer speech data; obtaining facial feature of a virtual image based on at least two reference face images obtained from the user; and obtaining the virtual image corresponding to the current frame of the answer speech data based on the face mask image containing face feature points and the facial feature. The application can automatically realize the fusion of facial features of multiple people, avoid legal risks caused by using facial features of the same person, realize personalized and realistic virtual image design without human intervention, is time-saving and low-cost, can output lip shapes and expressions matched with speech, and improves the expressiveness of the virtual image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio-video synthesis, and more particularly, to a virtual image generation method, a virtual image generation system and a computing device. BACKGROUND

[0002] In recent years, mixed reality products have gradually entered the public eye. As an important information expression carrier in the virtual world, virtual images have bloomed in online (for example, smart phone assistants, virtual e-commerce anchors, virtual stars, etc.) and offline (for example, intelligent question and answer integrated machines in banks, subways, hospitals, etc.) and are showing an expansion trend.

[0003] Currently, there are mainly two implementation methods for virtual image generation. The first method is to use CG (Computer Graphics) technology modeling. If individualization is to be achieved, this technology needs to be modified from the modeling stage, which takes a long time to produce and has high labor costs. Moreover, the rendering process has high requirements for hardware devices (for example, high-end graphics cards). In terms of rendering visual effects such as virtual image realism and light and shadow complexity, it is difficult to generate satisfactory virtual images using hardware devices with limited performance (for example, mobile phones). The second method is to use audio-video synthesis technology. This technology needs to collect real human audio-video data and go through post-processing, AI model training and engine deployment steps. It takes about 1 month to complete the entire process, which is time-consuming and costly, and there is a legal risk of infringing on portrait rights and misusing biological information.

[0004] Therefore, there is a need for a new virtual image generation method, a virtual image generation system and a computing device to solve the above problems. SUMMARY

[0005] A series of simplified concepts are introduced in the summary section, which will be further described in detail in the specific embodiments section. The summary section of the present application does not mean to attempt to limit the key features and necessary technical features of the claimed technical solutions, nor to attempt to determine the protection scope of the claimed technical solutions.

[0006] According to an aspect of the present application, a virtual image generation method is provided, which includes: obtaining speech features of answer speech data corresponding to input speech data based on the input speech data obtained from a user; obtaining a face mask image containing face feature points based on the speech features of each frame of the answer speech data; obtaining facial feature of a virtual image based on at least two reference face images obtained from a user; and obtaining the virtual image corresponding to the current frame of the answer speech data based on the face mask image containing face feature points and the facial feature.

[0007] In one embodiment, wherein the speech feature corresponding to the input speech data is obtained based on the input speech data obtained from the user, the method comprises: obtaining the answer speech data corresponding to the input speech data based on the input speech data obtained from the user; and obtaining the speech feature corresponding to the answer speech data based on the answer speech data.

[0008] In one embodiment, wherein the speech feature corresponding to the answer speech data is obtained based on the answer speech data, the method comprises: extracting normalized spectral coefficients of the answer speech data from the answer speech data; obtaining semantic features of the answer speech data based on the normalized spectral coefficients; obtaining emotion features of the answer speech data based on the normalized spectral coefficients; and fusing the semantic features and the emotion features to obtain the speech feature of the answer speech data.

[0009] In one embodiment, wherein the face mask image containing the face feature points is obtained based on the speech feature of each frame of the answer speech data, the method comprises: obtaining the face feature points corresponding to the answer speech data based on the speech feature of each frame of the answer speech data; and combining the face feature points with the stored face mask image to obtain the face mask image containing the face feature points.

[0010] In one embodiment, wherein the face feature points corresponding to the answer speech data are obtained based on the speech feature of each frame of the answer speech data, the method comprises: obtaining orthogonal basis vector coefficients of the answer speech data based on the speech feature of the answer speech data; and performing inverse normalization processing on the orthogonal basis vector coefficients of the answer speech data to obtain the face feature points corresponding to the answer speech data.

[0011] In one embodiment, wherein the facial feature for the virtual image is obtained based on at least two reference face images obtained from the user, the method comprises: extracting facial semantic features of the face from each reference face image; performing facial segmentation on each reference face image to obtain a binary facial mask image of the facial features in the reference face image; calculating an average feature vector of each facial feature from the facial semantic features using the facial mask image; and fusing the average feature vectors of the facial features in each of the reference face images to obtain the facial feature for the virtual image.

[0012] In an embodiment, wherein the facial features for the virtual avatar are obtained based on the reference face image, further comprising: performing face alignment on the reference face image with a preset standard face image to obtain an aligned reference face image before extracting the facial semantic features of the face from the reference face image; and extracting the facial semantic features of the face from the aligned reference face image.

[0013] In an embodiment, wherein the facial features for the virtual avatar are obtained based on the reference face image, further comprising: performing moderate inflation on the facial feature mask image to obtain an inflated facial feature mask image before calculating the average feature vector of each facial feature using the facial feature mask image; and calculating the average feature vector of each facial feature using the inflated facial feature mask image.

[0014] In an embodiment, wherein the virtual avatar corresponding to the answer speech data of the current frame is further based on the virtual avatar of the previous frame.

[0015] In an embodiment, wherein the facial features include eye features, eyebrow features, nose features, and mouth features.

[0016] In an embodiment, the virtual avatar generation method is implemented using a neural network.

[0017] According to another aspect of the present application, there is provided a virtual avatar generation system, the system comprising: a processor configured to perform the virtual avatar generation method as described above using a neural network; and a memory configured to store network parameters of the neural network.

[0018] According to yet another embodiment of the present application, there is provided a computing device comprising a memory and a processor, the memory having stored thereon a computer program which, when executed by the processor, causes the processor to perform the method as described above.

[0019] According to still another embodiment of the present application, there is provided a computer readable medium having stored thereon a computer program which, when executed by a processor, performs the method as described above.

[0020] According to the virtual avatar generation method, the virtual avatar generation system, and the computing device according to embodiments of the present application, the fusion of facial features of multiple people is automatically implemented, legal risks caused by using facial features of the same person are avoided, personalized and realistic virtual avatar design is achieved without human intervention, time consumption is short, cost is low, and lip shapes and expressions matching the speech can be output, thereby improving the expressiveness of the virtual avatar. BRIEF DESCRIPTION OF DRAWINGS

[0021] The following drawings for the present application are hereby incorporated into this specification as part of the present application for the purpose of understanding the present application. The embodiments of the present application and their description shown in the drawings are used to explain the principles of the present application.

[0022] In the drawings:

[0023] Figure 1 A schematic structural block diagram of an electronic device for implementing a virtual image generation method, a virtual image generation system and a computing device according to an embodiment of the present application.

[0024] Figure 2 An exemplary step flow chart of a virtual image generation method according to an embodiment of the present application.

[0025] Figure 3 A schematic diagram of a face mask image containing facial feature points according to an embodiment of the present application is shown.

[0026] Figure 4 A schematic diagram of obtained facial features for a virtual image according to an embodiment of the present application is shown.

[0027] Figure 5 A schematic structural block diagram of a virtual image generation system according to an embodiment of the present application is shown.

[0028] Figure 6 A schematic structural block diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] In order to make the objects, technical solutions and advantages of the present application more obvious, the following will describe the example embodiments according to the present application in detail with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the protection scope of the present application.

[0030] As described above, the existing virtual image generation method has problems such as inability to achieve personalization, high cost, infringement of portrait rights, etc.

[0031] Therefore, in order to solve the above problems, the present application provides a virtual image generation method, the method comprising: obtaining speech features of answer speech data corresponding to input speech data based on the input speech data obtained from a user; obtaining a face mask image containing face feature points based on the speech features of each frame of the answer speech data; obtaining facial feature of a face in each reference face image based on each reference face image obtained from a user; and obtaining the virtual image corresponding to the current frame of the answer speech data based on the face mask image containing face feature points and the facial feature.

[0032] According to the virtual image generation method of the present application, the fusion of multiple facial features is automatically realized, the legal risks caused by using the same facial features are avoided, personalized and realistic virtual image design can be realized without human intervention, the time consumption is short, the cost is low, the lip shape and expression matched with the speech can be output, and the expressiveness of the virtual image is improved.

[0033] The knowledge graph construction method, the knowledge graph construction system and the computing device according to the present application will be described in detail below with reference to specific embodiments.

[0034] First, the electronic device 100 for implementing the knowledge graph construction method, the knowledge graph construction system and the computing device according to the embodiments of the present application will be described with reference to Figure 1

[0035] In one embodiment, the electronic device 100 can be, for example, a notebook computer, a desktop computer, a tablet computer, a learning machine, a mobile device (such as a smart phone, a telephone watch, etc.), an embedded computer, a tower server, a rack server, a blade server, or any other suitable electronic device.

[0036] In one embodiment, the electronic device 100 can include at least one processor 102 and at least one memory 104.

[0037] The memory 104 can be a volatile memory such as a random access memory (RAM), a cache memory, a dynamic random access memory (DRAM) (including stacked DRAM), or a high bandwidth memory (HBM), etc., or a non-volatile memory such as a read only memory (ROM), a flash memory, a 3D Xpoint, etc. In one embodiment, some parts of the memory 104 can be volatile memory, while another part can be non-volatile memory (for example, using a two-level memory hierarchy). The memory 104 is used to store a computer program which, when executed, can implement the client functions (implemented by the processor) in the embodiments of the present application described below and / or other desired functions. ​

[0038] The processor 102 can be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, or other processing units that have data processing and / or instruction execution capabilities. The processor 102 can be communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices via a communication bus. In one embodiment, the communication bus can be implemented using any suitable protocol, such as peripheral component interconnect (PCI), peripheral component interconnect express (PCIe), accelerated graphics port (AGP), HyperTransport, or any other bus or point-to-point communication protocol(s).

[0039] The electronic device 100 can also include an input device 106 and an output device 108. The input device 106 is a device for receiving user input, which can include a keyboard, a mouse, a touchpad, a microphone, etc. In addition, the input device 106 can also be any interface that receives information. The output device 108 can output various information (e.g., images or sounds) to the outside (e.g., a user), which can include one or more of a display, a speaker, etc. In addition, the output device 108 can also be any other device with output capabilities, such as a printer, etc.

[0040] Reference will now be made to Figure 2 An exemplary step flowchart of a virtual avatar generation method 200 according to an embodiment of the present application is described below. As shown in FIG. 2, the virtual avatar generation method 200 can include the following steps: Figure 2

[0041] In step S210, based on the input speech data obtained from the user, the speech features of the answer speech data corresponding to the input speech data are obtained.

[0042] In step S220, based on the speech features of each frame of the answer speech data, a face mask image containing face feature points is obtained (as shown in FIG. 3). Figure 3

[0043] In step S230, based on at least two reference face images obtained from the user, the facial feature of the virtual avatar is obtained (as shown in FIG. 4). Figure 4

[0044] In step S240, based on the face mask image containing face feature points and the facial feature of the virtual avatar, the virtual avatar corresponding to the current frame of the answer speech data is obtained.

[0045] In an embodiment of the present application, the virtual avatar generation method 200 can be implemented using a neural network.

[0046] ​​​In the embodiments of the present application, the virtual image can include a virtual human, a virtual animated character, a virtual animal or other virtual image capable of being presented by the electronic device, and the present application does not make any limitation thereto.

[0047] In the embodiments of the present application, the input voice data can be audio, video and other data obtained from the user, and the present application does not make any limitation thereto.

[0048] In the embodiments of the present application, the step of obtaining the voice feature of the answer voice data corresponding to the input voice data obtained from the user in step S210 can include: obtaining the answer voice data corresponding to the input voice data obtained from the user; and obtaining the voice feature corresponding to the answer voice data based on the answer voice data.

[0049] In the embodiments of the present application, the step of obtaining the answer voice data corresponding to the input voice data obtained from the user can be implemented using any natural language processing model known in the art, such as a statistical language model, an N-gram model, a neural probabilistic language model, MM (Markov Model) and HMM (Hidden Markov Model), a maximum entropy model, SVM (Support Vector Machine), CRF (Conditional Random Field) and the like, and the present application does not make any limitation thereto. Exemplarily, the input voice data can be divided into several voice segments with lengths varying from 2 seconds to 20 seconds, for example, using a voice endpoint detection technology, and each segment of audio is input into the above-mentioned natural voice processing model, thereby obtaining several answer voice segments corresponding to the input voice segments.

[0050] In the embodiments of the present application, the step of obtaining the voice feature corresponding to the answer voice data based on the answer voice data can include: extracting the normalized spectral coefficient of the answer voice data from the answer voice data; obtaining the semantic feature Af of the answer voice data based on the normalized spectral coefficient; obtaining the emotional feature Ae of the answer voice data based on the normalized spectral coefficient; and fusing the semantic feature Af and the emotional feature Ae to obtain the voice feature Am of the answer voice data. Wherein, the semantic feature Af is used to control the synthesis of the lip shape of the virtual image, and the emotional feature Ae is used to control the synthesis of the facial expression of the virtual image.

[0051] In the embodiments of the present application, the step of extracting the normalized spectral coefficient of the answer voice data from the answer voice data can include extracting the Mel-frequency cepstral coefficient (MFCC) of the answer voice data, which is used to represent the spectral coefficient of the answer voice data, and normalizing the Mel-frequency cepstral coefficient, thereby obtaining the normalized spectral coefficient of the answer voice data. Exemplarily, the Mel-frequency cepstral coefficient with a dimension of 28 at 100 HZ, for example, can be extracted.

[0052] In an embodiment of the present application, the normalized spectral coefficients of the answer speech data can be input into a pre-trained speech semantic recognition model to obtain semantic features Af of the answer speech data from the normalized spectral coefficients, for example, GMM-HMM (Gaussian Mixture Model-Hidden Markov Model), RNN (Recurrent Neural Network, for example, LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit)), FSMN (Feedforward Fully Connected Network), DFSMN (Deep Feedforward Fully Connected Network), etc., which are not limited in the present application.

[0053] In an embodiment of the present application, the normalized spectral coefficients of the answer speech data can be input into a pre-trained speech semantic recognition model to obtain semantic features Af of the answer speech data from the normalized spectral coefficients, for example, GMM-HMM (Gaussian Mixture Model-Hidden Markov Model), RNN (Recurrent Neural Network, for example, LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit)), FSMN (Feedforward Fully Connected Network), DFSMN (Deep Feedforward Fully Connected Network), etc., which are not limited in the present application.

[0054] In an embodiment of the present application, the semantic features can be, for example, semantic features with a dimension of 512 at 25HZ, and the emotional features can be, for example, emotional features with a dimension of 128.

[0055] In an embodiment of the present application, the step of fusing the semantic features and the emotional features can include splicing, adding, subtracting or other fusion operations of the semantic features and the emotional features, which are not limited in the present application.

[0056] In an embodiment of the present application, based on the speech features of each frame of answer speech data, obtaining the face mask image containing the face feature points can include: based on the speech features of each frame of answer speech data, obtaining the face feature points corresponding to the answer speech data; and combining the face feature points with the stored face mask image to obtain the face mask image containing the face feature points.

[0057] In an embodiment of the present application, the speech features of the answer speech data can be input into a trained recurrent neural network (RNN), for example, a long short-term memory network (LSTM), a gated recurrent unit (GRU), etc., to obtain the face feature points corresponding to the answer speech data.

[0058] In an embodiment of the present application, based on the speech features of each frame of answer speech data, obtaining the face feature points corresponding to the answer speech data can include: based on the speech features of the answer speech data, obtaining the orthogonal basis vector coefficients of the answer speech data; and performing inverse normalization processing on the orthogonal basis vector coefficients of the answer speech data to obtain the face feature points corresponding to the answer speech data.

[0059] In the embodiment of the present application, a two-layer LSTM network can be built and trained to answer the speech feature of the speech data based on each frame to obtain the facial feature points corresponding to the speech feature. The speech feature of the answer speech data is input into the first layer of the LSTM network, and the first layer obtains the orthogonal basis vector coefficient of the answer speech data from the speech feature. The orthogonal basis vector coefficient of the answer speech data is input into the second layer of the LSTM network, and the second layer performs inverse normalization processing on the orthogonal basis vector coefficient to obtain the facial feature points corresponding to the answer speech data.

[0060] In the embodiment of the present application, the orthogonal basis vector coefficient can be, for example, a PCA (Principal Component Analysis) coefficient, and can also be a MDS (Multidimensional Scaling) coefficient, an LDA (Linear Discriminant Analysis) coefficient, an Isomap (Isometric Mapping) coefficient, an LLE (Local Linear Embedding) coefficient, and the like. The present application does not limit the above.

[0061] In the embodiment of the present application, the number of facial feature points can be set as needed, for example, 64, and the present application does not limit the above.

[0062] In the embodiment of the present application, the facial mask image can be obtained according to the contour of the facial feature points obtained during training, and can also be obtained by removing the facial information of the facial image obtained during training, and the like. The present application does not limit the above.

[0063] In the embodiment of the present application, the number of reference facial images can be set as needed, for example, 2, 3, 4, 5, 6, and the like. The present application does not limit the above.

[0064] In the embodiment of the present application, the facial feature can include eye feature, eyebrow feature, nose feature, and mouth feature, and can also include ear feature. The present application does not limit the above.

[0065] In the embodiment of the present application, the step of obtaining the facial feature of the virtual image based on at least two reference facial images in step S230 can include the following steps:

[0066] In step a1, the facial semantic feature of the face in each reference facial image is extracted.

[0067] In step a2, the face segmentation is performed on each reference facial image to obtain the binary facial mask image of the facial features in the reference facial image.

[0068] In step a3, the average feature vector of each facial feature is calculated from the facial semantic feature using the facial mask image.

[0069] In step a4, the average feature vector of the facial features in each reference facial image is fused to obtain the facial feature of the virtual image.

[0070] In an embodiment of the present application, a trained face recognition network can be used to extract the facial semantic features of the face from each reference facial image. Illustratively, the face recognition network can use any image semantic feature extraction method known in the art to extract facial semantic features, such as PCA (Principal Component Analysis), ICA (Independent Component Analysis), ISA (Independent Subspace Analysis), etc., which are not limited by the present application. Illustratively, the facial semantic features can be extracted using an arcface face recognition network based on a resnet100 framework. Illustratively, the extracted facial semantic features can be facial semantic features with a dimension of 14*14*256.

[0071] In an embodiment of the present application, before extracting the facial semantic features from the reference facial image, a step of performing face alignment processing on the reference facial image with a preset standard facial image can also be included to obtain an aligned reference facial image, and the facial semantic features are extracted from the aligned reference facial image.

[0072] In an embodiment of the present application, an exemplary method of performing face alignment on the reference facial image with the preset standard facial image can include: setting 5 feature points (2 pupil points, 1 nose tip point, 2 corner points) at a certain resolution (e.g., 256) as the feature points TP5 of the standard face, using a feature point detection model (e.g., mtcnn (multi-task convolutional neural network) and the like) to extract the corresponding 5 coordinate points P5 in the reference facial image, using the least squares method to calculate the affine transformation matrix H of P5 to TP5, using the affine transformation matrix H to deform the pixels of the reference facial image to obtain the aligned reference facial image.

[0073] In an embodiment of the present application, a trained facial semantic segmentation network can be used to segment the face of the reference facial image to obtain a binary facial feature mask image of the facial features in the reference facial image. Illustratively, the facial semantic segmentation network can be obtained by training FCN (Fully Convolutional Networks), dilated convolution (dilated convolution), U-net, etc., which are not limited by the present application.

[0074] In an embodiment of the present application, the average feature vector can be a feature vector with a dimension of 1*1*256, etc., which is not limited by the present application.

[0075] In the embodiments of the present application, before calculating the average feature vector of each facial feature using the facial feature mask image, the step of: moderately dilating the facial feature mask image to obtain a dilated facial feature mask image, and using the dilated facial feature mask image to calculate the average feature vector of each facial feature, can be further included. Moderately dilating the facial feature mask image can ensure that details such as double eyelids and eyelashes are not lost, making the obtained facial feature mask image more realistic.

[0076] In the embodiments of the present application, the fusion of the average feature vectors of the facial features in each reference facial image can include: splicing, adding, subtracting or other fusion operations on the average feature vectors of the facial features, which are not limited by the present application.

[0077] In the embodiments of the present application, the obtained facial feature for the virtual image can be a feature vector with dimensions of 1*1*768, etc., which is not limited by the present application.

[0078] In the embodiments of the present application, in step S240, based on the facial mask image containing facial feature points and the facial features for the virtual image, the virtual image corresponding to the answer speech data of the current frame can be obtained, which can be realized by using a video synthesis network, such as a trained generative adversarial network (GAN) or the like. For example, in addition to inputting the facial mask image containing facial feature points and the facial features for the virtual image into the GAN, the virtual image of the previous frame can also be inputted into the GAN, so that the virtual image corresponding to the answer speech data of the current frame is also based on the virtual image of the previous frame.

[0079] In the embodiments of the present application, the generative adversarial network GAN can include a generator network G and a discriminator network D, wherein the generator network G is used to synthesize the virtual image of the current frame, and the discriminator network D is used to determine whether the synthesized virtual image meets the expectation.

[0080] In the embodiments of the present application, the generator network G can be composed of an encoder network E, a feature transformation network Res and a decoder network De. Wherein, the encoder network E can be composed of 3 convolutional layers and 3 pooling layers, the feature transformation network Res can be composed of 5 residual networks, and the decoder network De can be composed of 3 up-sampling layers.

[0081] In the embodiments of the present application, the AdaIN (adaptive instance normalization) method can be used to input the facial mask image containing facial feature points, the facial features for the virtual image and the virtual image of the previous frame together into each network layer of the generator network G.

[0082] In the embodiments of the present application, the discriminator network D can include an image discriminator Di and a video discriminator Dt, wherein the image discriminator Di is used to discriminate whether the synthesized virtual image conforms to the real distribution, and the video discriminator Dt is used to discriminate whether the synthesized multiple virtual images have time continuity.

[0083] The images of the generated virtual images in each frame are arranged in time sequence, and a video frame sequence of the virtual image is obtained.

[0084] In the embodiments of the present application, the user can change the input reference image at any time, and the generated virtual image will also change accordingly, greatly improving the flexibility and diversity of the synthesized virtual image.

[0085] According to the virtual image generation method of the present application, the fusion of multiple facial features is automatically realized, the legal risks brought by using the same facial features are avoided, personalized and realistic virtual image design can be realized without human intervention, the time consumption is short, the cost is low, and the lip shape and expression matched with the voice can be output, thereby improving the expressiveness of the virtual image.

[0086] The following briefly introduces the simple steps of the training process of the neural network for performing the above-mentioned virtual image generation method 200:

[0087] Step 1, constructing a training data set.

[0088] Exemplarily, the training data set includes a basic image audio-visual data of no less than 10 minutes, and high-definition face data of different people of no less than 50,000.

[0089] Step 2, processing the training data.

[0090] Firstly, the collected audio data is subjected to noise reduction processing to make the human voice clear, and a speech endpoint detection technology is used to cut the audio data into audio segments with lengths varying from 2 seconds to 20 seconds, obtaining a plurality of audio segments. For each audio segment, for example, 100HZ, 28-dimensional mel-frequency cepstral coefficients (MFCC) are extracted, the mel-frequency cepstral coefficients are taken as the input of a pre-trained speech semantic recognition model, and for example, 25HZ, 512-dimensional semantic features are obtained, denoted as Af, which are used to control the synthesis of the lip shape of the virtual image. At the same time, the mel-frequency cepstral coefficients are taken as the input of a speech emotion recognition network, and for example, 128-dimensional emotion features are extracted, denoted as Ae, which are used to control the synthesis of the facial expression of the virtual image. The two kinds of features are spliced to obtain fused speech features Am.

[0091] Secondly, the collected video data is cropped according to the same timestamp as the audio data, and decoded frame by frame into images. The position of the face frame is determined using a face detector, and the face image is cropped, denoted as Ir(0), Ir(t), …, Ir(T). In order to improve the robustness of the neural network, the M face images R(0), R(m), …, R(M) in the training data and the face images Ir(0), Ir(t), …, Ir(T) cropped in the previous step are aligned, and the alignment process is as follows:

[0092] Set 5 feature points (2 pupil points, 1 nose tip point, and 2 corner points) at 256 resolution as standard image feature points TP5, and use mtcnn or other feature point detection tools to extract the 5 coordinate points in all face images Ir(0), Ir(t), …, Ir(T), R(0), R(m), …, R(M), denoted as P5(i), i∈[0,T+M-1]. The affine transformation matrix H(i) of P5(i) to TP5 is calculated using the least squares method, i∈[0,T+M-1], and the pixels of Ir(t), R(m) are deformed using the matrix H to obtain the aligned face images I align (t), R align (m).

[0093] Use the face semantic segmentation network based on the Xception39 framework to extract the face segmentation results of all the aligned face images, and separate the binary mask images M align (m), m∈[0,M-1] corresponding to the eyes, nose, and mouth regions. In order to not lose the details such as double eyelids and eyelashes, the mask image needs to be moderately dilated, and the dilated mask image is denoted as M(m).

[0094] Then use the Dlib face feature detector to obtain 68 face feature points of each face image in the cropped face images Ir(0), Ir(t), …, Ir(T). Rotate and normalize the 68 feature points according to a pair of feature points of the eyes, and translate and normalize the 68 feature points around the nose tip point to remove the influence of head movement. Use dimension reduction algorithm (e.g., principal component analysis (PCA)) to reduce the normalized feature points (e.g., dimension 132) to 64 dimensions, denoted as Lp(t), t∈[0,T-1]. At the same time, the positions of the 68 feature points on the aligned face image I align (t) are obtained using rotation denormalization, translation denormalization, and PCA denormalization according to the standard image feature points TP5, and the 68 feature points are superimposed on the face mask image to obtain the paired image of the face mask image containing the face feature points and the corresponding face image, denoted as I in (t) and I lb(t), t∈[0, T-1]. Wherein the face mask image can be obtained according to the contour of the eigenface, and the eigenface can also be obtained by erasing the facial information.

[0095] Thirdly, driving feature point generation.

[0096] Build a two-layer standard LSTM (Long Short-Term Memory) network, take the fused speech features Am(t), t∈[0, T-1] as input, and take the reduced PCA coefficients Lp(t), t∈[0, T-1] as supervision labels during training, and use Loss function for training, after several iterations (for example, 40 times), the loss decreases to the level of 1 e-3 , then the training is completed.

[0097] Fourthly, video synthesis.

[0098] Randomly select at least two from the face-aligned face images R align (0), R align (1), …, R align (M) as reference face images, for example, select three face images R1, R2, R3, respectively pass through the face recognition network F R (for example, use the arcface face recognition network based on the resnet100 framework), get the face semantic features F1, F2, F3 (for example, 14*14*256 dimensions), and use the expanded mask M(m) to calculate the average feature vector (for example, 1*1*256 dimensions) of the facial feature (for example, eyebrows, eyes, nose, mouth) region, denoted as V e ,V n ,V l , splice these average feature vectors to get the fusion feature V (for example, 1*1*768 dimensions), that is, the facial feature of the virtual image, and input the fusion feature V, the above obtained paired image and the image I p (t-1) generated by the AdaIN method into the video synthesis network.

[0099] In order to make the neural network have the facial feature fusion ability and improve the continuity of the video, the loss function of the network is designed as follows:

[0100]

[0101] Specifically,

[0102] L id =1-cos(F R (I p (t), F R (R1)), wherein I p(t) is the image of the virtual avatar generated for the current frame. It is expected that the similarity between the generated virtual avatar image and the reference face image is close to 1.

[0103] It is expected that the pixel error between the generated virtual avatar image and the corresponding face image is minimized.

[0104] where VGG denotes a face feature detector.

[0105] is the standard WGAN loss for the video synthesis network.

[0106] L local = 3 - cos(F R (I p (t)) * M1(t), F R (R1) * M1(t)) - cos(F R (I p (t)) * M2(t), F R (R2) * M2(t)) - cos(F R (I p (t)) * M3(t), F R (R3) * M3(t))

[0107] The neural network trained through the above steps can implement the virtual avatar generation method 200 in the embodiments of the present application.

[0108] Reference Figure 5 , Figure 5 A schematic structural block diagram of a virtual avatar generation system 500 according to an embodiment of the present application is shown. As Figure 5 shown, the virtual avatar generation system 500 can include a processor 510 and a memory 520.

[0109] The processor 510 is configured to use one or more neural networks to perform the virtual avatar generation method 200 as described above.

[0110] Those skilled in the art can understand the specific operations of the virtual avatar generation system 500 according to the embodiments of the present application in combination with the foregoing description, and for the sake of brevity, specific details are not repeated here, and only some main operations of the processor 510 are described as follows:

[0111] Based on the input speech data obtained from the user, obtain the speech features of the answer speech data corresponding to the input speech data;

[0112] Based on the speech features of each frame of the answer speech data, obtain a face mask image containing face feature points;

[0113] obtain facial features for the virtual image based on at least two reference facial images obtained from a user; and

[0114] obtain the virtual image corresponding to the answer speech data of the current frame based on the facial mask image containing facial feature points and the facial features.

[0115] Exemplarily, the processor 510 can be any processing device known in the art, such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, a field programmable gate array (FPGA), etc., and the present application is not limited thereto.

[0116] The memory 520 is configured to store network parameters of the neural network. Exemplarily, the memory 520 can be a RAM, a ROM, an EEPROM, a flash memory, or other storage technologies, a CD-ROM, a digital versatile disc (DVD) or other optical disc storage devices, a magnetic tape cassette, a magnetic tape, a magnetic disc storage device or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by the processor 510.

[0117] The virtual image generation system according to the present application automatically realizes the fusion of multiple facial features, avoids the legal risks caused by using the same facial features, realizes personalized and realistic virtual image design without human intervention, is time-saving and low-cost, and can output lip shapes and expressions matched with the speech, thereby improving the expressiveness of the virtual image.

[0118] Reference Figure 6 , Figure 6 A schematic structural block diagram of a computing device 600 according to an embodiment of the present application is shown. As shown, the computing device 600 can include a memory 610 and a processor 620, wherein the memory 610 stores a computer program, and the computer program, when executed by the processor 620, causes the processor 620 to perform the virtual image generation method 200 as described above. Figure 6 Those skilled in the art can understand the specific operations of the computing device 600 according to the embodiments of the present application in combination with the foregoing description, and for the sake of brevity, specific details are not described here, and only some main operations of the processor 620 are described as follows:

[0119] obtain speech features of answer speech data corresponding to the input speech data based on the input speech data obtained from a user;

[0120] obtain a facial mask image containing facial feature points based on the speech features of each frame of the answer speech data;

[0121]

[0122] ​obtaining facial features for the virtual image based on at least two reference face images obtained from a user; and

[0123] obtaining the virtual image corresponding to the answer speech data of the current frame based on the face mask image containing the facial feature points and the facial features.

[0124] The computing device according to the present application automatically realizes the fusion of multiple facial features, avoids the legal risks caused by using the same facial features, and realizes personalized and realistic virtual image design without human intervention, which is time-saving, low-cost, and can output lip shapes and expressions matching the speech, thereby improving the expressiveness of the virtual image.

[0125] According to an embodiment of the present application, a computer readable medium is also provided, which stores a computer program. The computer program, when executed, performs the virtual image generation method 200 as described in the above embodiments. Any tangible, non-transitory computer readable medium can be used, including magnetic storage devices (hard disk, floppy disk, etc.), optical storage devices (CD-ROM, DVD, Blu-ray disc, etc.), flash memory, and / or the like. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to form a machine, so that the instructions executed on the computer or other programmable data processing apparatus can generate a device that implements the specified function. These computer program instructions can also be stored in a computer readable storage medium, which can instruct the computer or other programmable data processing apparatus to operate in a specific way, so that the instructions stored in the computer readable storage medium can form a manufactured product, including an implementation device that implements the specified function. Computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus can provide steps for implementing the specified function.

[0126] Although example embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the above-described example embodiments are merely exemplary and are not intended to limit the scope of the present application. Those of ordinary skill in the art can make various changes and modifications without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as claimed in the appended claims.

[0127] In the specification provided herein, a large number of specific details are described. However, it can be understood that embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of the present specification.

[0128] Similarly, it is to be understood that the embodiments of the present application can be altered, combined, or modified and constitute modifications and variations of the application and are therefore intended to be within the scope of the claims. It is intended that each of the individual features described above and those not specifically stated can be employed separately or in any combination by those of ordinary skill in the art. As will be realized, the application is capable of modifications in various obvious aspects, all without departing from the scope of the present application. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature and explanations made are to support compliance with the patent statutes and are not intended to limit the scope of the application to exemplified embodiments. The claims set forth below are what is desired to be protected by legal rights and are intended to be the measure of the application, although the scope of the application can require further modification in form and detail.

[0129] Those skilled in the art will appreciate that all features described herein (including all features and processes described in the claims, abstract and drawings) can be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Each feature disclosed in this specification, including any "preferred" features, can be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

[0130] Furthermore, those skilled in the art will recognize that references in the specification to "one embodiment", "an embodiment", "an example embodiment", mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily referring to a single, "one embodiment".

[0131] It is to be understood that the embodiments and variations thereof described herein are merely exemplary embodiments of the application and are not intended to limit the scope of the application to these embodiments or such variations. It is contemplated that in these respects there is a potential for numerous adaptations and modifications which, if constructed according to the application and falling within the scope of the application, will be encompassed within the spirit of the application. Accordingly, reference should be made solely to the appended claims in construing the scope of the application. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, any reference signs placed between parentheses shall not be construed as limiting the claim. The use of the word "about" in relation to a numerical value specifies the precision of the value and not the exactness of the value. The use of the word "first", "second", and "third" does not connote any order, quantity, creation or importance, but rather are used to distinguish one element from another. The use of the word "one" does not exclude the presence of more than one such element or step.

[0132] The above merely describes specific embodiments or specific implementation of the present application, and the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. The protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating virtual avatars, characterized in that, The method includes: Based on the input voice data obtained from the user, the voice features of the response voice data corresponding to the input voice data are obtained; Based on the speech features of the response speech data, the orthogonal basis vector coefficients of the response speech data are obtained; the orthogonal basis vector coefficients of the response speech data are denormalized to obtain the facial feature points corresponding to the response speech data; and the facial feature points are combined with the stored facial mask image to obtain a facial mask image containing the facial feature points. Based on at least two reference face images obtained from the user, facial features for a virtual avatar are obtained, including: extracting facial semantic features from each reference face image; performing facial segmentation on each reference face image to obtain a binarized facial feature mask image of the facial features in the reference face image; calculating the average feature vector of each facial feature from the facial semantic features using the facial feature mask image; and fusing the average feature vectors of the facial features in each of the reference face images to obtain facial features for the virtual avatar; wherein, before calculating the average feature vector of each facial feature using the facial feature mask image, the facial feature mask image is dilated to obtain a dilated facial feature mask image; and the dilated facial feature mask image is used to calculate the average feature vector of each facial feature; and Based on the face mask image containing facial feature points and the facial features, the virtual image corresponding to the response voice data of the current frame is obtained.

2. The method as described in claim 1, characterized in that, The speech features of the response speech data corresponding to the input speech data obtained from the user are obtained, including: Based on the input voice data obtained from the user, response voice data corresponding to the input voice data is obtained; Based on the response speech data, the speech features corresponding to the response speech data are obtained.

3. The method as described in claim 2, characterized in that, Based on the response speech data, the speech features corresponding to the response speech data are obtained, including: Extract the normalized spectral coefficients of the response speech data; The semantic features of the response speech data are obtained based on the normalized spectral coefficients; The emotional features of the response speech data are obtained based on the normalized spectral coefficients; and The semantic features and the emotional features are fused to obtain the speech features of the response speech data.

4. The method as described in claim 1, characterized in that, The process of obtaining facial features for the virtual avatar based on the reference face image also includes: Before extracting the semantic features of the face from the reference face image, the reference face image is aligned with a preset standard face image to obtain an aligned reference face image; and Facial semantic features are extracted from the aligned reference face image.

5. The method as described in claim 1, characterized in that, The virtual avatar corresponding to the response voice data of the current frame is also based on the virtual avatar of the previous frame.

6. The method as described in claim 1, characterized in that, The facial features mentioned include eye features, eyebrow features, nose features, and mouth features.

7. The method as described in claim 1, characterized in that, The virtual avatar generation method is implemented using a neural network.

8. A virtual avatar generation system, characterized in that, The system includes: A processor for using a neural network to perform the virtual avatar generation method as described in any one of claims 1-7; A memory for storing the network parameters of the neural network.

9. A computing device, characterized in that, The computing device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable medium, characterized in that, The computer-readable medium stores a computer program that, when executed, performs the method as described in any one of claims 1-7.

Images