A deep learning-based speech-driven face generation method
By using deep learning technology, combined with speech recognition and progressive generative adversarial networks such as StyleGAN2 and CLIP models, the problem of facial feature and expression restoration in speech-driven face generation is solved, achieving accurate face generation and automated processing of diverse facial changes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2022-09-27
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies struggle to generate accurate simulated portraits directly from voice, particularly in terms of facial features and expressions, and also fail to handle facial changes at different ages.
Using a deep learning-based approach, speech recognition is converted into text information. A progressive generative adversarial network combining StyleGAN2 and CLIP models is used to train the generator and encoder, thereby achieving a mapping from text to image and generating a face that matches the speech description.
It enables the automatic generation of accurate facial images based on speech, including simulation of facial features and expressions, and can handle facial changes at different ages, simplifying the generation process.
Smart Images
Figure CN115861461B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of pattern recognition technology, and in particular to a speech-driven face generation method based on deep learning. Background Technology
[0002] The emergence of voice-driven face generation technology stems from the urgent need for simulated portraits. If computers can provide assistance in creating more accurate portraits, it will greatly help in the simulation of portraits.
[0003] Simulated portraits strive for a close resemblance, ensuring precise similarity in features such as the eyes, nose, mouth, and eyebrows. Beyond accurately reproducing the appearance, they also emphasize the simulation of expressions and the changes in facial features at different ages. The ability to automatically generate a portrait containing relevant descriptive information based solely on verbal description has significant practical implications. Summary of the Invention
[0004] This application provides a voice-driven face generation method based on deep learning, the technical purpose of which is to automatically and accurately generate faces including corresponding features based on spoken information.
[0005] The above-mentioned technical objective of this application is achieved through the following technical solution:
[0006] A deep learning-based speech-driven face generation method includes:
[0007] S1: Recognize the speech signal and convert it into corresponding text information;
[0008] S2: Invert the generator of the StyleGAN2 model to obtain the image encoder, and train the text encoder of the CLIP model to make the text vector w l and image vector w v If the distance is minimized, a trained text encoder is obtained. The image encoder and the trained text encoder are then connected to the synthetic network of the StyleGAN2 model to form a progressive generative adversarial network.
[0009] S3: Input the text information into the progressive generative adversarial network to obtain the latent code W corresponding to the text information;
[0010] S4: Input the latent encoding W into the generator of the StyleGAN2 model to generate a face.
[0011] The beneficial effects of this application are as follows: This application makes targeted improvements to the system framework for the input type that drives the face, and innovatively adds a speech recognition module to directly obtain the basic facial features of the face through speech recognition, which simplifies the face generation process; at the same time, it makes full use of the similarity of the mapping relationship between different face generation models in the StyleGAN2 model, and can generate a face shape of a certain style in a targeted manner. Attached Figure Description
[0012] Figure 1 This is a flowchart of the method described in this application;
[0013] Figure 2 This is a facial rendering generated in a specific embodiment. Detailed Implementation
[0014] The technical solution of this application will be described in detail below with reference to the accompanying drawings.
[0015] like Figure 1 As shown, the deep learning-based speech-driven face generation method described in this application includes:
[0016] S1: Recognize the speech signal and convert it into corresponding text information.
[0017] Specifically, the aforementioned speech recognition process is implemented through a feature extraction network, an acoustic model, a language model, a dictionary, and decoding. Furthermore, to extract features more effectively, the acquired sound signal requires preprocessing such as filtering and framing to appropriately extract the audio signal to be analyzed from the original signal. The audio to be detected should include basic facial features of a specific person, including but not limited to gender, age, nose shape, lip thickness, hair color and length, and whether glasses are worn.
[0018] As a specific implementation, the voice signal is recognized based on the Baidu API, and the API is called through the API Key and Secret Key.
[0019] S2: Invert the generator of the StyleGAN2 model to obtain the image encoder, and train the text encoder of the CLIP model to make the text vector w l and image vector w v If the distance is minimized, a trained text encoder is obtained. The image encoder and the trained text encoder are then connected to the synthetic network of the StyleGAN2 model to form a progressive generative adversarial network.
[0020] Specifically, the inversion approach involves introducing an additional image encoder E. v via image encoder E vThe real image x is then encoded so that it maps to Z. s Nearby, represented as:
[0021]
[0022] Among them, Z s G(.) represents the vector Z in space s; G(.) represents the generator of the StyleGAN2 model; G(Z s ) indicates that the generator of the StyleGAN2 model is based on the vector Z s Generate an image; E v (G(Z s )) indicates that the image encoder E v For image G(Z) s Then encode it.
[0023] Then, additional text information of the real image x is added, and this text information is mapped to the latent space w, thereby obtaining the semantic information of the real image x, represented as:
[0024]
[0025]
[0026] Where F(.) represents the feature extraction network VGG; ||·||2 represents the L2 distance; The formula for calculating variance is represented by D. v λ1, λ2, and λ3 represent the image discriminator; λ1, λ2, and λ3 all represent hyperparameters.
[0027] The image encoder E is evaluated using equations (2) to (3). v Train it to infer Z from image x. s The ability to obtain the image encoder.
[0028] Training the text encoder of the CLIP model results in text vector w l and image vector w v The text encoder with the smallest distance is obtained, including:
[0029] By encoding text and images separately using the CLIP model, the text is transformed into a vector w in the latent space w through the text encoder of the CLIP model. l The image is processed by the image encoder of the CLIP model to obtain the vector w in the latent space w. v Training the text encoder of the CLIP model makes w l and w v The distance is minimized, and the trained text encoder is obtained through equation (4), which is expressed as:
[0030]
[0031] Among them, E l Indicates a text encoder; p i This represents the weights of the i-th input layer of the generator.
[0032] Equation (4) is the core idea of training a text encoder, w v For image encoder E v The vector obtained by mapping the image to space w; w l This represents the vector obtained by mapping text to the w space through the text encoder (the encoder built into the CLIP model). For each input layer of the StyleGAN2 model, w is... l and w v The distance between them should be minimized, meaning that the w mapped to the same pair of text and image in the latent space should be as similar as possible.
[0033] S3: Input the text information into the progressive generative adversarial network to obtain the latent code W corresponding to the text information.
[0034] S4: Input the latent encoding W into the generator of the StyleGAN2 model to generate a face.
[0035] In a specific embodiment, the latent encoding W is a (18, 512)-dimensional vector. The latent encoding W is mapped from the (18, 512)-dimensional vector to a (1024, 1024, 3)-dimensional vector using the generator of the StyleGAN2 model. To meet the requirements of the simulated portrait task, an Asian-style face model is specifically trained using a dataset of faces with Asian facial features. The specific method for face generation is to input the (18, 512)-dimensional latent encoding W of the bridged text obtained in step S3 into the generator of the StyleGAN2 model loaded with the parameters of the Asian facial model, automatically generating face images with Asian facial features.
[0036] As a specific embodiment, the StyleGAN2 model described in this application is trained using a dataset of 10,000 Asian-style faces. The StyleGAN2 model's network structure consists of two parts. The first is the Mapping network, which generates intermediate hidden variables w from the hidden variable z. This w is used to control the style of the generated image. The second is the Synthesis network, which generates the image. Each sub-network layer is input with A and B. A is an affine transformation obtained by transforming w, used to control the style of the generated image. B is the transformed random noise used to enrich the details of the generated image. That is, each convolutional layer can adjust the style based on the input A and adjust the details through B.
[0037] Taking the method of generating an image of a middle-aged man using speech, characterized by features such as a middle-aged man, long straight gray hair, a beard, a wide nose, and a wide chin as an example:
[0038] First, input facial features in voice form, such as "middle-aged male, long straight gray hair, with a beard, wide nose, and wide chin." The system automatically collects the input audio and outputs the voice as text.
[0039] Secondly, text information containing the man's facial features is input into a text-driven generation network, combining StyleGAN and CLIP methods. The CLIP model is used as a loss network to return the latent encoding W.
[0040] Finally, the parameters of the StyleGAN2 model trained on the Asian-style face dataset are loaded to obtain face images that match the text information. Because different models share similar mapping relationships from (18, 512)-dimensional vectors to (1024, 1024, 3)-dimensional vectors, the input latent encoding W yields a face image of a middle-aged Asian man with features such as long, straight, graying hair, a beard, a wide nose, and a wide chin. Figure 2 As shown.
[0041] The above are exemplary embodiments of this application, and the scope of protection of this application is defined by the claims and their equivalents.
Claims
1. A speech-driven face generation method based on deep learning, characterized in that, include: S1: Recognize the speech signal and convert it into corresponding text information; S2: Invert the generator of the StyleGAN2 model to obtain the image encoder, and train the text encoder of the CLIP model to make the text vectors... and image vectors If the distance is minimized, a trained text encoder is obtained. The image encoder and the trained text encoder are then connected to the synthetic network of the StyleGAN2 model to form a progressive generative adversarial network. S3: Input the text information into the progressive generative adversarial network to obtain the latent code W corresponding to the text information; S4: Input the latent encoding W into the generator of the StyleGAN2 model to generate a face; In step S2, the generator of the StyleGAN2 model is inverted to obtain the image encoder, including: Introducing an image encoder Through the image encoder The real image x is then encoded so that it is mapped to... Nearby, represented as: ；（1） in, Let Z represent a vector in space s; This represents the generator of the StyleGAN2 model; This indicates that the generator of the StyleGAN2 model is based on vectors. Generate an image; Indicates via image encoder For images Then encode it; By adding additional textual information from the real image x and mapping this textual information to the latent space w, the semantic information of the real image x can be obtained, represented as: ；（2） ；（3） Where F(.) represents the feature extraction network VGG; Indicates L2 distance; This represents the formula for calculating variance. Represents an image discriminator; , and All represent hyperparameters; The image encoder is processed by equations (2) to (3). Train it to infer from real image x The ability to obtain the image encoder.
2. The method as described in claim 1, characterized in that, Step S1 is implemented through a feature extraction network, an acoustic model, a language model, a dictionary, and decoding.
3. The method as described in claim 2, characterized in that, Before feeding the speech signal into the feature extraction network for feature extraction, the speech signal is preprocessed, including filtering and framing.
4. The method as described in claim 1, characterized in that, The latent encoding W is a (18, 512) dimensional vector. The latent encoding W establishes a mapping relationship from the (18, 512) dimensional vector to the (1024, 1024, 3) dimensional vector through the generator of the StyleGAN2 model.
5. The method as described in claim 1, characterized in that, The StyleGAN2 model was trained using a dataset of 10,000 Asian-style faces.
6. The method as described in claim 1, characterized in that, In step S2, the text encoder of the CLIP model is trained to make the text vectors and image vectors The text encoder with the smallest distance is obtained, including: By encoding text and images separately using the CLIP model, the text is transformed into a vector of the latent space w through the text encoder of the CLIP model. The image is processed by the image encoder of the CLIP model to obtain the vector of the latent space w. Training the text encoder of the CLIP model enables and The distance is minimized, and the trained text encoder is obtained through equation (4), which is expressed as: ；（4） in, Indicates a text encoder; This represents the weights of the i-th input layer of the generator.
7. The method as described in claim 1, characterized in that, In step S1, the voice signal is recognized based on the Baidu API, and the API is called using the API Key and Secret Key.