Face generation method and device, electronic equipment and storage medium
Through the improvement of speaking face synthesis technology, using noise-added processing and encoding processing combined with diffusion model, the problems of inaccuracy and stiffness of human faces in the existing technology are solved, and high-quality matching of speech and facial features are achieved, and the generated animations are achieved. More vivid and voice-compatible.
Patent Information
- Application Number
- CN202411203932.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-06
AI Technical Summary
The existing speaking face synthesis technology relies on a large number of training samples, and the generated faces are inaccurate and stiff, making it difficult to achieve high-quality matching of speech and facial features.
By obtaining the current face data of the target object, performing noise addition and encoding processing, combining the target voice data and face key point embedding data, image merging and denoising processing are used to generate the target face image.
The accuracy and naturalness of face generation are improved, and the generated face image animation is more vivid, conforming to voice content and facial identity information, rich facial movements and expressions, and strong generalization ability.
Smart Images

Figure CN119090988A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a face generation method and device, electronic device, and storage medium. Background Art
[0002] Face generation is a computer graphics and artificial intelligence technology that can be used to generate images that match facial features. Face generation technology can be applied to speaking face synthesis, thereby synthesizing images or animations that match speech content, so as to achieve the conversion of speech content and face images into face animations. Speaking face synthesis technology can be used in video games, virtual hosts, voice assistants and other scenarios. The current speaking face synthesis technology is mainly based on deep learning technology, which requires a large number of training samples, and the generated faces are inaccurate and relatively rigid. Summary of the invention
[0003] The main purpose of the embodiments of the present application is to propose a face generation method and device, an electronic device, and a storage medium, aiming to improve the accuracy of face generation.
[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a face generation method, the method comprising:
[0005] Acquire current facial data of a target object; wherein the current facial data includes a first facial image, a second facial image, and an occluded facial image, wherein the first facial image has a first head posture of the target object, the second facial image has a second head posture of the target object, the first head posture and the second head posture are different, and the occluded facial image is an image obtained by occluding the mouth of the first facial image;
[0006] Performing noise processing on the first facial image to obtain a noisy facial image having the first head posture;
[0007] Encoding the blocked face image to obtain a first embedded image with the first head posture, and encoding the second face image to obtain a second embedded image with the second head posture;
[0008] Merging the noisy face image, the first embedded image, and the second embedded image to obtain a merged face image;
[0009] Acquire target speech data, and encode the target speech data to obtain target speech embedded data;
[0010] The face merged image is reconstructed based on the target speech embedding data and the target face key point embedding data to obtain a target face image; wherein the target face key point embedding data is obtained based on encoding the face key points of the target object.
[0011] In some embodiments, the step of performing face reconstruction on the face merged image based on the target speech embedding data and the target face key point embedding data to obtain a target face image includes:
[0012] Extracting speech features from the target speech embedding data to obtain a speech feature sequence;
[0013] Extract key point features from the target face key point embedding data to obtain a key point feature sequence;
[0014] The face merged image is denoised based on the speech feature sequence and the key point feature sequence to obtain the target face image.
[0015] In some embodiments, the denoising process is performed on the face merged image based on the speech feature sequence and the key point feature sequence to obtain the target face image, including:
[0016] Obtaining a pre-trained target diffusion model; wherein the target diffusion model includes an attention network layer with a cross-attention mechanism;
[0017] Based on the attention network layer, cross attention calculation is performed on the speech feature sequence and the key point feature sequence to obtain image attention weight data;
[0018] The face merged image is denoised based on the image attention weight data to obtain the target face image.
[0019] In some embodiments, the target diffusion model further includes a noise adding network layer, and the step of performing noise adding processing on the first face image to obtain the noisy face image includes:
[0020] Mapping the first face image to the target latent space through a preset image encoder;
[0021] In the target latent space, the first face image is subjected to noise addition processing based on the noise addition network layer to obtain the noisy face image.
[0022] In some embodiments, the method further includes training the target diffusion model, specifically including:
[0023] Acquire a facial image sample of a sample object; wherein the facial image sample includes a first facial sample, a second facial sample, and an occluded facial sample, wherein the first facial sample is an image having a first posture of the sample object, the second facial sample is an image having a second posture of the sample object, the first posture and the second posture are both head postures of the sample object and the first posture and the second posture are different, and the occluded facial sample is an image obtained by occluding the mouth of the first facial sample;
[0024] Performing noise processing on the first face sample to obtain a noisy face sample;
[0025] Encoding the blocked face sample to obtain a first embedded sample, and encoding the second face sample to obtain a second embedded sample;
[0026] Merging the noisy face sample, the first embedded sample, and the second embedded sample to obtain a merged face sample;
[0027] Acquire a speech sample, and perform encoding processing on the speech sample to obtain a speech embedding sample;
[0028] Acquire the facial key points of the sample object to obtain facial key point feature samples, and encode the facial key point feature samples to obtain facial key point embedded samples;
[0029] The speech embedding sample is used as the key of the attention network layer, and the facial key point embedding sample is used as the value of the attention network layer to denoise the face merged sample to obtain a target denoised image; wherein the target denoised image is used as the face image sample.
[0030] In some embodiments, merging the noisy face image, the first embedded image, and the second embedded image to obtain a merged face image includes:
[0031] Merging the first head posture and the second head posture to obtain posture merging information;
[0032] Adding the posture merging information and the target identity information of the target object to the noisy face image to obtain a noisy merged image; wherein the target identity information is used to indicate the identity of the target object;
[0033] The noisy merged image, the first embedded image, and the second embedded image are merged to obtain the face merged image.
[0034] In some embodiments, before performing face reconstruction on the face merged image based on the target speech embedding data and the target face key point embedding data to obtain the target face image, the method further includes:
[0035] Acquire a third facial image of the target object; wherein the first facial image or the second facial image is used as the third facial image;
[0036] Extracting facial key points from the third facial image to obtain target key point features;
[0037] Coordinate encoding is performed on the target key point features to obtain the target face key point embedding data.
[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a face generation device, the device comprising:
[0039] A face data acquisition module, configured to acquire current face data of a target object; wherein the current face data includes a first face image, a second face image, and an occluded face image, wherein the first face image has a first head posture of the target object, the second face image has a second head posture of the target object, the first head posture and the second head posture are different, and the occluded face image is an image obtained by occluding the mouth of the first face image;
[0040] An image denoising module, configured to perform denoising processing on the first face image to obtain a denoised face image having the first head posture;
[0041] An image encoding module, configured to encode the blocked face image to obtain a first embedded image having the first head posture, and encode the second face image to obtain a second embedded image having the second head posture;
[0042] An image superposition module, used for merging the noisy face image, the first embedded image, and the second embedded image to obtain a face merged image;
[0043] The voice data processing module is used to: obtain target voice data, and encode the target voice data to obtain target voice embedded data;
[0044] A face synthesis module is used to reconstruct the face merged image based on the target speech embedding data and the target face key point embedding data to obtain a target face image; wherein the target face key point embedding data is obtained based on encoding the face key points of the target object.
[0045] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the method described in the first aspect is implemented.
[0047] The face generation method and device, electronic device, and storage medium proposed in the embodiments of the present application obtain current face data of a target object, where the current face data includes a first face image with a first head posture, a second face image with a second head posture, and an occluded face image obtained by occluding the mouth of the first face image, performs noise processing on the first face image to obtain a noisy face image with a first head posture, performs encoding processing on the occluded face image to obtain a first embedded image with the first head posture, performs encoding processing on the second face image to obtain a second embedded image with the second head posture, and performs image merging on the noisy face image, the first embedded image, and the second embedded image to obtain a face merged image, and performs face reconstruction on the face merged image based on target speech embedding data and target face key point embedding data to obtain a target face image, thereby improving the accuracy of face synthesis. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a flow chart of a face generation method provided in an embodiment of the present application;
[0049] Figure 2 yes Figure 1 Flowchart of step 104 in;
[0050] Figure 3 is another flow chart of the face generation method provided in an embodiment of the present application;
[0051] Figure 4 yes Figure 1 Flowchart of step 106 in;
[0052] Figure 5 yes Figure 4 Flow chart of step 403 in;
[0053] Figure 6 is a schematic diagram of the structure of a face generation device provided in an embodiment of the present application;
[0054] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0056] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0058] First, some nouns involved in this application are analyzed:
[0059] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0060] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information face generation, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.
[0061] Cross-attention mechanism: The cross-attention mechanism is a type of attention mechanism, which is usually used in the fields of NLP and deep learning. It is used to imitate the focusing behavior of humans when processing information. It can help the model focus on the most relevant information when processing long sequences. The cross-attention mechanism can process two related but different sequences and is often used in multimodal tasks, such as for the joint processing of images and text.
[0062] Face generation is a computer graphics and artificial intelligence technology that can be used to generate images that match facial features. Face generation technology can be applied to speaking face synthesis, thereby synthesizing images or animations that match speech content, so as to achieve the conversion of speech content and face images into face animations. Speaking face synthesis technology can be used in video games, virtual hosts, voice assistants and other scenarios. The current speaking face synthesis technology is mainly based on deep learning technology, which requires a large number of training samples, and the generated faces are inaccurate and relatively rigid.
[0063] Based on this, the embodiments of the present application provide a face generation method and device, an electronic device, and a storage medium, aiming to improve the accuracy of face synthesis.
[0064] The face generation method and device, electronic device, and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the face generation method in the embodiments of the present application is described.
[0065] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0066] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0067] The face generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The face generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the face generation method, etc., but is not limited to the above forms.
[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0069] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics such as the user's face, user's voice, user information, etc., the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0070] Figure 1 is an optional flowchart of the face generation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 101 to 106.
[0071] Step 101, obtaining current facial data of a target object; wherein the current facial data includes a first facial image, a second facial image, and an occluded facial image, the first facial image has a first head posture of the target object, the second facial image has a second head posture of the target object, the first head posture and the second head posture are different, and the occluded facial image is an image obtained by occluding the mouth of the first facial image;
[0072] Step 102, performing noise processing on the first face image to obtain a noisy face image;
[0073] Step 103, encoding the blocked face image to obtain a first embedded image, and encoding the second face image to obtain a second embedded image;
[0074] Step 104, merging the noisy face image, the first embedded image, and the second embedded image to obtain a merged face image;
[0075] Step 105, obtaining target speech data, and encoding the target speech data to obtain target speech embedded data;
[0076] Step 106, reconstructing the face merged image based on the target speech embedding data and the target face key point embedding data to obtain a target face image; wherein the target face key point embedding data is obtained based on encoding the face key points of the target object.
[0077] In steps 101 to 106 illustrated in the embodiment of the present application, current facial data of the target object is obtained, wherein the current facial data includes a first facial image with a first head posture, a second facial image with a second head posture, and an occluded facial image obtained by occluding the mouth of the first facial image, the first facial image is subjected to noise processing to obtain a noisy facial image with a first head posture, the occluded facial image is subjected to encoding processing to obtain a first embedded image with a first head posture, the second facial image is subjected to encoding processing to obtain a second embedded image with a second head posture, and the noisy facial image, the first embedded image, and the second embedded image are subjected to image merging to obtain a merged face image, and the merged face image is subjected to face reconstruction based on target speech embedding data and target facial key point embedding data to obtain a target face image, thereby improving the accuracy of face generation.
[0078] In step 101 of some embodiments, the target object may be a person required for synthesizing a human face. For example, in a financial scenario, the target object may be an insurance agent or a department manager of an insurance agent. In another example, in the field of digital medicine, the target object may be the patient's attending physician or the head nurse in charge of the patient. In other fields, the target object may also be a host, sports star, game anchor, etc.
[0079] The current facial data of the target object includes at least three images, namely: at least one first facial image, at least one second facial image and at least one occluded facial image, wherein the first facial image has the first head posture of the target object, the second facial image has the second head posture of the target object, and the first head posture and the second head posture are different, and the first head posture and the second head posture are used to characterize the state of the head of the target object, such as the rotation angle of the head, the facing direction of the head, etc. In an actual application scenario, the first head posture of the first facial image can be, for example, the target object looking to the left, and the second head posture of the second facial image can be, for example, the target object looking to the right; the difference between the occluded facial image and the first facial image is that the mouth of the target object in the occluded facial image is an occluded mouth, that is, the mouth of the target object in the first facial image is occluded to obtain the occluded facial image.
[0080] The first facial image and the second facial image of the embodiment of the present application are derived from the same target object. In some embodiments, the first facial image and the second facial image in the current facial data can be obtained by photographing the target object by a photographing device, wherein the photographing device can be a camera, a mobile phone, or a still camera, etc.; the occluded facial image in the current facial data can be obtained by performing mouth occlusion processing on the first facial image. In other embodiments, the current facial data of the target object can be pre-stored in a database and directly obtained from the database through step 101. The embodiment of the present application does not limit the current facial data of the target object.
[0081] In step 102 of some embodiments, the first facial image may be subjected to noise addition processing based on a target diffusion model, thereby obtaining a noisy facial image.
[0082] A denoising diffusion model of the target diffusion model adopted in the embodiment of the present application is a type of generative model. For the original data input into the model, the model can learn the probability distribution of the original data and generate new samples similar to the original data. The core idea of the denoising diffusion model is to generate new samples in the noise diffusion process, wherein noise diffusion is a random process, and during the noise diffusion process, the noise gradually propagates and affects the distribution of the data. By adding noise in the diffusion process and removing the noise inversely, the denoising diffusion model can generate new samples similar to the original data. The processing process of the target diffusion model mainly includes: diffusion process and inverse diffusion process, wherein the diffusion process starts from an original data, gradually introduces noise, so that it gradually approaches the distribution of the target data (new sample) at each step, the original data is the initial noise sample, and the target data is the new sample; wherein the noise in the diffusion process can be Gaussian noise or other types of noise. Among them, the inverse diffusion process is also a denoising process, which removes the noise inversely on the basis of the diffusion process until a new sample similar to the original data is restored.
[0083] The embodiment of the present application uses a deep neural network to introduce and remove noise. The deep neural network can be trained by back propagation and maximum likelihood estimation to optimize model parameters so as to better approximate the distribution of real data. Compared with other generative models, the denoising diffusion model used in this application performs better in image synthesis tasks, for example, it is more stable and easier to train when generating high-resolution images.
[0084] In some embodiments, step 102 may include, but is not limited to, including:
[0085] Mapping the first face image to the target latent space through a preset image encoder;
[0086] In the target latent space, the first face image is denoised based on the target diffusion model to obtain a noisy face image.
[0087] In this embodiment, by performing noise processing on the first face image in the target latent space, the computational consumption can be reduced and the running speed can be improved.
[0088] In the embodiment of the present application, the process of performing noise addition processing on the first face image is not limited, and reference may be made to the principle of image noise addition using a denoising diffusion probability model (DDPM).
[0089] In step 103 of some embodiments, the occluded face image can be encoded based on the image encoder to map the occluded face image to a latent space, thereby obtaining a first embedded image; similarly, the second face image can be encoded based on the image encoder to map the second face image to a latent space, thereby obtaining a second embedded image.
[0090] In some embodiments, the image encoding used in step 102 and step 103 is a pre-trained encoder, and the entire encoder is frozen during training and does not participate in gradient descent. The image encoder can be regarded as an encoder of a vector quantization generative adversarial network. In vector quantization, the image is regarded as composed of many small blocks, which can be regarded as the basic elements of the image, and these small blocks are also a kind of vector; the goal of vector quantization is to map these small blocks to the most similar codebook vectors to achieve encoding and decoding of the image; the core of vector quantization lies in the design of the codebook and the selection of the codebook vector. Generative adversarial network is an algorithm for generating realistic images. The generative adversarial network includes a generator and a discriminator, wherein the generator is responsible for creating false images, and the discriminator is responsible for distinguishing between real images and false images; during the training process, the generator and the discriminator compete with each other, the generator tries to deceive the discriminator, and the discriminator tries to distinguish between real images and false images. As the training proceeds, the generator is gradually able to generate more and more realistic images. The image encoder of the embodiment of the present application combines the technical principles of vector quantization and generative adversarial networks. Vector quantization decomposes the input image into small blocks and represents them with the closest codebook vector; the codebook vector is used as the input of the generator to gradually generate realistic images. Therefore, the image encoder of the embodiment of the present application can learn the potential structure and texture of the image and generate new image samples in a better way.
[0091] See also Figure 2 In some embodiments, step 104 may include, but is not limited to:
[0092] Step 201, merging the first head posture and the second head posture to obtain posture merging information;
[0093] Step 202, adding the posture merging information and the target identity information of the target object to the noisy face image to obtain a noisy merged image; wherein the target identity information is used to indicate the identity of the target object;
[0094] Step 203: merge the noisy merged image, the first embedded image, and the second embedded image to obtain a face merged image.
[0095] In step 201 and step 202 of some embodiments, by merging the first head posture and the second head posture, the head posture information of the target object and the target identity information of the target object are integrated into the noisy face image, and a target face image that conforms to the identity and head posture of the target object can be generated during subsequent denoising and reconstruction, thereby improving the accuracy of face generation. The target identity information can be the name of the target object, for example, the target identity information of target object A is Zhang San, the target identity information of target object B is Li Si, and the target identity information of target object C is Wang Wu.
[0096] In step 203 of some embodiments, by merging the noisy merged image, the first embedded image, and the second embedded image in the target latent space to obtain a face merged image, the computational cost can be reduced and the running speed can be improved.
[0097] In an embodiment of the present application, at least two images of the same target object with different head postures and two different images of the same target object with and without a mouth are required, so that the resulting merged face image contains information such as the identity, head posture, and mouth appearance of the target object. This information helps to generate a facial image that is more consistent with the facial identity, head posture, and mouth appearance (a mouth appearance that better matches the speech content) during face reconstruction.
[0098] In some embodiments, step 105 may include, but is not limited to:
[0099] Obtain target voice data;
[0100] Based on the preset speech encoder, the target speech data is mapped to the target latent space to obtain the target speech embedding data.
[0101] In step 105, the acquired target voice data is used for subsequent face synthesis. In a financial scenario, if the target object is an insurance agent, the target voice data is voice data used to explain the insurance plan. For example, the content of the target voice data may be: Hello Mr. Zhang, based on your current family situation and financial situation, we recommend that you purchase insurance product M and financial product N.
[0102] In an embodiment of the present application, the target speech data can be encoded based on a speech encoder to map the target speech data to a target latent space, thereby obtaining target speech embedding data. In one application scenario, the speech encoder can be a pre-trained DeepSpeech model based on a recurrent neural network, which helps to subsequently generate a face that is more consistent with the speech content, so that the facial features of the generated face are more consistent with the speech content.
[0103] In the embodiment of the present application, both the image and the speech are mapped to the latent space through the corresponding encoder, so that the corresponding image denoising and denoising processing and face reconstruction processing are performed in the latent space, which can reduce the computing consumption and improve the running speed.
[0104] In some other embodiments, step 105 may also include, but is not limited to:
[0105] Acquire speech emotion data; wherein the speech emotion data is used to characterize an emotion category, and the emotion category may, for example, represent an emotion such as surprise, happiness, sadness, joy, anger, tension, etc.;
[0106] The speech emotion data is fused with the target speech data to update the target speech data. The updated target speech data includes the above emotion category, so that the emotion category can be included in the synthesized target face image.
[0107] In this embodiment, the speech emotion data can be edited and configured by the user end without relying on the emotion category originally expressed by the target speech data. For example, the target speech data originally expressed the emotion category of sadness, but the user end can configure a new emotion category of happiness, thereby expressing the target speech data that was originally sad through the emotion category of happiness. In some application scenarios, by reconfiguring the emotion category, for example, configuring it to express an emotion category opposite to the original emotion category, a self-deprecating or humorous way of expression can be achieved, which can enhance the user experience.
[0108] See also Figure 3 In some embodiments, before step 106, the face generation method further includes: obtaining target face key point embedding data, specifically including:
[0109] Step 301, obtaining a third face image of the target object; wherein the first face image or the second face image is used as the third face image;
[0110] Step 302, extracting facial key points from the third facial image to obtain target key point features;
[0111] Step 303: Coordinate encoding of the target key point features is performed to obtain target face key point embedding data.
[0112] In step 301 of some embodiments, the third face image and the first face image are from the same target object, and the third face image and the second face image are from the same target object. The first face image can be used as the third face image, and the second face image can be used as the third face image. In other application scenarios, other images obtained by photographing the target object can also be used as the third face image, and this application does not limit this. The third face image can be used to determine the facial key points of the target object.
[0113] In step 302 of some embodiments, the face of the target object includes 68 facial key points, which may include, for example, the right corner of the left eye, the left corner of the right eye, the center of the left eye, the outer side of the left eye, the upper left corner of the left eyebrow, the upper part of the left ear, the middle part of the left ear, the lower part of the left ear, etc. By extracting facial key points from the 68 key points of the third face image, target key point features are obtained.
[0114] In step 303 of some embodiments, the target key point features are coordinate encoded based on a preset coordinate encoder to map the target key point features to a target latent space, thereby obtaining target face key point embedding data. In an application scenario, the coordinate encoder is a multi-layer perception mechanism, through which more accurate features and information can be extracted, thereby mining target face key point embedding data that is more consistent with face key point features.
[0115] Through step 301 to step 303, the target key point features of the target object are mapped to the target latent space, which facilitates the subsequent face reconstruction processing in the target latent space and can reduce computational consumption.
[0116] See also Figure 4 In some embodiments, step 106 may include, but is not limited to:
[0117] Step 401, extracting speech features from target speech embedding data to obtain a speech feature sequence;
[0118] Step 402, extracting key point features from the target face key point embedding data to obtain a key point feature sequence;
[0119] Step 403: De-noising the face merged image based on the speech feature sequence and the key point feature sequence to obtain a target face image.
[0120] In step 401 of some embodiments, the target speech embedding data is vectorized speech data, and a speech sequence can be obtained through the target speech embedding data to facilitate cross-attention calculation in step 403.
[0121] In step 402 of some embodiments, the target face key point embedding data is vectorized image data, and an image sequence of key points can be obtained through the target face key point embedding data to facilitate cross-attention calculation in step 403.
[0122] See also Figure 5 In some embodiments, step 403 may include, but is not limited to:
[0123] Step 501, obtaining a pre-trained target diffusion model; wherein the target diffusion model includes an attention network layer with a cross-attention mechanism;
[0124] Step 502, performing cross attention calculation on the speech feature sequence and the key point feature sequence based on the attention network layer to obtain image attention weight data;
[0125] Step 503: De-noise the face merged image based on the image attention weight data to obtain a target face image.
[0126] In steps 501 and 502 of some embodiments, the target diffusion model is obtained by pre-training, and the target diffusion model includes an attention network layer with a cross-attention mechanism, through which the key point feature sequence and the speech feature sequence can be jointly processed to perform cross-attention calculation to obtain image attention weight data. This application does not limit the cross-attention calculation.
[0127] In some embodiments, step 503 includes performing denoising and decoding on the face merged image, and both the denoising and decoding are performed in the target latent space, which can reduce computational consumption and improve running speed. The target face image obtained by step 503 includes a target mouth movement, which matches the speech text of the speech feature sequence.
[0128] Through steps 501 to 503 of the above embodiment, denoising of the face merged image can be achieved based on the cross-attention mechanism of the target diffusion model, and the denoising and noise addition processing in the embodiment of the present application are both completed in the target latent space, which can reduce the overall computational consumption and improve the overall running speed.
[0129] In steps 401 to 403 of the embodiment of the present application, the joint processing of the speech sequence and the key point image sequence can be achieved through the cross-attention mechanism to obtain the corresponding weight data, thereby achieving denoising of the face merged image through the weight data, and the denoising is completed in the target latent space, which can reduce the overall computational consumption.
[0130] In some embodiments, the target diffusion model further includes a noise adding network layer, and step 102 specifically includes:
[0131] Mapping the first face image to the target latent space through a preset image encoder;
[0132] In the target latent space, the first face image is subjected to noise processing based on the noise adding network layer to obtain a noisy face image. In step 102, by performing noise adding processing on the first face image in the target latent space, the computational consumption can be reduced and the operation speed can be improved.
[0133] In one application scenario, the face generation method may include:
[0134] Acquire current face data of target object A; wherein the current face data includes a first face image m1, a second face image m2, and an occluded face image m3; wherein the first face image m1 has a first head posture p1 of target object A, the second face image m2 has a second head posture p2 of target object A, the first head posture p1 and the second head posture p2 are different, and the occluded face image m3 is an image obtained by occluding the mouth of the first face image m1;
[0135] The first face image m1 is mapped to the target latent space by a preset image encoder IE1, and in the target latent space, the first face image m1 is subjected to noise processing based on the target diffusion model to obtain a noisy face image mq;
[0136] The occluded face image m3 is mapped to the target latent space by a preset image encoder IE2, so as to encode the occluded face image m3 to obtain a first embedded image e1; the second face image m2 is mapped to the target latent space by a preset image encoder IE3, so as to encode the second face image m2 to obtain a second embedded image e2; wherein the image encoder IE1, the image encoder IE2, and the image encoder IE3 may be the same encoder;
[0137] Superimpose the noisy face image mq, the first embedded image e1, and the second embedded image e2 to obtain a merged face image mc;
[0138] Obtain target speech data S, and map the target speech data S to a target latent space through a preset audio encoder IE4, so as to encode the target speech data S and obtain target speech embedding data Se;
[0139] Obtaining facial key points of target object A to obtain facial key point feature samples, and encoding the facial key point feature samples to obtain facial key point embedding samples;
[0140] Acquire a third face image m4 of the target object; wherein the first face image m1 or the second face image m2 is used as the third face image m4; extract face key points from the third face image m4 to obtain target key point features; coordinate encode the target key point features to obtain target face key point embedding data Fe;
[0141] Based on the target speech embedding data Se and the target face key point embedding data Fe, the face merged image mc is reconstructed to obtain the target face image; wherein, the target face key point embedding data is obtained by encoding the face key points of the target object. The obtained target face image is used as the first frame image fx1, and the mouth of the first frame image fx1 is blocked, and the obtained image is used as the blocked face image m3, and the above steps are repeated to obtain the second frame image fx2, the third frame image fx3...the nth frame image fxn. According to the above first frame image fx1, the second frame image fx2, the third frame image fx3...the nth frame image fxn, the image animation of the target object can be obtained, and the animation matches the content of the target speech data S, thereby realizing the synthesis of speech and face image, ensuring the smoothness of the image, and no jitter, and the animation of the synthesized face image is more vivid, more in line with the face identity information, head posture and mouth appearance, and the facial movements and expressions are richer and more able to match the speech content, and the generalization ability is stronger.
[0142] In some embodiments, the face generation method further includes training a target diffusion model, specifically including:
[0143] Acquire a facial image sample of a sample object; wherein the facial image sample includes a first facial sample, a second facial sample, and an occluded facial sample; wherein the first facial sample is an image having a first posture of the sample object, the second facial sample is an image having a second posture of the sample object, the first posture and the second posture are both head postures of the sample object and the first posture and the second posture are different, and the occluded facial sample is an image obtained by occluding the mouth of the first facial sample;
[0144] Performing noise processing on the first face sample to obtain a noisy face sample;
[0145] Encoding the occluded face sample to obtain a first embedded sample, and encoding the second face sample to obtain a second embedded sample;
[0146] Merging the noisy face sample, the first embedded sample, and the second embedded sample to obtain a merged face sample;
[0147] Acquire a speech sample, and encode the speech sample to obtain a speech embedding sample;
[0148] Acquire facial key points of the sample object to obtain facial key point feature samples, and encode the facial key point feature samples to obtain facial key point embedded samples;
[0149] The speech embedding sample is used as the key of the attention network layer, and the face key point embedding sample is used as the value of the attention network layer to denoise the face merged sample to obtain a target denoised image; wherein the target denoised image is used as the face image sample.
[0150] In the embodiment of the present application, the speech embedding sample is used as the key in the cross-attention mechanism, and the face key point embedding sample is used as the value in the cross-attention mechanism to guide the denoising process, thereby obtaining the target face sample. The target face sample obtained by the embodiment of the application can be used as a face image sample to generate a new sample.
[0151] The first embedding sample, the second embedding sample, and the facial key point embedding sample in the embodiment of the present application are vectorized image data, and the speech embedding sample is vectorized speech data.
[0152] The implementation principle of the training target diffusion model in this embodiment of the application is basically similar to the implementation principle of the above steps 101 to 106, which will not be repeated here.
[0153] The face generation method in the embodiments of the present application can be applied to scenarios such as intelligent customer service assistants and online intelligent insurance agents in the field of financial technology, and can also be applied to scenarios such as online doctor face generation and intelligent consultation assistants in the field of digital medicine. It can also be applied to scenarios such as video games and virtual hosts to make human-computer interaction more natural and vivid.
[0154] In the face generation method of the embodiment of the present application, the identity information, head posture and mouth appearance corresponding to the face are added to the noisy face image. In the denoising process, the identity information, head posture and mouth appearance will help reconstruct the target face image that conforms to the face identity information, head posture and mouth appearance. By splicing the target face image of each frame, an animation of the face image of the target object can be obtained, and the presentation of the animation changes the action and expression according to the voice content. Compared with the current face generation technology, the animation of the face image generated by the face generation method of the present application is more vivid and has a stronger generalization ability.
[0155] The face generation method provided in the embodiment of the present application can be used in dialogue scenarios between intelligent customer service and users. For example, in a financial technology scenario, the target object can be the intelligent customer service of an insurance company. The face animation generated by the face generation method in the embodiment of the present application is displayed on the screen of the smart terminal, so that a dialogue is conducted with the user through the smart terminal through the face animation. For example, the face animation can realize the explanation of insurance products.
[0156] See also Figure 6 The embodiment of the present application also provides a face generation device, which can implement the above face generation method, and the device includes:
[0157] A face data acquisition module, used to acquire current face data of a target object; wherein the current face data includes a first face image, a second face image, and an occluded face image, wherein the first face image has a first head posture of the target object, the second face image has a second head posture of the target object, the first head posture and the second head posture are different, and the occluded face image is an image obtained by occluding the mouth of the first face image;
[0158] An image denoising module, configured to perform denoising processing on the first face image to obtain a denoised face image with a first head posture;
[0159] An image encoding module, configured to encode the blocked face image to obtain a first embedded image with a first head posture, and encode the second face image to obtain a second embedded image with a second head posture;
[0160] An image superposition module is used to merge the noisy face image, the first embedded image, and the second embedded image to obtain a face merged image;
[0161] The voice data processing module is used to: obtain target voice data, and encode the target voice data to obtain target voice embedded data;
[0162] The face synthesis module is used to reconstruct the face merged image based on the target speech embedding data and the target face key point embedding data to obtain the target face image; wherein the target face key point embedding data is obtained based on encoding the face key points of the target object.
[0163] In some embodiments, the image overlay module may be specifically used to implement:
[0164] Merging the first head posture and the second head posture to obtain posture merging information;
[0165] Adding the posture merging information and the target identity information of the target object to the noisy face image to obtain a noisy merged image; wherein the target identity information is used to indicate the identity of the target object;
[0166] The noisy merged image, the first embedded image, and the second embedded image are merged to obtain a face merged image.
[0167] Specifically, the image overlay module can be used to implement the above steps 201 to 203, which will not be described in detail here.
[0168] In some embodiments, the face generation device may also be used to implement:
[0169] Acquire a third facial image of the target object; wherein the first facial image or the second facial image is used as the third facial image;
[0170] Extracting facial key points from the third face image to obtain target key point features;
[0171] The target key point features are coordinate encoded to obtain the target face key point embedding data.
[0172] Specifically, the face generation device can be used to implement the above steps 301 to 303, which will not be described in detail here.
[0173] In some embodiments, the image overlay module may be specifically used to implement:
[0174] Extract speech features from the target speech embedding data to obtain a speech feature sequence;
[0175] Extract key point features from the target face key point embedding data to obtain a key point feature sequence;
[0176] The face merged image is denoised based on the speech feature sequence and the key point feature sequence to obtain the target face image.
[0177] Specifically, the image overlay module can be used to implement the above steps 401 to 403, which will not be described in detail here.
[0178] In some embodiments, the image overlay module is used to implement "denoising the face merged image based on the speech feature sequence and the key point feature sequence to obtain the target face image", which may specifically include but is not limited to:
[0179] Obtain a pre-trained target diffusion model; wherein the target diffusion model includes an attention network layer with a cross-attention mechanism;
[0180] Based on the attention network layer, cross-attention calculation is performed on the speech feature sequence and the key point feature sequence to obtain the image attention weight data;
[0181] The face merged image is denoised based on the image attention weight data to obtain the target face image.
[0182] Specifically, the image overlay module can be used to implement the above steps 501 to 503, which will not be described in detail here.
[0183] The specific implementation of the face generation device is basically the same as the specific implementation of the above-mentioned face generation method, and will not be repeated here.
[0184] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned face generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0185] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0186] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0187] The memory 702 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the face generation method of the embodiment of this application;
[0188] Input / output interface 703, used to implement information input and output;
[0189] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);
[0190] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);
[0191] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .
[0192] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned face generation method is implemented.
[0193] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0194] The face generation method and device, electronic device, and storage medium provided by the embodiments of the present application obtain current face data of a target object, where the current face data includes a first face image with a first head posture, a second face image with a second head posture, and an occluded face image obtained by occluding the mouth of the first face image, performs noise processing on the first face image to obtain a noisy face image with the first head posture, performs encoding processing on the occluded face image to obtain a first embedded image with the first head posture, performs encoding processing on the second face image to obtain a second embedded image with the second head posture, and performs image merging on the noisy face image, the first embedded image, and the second embedded image to obtain a face merged image, and performs face reconstruction on the face merged image based on target speech embedding data and target face key point embedding data to obtain a target face image, thereby improving the accuracy of face generation. The embodiments of the present application can realize the synthesis of speech and facial images, ensure the smoothness of the image, and prevent jitter. The animation of the synthesized facial image is more vivid, more consistent with the facial identity information, head posture and mouth appearance, the facial movements and expressions are richer and more able to match the speech content, and the generalization ability is stronger.
[0195] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0196] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0197] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0198] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0199] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0200] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0201] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0202] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0203] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0204] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0205] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A face generation method, It is characterized in that The method comprises: Acquire current facial data of a target object; wherein the current facial data includes a first facial image, a second facial image, and an occluded facial image, wherein the first facial image has a first head posture of the target object, the second facial image has a second head posture of the target object, the first head posture and the second head posture are different, and the occluded facial image is an image obtained by occluding the mouth of the first facial image; Performing noise processing on the first facial image to obtain a noisy facial image having the first head posture; Encoding the blocked face image to obtain a first embedded image with the first head posture, and encoding the second face image to obtain a second embedded image with the second head posture; Merging the noisy face image, the first embedded image, and the second embedded image to obtain a merged face image; Acquire target speech data, and encode the target speech data to obtain target speech embedded data; The face merged image is reconstructed based on the target speech embedding data and the target face key point embedding data to obtain a target face image; wherein the target face key point embedding data is obtained based on encoding the face key points of the target object.
2. The method according to claim 1, It is characterized in that The step of performing face reconstruction on the face merged image based on the target speech embedding data and the target face key point embedding data to obtain a target face image comprises: Extracting speech features from the target speech embedding data to obtain a speech feature sequence; Extract key point features from the target face key point embedding data to obtain a key point feature sequence; The face merged image is denoised based on the speech feature sequence and the key point feature sequence to obtain the target face image.
3. The method according to claim 2, It is characterized in that The denoising process is performed on the face merged image based on the speech feature sequence and the key point feature sequence to obtain the target face image, including: Obtaining a pre-trained target diffusion model; wherein the target diffusion model includes an attention network layer with a cross-attention mechanism; Based on the attention network layer, cross attention calculation is performed on the speech feature sequence and the key point feature sequence to obtain image attention weight data; The face merged image is denoised based on the image attention weight data to obtain the target face image.
4. The method according to claim 3, It is characterized in that The target diffusion model further includes a noise adding network layer, and the step of performing noise adding processing on the first face image to obtain a noisy face image includes: Mapping the first face image to the target latent space through a preset image encoder; In the target latent space, the first face image is subjected to noise addition processing based on the noise addition network layer to obtain the noisy face image.
5. The method according to claim 3, It is characterized in that The method further includes training the target diffusion model, specifically including: Acquire a facial image sample of a sample object; wherein the facial image sample includes a first facial sample, a second facial sample, and an occluded facial sample; wherein the first facial sample is an image having a first posture of the sample object, the second facial sample is an image having a second posture of the sample object, the first posture and the second posture are both head postures of the sample object and the first posture and the second posture are different, and the occluded facial sample is an image obtained by occluding the mouth of the first facial sample; Performing noise processing on the first face sample to obtain a noisy face sample; Encoding the blocked face sample to obtain a first embedded sample, and encoding the second face sample to obtain a second embedded sample; Merging the noisy face sample, the first embedded sample, and the second embedded sample to obtain a merged face sample; Acquire a speech sample, and perform encoding processing on the speech sample to obtain a speech embedding sample; Acquire the facial key points of the sample object to obtain facial key point feature samples, and encode the facial key point feature samples to obtain facial key point embedded samples; The speech embedding sample is used as the key of the attention network layer, and the facial key point embedding sample is used as the value of the attention network layer to denoise the face merged sample to obtain a target denoised image; wherein the target denoised image is used as the face image sample.
6. The method according to any one of claims 1 to 5, It is characterized in that The step of merging the noisy face image, the first embedded image, and the second embedded image to obtain a merged face image includes: Merging the first head posture and the second head posture to obtain posture merging information; Adding the posture merging information and the target identity information of the target object to the noisy face image to obtain a noisy merged image; wherein the target identity information is used to indicate the identity of the target object; The noisy merged image, the first embedded image, and the second embedded image are merged to obtain the face merged image.
7. The method according to any one of claims 1 to 5, It is characterized in that Before performing face reconstruction on the face merged image based on the target speech embedding data and the target face key point embedding data to obtain the target face image, the method further includes: Acquire a third facial image of the target object; wherein the first facial image or the second facial image is used as the third facial image; Extracting facial key points from the third facial image to obtain target key point features; Coordinate encoding is performed on the target key point features to obtain the target face key point embedding data.
8. A face generation device, It is characterized in that The device comprises: A face data acquisition module, configured to acquire current face data of a target object; wherein the current face data includes a first face image, a second face image, and an occluded face image, wherein the first face image has a first head posture of the target object, the second face image has a second head posture of the target object, the first head posture and the second head posture are different, and the occluded face image is an image obtained by occluding the mouth of the first face image; An image denoising module, configured to perform denoising processing on the first face image to obtain a denoised face image having the first head posture; An image encoding module, configured to encode the blocked face image to obtain a first embedded image having the first head posture, and encode the second face image to obtain a second embedded image having the second head posture; An image superposition module, used for merging the noisy face image, the first embedded image, and the second embedded image to obtain a face merged image; The voice data processing module is used to: obtain target voice data, and encode the target voice data to obtain target voice embedded data; A face synthesis module is used to reconstruct the face merged image based on the target speech embedding data and the target face key point embedding data to obtain a target face image; wherein the target face key point embedding data is obtained based on encoding the face key points of the target object.
9. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
