Character speaking video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a character speaking video generation method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for the fields of financial science and technology and medical science and technology. The method comprises the following steps: acquiring original voice data and an emotion prompt text, and acquiring an original figure image containing a target figure; performing feature extraction on the original voice data to obtain original audio features; performing feature extraction on the emotion prompt text to obtain emotion guidance embedded features; performing feature extraction on the original character image to obtain character facial features; performing expression generation on the original audio features, the character facial features and the emotion guidance embedded features through a preset target expression generation model to obtain a target expression sequence; and performing video generation processing based on the original voice data and the target expression sequence to obtain a target speaking video containing a speaker which is a digital person of the target person. According to the embodiment of the invention, the quality of the character speaking video can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of financial technology and medical technology. In particular, it relates to a method and apparatus for generating videos of people speaking, an electronic device, and a storage medium. Background Technology

[0002] Voice-acting video generation is a multimodal interactive technology based on artificial intelligence. It combines static images with dynamic audio using deep learning algorithms to drive natural movements of the head and face in the image, achieving the effect of a person speaking. This technology can be applied to various scenarios. For example, in fintech, it can transform complex financial product introductions, operational demonstrations, or personalized financial advice into video format. In healthcare, it can create virtual avatars to provide intelligent triage, health consultations, and psychological counseling.

[0003] Currently, the main method used is audio-driven video generation of spoken characters. However, in practical applications, the inability to accurately capture the emotional information contained in the audio results in emotional deficiencies or inaccurate emotional expression in the generated spoken character videos, thus affecting the overall quality of the videos.

[0004] Therefore, how to improve the quality of videos featuring people speaking has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for generating videos of people speaking, with the aim of improving the quality of such videos.

[0006] To achieve the above objectives, a first aspect of this application proposes a method for generating a video of a person speaking, the method comprising: Acquire raw speech data and emotional prompt text, and acquire raw person images containing the target person; Feature extraction is performed on the raw speech data to obtain the raw audio features; Feature extraction is performed on the emotional prompt text to obtain emotional guidance embedding features; Feature extraction is performed on the original image of the person to obtain the facial features; The target expression sequence is obtained by generating expressions from the original audio features, the facial features of the person, and the emotion-guided embedding features using a preset target expression generation model. Based on the original speech data and the target facial expression sequence, video generation processing is performed to obtain a target speaking video containing the speaker; wherein the speaker is a digital person of the target person.

[0007] In some embodiments, the step of generating a target expression sequence by using a preset target expression generation model to generate expressions from the original audio features, the facial features of the person, and the emotion-guided embedding features includes: The target expression generation model is used to predict expressions from the original audio features to obtain an expression prediction sequence. The target expression generation model adjusts the emotion of the predicted expression sequence based on the emotion-guided embedding features to obtain the original expression sequence. The target expression sequence is obtained by modifying the facial features of the person based on the original expression sequence using the target expression generation model.

[0008] In some embodiments, the target facial expression generation model is trained in the following manner: Obtain a video generation dataset, wherein the video generation dataset includes sample video data, sample audio data, and sample sentiment type annotation data; Feature extraction is performed on the sample audio data to obtain sample audio features; Feature extraction is performed on the labeled data of the sample sentiment types to obtain the sample sentiment embedding features; Feature extraction is performed on the sample video data to obtain sample facial features; The sample audio features, the sample emotion embedding features, and the sample facial features are used to generate facial expressions using a preset original expression generation model to obtain a sample expression prediction sequence. Based on the sample expression prediction sequence, the sample video data, the sample audio features, and the sample facial features, loss calculation is performed to obtain video synthesis loss data; The original expression generation model is optimized based on the video synthesis loss data to obtain the target expression generation model.

[0009] In some embodiments, the step of calculating loss based on the sample expression prediction sequence, the sample video data, the sample audio features, and the sample facial features to obtain video synthesis loss data includes: Loss calculations are performed on the sample expression prediction sequence, the sample video data, and the sample facial features to obtain expression prediction loss data; Loss calculations are performed on the sample facial expression prediction sequence and the sample audio features to obtain audio-video synchronization loss data; The video synthesis loss data is obtained by aggregating and calculating the expression prediction loss data and the audio-video synchronization loss data.

[0010] In some embodiments, the step of calculating the loss of the sample expression prediction sequence, the sample video data, and the sample facial features to obtain expression prediction loss data includes: Facial expression features are extracted from the sample video data to obtain a sequence of facial expressions from the sample video. Facial features are extracted from the sample expression prediction sequence to obtain predicted facial features; The norm is calculated based on the sample facial expression prediction sequence and the sample video facial expression sequence to obtain the facial expression prediction difference norm. The norm of the sample face features and the predicted face features is calculated to obtain the face feature difference norm; The expression prediction loss data is obtained by aggregating the expression prediction difference norm and the facial feature difference norm.

[0011] In some embodiments, the step of calculating the loss on the sample facial expression prediction sequence and the sample audio features to obtain audio-video synchronization loss data includes: Visual features are extracted from the sample facial expression prediction sequence to obtain the predicted visual features; The loss is calculated based on the predicted visual features and the sample audio features to obtain the audio-video synchronization loss data.

[0012] In some embodiments, the step of extracting features from the original speech data to obtain original audio features includes: The original speech data is subjected to noise reduction processing to obtain noise-reduced speech data; The noise-reduced speech data is encoded to obtain speech encoded data; Semantic extraction is performed on the speech encoded data to obtain the original semantic features; The speech coding data is acoustically extracted to obtain the original acoustic features; The original semantic features and the original acoustic features are concatenated to obtain the original audio features.

[0013] To achieve the above objectives, a second aspect of this application provides an apparatus for generating a video of a person speaking, the apparatus comprising: The data acquisition module is used to acquire raw voice data, emotional prompt text, and raw human images containing the target person. The speech feature extraction module is used to extract features from the original speech data to obtain the original audio features; The text feature extraction module is used to extract features from the sentiment prompt text to obtain sentiment guidance embedding features; The image feature extraction module is used to extract features from the original human image to obtain facial features. The expression generation module is used to generate expressions from the original audio features, the facial features of the person, and the emotion-guided embedding features using a preset target expression generation model, thereby obtaining a target expression sequence. The video synthesis module is used to perform video generation processing based on the original voice data and the target facial expression sequence to obtain a target speaking video containing a speaker; wherein the speaker is a digital person of the target person.

[0014] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0016] This application proposes a method, apparatus, electronic device, and storage medium for generating spoken video. It acquires raw speech data and emotional prompt text to obtain an original image of the target person. Feature extraction is performed on the raw speech data to obtain raw audio features, capturing dynamic information such as rhythm and pitch. Feature extraction is performed on the emotional prompt text to obtain emotional guidance embedding features, which guide subsequent expression synthesis. Feature extraction is performed on the original image of the target person to obtain facial features, preserving personalized facial details and enhancing identity consistency. Next, a pre-trained target expression generation model is used to generate expressions based on the raw audio features, facial features, and emotional guidance embedding features, resulting in a target expression sequence that conforms to the speech rhythm and is rich in emotional variation. Finally, video generation processing is performed based on the raw speech data and the target expression sequence to obtain a target speaking video containing the speaker; the speaker is a digital human of the target person, achieving natural coordination of speech, expression, and head movements, making the target speaking video more realistic and expressive, and improving the quality of the generated spoken video. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method for generating a speaking video provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3This is a flowchart of a method for generating a speaking video according to another embodiment of this application; Figure 4 yes Figure 3 The flowchart of step S306 in the process; Figure 5 yes Figure 4 The flowchart of step S401 in the text; Figure 6 yes Figure 4 The flowchart of step S402 in the document; Figure 7 yes Figure 1 The flowchart of step S105 in the process; Figure 8 This is a schematic diagram of the structure of the speech video generation device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0022] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0023] Audio-driven speaking head technology is a multimodal interaction technology based on artificial intelligence. By analyzing the rhythm, pitch, and emotional characteristics of speech signals, it combines static images with dynamic speech to dynamically control the facial expressions, lip movements, and head posture of a static human avatar, generating realistic dynamic videos synchronized with the audio to achieve the effect of a person speaking. Audio-driven speaking head technology is widely used in fields such as virtual anchors, film and television dubbing, and remote conferencing, including digital human animation, visual dubbing, and video content creation.

[0024] Audio-driven speech-generating technology can be used to generate spoken video, applicable to various scenarios. For example, in fintech, it can transform complex financial product introductions, operational demonstrations, or personalized financial advice into video format. Alternatively, it can be used to build intelligent financial robots that achieve highly realistic human-computer interaction, accurately understand customer needs, and provide personalized services. In healthcare, it can create virtual avatars for intelligent triage, health consultations, psychological counseling, medical education, and training.

[0025] Currently, the main method used is audio-driven video generation of spoken characters. However, in practical applications, the inability to accurately capture the emotional information contained in the audio leads to issues such as emotional deficiencies, inaccurate emotional expression, and lip-syncing mismatch in the generated spoken character videos, thus affecting the overall quality of the videos.

[0026] Based on this, embodiments of this application provide a method and apparatus for generating videos of people speaking, an electronic device, and a storage medium, aiming to improve the quality of videos of people speaking.

[0027] The method, apparatus, electronic device, and storage medium for generating videos of people speaking provided in this application are specifically described through the following embodiments. First, the method for generating videos of people speaking in this application is described.

[0028] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0029] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0030] The method for generating a speaking video provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speaking video generation method, but is not limited to the above forms.

[0031] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0032] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0033] Figure 1 This is an optional flowchart of the method for generating a speaking video provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0034] Step S101: Obtain raw voice data, emotional prompt text, and raw image of the target person; Step S102: Extract features from the original speech data to obtain the original audio features; Step S103: Extract features from the sentiment prompt text to obtain sentiment guidance embedding features; Step S104: Extract features from the original image of the person to obtain facial features; Step S105: Generate expression using the original audio features, facial features, and emotion-guided embedding features through a preset target expression generation model to obtain the target expression sequence; Step S106: Based on the original speech data and the target facial expression sequence, video generation processing is performed to obtain the target speaking video containing the speaker; wherein, the speaker is a digital person of the target person.

[0035] Steps S101 to S106, as illustrated in this embodiment, involve acquiring original speech data and emotional prompt text to obtain an original image of the target person. Feature extraction is performed on the original speech data to obtain original audio features, capturing dynamic information such as rhythm and pitch. Feature extraction is performed on the emotional prompt text to obtain emotional guidance embedding features, which guide the subsequent expression synthesis. Feature extraction is performed on the original image of the target person to obtain facial features, preserving the target person's personalized appearance details and enhancing identity consistency. Next, a pre-trained target expression generation model is used to generate expressions based on the original audio features, facial features, and emotional guidance embedding features, resulting in a target expression sequence that conforms to the speech rhythm and is rich in emotional variation. Finally, video generation processing is performed based on the original speech data and the target expression sequence to obtain a target speaking video containing the speaker; wherein the speaker is a digital human of the target person, achieving natural coordination of speech, expression, and head movements, making the target speaking video more realistic and expressive, and improving the quality of the generated character video.

[0036] In step S101 of some embodiments, the original speech data is an audio file recorded by a specific speaker, which includes information such as the speaker's speech content, tone, and rhythm.

[0037] Emotional cue text is text describing the emotions contained in the raw speech data (such as "happy", "angry", etc.) and is used to guide the emotional direction of facial expression generation.

[0038] The original image of the person is an image containing the face or upper body of the target person. It is used to extract the facial features of the target person for expression synthesis, and finally synthesize a speaking video.

[0039] It should be noted that the target person can be a virtual person (such as an anime character) or a real person image that has been authorized for use.

[0040] For example, in fintech scenarios, raw voice data can be audio data for financial product introductions, operation process demonstrations, or personalized financial advice.

[0041] In medical technology scenarios, raw voice data can be used for intelligent triage, health consultation scripts, psychological counseling scripts, and audio data for medical education and training.

[0042] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S205: Step S201: Perform noise reduction processing on the original speech data to obtain noise-reduced speech data; Step S202: Encode the denoised speech data to obtain coded speech data; Step S203: Semantic extraction is performed on the speech coding data to obtain the original semantic features; Step S204: Perform acoustic extraction on the speech coding data to obtain the original acoustic features; Step S205: Perform feature concatenation on the original semantic features and the original acoustic features to obtain the original audio features.

[0043] Steps S201 to S205, as illustrated in this embodiment, involve denoising the original speech data to obtain denoised speech data, improving speech purity and providing a clear data foundation for subsequent processing. Next, the denoised speech data is encoded to obtain coded speech data, reducing data complexity while retaining key information. Further, semantic extraction is performed on the coded speech data to obtain original semantic features, capturing language content and intent; acoustic extraction is also performed on the coded speech data to obtain original acoustic features, extracting physical attributes such as pitch and rhythm, achieving multi-dimensional information extraction. Finally, the original semantic features and original acoustic features are concatenated to obtain decoupled original audio features, which helps enhance feature richness and provides more accurate and comprehensive input for subsequent speech head generation, thereby improving the realism and expressiveness of the spoken video.

[0044] In step S201 of some embodiments, since the original speech data is an unprocessed audio file, it contains interference factors such as environmental noise (e.g., background music, device noise), breathing sounds, etc. Therefore, noise reduction models (e.g., RNNoise, Wav-U-net, etc.) are used to eliminate the noise in the original speech data, preserving clear human voices to obtain denoised speech data. This helps reduce the contamination of semantic and acoustic features by noise and improves the accuracy of feature extraction.

[0045] In step S202 of some embodiments, continuous denoised speech data is converted into discrete feature vectors (such as Mel spectrum, MFCC) by a pre-trained audio coding model to obtain speech coding data. This can compress data dimensionality while retaining key information, reduce data complexity, and improve audio processing efficiency.

[0046] In step S203 of some embodiments, the language content, intent and emotion in the speech coding data are identified by a natural language processing model (such as BERT, Wav2Vec, etc.) to obtain the original semantic features.

[0047] In step S204 of some embodiments, the physical properties (including but not limited to pitch, energy, and speech rate) of the speech coding data are analyzed by signal processing or deep learning models (such as OpenSmile and Tacotron) to obtain the original acoustic features.

[0048] In step S205 of some embodiments, the original semantic features and the original acoustic features are spliced ​​together by dimension to obtain the original audio features, which can integrate the mixed features of language content and physical expression to improve the naturalness of facial expressions and mouth movements when generating the speaking head.

[0049] In step S103 of some embodiments, the emotional prompt text is feature extracted by a pre-trained CLIP model, and the contextual emotional information is fused to obtain emotional guidance embedding features, which are used to make the expression generation conform to the emotional logic (such as frowning when "angry"), avoiding the mechanical expression caused by relying solely on speech.

[0050] In step S104 of some embodiments, a face recognition model (such as a 3DMM model, FLAME model, etc.) is used to extract features from the original person image to obtain the person's facial features. High-dimensional features such as the key points, textures, and contours of the target person's face are preserved, so that the synthesized speaking head is consistent with the facial features of the target person.

[0051] Specifically, facial features include static features such as facial contours, the shape and position of facial features, as well as dynamic features such as facial muscle movements. These features determine the basic shape of a person's face and the way their expressions change.

[0052] Prior to step S105 in some embodiments, the method for generating a speaking video further includes a pre-trained target expression generation model, which is used to generate expressions based on the original audio features, facial features of the person, and emotion-guided embedding features to obtain a target expression sequence.

[0053] Please see Figure 3 In some embodiments, the training process of the target expression generation model may include, but is not limited to, steps S301 to S307: Step S301: Obtain the video generation dataset, which includes sample video data, sample audio data, and sample sentiment type annotation data; Step S302: Extract features from the sample audio data to obtain sample audio features; Step S303: Extract features from the labeled data of sample sentiment types to obtain the embedded features of sample sentiment. Step S304: Extract features from the sample video data to obtain sample face features; Step S305: Generate facial expressions from sample audio features, sample emotion embedding features, and sample facial features using a preset original facial expression generation model to obtain a sample facial expression prediction sequence. Step S306: Based on the sample expression prediction sequence, sample video data, sample audio features and sample face features, loss calculation is performed to obtain video synthesis loss data; Step S307: Optimize the original expression generation model based on the video synthesis loss data to obtain the target expression generation model.

[0054] Steps S301 to S307 of this embodiment provide a rich sample foundation for model training by acquiring a video generation dataset including sample video data, sample audio data, and sample emotion type annotation data. Feature extraction is performed on the sample audio data to obtain sample audio features, which can capture dynamic information such as rhythm and pitch of speech. Feature extraction is performed on the sample emotion type annotation data to obtain sample emotion embedding features, enhancing the accuracy of emotion expression. Feature extraction is performed on the sample video data to obtain sample facial features, preserving facial structure information. An emoticon generation model is used to generate emoticons from the sample audio features, sample emotion embedding features, and sample facial features using a preset original emoticon generation model, resulting in a sample emoticon prediction sequence. Loss calculation is performed based on the sample emoticon prediction sequence, sample video data, sample audio features, and sample facial features to obtain video synthesis loss data. Finally, the original emoticon generation model is optimized based on the video synthesis loss data to obtain a target emoticon generation model with strong generalization ability and high emoticon synthesis accuracy, which helps improve the accuracy of synthesizing spoken video.

[0055] In step S301 of some embodiments, the video generation dataset is a pre-collected and labeled dataset used to train the facial expression generation model.

[0056] The video generation dataset is a triplet containing sample video data, sample audio data, and sample sentiment type annotation data. The sample video data comes from publicly available resources (such as VoxCeleb2 and MEAD) and covers diverse identities, emotions, and perspectives. The sample audio data consists of the spoken audio of the person in the video to be synthesized; this audio can be the spoken audio corresponding to the sample video data. The sample sentiment type annotation data consists of sentiment labels, such as happy, angry, and sad, annotated by humans or artificial intelligence models, used to guide the model in learning the mapping between emotions and expressions.

[0057] In other embodiments, the sample audio data may also be audio of people speaking obtained from other sources.

[0058] In step S302 of some embodiments, sample audio data is denoised, encoded, semantically extracted, and acoustically extracted to obtain sample semantic features and sample acoustic features. These features are then concatenated to obtain sample audio features, which can accurately capture dynamic changes (such as speech rate and pitch) in the sample audio data. This helps to enhance feature richness and provides more accurate and comprehensive input for subsequent speech head generation. Specifically, the specific implementation of step S302 is basically the same as the implementation shown in steps S201 to S205 above, and will not be repeated here.

[0059] In step S303 of some embodiments, the sample emotion type annotation data is used to extract features by using a pre-trained CLIP model, and the contextual emotion information is fused to obtain the sample emotion embedding features, which are used to guide expression synthesis.

[0060] In step S304 of some embodiments, the sample video data is first segmented into segments with a fixed number of frames (e.g., 64 frames / segment), and an enhanced 3D latent keypoint representation is extracted for each frame using a face recognition model (e.g., 3DMM model, FLAME model, etc.) to obtain the 3D latent keypoint representation. Next, neutral facial features that do not contain rich emotions are selected from the 3D latent keypoint representation to obtain the sample facial features.

[0061] Specifically, subtle facial features that do not contain rich emotions can be captured by removing deformable prior loss, using MEAD paired data and influence network dataset to enhance representations from 3D potential keypoint representations.

[0062] In step S304 of some embodiments, the sample audio features, sample emotion embedding features and sample facial features are processed by a preset original expression generation model to generate expressions, generate a sequence of facial action parameters (such as eyebrow height, corner of mouth angle and lip shape), and obtain a sample expression prediction sequence. This achieves the coordinated driving of speech, emotion and facial structure to generate natural and emotionally consistent expressions.

[0063] Please see Figure 4 In some embodiments, step S306 may include, but is not limited to, steps S401 to S403: Step S401: Calculate the loss of the sample expression prediction sequence, sample video data, and sample facial features to obtain expression prediction loss data; Step S402: Calculate the loss of the sample facial expression prediction sequence and sample audio features to obtain audio-video synchronization loss data; Step S403: Aggregate and calculate the expression prediction loss data and audio-visual synchronization loss data to obtain video synthesis loss data.

[0064] Steps S401 to S403, as illustrated in this embodiment, involve calculating the loss of sample expression prediction sequences, sample video data, and sample facial features to obtain expression prediction loss data. This allows for accurate evaluation of the realism and detail reproduction of the generated expressions. Loss calculation is then performed on the sample expression prediction sequences and sample audio features to obtain audio-video synchronization loss data, ensuring strict matching between lip movements and speech rhythm. Finally, the expression prediction loss data and audio-video synchronization loss data are aggregated to obtain video synthesis loss data. This facilitates subsequent model optimization of the original expression generation model based on the video synthesis loss data, improving the model's generalization and synthesis capabilities.

[0065] See Figure 5 In some embodiments, step S401 may include, but is not limited to, steps S501 to S505: Step S501: Extract facial expression features from the sample video data to obtain the facial expression sequence of the sample video; Step S502: Extract facial features from the sample expression prediction sequence to obtain the predicted face features; Step S503: Calculate the norm based on the sample expression prediction sequence and the sample video expression sequence to obtain the expression prediction difference norm; Step S504: Calculate the norm of the sample face features and the predicted face features to obtain the face feature difference norm; Step S505: Aggregate and calculate based on the expression prediction difference norm and the facial feature difference norm to obtain expression prediction loss data.

[0066] Steps S501 to S505, as illustrated in this embodiment, extract facial features from sample video data to obtain sample video facial expression sequences, accurately capturing the dynamic changes of real facial expressions in the video. Facial feature extraction is performed on the sample facial expression prediction sequences, transforming them into quantifiable feature data to obtain predicted facial features. Norm calculation is performed on the sample facial expression prediction sequences and sample video facial expression sequences to obtain the facial prediction difference norm, which can intuitively measure the degree of deviation between the sample facial expression prediction sequences and sample video facial expression sequences. Norm calculation is performed on the sample facial features and predicted facial features to obtain the facial feature difference norm, which can evaluate the prediction effect from the perspective of facial features. Finally, the facial prediction difference norm and the facial feature difference norm are aggregated to obtain facial prediction loss data, integrating the differences between the facial expression sequences and facial features. This allows for a more comprehensive and accurate evaluation of the performance of the facial prediction model, thereby guiding model optimization and improving prediction accuracy.

[0067] In step S501 of some embodiments, facial geometric features such as eyebrow height, mouth corner curvature, and eye opening and closing are extracted from each frame of the sample video data by a pre-trained deep learning model (such as OpenFace or 3DMM) and converted into a computable numerical vector, thereby converting the sample video data into a sample video expression sequence.

[0068] In step S502 of some embodiments, an enhanced 3D latent keypoint representation is extracted for each frame in the sample expression prediction sequence using a face recognition model (such as the 3DMM model, FLAME model, etc.) to obtain the predicted face features.

[0069] Specifically, the calculation process for the facial expression prediction loss data is shown in formula (1): (1); in, For facial expression prediction loss data, This represents the range of values ​​for a video frame. Predict the expression sequence for the sample. The sample video contains facial expression sequences. To predict facial features, The facial features of the sample are shown.

[0070] It should be noted that, This represents the Euclidean norm of x, i.e., the 2-norm. It represents the square of the 2-norm. For the difference norm of facial expression prediction, It is the facial feature difference norm.

[0071] Please see Figure 6 In some embodiments, step S402 includes, but is not limited to, steps S601 to S602: Step S601: Extract visual features from the sample expression prediction sequence to obtain the predicted visual features; Step S602: Based on the predicted visual features and sample audio features, the loss is calculated to obtain audio-visual synchronization loss data.

[0072] Steps S601 to S602, as illustrated in this embodiment, involve extracting visual features from the sample facial expression prediction sequence, transforming the sequence into feature data that reflects visual performance, and obtaining predicted visual features. Loss calculation is performed based on the predicted visual features and sample audio features to obtain audio-visual synchronization loss data. This data accurately measures the difference in synchronization between audio and video, helping to improve the audio-visual synchronization quality of the model-generated content.

[0073] In step S601 of some embodiments, high-dimensional features, including spatial structure (such as facial contours and features) and temporal dynamics (such as the rate of change of expression), are extracted from the sample expression prediction sequence using a deep learning model (such as CNN or Transformer) to quantify the visual performance of the expression and obtain predicted visual features.

[0074] Specifically, the calculation process for audio and video synchronization loss data is shown in formula (2): (2); in, To predict visual features, These are the audio features of the sample.

[0075] In step S403 of some embodiments, a weighted summation method can be used to calculate the weighted summation of the expression prediction loss data and the audio-visual synchronization loss data to obtain the video synthesis loss data, thereby jointly optimizing the expression realism and synchronization and avoiding local optima caused by a single loss.

[0076] In step S307 of some embodiments, methods such as backpropagation, gradient descent, momentum update, and LoRA can be used to adjust the parameters of the original expression generation model based on video synthesis loss data, thereby optimizing the model's learning ability and generalization ability to obtain the target expression generation model.

[0077] Please see Figure 7 In some embodiments, step S105 may include, but is not limited to, steps S701 to S703: Step S701: Use the target expression generation model to predict the expression from the original audio features to obtain the expression prediction sequence; Step S702: The expression prediction sequence is adjusted based on the emotion-guided embedding features by the target expression generation model to obtain the original expression sequence; Step S703: Based on the original expression sequence, the facial features of the person are corrected using the target expression generation model to obtain the target expression sequence.

[0078] Steps S701 to S703 as shown in the embodiments of this application involve predicting facial expressions from the original audio features using a target facial expression generation model to obtain a predicted facial expression sequence. Then, the predicted facial expression sequence is adjusted based on emotion-guided embedding features to obtain an original facial expression sequence. Finally, the facial features of the person are corrected based on the original facial expression sequence to obtain a target facial expression sequence. This ultimately achieves accurate and emotionally appropriate facial expression generation that matches the person's facial features.

[0079] In step S701 of some embodiments, the target expression generation model performs expression prediction by combining the original semantic features and original acoustic features in the original audio features, as well as the emotion-guided embedding features, to obtain an expression prediction sequence. This ensures that the generated expression prediction sequence is synchronized in terms of lip movement and temporal smoothing. Furthermore, the target expression generation model iteratively refines the expression sequence through a denoising diffusion process.

[0080] In step S702 of an earlier embodiment, the target expression generation model predicts relevant emotional deformations based on emotion-guided embedding features, thereby adjusting the expression prediction sequence to obtain an expression deformation sequence, which can ensure the linear additive nature of emotional expression.

[0081] Next, the facial expression deformation sequence and the facial expression prediction sequence are concatenated to obtain the original facial expression sequence.

[0082] It should be noted that the original facial expression sequence is an emotion-adjusted facial expression sequence, which can more accurately reflect facial expression changes under specific emotions compared to the facial expression prediction sequence.

[0083] In step S703 of some embodiments, the target expression generation model precisely adjusts and optimizes the facial features of the character based on the original expression sequence, so that the character's face presents an expression state consistent with the original expression sequence. The resulting target expression sequence can accurately reflect the sequence of specific expressions and emotions, and can be directly applied to scenarios such as virtual character display and film and television special effects.

[0084] In step S106 of some embodiments, the temporal features of the original speech data are time-aligned with the frame-level data of the target facial expression sequence to ensure accurate matching between lip movements and speech pronunciation. Next, facial images are rendered frame-by-frame using a generative model (such as a GAN or a diffusion model). Finally, the rendered single-frame images are stitched together in chronological order to generate a continuous video stream, resulting in the target speaking video containing the speaker.

[0085] It should be noted that the speaker is a digital person representing the target.

[0086] The method for generating human facial expressions in this application, compared to traditional emotionless methods, can generate speaking head videos containing realistic emotional expressions. The target facial expression sequence incorporates emotion-guided embedding features, enabling the digital human to not only synchronize speech but also express corresponding emotions (such as frowning when angry or smiling when happy). For example, when inputting the emotional text "I am happy," the model will generate dynamic expressions such as upturned corners of the mouth and squinting eyes, rather than mechanical repetition, which helps improve the naturalness of the expressions and significantly enhances the realism of the generated video and the naturalness of human communication. It is suitable for scenarios such as digital human animation and visual dubbing. In addition, by aligning the precise timing of pronunciation (such as the start time of vowels / consonants) provided by the original speech data with the facial expression sequence, the "lip-syncing" problem caused by time misalignment in traditional methods can be avoided.

[0087] In addition, the facial expression generation method for speaking provided in this application supports zero-sample facial expression editing: by using text-driven editing, the dependence on emotion training videos is eliminated, ensuring that the generated results outperform existing technologies in diversity metrics (such as PSNR, SSIM, FID) and show good generalization ability.

[0088] Furthermore, it can also support the generation of videos of any length, breaking through the length limitations of traditional methods, making it suitable for long interactive scenarios and increasing scalability.

[0089] Please see Figure 8 This application also provides a device for generating a video of a person speaking, which can implement the above-described method for generating a video of a person speaking. The device includes: The data acquisition module 801 is used to acquire raw voice data, emotional prompt text, and raw image of the target person. The speech feature extraction module 802 is used to extract features from the original speech data to obtain the original audio features; The text feature extraction module 803 is used to extract features from the sentiment prompt text to obtain sentiment guidance embedding features; The image feature extraction module 804 is used to extract features from the original human image to obtain the facial features of the human. The expression generation module 805 is used to generate expressions from the original audio features, facial features, and emotion-guided embedding features using a preset target expression generation model, thereby obtaining a target expression sequence. The video synthesis module 806 is used to perform video generation processing based on the original speech data and the target facial expression sequence to obtain a target speaking video containing the speaker; wherein the speaker is a digital human of the target person.

[0090] The specific implementation of the character speaking video generation device is basically the same as the specific implementation of the character speaking video generation method described above, and will not be repeated here.

[0091] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for generating a video of a person speaking. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0092] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the method for generating a talking video according to the embodiments of this application. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0093] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating a video of a person speaking.

[0094] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] The method, apparatus, electronic device, and storage medium for generating spoken video provided in this application acquire raw speech data and emotional prompt text to obtain an original image of a target person. Feature extraction is performed on the raw speech data to obtain raw audio features, capturing dynamic information such as rhythm and pitch. Feature extraction is performed on the emotional prompt text to obtain emotional guidance embedding features, which guide the subsequent expression synthesis. Feature extraction is performed on the original image of the target person to obtain facial features, preserving the target person's personalized appearance details and enhancing identity consistency. Next, a pre-trained target expression generation model is used to generate expressions based on the raw audio features, facial features, and emotional guidance embedding features, resulting in a target expression sequence that conforms to the speech rhythm and is rich in emotional variation. Finally, video generation processing is performed based on the raw speech data and the target expression sequence to obtain a target speaking video containing the speaker; wherein the speaker is a digital human of the target person, achieving natural coordination of speech, expression, and head movements, making the target speaking video more realistic and expressive, and improving the quality of the generated spoken video.

[0096] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0097] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0098] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0100] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0101] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0103] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The software tools or components not belonging to our company that appear in the embodiments of this application are for illustrative purposes only and do not represent actual use.

[0107] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating a video of a person speaking, characterized in that, The method includes: Acquire raw speech data and emotional prompt text, and acquire raw person images containing the target person; Feature extraction is performed on the raw speech data to obtain the raw audio features; Feature extraction is performed on the emotional prompt text to obtain emotional guidance embedding features; Feature extraction is performed on the original image of the person to obtain the facial features; The target expression sequence is obtained by generating expressions from the original audio features, the facial features of the person, and the emotion-guided embedding features using a preset target expression generation model. Based on the original speech data and the target facial expression sequence, video generation processing is performed to obtain a target speaking video containing the speaker; wherein the speaker is a digital person of the target person.

2. The method according to claim 1, characterized in that, The step involves generating a target expression sequence by using a preset target expression generation model to analyze the original audio features, the facial features of the person, and the emotion-guided embedding features, resulting in the following: The target expression generation model is used to predict expressions from the original audio features to obtain an expression prediction sequence. The target expression generation model adjusts the emotion of the predicted expression sequence based on the emotion-guided embedding features to obtain the original expression sequence. The target expression sequence is obtained by modifying the facial features of the person based on the original expression sequence using the target expression generation model.

3. The method according to claim 1, characterized in that, The target facial expression generation model is trained in the following way: Obtain a video generation dataset, wherein the video generation dataset includes sample video data, sample audio data, and sample sentiment type annotation data; Feature extraction is performed on the sample audio data to obtain sample audio features; Feature extraction is performed on the labeled data of the sample sentiment types to obtain the sample sentiment embedding features; Feature extraction is performed on the sample video data to obtain sample facial features; The sample audio features, the sample emotion embedding features, and the sample facial features are used to generate facial expressions using a preset original expression generation model to obtain a sample expression prediction sequence. Based on the sample expression prediction sequence, the sample video data, the sample audio features, and the sample facial features, loss calculation is performed to obtain video synthesis loss data; The original expression generation model is optimized based on the video synthesis loss data to obtain the target expression generation model.

4. The method according to claim 3, characterized in that, The loss calculation based on the sample expression prediction sequence, the sample video data, the sample audio features, and the sample facial features yields video synthesis loss data, including: Loss calculations are performed on the sample expression prediction sequence, the sample video data, and the sample facial features to obtain expression prediction loss data; Loss calculations are performed on the sample facial expression prediction sequence and the sample audio features to obtain audio-video synchronization loss data; The video synthesis loss data is obtained by aggregating and calculating the expression prediction loss data and the audio-video synchronization loss data.

5. The method according to claim 4, characterized in that, The step of calculating the loss of the sample expression prediction sequence, the sample video data, and the sample facial features to obtain expression prediction loss data includes: Facial expression features are extracted from the sample video data to obtain a sequence of facial expressions from the sample video. Facial features are extracted from the sample expression prediction sequence to obtain predicted facial features; The norm is calculated based on the sample facial expression prediction sequence and the sample video facial expression sequence to obtain the facial expression prediction difference norm. The norm of the sample face features and the predicted face features is calculated to obtain the face feature difference norm; The expression prediction loss data is obtained by aggregating the expression prediction difference norm and the facial feature difference norm.

6. The method according to claim 4, characterized in that, The loss calculation on the sample facial expression prediction sequence and the sample audio features to obtain audio-video synchronization loss data includes: Visual features are extracted from the sample facial expression prediction sequence to obtain the predicted visual features; The loss is calculated based on the predicted visual features and the sample audio features to obtain the audio-video synchronization loss data.

7. The method according to any one of claims 1 to 6, characterized in that, The step of extracting features from the original speech data to obtain original audio features includes: The original speech data is subjected to noise reduction processing to obtain noise-reduced speech data; The noise-reduced speech data is encoded to obtain speech encoded data; Semantic extraction is performed on the speech encoded data to obtain the original semantic features; The speech coding data is acoustically extracted to obtain the original acoustic features; The original semantic features and the original acoustic features are concatenated to obtain the original audio features.

8. A device for generating a video of a person speaking, characterized in that, The device includes: The data acquisition module is used to acquire raw voice data, emotional prompt text, and raw human images containing the target person. The speech feature extraction module is used to extract features from the original speech data to obtain the original audio features; The text feature extraction module is used to extract features from the sentiment prompt text to obtain sentiment guidance embedding features; The image feature extraction module is used to extract features from the original human image to obtain facial features. The expression generation module is used to generate expressions from the original audio features, the facial features of the person, and the emotion-guided embedding features using a preset target expression generation model, thereby obtaining a target expression sequence. The video synthesis module is used to perform video generation processing based on the original voice data and the target facial expression sequence to obtain a target speaking video containing a speaker; wherein the speaker is a digital person of the target person.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images