Video data generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video data generation method and device, electronic equipment and a storage medium, and the method comprises the steps: firstly, obtaining audio data, and carrying out the feature extraction of the audio data, and obtaining the features of the audio data; then, performing data processing based on the audio data by using a preset cue word and a large language model to obtain an emotion category intensity feature; secondly, acquiring reference video data, and performing feature extraction on the reference video data to obtain video data features; then, performing mask processing on the digital face image in the reference video data to generate a face image mask variable; and finally, performing video data generation based on the audio data features, the video data features, the emotion category intensity features and the face image mask variables to obtain target video data, thereby remarkably improving the simulation degree and expressive force of the digital human in the finally generated video.

Description

Technical Field

[0001] This application relates to the field of visual data processing technology, and in particular to video data generation methods, apparatus, electronic devices and storage media. Background Technology

[0002] In the field of AI-generated content (AIGC), 2D virtual digital humans, as an image generation technology that uses audio to drive mouth movements, have been widely applied in scenarios such as virtual companionship, live-streaming e-commerce, and enterprise customer service. In related video image generation technologies, inference performance is typically improved through processing in latent space. The core process involves extracting audio features and image features from a reference video separately, then fusing them into a network structure such as UNet3D. Finally, a decoder synthesizes mouth movements synchronized with the audio, achieving audio-visual synchronized video generation.

[0003] However, existing technologies mainly focus on the precise synchronization of audio and lip movements. Their generation process is limited to synthesizing mouth movements, while ignoring the coordinated changes of other facial areas such as eyebrows and eyes. This results in the generated digital human having relatively simple and rigid expressions, unable to express corresponding dialogue content with appropriate emotions, thus leading to poor realism and expressiveness of the digital human in the generated video data. Summary of the Invention

[0004] This application provides a video data generation method, apparatus, electronic device, and storage medium, which can improve the realism and expressiveness of digital humans in the generated video data.

[0005] To achieve the above objectives, a first aspect of this application proposes a video data generation method, the method comprising: Acquire audio data and extract features from the audio data to obtain audio data features; Using preset prompt words and a large language model, data processing is performed on the audio data to obtain emotion category intensity features; Acquire reference video data and extract features from the reference video data to obtain video data features; The digital face image in the reference video data is masked to generate a face image mask variable; Video data is generated based on the audio data features, the video data features, the emotion category intensity features, and the face image mask variables to obtain the target video data.

[0006] In some embodiments, the step of processing the audio data using preset cue words and a large language model to obtain emotion category intensity features includes: The audio data is subjected to speech recognition to obtain audio text information; Based on the preset prompt words, the audio text information is input into the large language model for data processing to obtain the emotion category intensity features.

[0007] In some embodiments, the step of inputting the audio text information into the large language model for data processing based on preset prompt words to obtain the emotion category intensity features includes: Based on preset prompt words, the audio text information is input into the large language model for data processing. From multiple preset emotion categories, a target emotion category is selected and the emotion category code corresponding to the target emotion category is output. From a preset continuous emotion intensity range, a target continuous emotion intensity is selected and the continuous intensity code corresponding to the target continuous emotion intensity is output. The emotion category intensity feature is obtained by weighting and superimposing the emotion category code and the continuous intensity code, and then adding a bias factor.

[0008] In some embodiments, the step of extracting features from the reference video data to obtain video data features includes: Extract image frame data from multiple consecutive moments from the reference video data; Alignment transformation is performed on each image frame data based on the alignment transformation matrix to obtain an aligned face image; The aligned face image is input into an image encoder for encoding to obtain the video data features.

[0009] In some embodiments, the step of performing masking processing on the digital face image in the reference video data to generate face image mask variables includes: Select sequence elements of a first and a second numerical range from the preset sequence of facial key point coordinates as the upper part of the coordinate sequence, and select sequence elements of a third numerical range from the sequence of facial key point coordinates as the lower part of the coordinate sequence. The first element of the third numerical range is the last element of the first numerical range, and the last element of the third numerical range is the first element of the second numerical range. The upper part of the face mask is obtained by processing the bounding box based on the upper part of the coordinate sequence and the preset scaling factor. Based on the lower part coordinate sequence and the preset scaling factor, convex hull processing is performed to obtain the lower part face mask; The face image mask variable is obtained by vertically stitching the upper part of the face mask and the lower part of the face mask together.

[0010] In some embodiments, the step of generating target video data based on the audio data features, the video data features, the emotion category intensity features, and the face image mask variables includes: Based on the audio data features, video data features, emotion category intensity features, and face image mask variables, data processing is performed using an input diffusion model to obtain image latent variables; The latent variables of the image are input into the image decoder for decoding processing to obtain the decoded data features; Based on the alignment transformation matrix, the features of the decoded data are subjected to inverse alignment transformation to obtain post-processed image data; The target video data is obtained based on all the post-processed image data.

[0011] In some embodiments, the data processing based on the audio data features, the video data features, the emotion category intensity features, and the face image mask variable input diffusion model to obtain image latent variables includes: In the diffusion model, the audio data features and the emotion category intensity features are concatenated to obtain audio emotion features, and the video data features, the face image mask variables, and random noise are concatenated to obtain video mask features. The audio emotion features and the video mask features are concatenated using tensors to obtain an initial tensor. Based on the initial tensor and time step, noise prediction is performed to obtain the predicted noise; Starting from the last time step, each time step is used as a denoising time step in sequence; In the denoising time step, the initial tensor corresponding to the denoising time step is iteratively denoised based on the initial tensor corresponding to the next time step, the denoising time step, the audio emotion features, and the predicted noise to obtain a denoised tensor, and the denoised tensor is used as the new initial tensor corresponding to the denoising time step. After the initial tensors corresponding to all the time steps are denoised, the image latent variables are obtained based on the initial tensor corresponding to the first time step.

[0012] To achieve the above objectives, a second aspect of this application provides a video data generation apparatus, the apparatus comprising: An audio data acquisition module is used to acquire audio data and extract features from the audio data to obtain audio data features; The emotion feature acquisition module is used to process the audio data using preset prompt words and a large language model to obtain emotion category intensity features. The video feature acquisition module is used to acquire reference video data and extract features from the reference video data to obtain video data features; An image mask generation module is used to perform masking processing on the digital face image in the reference video data to generate face image mask variables; The video data generation module is used to generate target video data based on the audio data features, the video data features, the emotion category intensity features, and the face image mask variables.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the video data generation method as described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the video data generation method as described in the first aspect.

[0015] The video data generation method, apparatus, electronic device, and storage medium proposed in this application include: First, acquiring audio data and extracting features from the audio data to obtain audio data features; then, using preset prompt words and a large language model, processing the audio data to obtain emotion category intensity features; second, acquiring reference video data and extracting features from the reference video data to obtain video data features; then, performing masking processing on the digital face image in the reference video data to generate face image mask variables; finally, generating video data based on the audio data features, video data features, emotion category intensity features, and face image mask variables to obtain target video data. This application's embodiments, by introducing emotion category intensity features generated by a large language model based on audio content, overcome the limitations of existing technologies that only focus on lip-sync, enabling digital human facial expressions to dynamically match the inherent emotions of the dialogue, achieving audio-visual synchronization at the emotional level. Secondly, by designing a facial image mask variable that can cover key emotional expression areas such as eyes and eyebrows, it ensures that emotional expression is global and collaborative, rather than limited to a single change in the mouth area. This allows for the generation of digital human videos with richer and more natural expressions, effectively solving the problem of monotonous and rigid digital human expressions in existing technologies, and thus significantly improving the realism and expressiveness of the digital human in the final generated video.

[0016] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0017] Figure 1 This is a flowchart of a video data generation method provided in an embodiment of this application.

[0018] Figure 2 yes Figure 1 The flowchart for step 102.

[0019] Figure 3 yes Figure 2 The flowchart for step 202.

[0020] Figure 4 yes Figure 1 The flowchart for step 103.

[0021] Figure 5 yes Figure 1 The flowchart for step 104.

[0022] Figure 6 This is a schematic diagram of a face mask provided in another embodiment of this application.

[0023] Figure 7 This is another embodiment of the present application that provides a facial landmark map.

[0024] Figure 8 This is a diagram showing the effect of face mask overlay provided in another embodiment of this application.

[0025] Figure 9 yes Figure 1 The flowchart for step 105.

[0026] Figure 10 yes Figure 9 The flowchart for step 901.

[0027] Figure 11 This is a schematic diagram of a data flow for video data generation provided in another embodiment of this application.

[0028] Figure 12 This is a schematic diagram illustrating a concept for video data generation, provided in another embodiment of this application.

[0029] Figure 13 This is a schematic diagram of the structure of a video data generation device provided in an embodiment of this application.

[0030] Figure 14This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0034] In the field of AI-generated content (AIGC), 2D virtual digital humans, as an image generation technology that uses audio to drive mouth movements, have been widely applied in scenarios such as virtual companionship, live-streaming e-commerce, and enterprise customer service. In related video image generation technologies, inference performance is typically improved through processing in latent space. The core process involves extracting audio features and image features from a reference video separately, then fusing them into a network structure such as UNet3D. Finally, a decoder synthesizes mouth movements synchronized with the audio, achieving audio-visual synchronized video generation.

[0035] However, existing technologies mainly focus on the precise synchronization of audio and lip movements. Their generation process is limited to synthesizing mouth movements, while ignoring the coordinated changes of other facial areas such as eyebrows and eyes. This results in the generated digital human having relatively simple and rigid expressions, unable to express corresponding dialogue content with appropriate emotions, thus leading to poor realism and expressiveness of the digital human in the generated video data.

[0036] To improve the realism and expressiveness of digital humans in generated video data, this application introduces emotion category intensity features generated by a large language model based on audio content. This overcomes the limitation of existing technologies that only focus on lip-sync, enabling the digital human's facial expressions to dynamically match the inherent emotions of the dialogue, achieving audio-visual synchronization at the emotional level. Secondly, by designing a facial image mask variable that can cover key emotional expression areas such as eyes and eyebrows, it ensures that the expression of emotions is global and coordinated, rather than limited to a single change in the mouth area. This allows for the generation of digital human videos with richer and more natural expressions, effectively solving the problem of monotonous and rigid digital human expressions in existing technologies, and thus significantly improving the realism and expressiveness of the digital human in the final generated video.

[0037] The video data generation method, apparatus, electronic device, and storage medium in the embodiments of this application will be described in detail below. First, the video data generation method in the embodiments of this application will be described. (Refer to...) Figure 1 This is an optional flowchart of the video data generation method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 101 to 105. It is also understood that this embodiment... Figure 1 The order of steps 101 to 105 is not specifically limited; the order of steps can be adjusted or certain steps can be added or removed according to actual needs. The video data generation method provided in this application can be applied to any control system with computing resources (such as servers, computers, etc.).

[0038] Step 101: Obtain audio data and extract features from the audio data to obtain audio data features.

[0039] Step 101 will be described in detail below.

[0040] In some embodiments, upon responding to a video data generation request, audio data as the input source is first required. This audio data can be a real-time speech stream or a pre-recorded audio file. To enable the model to understand and process these sound signals, the audio data needs to be processed... Feature extraction is then performed, typically using an audio feature extractor, such as the Whisper model mentioned in the technical solution, to transform the raw audio waveform data (Audio Chunk) into a high-dimensional mathematical representation, i.e., audio data features. This audio data feature is a feature vector that contains information such as speech content, timbre, and rhythm. It is the foundation for subsequent digital human lip-reading and facial expression generation.

[0041] It is understood that the audio data in this embodiment of the application is 10 consecutive audio frames extracted from streaming audio. (When the digital human synthesizes video at fps=25, each image frame corresponds to 2 audio frames.) Whisper is used as the audio feature extractor, denoted as... Audio feature values ​​are extracted based on the whisper module, denoted as... The audio data characteristics are obtained by formula (1) as follows.

[0042] (1) Where M is the length of a single audio frame; For audio feature dimensions.

[0043] Step 102: Using preset prompts and a large language model, perform data processing based on audio data to obtain emotion category intensity features.

[0044] Step 102 is described in detail below.

[0045] In some embodiments, to enable digital humans to express emotions, this application introduces a large language model for processing. This step utilizes preset prompts to guide the large language model based on the audio data acquired in the previous step. (Typically, it involves performing deep semantic understanding and sentiment analysis on the text information after ASR speech recognition.) The large language model outputs a structured sentiment category intensity feature based on the text content, from preset sentiment categories (such as happy, sad, etc.) and continuous sentiment intensity ranges (such as floating-point numbers from 0 to 1). This feature, through a specific encoding method, fuses discrete emotion categories and continuous emotion intensities into a unified vector, providing crucial conditional input for the subsequent refined emotion control of the model. This will be described in further detail below.

[0046] Reference Figure 2 Using preset prompts and a large language model, data processing is performed on audio data to obtain emotion category intensity features, including the following steps 201 to 202.

[0047] Step 201: Perform speech recognition on the audio data to obtain audio text information.

[0048] Step 202: Based on preset prompts, input the audio text information into the large language model for data processing to obtain the emotion category intensity features.

[0049] Steps 201 to 202 are described in detail below.

[0050] In some embodiments, in order for a large language model to understand speech content, it is first necessary to process the audio data. Speech recognition is performed. This process is typically executed by an Automatic Speech Recognition (ASR) module. This module's role is to process the input raw audio signal... Whether it's real-time audio streaming or offline audio files, the audio is converted into a text format that computers can process, thus obtaining audio-text information. This step is a prerequisite for subsequent natural language understanding and sentiment analysis. It transforms human speech into machine-readable text sequences, laying the foundation for analyzing emotions at the content level.

[0051] After obtaining the audio text information, it is input into the Large Language Model (LLM) for data processing based on preset prompts. These preset prompts are pre-designed instructions that guide the LLM to perform specific tasks, namely, analyzing and embedding sentiment tags while generating response text. LLM, with its powerful semantic understanding capabilities, parses the inherent sentiment of the text and outputs structured data in a preset format, containing sentiment categories (such as "happy" or "sad") and corresponding sentiment intensities (such as continuous values ​​from 0 to 1), ultimately yielding sentiment category intensity features. This feature is a quantized vector that includes emotion type and emotion intensity, providing a direct driving signal for subsequent fine-grained control of digital human expressions, as described below.

[0052] Reference Figure 3 Based on preset prompts, the audio text information is input into a large language model for data processing to obtain emotion category intensity features, including the following steps 301 to 302.

[0053] Step 301: Based on the preset prompt words, input the audio text information into the large language model for data processing, select the target emotion category from multiple preset emotion categories, and output the emotion category code corresponding to the target emotion category, and select the target continuous emotion intensity from the preset continuous emotion intensity range, and output the continuous intensity code corresponding to the target continuous emotion intensity.

[0054] Step 302: Perform weighted superposition processing based on emotion category encoding and continuous intensity encoding, and add a bias factor to obtain emotion category intensity features.

[0055] Steps 301 to 302 are described in detail below.

[0056] In some embodiments, to parse and quantify sentiment information from text, audio text information is input into a large language model for data processing based on preset prompts. Guided by the preset prompts, the large language model performs semantic and sentiment analysis on the input text and executes two parallel tasks: First, it selects the target sentiment category that best matches the text content from a set of preset sentiment categories (e.g., a set of eight categories including "angry," "happy," "sad," etc.) and outputs the vectorized representation of that category, i.e., the sentiment category code corresponding to the target sentiment category. Second, the model evaluates the intensity of the emotion and selects a specific target continuous sentiment intensity value from a preset continuous sentiment intensity range (e.g., a normalized range from 0 to 1, representing from no intensity to maximum intensity), and converts it into a vector representation using techniques such as timestep embedding, i.e., outputting the continuous intensity code corresponding to the target continuous sentiment intensity.

[0057] Understandably, this application models emotional semantics. Emotional tags are divided into eight categories: anger, disgust, fear, happiness, neutrality, sadness, surprise, and contempt. Each category has three levels, representing the intensity of different emotions. A large-scale model prompt is designed, outputting the response text along with the emotion C and intensity S. The set of emotion categories is defined as shown in formula (2).

[0058] (2) The embedding layer of the emotion category is shown in formula (3) below.

[0059] (3) Therefore, the emotion category code corresponding to each emotion category is shown in the following formula (4).

[0060] (4) in To use the nn.Enbedding operation to vectorize the categories; d is the dimension of the category vector, and we take d=1280; For the emotion embedding module, the input emotion label is i; This represents the dimension of each emotion embedding.

[0061] In addition, the corresponding emotional intensity level This corresponds to three different levels: mild, moderate, and intense. To ensure continuity in emotional intensity, the three different levels are normalized to a continuous [0,1] interval (i.e., a continuous emotional intensity interval). 0 represents no intensity, and 1 represents maximum intensity. Continuous intensity encoding is performed using the timestep embedding method as shown in formula (5).

[0062] (5) In one example, the preset prompt could be: {You are an AI assistant to a virtual digital human. Your task is to analyze and rewrite the text provided by the user, embedding structured sentiment tags within it.}

[0063] Instructions: Emotion Classification: You must choose from the following 8 preset emotion categories: anger, disgust, fear, happiness, neutrality, sadness, surprise, and contempt.

[0064] Emotion Intensity: Assign a continuous emotion intensity value to each emotion tag you select, within a closed interval of [0.0, 1.0]. 0.0 represents no intensity, and 1.0 represents the strongest intensity.

[0065] Output format: Must follow the format <emotion label:intensity value>, and place this label before the sentence or clause expressing the corresponding emotion. For statements with a neutral emotion, use <neutral:0.0>.

[0066] Processing text: Now, please process the following user text: [Insert user's audio text information here]"}.

[0067] In another example, the preset prompt could be concisely written as: {Task: Add an emotion label and intensity value before each clause of the following text. The emotion label must be selected from [angry, disgusted, afraid, happy, neutral, sad, surprised, contemptuous]. The intensity value is a floating-point number between 0.0 and 1.0. The format is <label:intensity>. Text: [Insert user's audio text information here]}.

[0068] To fuse the discrete emotion category information and continuous intensity information obtained in the previous step into a unified feature usable by downstream models, a weighted superposition process is performed based on the emotion category encoding and continuous intensity encoding, plus a bias factor, to obtain the emotion category intensity feature. This weighted superposition process involves multiplying the emotion category encoding and continuous intensity encoding by their respective weight matrices (including the weight matrix for emotion classification). Weight matrix of emotion intensity The results are then summed, a process that allows the model to learn the importance of each emotion category and intensity during training. Finally, a bias factor is added. This is a learnable parameter used to fine-tune the final output. The entire process is accomplished through a linear layer containing an activation function (such as ReLU), ultimately generating a single feature vector that includes both sentiment type and sentiment intensity, i.e., the sentiment category intensity feature. As shown in formula (6) below.

[0069] (6) in, The intensity is vectorized using the TimestepEmbedding operation; ReLU is a neural network operator.

[0070] In one example, suppose the audio text after speech recognition is: "Great! We finally won the game!". This text is input into a large language model guided by preset prompts. The large language model analyzes the text, first selecting "happy" as the target emotion category that best matches the sentiment of the text from a set of preset emotion categories (such as happy, sad, surprised, etc.), and outputting a predefined vectorized representation of this category, i.e., the emotion category code corresponding to the target emotion category. Next, the model determines that this is a strong emotional expression based on words such as "great!" and "finally" in the text. Therefore, it selects a high target continuous emotion intensity value, such as 0.9, from a preset continuous emotion intensity range (e.g., [0.0, 1.0]), and converts it into a vector through time-step embedding, i.e., outputting the continuous intensity code corresponding to the target continuous emotion intensity.

[0071] The system acquires the emotion category code representing "happiness" and the continuous intensity code representing an intensity of 0.9. Then, it performs a weighted summation process based on these two codes, multiplying them separately by their respective weight matrices and then summing the results. This process allows the model to learn the relative importance of different emotions and intensities. A learnable bias factor is then added for fine-tuning. Finally, the entire calculation result (e.g., through a ReLU activation function) generates a unified, high-dimensional vector. This vector represents the final emotion category intensity feature, containing both the "happiness" emotion type and the high intensity information of "0.9," which can be directly used to guide subsequent video generation models.

[0072] Through steps 301 to 302 above, a technical solution for refined and structured modeling of emotions is realized. It can not only identify the basic emotion categories contained in the text, but also quantitatively evaluate the continuous intensity of the emotion. This avoids the limitation of traditional methods that can only be discretely and roughly divided into emotion states. By weightedly fusing emotion category encoding and continuous intensity encoding, the final generated emotion category intensity feature can provide a rich, accurate and highly controllable emotion guidance signal for the subsequent video generation model in a unified mathematical form. This plays a crucial role in generating digital human videos with subtle facial expressions and realistic emotional expression.

[0073] Through steps 201 to 202 above, deep emotional information can be effectively extracted from the original audio data. First, speech recognition is used to convert the sound into text, solving the problem of machine understanding of speech. Then, using a large language model and customized prompts, the specific emotion category and its intensity are accurately parsed from the text content and quantified into emotion category intensity features that can be used by computer models. This allows the expression driving of digital humans to no longer rely solely on the physical characteristics of the audio, but to be dominated by the actual emotion of the content, providing core technical support for achieving highly realistic and infectious digital human interaction.

[0074] Step 103: Obtain reference video data and extract features from the reference video data to obtain video data features.

[0075] Step 103 will be described in detail below.

[0076] In some embodiments, to generate the basic image of a digital human, a reference video is needed, which typically contains a static or dynamic portrait of the digital human. Subsequently, feature extraction is performed on the reference video data. Specifically, consecutive image frames are selected from the video, and these high-resolution image frames are compressed into a low-dimensional latent space using the encoding part of an image encoder, such as a Variational Autoencoder (VAE), to obtain video data features (Zref). This video data feature is a compact representation of the original face image, preserving core information such as identity, lighting, and background, and significantly reducing the complexity of subsequent computations.

[0077] It is understood that the reference video data in this application is selected from the reference video frames, consisting of 5 consecutive image frames at time t. Here, five consecutive frames are selected, which includes temporal information while minimizing computational cost.

[0078] The following section further describes how to extract features from the reference video data.

[0079] Reference Figure 4 The reference video data is used to extract features to obtain video data features, including the following steps 401 to 403.

[0080] Step 401: Extract image frame data from multiple consecutive moments from the reference video data.

[0081] Step 402: Perform alignment transformation on each image frame data based on the alignment transformation matrix to obtain an aligned face image.

[0082] Step 403: Input the aligned face image into the image encoder for encoding processing to obtain video data features.

[0083] Steps 401 to 403 are described in detail below.

[0084] In some embodiments, to obtain the basic visual information driving the generation of the digital human, it is necessary to extract image frame data from multiple consecutive moments in the reference video data. The reference video data is typically a short clip containing a clear, frontal view of the target digital human. This step involves selecting a point in time and extracting several consecutive frames before and after that point, such as the five consecutive frames mentioned in the technical solution. The purpose of selecting consecutive frames rather than single frames is to capture subtle facial dynamics and temporal information, which helps to generate more stable and coherent video results, while also providing richer contextual information for subsequent processing.

[0085] After acquiring the raw image frame data, in order to eliminate the interference caused by changes in head pose, position, and size, it is necessary to perform an alignment transformation based on the matrix. For each image frame data Alignment transformation is then performed. This process typically begins by detecting facial keypoints in each frame of the image, and then calculates an alignment transformation matrix. This matrix records the operational parameters required to rotate, scale, and translate the current face to a standard pose (e.g., eyes level, face centered). After applying this matrix for transformation, a standardized aligned face image is obtained. As shown in formula (7) below.

[0086] (7) in, For image alignment operations, the corresponding inverse image alignment transform is: This alignment transformation ensures that the faces input to subsequent models are consistent in pose and size, thereby reducing the learning difficulty of the model and improving the generation quality.

[0087] To convert high-dimensional image data into a more easily processed low-dimensional representation, it is necessary to align face images. Input image encoder for encoding processing The image encoder here is typically the encoding part (Encoder) of a Variational Autoencoder (VAE). It's a deep neural network that learns how to compress an input aligned face image into a compact latent space and output a low-dimensional vector containing the core features of the image, ultimately yielding the video data features. As shown in formula (8) below.

[0088] (8) in The encoding process of the image encoder is as follows: The corresponding decoding process of the image decoder is as follows: ; Latent space dimension; video data features It retains key information such as the digital human's identity, facial lighting, and environment, while discarding pixel-level redundant details, greatly improving the efficiency of subsequent processing.

[0089] Through steps 401 to 403 above, an efficient and robust video feature extraction process was constructed. First, temporal information was preserved by extracting consecutive frames. Then, alignment transformation was used to standardize facial poses, solving the problem of variable head poses in practical applications. Finally, visual information was compressed into low-dimensional video data features through an image encoder. This series of processes not only significantly reduced the computational burden of the subsequent generation model, but also ensured the consistency of identity and high visual quality of the final generated digital human video by providing a stable and standardized identity feature input, laying a solid foundation for achieving realistic facial animation.

[0090] Step 104: Perform masking on the digital face image in the reference video data to generate face image mask variables.

[0091] Step 104 will be described in detail below.

[0092] In some embodiments, to ensure that emotions are expressed in key facial areas, the digital facial image in the reference video data needs to be masked. This step does not use a traditional, simple facial mask; instead, it generates an asymmetric facial image mask variable (Imask) based on preset facial landmark coordinates using an innovative combination method. This mask is a binary matrix with a specifically designed shape that precisely covers the core emotional expression areas, including the mouth, eyes, and eyebrows, while removing interference from irrelevant parts such as hair and forehead. The facial image mask variable generated in this way guides the subsequent generation process to focus emotional changes on the facial areas most expressive of the emotion.

[0093] It is understandable that the richness of emotional expression is closely related to the mouth, eyes, and eyebrows. To include the mouth and eyebrows while removing redundant hair and forehead information, this application creates a face image mask variable based on the 68-point bounding box of a standard face landmark. The face image mask variable is composed of two concatenated parts, as described below.

[0094] Reference Figure 5 The digital face image in the reference video data is masked to generate a face image mask variable, including the following steps 501 to 504.

[0095] Step 501: Select sequence elements of the first and second numerical ranges from the preset facial key point coordinate sequence as the upper part coordinate sequence, and select sequence elements of the third numerical range from the facial key point coordinate sequence as the lower part coordinate sequence. The first element of the third numerical range is the last element of the first numerical range, and the last element of the third numerical range is the first element of the second numerical range.

[0096] Step 502: Perform bounding box processing based on the upper part coordinate sequence and the preset scaling factor to obtain the upper part face mask.

[0097] Step 503: Perform convex hull processing based on the lower part coordinate sequence and the preset scaling factor to obtain the lower part face mask.

[0098] Step 504: Vertically stitch the upper and lower face masks to obtain a face image mask, and obtain face image mask variables based on the face image mask.

[0099] Steps 501 to 504 are described in detail below.

[0100] In some embodiments, firstly, a selection needs to be made from a preset sequence of facial keypoint coordinates, i.e., a standard point set (e.g., the dlib 68-point model) that marks the positions of facial features (such as eyes, nose, mouth, contours, etc.). Specifically, sequence elements from the first numerical range (points 1 to 4) and the second numerical range (points 14 to 17) are selected to form the upper coordinate sequence. Simultaneously, sequence elements from the third numerical range (points 4 to 14) are selected from this coordinate sequence to form the lower coordinate sequence. This selection method ensures that the start and end points of the lower coordinate sequence (points 4 and 14) are precisely connected to the endpoints of the two ranges of the upper coordinate sequence, guaranteeing a seamless connection between the subsequently generated upper and lower masks.

[0101] Then, the upper half of the mask is generated using the coordinate points selected in the previous step. An outer bounding box is then created based on the coordinate sequence of the upper half and a preset scaling factor. () is shown in the following formula (9).

[0102] (9) in, That is, the upper part of the coordinate sequence. These are the coordinates of the key points on the i-th person's face. This is a scaling factor, specifically the factor by which the bounding box formed based on the coordinates of the facial key points expands outwards. Bounding Box Processing is an operation that calculates the smallest rectangular region that can completely enclose all given coordinate points. By performing this operation on a selected upper portion of the coordinate sequence (i.e., the jawline points on both sides of the facial contour), a rectangular box covering the entire core area of ​​the face is generated. A preset scaling factor (e.g., 1.2) is introduced to moderately expand the range of this rectangle, ensuring that the final upper portion of the face mask is sufficiently large. It can completely cover all key areas of emotional expression, from eyebrows and eyes to mouth.

[0103] Then, to generate the lower half of the mask, forming a smooth curve that conforms to the jawline of the human face, convex hull processing is performed based on the lower half's coordinate sequence and a preset scaling factor. As shown in formula (10) below.

[0104] (10) in, This refers to the lower coordinate sequence. Convex hull processing is a method for finding the smallest convex polygon that can enclose all given points, similar to wrapping points with a rubber band. When this processing is applied to the lower coordinate sequence (i.e., the set of points along the chin and jawline), it generates an arc-shaped or polygonal region that closely fits the jawline. Similarly, this region can be fine-tuned using a preset scaling factor to ensure that the resulting lower face mask is... It matches the lower half of the face perfectly.

[0105] Then, the independent parts generated in the first two steps are integrated into a complete face mask. The upper and lower face masks are vertically stitched together to obtain the final face image mask as shown in formula (11).

[0106] (11) This stitching operation combines the upper rectangular mask with the lower curved mask to form a U-shaped mask with a flat upper half and a rounded lower half. The final face image mask variable is a binary matrix, where the covered area is the effective area, and the uncovered areas (such as hair, neck, and background) are masked, which is used to precisely limit the model's attention range in subsequent generation processes.

[0107] Then, further, mask the face image. The data is input into a VAE encoder for encoding processing to obtain the face image mask features. As shown in formula (12) below.

[0108] (12) Among them, face image mask features Characterized as a face image mask Latent space variables.

[0109] Reference Figure 6 This is a schematic diagram of a face mask provided in an embodiment of this application. Figure 6 The image shown illustrates the effect of applying the face image mask variables generated in this application to actual digital face images. From... Figure 6 As can be seen, the mask successfully focuses image processing on the core facial areas, including the eyes, eyebrows, nose, and mouth, which are key to emotional expression. At the same time, it effectively masks irrelevant information such as hair, forehead, and background, laying the foundation for subsequent accurate emotional video generation.

[0110] Reference Figure 7 This is a facial landmark map provided in an embodiment of this application. Figure 7The image shows a sequence of 68 standard facial key point coordinates used to define the structure of a face. In this application, this preset coordinate sequence forms the basis for generating a facial image mask. The technical solution precisely constructs a mask outline that covers key expression areas by selecting coordinate points within a specific range of values ​​from this sequence (e.g., points 1-4 and 14-17 in the upper part, and points 4-14 in the lower part).

[0111] Reference Figure 8 This is a face mask overlay effect diagram provided in an embodiment of this application. Figure 8 The image shown is the final face image mask (Imask) generated in this application. It is a binary matrix whose unique U-shaped contour is formed by vertically stitching the processing results of the upper bounding box and the lower convex hull based on specific facial key points. In the video generation process, this mask is used to process the face image in the reference video, ensuring that the subsequent diffusion model can focus its attention on the facial regions that best reflect emotional changes.

[0112] Through steps 501 to 504 above, a novel and efficient method for generating face masks is realized. Unlike traditional methods, this method combines bounding box processing and convex hull processing, and based on specific facial key points, creates a customized mask that can accurately cover the entire core expression area from eyebrows and eyes to mouth, while also conforming to the jaw contour. This design not only effectively eliminates the interference of redundant information such as hair and background, but more importantly, it forces the subsequent diffusion model to focus the generation on the facial area most related to emotional expression, thus providing key technical support for generating digital human videos with rich expressions, subtle changes and high realism.

[0113] Step 105: Generate video data based on audio data features, video data features, emotion category intensity features, and face image mask variables to obtain target video data.

[0114] Step 105 is described in detail below.

[0115] In some embodiments, all key features generated in the preceding steps, i.e., audio data features, are further considered. Video data characteristics Emotion category intensity characteristics and face image mask variables The video data is generated by fusing and concatenating these feature vectors, using them as a unified input to a diffusion model. The diffusion model then iteratively denoises the random noise, gradually generating latent image variables that match the input conditions. The latent variable is then converted into image frames by the image decoder. Finally, by combining all the generated image frame sequences with the original audio, the target video data, which is highly synchronized with the audio content and emotion, can be obtained, as described below.

[0116] Reference Figure 9 The target video data is generated based on audio data features, video data features, emotion category intensity features, and face image mask variables, including the following steps 901 to 904.

[0117] Step 901: Based on the audio data features, video data features, emotion category intensity features, and face image mask variables, the data is processed by the diffusion model to obtain image latent variables.

[0118] Step 901 will be described in detail below.

[0119] In some embodiments, data processing is performed using a diffusion model based on audio data features, video data features, emotion category intensity features, and facial image mask variables. The diffusion model is a powerful generative model that generates data from pure noise by learning a progressive denoising process. In this embodiment, all the aforementioned features are used as conditions to guide this denoising process. The model fuses and concatenates these different features, and then iteratively denoises them in the latent space using a preset time series, gradually transforming a random noise tensor into a structured representation containing all conditional information (identity, lip movement, emotion), ultimately obtaining the image latent variables. The details are described below.

[0120] Reference Figure 10 Data processing is performed based on audio data features, video data features, emotion category intensity features, and facial image mask variables input diffusion model to obtain image latent variables, including the following steps 1001 to 1006.

[0121] Step 1001: In the diffusion model, the audio data features and the emotion category intensity features are concatenated to obtain the audio emotion features, and the video data features, the face image mask variable, and random noise are concatenated to obtain the video mask features.

[0122] Step 1002: Perform tensor concatenation between the audio emotion features and the video mask features to obtain the initial tensor.

[0123] Step 1003: Based on the initial tensor and time step, perform noise prediction to obtain the predicted noise.

[0124] Step 1004: Starting from the last time step, use each time step in sequence as a denoising time step.

[0125] Step 1005: In the denoising time step, based on the initial tensor corresponding to the next time step, the denoising time step, the audio emotion features, and the predicted noise, iteratively denoise the initial tensor corresponding to the denoising time step to obtain a denoised tensor, and use the denoised tensor as the new initial tensor corresponding to the denoising time step.

[0126] Step 1006: After the initial tensors corresponding to all time steps are denoised, the image latent variables are obtained based on the initial tensor corresponding to the first time step.

[0127] Steps 1001 to 1006 are described in detail below.

[0128] In some embodiments, in the diffusion model, the audio data features are first... and emotion category intensity features Splicing is then performed, which refers to concatenating two independent feature vectors along a specific dimension to form a longer, fused vector containing both types of information, thereby obtaining the audio emotion features. At the same time, the video data features representing identity will be used. The facial image mask features that define the generation region. and a random noise tensor The video mask features are then stitched together to obtain features that incorporate visual, spatial constraints, and initial randomness. This step divides the guidance information from different sources into two groups: one group consists of auditory conditions that drive content and emotion, and the other group consists of visual conditions that define identity and form.

[0129] To integrate all conditional information into a single input, it is necessary to incorporate audio emotion features. and video mask features Perform tensor concatenation. Tensor concatenation is an operation that connects two sets of features (audio emotion features and video mask features) obtained in the previous step into a higher-dimensional initial tensor as shown in formula (13).

[0130] (13) This initial tensor It is the final input for the diffusion model to predict noise. It contains all the guiding information needed to generate the target image, including "what to say", "what emotion", "what it looks like" and "in which region it is generated".

[0131] Then, based on the initial tensor and the current time step Perform noise prediction. Time step. This is a key concept in diffusion models, representing the different stages of data transitioning from clear to completely noisy. During denoising, the model (such as the UNET3D network in the technical solution) receives the current initial tensor (i.e., an image representation with some noise) and the corresponding time step as input. Its task is to analyze the noise distribution in the current tensor and output a predicted noise that is as consistent as possible with the added noise. As shown in formula (14) below.

[0132] (14) in, This is a diffusion processing method for a UNET3D diffusion network.

[0133] Furthermore, starting from the last time step, each time step will be used sequentially as a denoising time step. (This is also the time value for the denoising step k). The generation process of the diffusion model is a reverse process. It starts from a state consisting entirely of random noise (corresponding to the last time step) and gradually removes the noise step by step. Therefore, this step reverses the entire time series, defining a clear path of evolution from a high-noise state to a low-noise state, thus preparing for subsequent iterative processing.

[0134] Then, at each selected denoising time step In the middle, it will be based on the initial tensor corresponding to the next time step. Noise reduction time step Audio emotional characteristics and prediction noise The initial tensor corresponding to the denoising time step is iteratively denoised. Specifically, the model uses the tensor from the previous step (the later time step with more noise), combines it with the current time step and the audio emotion features used as guiding conditions, and subtracts the prediction noise predicted by the model in that step to calculate a new tensor with less noise, thus obtaining the denoised tensor. As shown in formula (15) below.

[0135] (15) in, This is the denoising process for the diffusion network; subsequently, this denoised tensor... It serves as the new initial tensor for the current denoising time step, for use in the next (less noisy) time step.

[0136] Finally, after all the initial tensors corresponding to all time steps have undergone denoising, i.e., after iterating from the last time step to the first time step (usually time 0), the image latent variables are obtained based on the initial tensor corresponding to the first time step. At this point, the initial tensor is almost noise-free and is a structured data representation in the latent space that clearly represents the face image that meets all input conditions; this is the final required image latent variable. .

[0137] Through steps 1001 to 1006 above, starting from pure random noise, at each time step, multiple features such as audio, emotion, identity, and mask are used as powerful guides to accurately predict and eliminate noise, gradually sculpting disordered noise into an image latent variable with highly structured information. This iterative generation mechanism, constrained by multiple conditions, is the core technical guarantee for the present invention to generate high-quality digital human videos that not only conform to a specific identity but also accurately synchronize lip movements and delicately express emotions.

[0138] Step 902: Input the latent variables of the image into the image decoder for decoding processing to obtain the decoded data features.

[0139] Step 903: Based on the alignment transformation matrix, perform inverse alignment transformation on the decoded data features to obtain post-processed image data.

[0140] Step 904: Obtain the target video data based on all post-processed image data.

[0141] Steps 902 to 904 are described in detail below.

[0142] In some embodiments, a compact representation of the image (i.e., image latent variables) is obtained from the diffusion model. After that, it needs to be converted back to a visible view. This step inputs the image latent variables into the image decoder for decoding. (), to obtain the features of the decoded data As shown in formula (16) below.

[0143] (16) An image decoder is typically the decoding part of a variational autoencoder (VAE) paired with the encoder previously used to encode the reference video. Its function is the opposite of the encoder; it upsamples and reconstructs low-dimensional latent image variables to recover pixel-level image details, thus obtaining decoded data features. These decoded data features are essentially a synthesized, but still normalized and aligned, image of a face.

[0144] To ensure that the generated standardized face images can be seamlessly embedded back into the scene of the original video, pose and size restoration is required. This step will be based on the alignment transformation matrix. , for decoded data features Perform inverse alignment transformation () to obtain post-processed image data As shown in formula (17) below.

[0145] (17) The alignment transformation matrix used here is the inverse of the transformation matrix previously calculated for face alignment during video data feature extraction. By applying this inverse transformation, the decoded aligned face image can be restored to its correct position, rotation angle, and size in the original video frame, ultimately resulting in post-processed image data that matches the original video background and resolution.

[0146] Since the aforementioned steps generate a single frame image at a single point in time, it is necessary to aggregate the image frames generated at all points in time. Based on all the post-processed image data, they are serialized and combined according to a preset video frame rate (e.g., 25 frames per second). Simultaneously, the original audio data is synchronized with it, ultimately synthesizing and outputting a complete video file that is synchronized with the visuals and audio and contains rich emotional expression—this is the target video data.

[0147] In the embodiments of this application, each batch of synthesized image frames and audio are streamed at a video frame rate of 25 FPS and an audio frame rate of 50 FPS.

[0148] Through steps 901 to 904 above, an end-to-end high-quality video generation process driven by multimodal features is realized. The core advantage of this process lies in leveraging the powerful generation capabilities of the diffusion model to deeply fuse various heterogeneous information such as lip movements determined by audio features, facial expressions controlled by emotional features, identity maintained by video features, and the area focused by the face mask. Finally, through post-processing steps such as decoding and inverse transformation, not only is a video highly synchronized with the audio content generated, but more importantly, the digital human in the video can exhibit content-driven, delicate, and natural facial expression changes, thereby effectively solving the problems of monotonous expressions and poor realism in existing digital human technologies.

[0149] Reference Figure 11 This is a schematic diagram of a data flow for video data generation provided in an embodiment of this application. For example... Figure 11 The diagram illustrates the internal workflow of the core "lip-sync" module of this invention. It describes how the various input data are processed and fused: the audio waveform extracts features through an audio encoder, and then a multilayer perceptron network (MLP) parses out emotion and intensity, embedding the resulting vectors; the video reference frames extract identity features through a VAE encoder. Finally, the audio features, emotion intensity features, identity features, and initial random noise are fed as conditional inputs into the central backbone network. A diffusion model denoising process generates a latent representation of the image, which is then restored by the VAE decoder into the final video frame with rich facial expressions.

[0150] Reference Figure 12 This application provides a schematic diagram illustrating a concept for video data generation. For example... Figure 12 The diagram illustrates the complete end-to-end flow of the 2D digital human emotion control method proposed in this invention. The process begins with the ASR module converting the user's speech into text. The text is then fed into an LLM (Large Language Model) for processing. The LLM not only generates response text but also outputs "emotion; intensity" features in parallel. Subsequently, the emotionally charged text is synthesized into emotional speech through a TTS module, and the "emotion; intensity" features, along with "audio features" extracted from this speech, are fed into the core "lip-sync synthesis" module. Finally, the video frames generated by the "lip-sync synthesis" module are merged with the audio generated by the TTS module to generate the final spoken video (.mp4).

[0151] The video data generation method, apparatus, electronic device, and storage medium proposed in this application include: first, acquiring audio data and extracting features from the audio data to obtain audio data features; then, performing speech recognition on the audio data to obtain audio text information; inputting the audio text information into a large language model for data processing based on preset prompt words; selecting a target emotion category from multiple preset emotion categories and outputting the emotion category code corresponding to the target emotion category; selecting a target continuous emotion intensity from a preset continuous emotion intensity range and outputting the continuous intensity code corresponding to the target continuous emotion intensity; performing weighted superposition processing based on the emotion category code and the continuous intensity code, and adding a bias factor to obtain the emotion category intensity features. First, acquire reference video data and extract image frame data from multiple consecutive moments within the reference video data. Perform alignment transformation on each image frame data based on the alignment transformation matrix to obtain aligned face images. Input the aligned face images into an image encoder for encoding to obtain video data features. Then, select sequence elements from the first and second numerical ranges of the preset facial key point coordinate sequence as the upper part coordinate sequence, and select sequence elements from the third numerical range of the facial key point coordinate sequence as the lower part coordinate sequence. The first element of the third numerical range is the last element of the first numerical range, and the last element of the third numerical range is the first element of the second numerical range. Based on the upper part coordinate sequence... The upper face mask is obtained by processing the bounding box of the coordinate sequence and a preset scaling factor. The lower face mask is obtained by processing the convex hull of the coordinate sequence and the preset scaling factor. The upper and lower face masks are then vertically stitched together to obtain the face image mask, and face image mask variables are derived from this mask. Finally, in the diffusion model, audio data features and emotion category intensity features are stitched together to obtain audio emotion features. Video data features, face image mask variables, and random noise are then stitched together to obtain video mask features. The audio emotion features and video mask features are then tensor-stitched together to obtain an initial tensor. Noise prediction is performed based on the initial tensor and the time step to obtain predicted noise. Starting from the last time step, each time step is sequentially used as a denoising time step. In each denoising time step, the initial tensor corresponding to the denoising time step is iteratively denoised based on the initial tensor of the next time step, the denoising time step, audio emotion features, and predicted noise, resulting in a denoised tensor. This denoised tensor is then used as the new initial tensor for the denoising time step. After denoising has been performed on the initial tensors of all time steps, the image latent variables are obtained based on the initial tensor of the first time step. These latent variables are then input into the image decoder for decoding, yielding decoded data features. Based on the alignment transformation matrix, the decoded data features are inversely aligned to obtain post-processed image data. Finally, based on all post-processed image data, the target video data is obtained.

[0152] This application's embodiments overcome the limitations of existing technologies that only focus on lip-sync by introducing emotion category intensity features generated by a large language model based on audio content. This allows digital human facial expressions to dynamically match the inherent emotions of the dialogue, achieving audio-visual synchronization at the emotional level. Secondly, by designing a facial image mask variable that covers key emotional expression areas such as the eyes and eyebrows, it ensures that emotional expression is global and collaborative, rather than limited to single changes in the mouth area. This enables the generation of digital human videos with richer and more natural expressions, effectively solving the problem of monotonous and rigid digital human expressions in existing technologies, and thus significantly improving the final... This technology enhances the realism and expressiveness of digital humans in generated videos. Furthermore, by weighted fusion of emotion category encoding and continuous intensity encoding, the resulting emotion category intensity features provide a rich, accurate, and highly controllable emotional guidance signal for subsequent video generation models in a unified mathematical form. This is crucial for generating digital human videos with nuanced facial expressions and realistic emotional expression. Additionally, speech recognition converts sound into text, solving the challenge of machine speech understanding. Then, using a large language model and customized prompts, specific emotion categories and their intensities are accurately extracted from the text content and quantified into emotion category intensity features usable by computer models. This allows digital human facial expressions to be driven not only by the physical characteristics of the audio but also by the actual emotion of the content, providing core technical support for achieving highly realistic and engaging digital human interaction. Finally, by extracting continuous frames to preserve temporal information and using alignment transformation to standardize facial poses, the technology addresses the issue of variable head poses in practical applications. Finally, an image encoder compresses visual information into low-dimensional video data features. This series of processes not only significantly reduces the computational burden of subsequent generation models but also ensures the final output by providing a stable and standardized identity feature input. The consistency in identity and high visual quality of the generated digital human videos lay a solid foundation for achieving realistic facial animation. In addition, by combining bounding box processing and convex hull processing, and based on specific facial key points, a customized mask is created that can accurately cover the entire core expression area from eyebrows, eyes to mouth, while also conforming to the jaw contour. This design not only effectively eliminates the interference of redundant information such as hair and background, but more importantly, it forces the subsequent diffusion model to focus the generation on the facial areas most related to emotional expression, thus providing key technical support for generating digital human videos with rich expressions, subtle changes and high realism.Furthermore, starting from pure random noise, at each time step, multiple features such as audio, emotion, identity, and mask are used as powerful guides to accurately predict and eliminate noise, gradually sculpting the disordered noise into a latent image variable with highly structured information. This iterative generation mechanism, constrained by multiple conditions, is the core technical guarantee for this invention to generate high-quality digital human videos that not only conform to a specific identity but also accurately synchronize lip movements and delicately express emotions. Finally, utilizing the powerful generation capabilities of the diffusion model, multiple heterogeneous information such as lip movements determined by audio features, facial expressions controlled by emotional features, identity maintained by video features, and the area focused by the face mask are deeply fused. Ultimately, through post-processing steps such as decoding and inverse transformation, not only is a video highly synchronized with the audio content generated, but more importantly, the digital human in the video can exhibit content-driven, delicate, and natural facial expression changes, thus effectively solving the problems of monotonous expressions and poor realism in existing digital human technologies.

[0153] This application also provides a video data generation apparatus that can implement the above-described video data generation method, see reference. Figure 13 The device 1300 includes: The audio data acquisition module 1310 is used to acquire audio data and extract features from the audio data to obtain audio data features; The emotion feature acquisition module 1320 is used to process audio data using preset prompt words and a large language model to obtain emotion category intensity features. The video feature acquisition module 1330 is used to acquire reference video data and extract features from the reference video data to obtain video data features; The image mask generation module 1340 is used to perform mask processing on the digital face image in the reference video data and generate face image mask variables; The video data generation module 1350 is used to generate video data based on audio data features, video data features, emotion category intensity features, and face image mask variables to obtain target video data.

[0154] In some embodiments, the emotion feature acquisition module 1320 is further configured to: Speech recognition is performed on the audio data to obtain audio text information; Based on preset prompts, audio text information is input into a large language model for data processing to obtain emotion category intensity features.

[0155] In some embodiments, the emotion feature acquisition module 1320 is further configured to: Based on preset prompt words, the audio text information is input into the large language model for data processing. From multiple preset emotion categories, the target emotion category is selected and the emotion category code corresponding to the target emotion category is output. From a preset continuous emotion intensity range, the target continuous emotion intensity is selected and the continuous intensity code corresponding to the target continuous emotion intensity is output. The emotion category intensity feature is obtained by weighting and superimposing the emotion category code and the continuous intensity code, and then adding a bias factor.

[0156] In some embodiments, the video feature acquisition module 1330 is further configured to: Extract image frame data from multiple consecutive moments from reference video data; Alignment transformation is performed on each image frame data based on the alignment transformation matrix to obtain aligned face images; The aligned face image is input into an image encoder for encoding to obtain video data features.

[0157] In some embodiments, the image mask generation module 1340 is further configured to: Select sequence elements from the first and second numerical ranges from the preset facial key point coordinate sequence as the upper part coordinate sequence, and select sequence elements from the third numerical range from the facial key point coordinate sequence as the lower part coordinate sequence. The first element of the third numerical range is the last element of the first numerical range, and the last element of the third numerical range is the first element of the second numerical range. The upper part of the face mask is obtained by processing the bounding box based on the upper part of the coordinate sequence and the preset scaling factor. The lower part of the face mask is obtained by performing convex hull processing based on the lower part of the coordinate sequence and the preset scaling factor. A face image mask is obtained by vertically stitching the upper and lower face masks, and face image mask variables are obtained based on the face image mask.

[0158] In some embodiments, the video data generation module 1350 is further configured to: Data processing is performed based on audio data features, video data features, emotion category intensity features, and face image mask variables input to a diffusion model to obtain image latent variables; The latent variables of the image are input into the image decoder for decoding processing to obtain the decoded data features; Based on the alignment transformation matrix, the features of the decoded data are subjected to inverse alignment transformation to obtain post-processed image data. The target video data is obtained based on all post-processed image data.

[0159] In some embodiments, the video data generation module 1350 is further configured to: In the diffusion model, audio data features and emotion category intensity features are concatenated to obtain audio emotion features, and video data features, face image mask variables, and random noise are concatenated to obtain video mask features. The audio emotion features and video mask features are concatenated into tensors to obtain the initial tensor. Noise prediction is performed based on the initial tensor and time step to obtain the predicted noise; Starting from the last time step, each time step is used as a denoising time step in sequence; In the denoising time step, the initial tensor corresponding to the denoising time step is iteratively denoised based on the initial tensor of the next time step, the denoising time step, the audio emotion features, and the predicted noise to obtain the denoised tensor, and the denoised tensor is used as the new initial tensor corresponding to the denoising time step. After denoising the initial tensors corresponding to all time steps, the image latent variables are obtained based on the initial tensor corresponding to the first time step.

[0160] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, the specific implementation of the video data generation device is basically the same as the specific implementation of the video data generation method described above, and will not be repeated here.

[0161] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the video data generation method described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0162] Please see Figure 14 , Figure 14 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1402 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1402 and is called and executed by the processor 1401 using the video data generation method of the embodiments of this application. The input / output interface 1403 is used to implement information input and output; The communication interface 1404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1405 transmits information between various components of the device (e.g., processor 1401, memory 1402, input / output interface 1403, and communication interface 1404); The processor 1401, memory 1402, input / output interface 1403 and communication interface 1404 are connected to each other within the device via bus 1405.

[0163] This application also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described video data generation method.

[0164] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0166] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0169] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0170] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0172] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating video data, characterized in that, The method includes: Acquire audio data and extract features from the audio data to obtain audio data features; Using preset prompt words and a large language model, data processing is performed on the audio data to obtain emotion category intensity features; Acquire reference video data and extract features from the reference video data to obtain video data features; The digital face image in the reference video data is masked to generate a face image mask variable. Video data is generated based on the audio data features, the video data features, the emotion category intensity features, and the face image mask variables to obtain the target video data.

2. The video data generation method according to claim 1, characterized in that, The process of using preset prompts and a large language model to process the audio data and obtain emotion category intensity features includes: The audio data is subjected to speech recognition to obtain audio text information; Based on the preset prompt words, the audio text information is input into the large language model for data processing to obtain the emotion category intensity features.

3. The video data generation method according to claim 2, characterized in that, The process of inputting the audio text information into the large language model based on preset prompt words for data processing to obtain the emotion category intensity features includes: Based on preset prompt words, the audio text information is input into the large language model for data processing. From multiple preset emotion categories, a target emotion category is selected and the emotion category code corresponding to the target emotion category is output. From a preset continuous emotion intensity range, a target continuous emotion intensity is selected and the continuous intensity code corresponding to the target continuous emotion intensity is output. The emotion category intensity feature is obtained by weighting and superimposing the emotion category code and the continuous intensity code, and then adding a bias factor.

4. The video data generation method according to claim 1, characterized in that, The step of extracting features from the reference video data to obtain video data features includes: Extract image frame data from multiple consecutive moments from the reference video data; Alignment transformation is performed on each image frame data based on the alignment transformation matrix to obtain an aligned face image; The aligned face image is input into an image encoder for encoding to obtain the video data features.

5. The video data generation method according to claim 1, characterized in that, The step of performing masking processing on the digital face image in the reference video data to generate face image mask variables includes: Select sequence elements of a first and a second numerical range from the preset sequence of facial key point coordinates as the upper part of the coordinate sequence, and select sequence elements of a third numerical range from the sequence of facial key point coordinates as the lower part of the coordinate sequence. The first element of the third numerical range is the last element of the first numerical range, and the last element of the third numerical range is the first element of the second numerical range. The upper part of the face mask is obtained by processing the bounding box based on the upper part of the coordinate sequence and the preset scaling factor. Based on the lower part coordinate sequence and the preset scaling factor, convex hull processing is performed to obtain the lower part face mask; A face image mask is obtained by vertically stitching the upper and lower face masks together, and the face image mask variables are obtained based on the face image mask.

6. The video data generation method according to claim 1, characterized in that, The process of generating target video data based on the audio data features, the video data features, the emotion category intensity features, and the face image mask variables includes: Based on the audio data features, video data features, emotion category intensity features, and face image mask variables, data processing is performed using an input diffusion model to obtain image latent variables; The latent variables of the image are input into the image decoder for decoding processing to obtain the decoded data features; Based on the alignment transformation matrix, the features of the decoded data are subjected to inverse alignment transformation to obtain post-processed image data; The target video data is obtained based on all the post-processed image data.

7. The video data generation method according to claim 6, characterized in that, The data processing based on the audio data features, video data features, emotion category intensity features, and facial image mask variables is performed using a diffusion model to obtain image latent variables, including: In the diffusion model, the audio data features and the emotion category intensity features are concatenated to obtain audio emotion features, and the video data features, the face image mask variables, and random noise are concatenated to obtain video mask features. The audio emotion features and the video mask features are concatenated using tensors to obtain an initial tensor. Based on the initial tensor and time step, noise prediction is performed to obtain the predicted noise; Starting from the last time step, each time step is used as a denoising time step in sequence; In the denoising time step, the initial tensor corresponding to the denoising time step is iteratively denoised based on the initial tensor corresponding to the next time step, the denoising time step, the audio emotion features, and the predicted noise to obtain a denoised tensor, and the denoised tensor is used as the new initial tensor corresponding to the denoising time step. After the initial tensors corresponding to all the time steps are denoised, the image latent variables are obtained based on the initial tensor corresponding to the first time step.

8. A video data generation device, characterized in that, The device includes: An audio data acquisition module is used to acquire audio data and extract features from the audio data to obtain audio data features; The emotion feature acquisition module is used to process the audio data using preset prompt words and a large language model to obtain emotion category intensity features. The video feature acquisition module is used to acquire reference video data and extract features from the reference video data to obtain video data features; An image mask generation module is used to perform masking processing on the digital face image in the reference video data to generate face image mask variables; The video data generation module is used to generate target video data based on the audio data features, the video data features, the emotion category intensity features, and the face image mask variables.

9. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the video data generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video data generation method according to any one of claims 1 to 7.

Images