Emotion-driven 2D supernatural digital human video generation system of multi-modal model

Abstract

The invention relates to the technical field of image recognition and processing, and discloses an emotion-driven 2D supernatural digital human video generation system of a multi-modal model, and the system comprises a voice input module which is used for receiving a voice signal of a user and then converting the voice signal into a digital signal; the voice preprocessing module is used for carrying out noise elimination and normalization processing on the voice signals; the voice emotion analysis module extracts emotion features based on the digital signals of the voice preprocessing module; the multi-modal generation module is used for mapping the emotional characteristics into digital human gestures and digital human population type actions; and the video generation module is used for synchronizing the gesture and the mouth shape action with the voice signal and generating a 2D digital human video. According to the method and the device, the speech emotion features are accurately mapped into the gestures and mouth shape actions of the digital human through the multi-modal generation model, the actions highly matched with the emotion types, the emotion intensity and the mood features are generated, so that the expression effect with rich emotions is visually presented, and the user experience is more realistic.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition and processing, and in particular to an emotion-driven 2D supernatural digital human video generation system of a multimodal model. Background Art

[0002] With the development of virtual digital human technology, the demand for digital humans in various application scenarios has gradually increased, including virtual customer service, virtual characters in the entertainment industry, education and training, etc. Most of the existing 2D digital human generation technologies generate videos of digital humans speaking through voice drive to meet users' needs for virtual interaction. However, these traditional 2D digital human generation technologies have certain limitations and it is difficult to achieve natural and smooth emotional expression, as shown below:

[0003] Most of the existing 2D digital human generation systems use similar voice-driven methods. The generated movements and expressions lack innovation and differentiation, resulting in a relatively simple generation effect. In addition, many current generation systems are relatively limited in emotional expression. The generated digital human videos are difficult to reflect the delicate features of the voice, such as emotions, pauses, and tone. As a result, the generated video expressions and movements cannot match the emotional information and lack real emotional expression effects.

[0004] In addition, existing technologies rely on sampled videos for generation, resulting in incoherence in the speech and gesture content of the generated videos. Gestures and actions need to be manually configured to match the speech content, which increases labor costs. The actions in the generated videos are often uncoordinated, with abrupt jumps and incoherence. This inconsistency affects the naturalness of the video and brings a bad viewing experience to users. Summary of the invention

[0005] In order to make up for the above shortcomings, the present invention provides an emotion-driven 2D super-natural digital human video generation system with a multimodal model, aiming to improve the problems of unnatural emotional expression, incoherent movements, complex manual configuration and lack of consistency in the existing 2D digital human generation technology.

[0006] In a first aspect, the present invention provides the following technical solution: a multimodal model emotion-driven 2D supernatural digital human video generation system, comprising:

[0007] A voice input module, used to receive the user's voice signal and convert it into a digital signal;

[0008] A speech preprocessing module is used to perform noise elimination and normalization processing on speech signals;

[0009] Speech emotion analysis module, which extracts emotional features based on the digital signal of the speech preprocessing module;

[0010] A multimodal generation module for mapping emotional features into digital human gestures and digital human-type movements;

[0011] The video generation module is used to synchronize gestures and lip movements with speech signals and generate 2D digital human videos.

[0012] Preferably, the emotional features extracted by the speech emotion analysis module include but are not limited to emotion category, emotion intensity and tone features.

[0013] Preferably, the speech preprocessing module includes:

[0014] A sampling and quantization unit for converting analog voice signals into discrete digital signals;

[0015] A short-time Fourier transform unit is used to convert discrete digital signals into frequency domain features and generate a time-frequency feature matrix;

[0016] The noise reduction unit is used to eliminate noise using a Wiener filter, and the function of the filter is:

[0017]

[0018] Where P S (f) and P N (f) represents the power spectral density of speech and noise respectively;

[0019] The normalization unit is used to normalize the processed signal to the interval [-1,1].

[0020] Preferably, the speech emotion analysis module includes:

[0021] A convolutional neural network feature extraction unit is used to receive the spectrum feature matrix from the speech preprocessing module and extract discrete digital signals therefrom. The convolutional neural network feature extraction unit extracts a feature map H by a convolution operation. i , and its calculation formula is:

[0022]

[0023] Among them, F represents the input spectrum feature matrix, W k represents the convolution kernel weight, b represents the bias term, and f is the activation function;

[0024] A bidirectional long short-term memory network modeling unit is used to receive the feature map H output by the convolutional neural network feature extraction unit and capture the emotional changes in the speech signal based on time series modeling. The bidirectional long short-term memory network modeling unit obtains the hidden state h by forward and backward time series processing. t , and its calculation formula is:

[0025] ht =σ(W f ·[h t-1 ,x t ]+b f )

[0026] Among them, W f is the time step weight matrix, b f is the bias term, x t is the input feature vector of the current time step, h t represents the hidden state of the current time step, σ represents the activation function, which is used to generate the nonlinear mapping of the emotion sequence;

[0027] Self-attention mechanism unit is used to fuse the sentiment features extracted by convolutional neural network and bidirectional long short-term memory network.

[0028] Preferably, the multimodal generation module includes:

[0029] Generative adversarial network generator, used to generate digital human gestures and digital human-type movements based on emotional features;

[0030] The discriminator is used to evaluate the generated actions;

[0031] Multi-head attention mechanism for combining emotional features with action generation;

[0032] The motion consistency optimization unit is used to minimize the motion differences of adjacent time steps through temporal interpolation and consistency constraints. Its consistency constraint loss is:

[0033]

[0034] Among them A t and A t+1 are action frames at adjacent time steps;

[0035] The multimodal fusion unit uses the multi-head attention mechanism of the Transformer structure to complete the multimodal fusion of emotional features and action features.

[0036] Preferably, the video generation module includes:

[0037] The dynamic time warping synchronization unit is used to align the speech signal with the gestures and lip movements through the dynamic time warping algorithm. The cumulative distance formula is:

[0038] D(i,j)=d(i,j)+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}

[0039] Where D(i,j) is the cumulative distance and d(i,j) is the local distance;

[0040] The video encoding and format conversion unit is used to encode, compress and convert the format of the generated 2D digital human video.

[0041] Preferably, the multi-head attention mechanism calculation formula adopted by the multimodal fusion unit is:

[0042] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0043] in, Q, K, V represent query, key and value matrices respectively, and W O is the output transformation matrix.

[0044] In a second aspect, the present invention provides the following technical solutions: a method for generating a 2D supernatural digital human video driven by an emotion of a multimodal model, and a system for generating a 2D supernatural digital human video driven by an emotion of a multimodal model, wherein the method comprises the following steps:

[0045] S1, receiving the user's voice signal, converting the analog signal into a discrete digital signal through sampling and quantization;

[0046] S2, performing noise elimination and normalization processing on the digital signal generated by the voice input module, wherein the noise elimination reduces the influence of environmental noise through the Wiener filter, and the normalization processing standardizes the signal within a specific range;

[0047] S3, based on the preprocessed signal output by the speech preprocessing module, using a convolutional neural network and a bidirectional long short-term memory network to extract emotional features, this step includes extracting local emotional information from spectral features, capturing emotional changes in time series through a bidirectional long short-term memory network, and weighting emotional features through a self-attention mechanism to generate an emotional feature matrix including emotional category, emotional intensity and tone features;

[0048] S4, inputting the emotion feature matrix generated by the speech emotion analysis module into the multimodal generation module, generating digital human gestures and lip movements matching the emotion features through a generative adversarial network, the multimodal generation module also includes a multi-head attention mechanism and a consistency optimization step, optimizing the timing consistency of the generated gestures and lip movements;

[0049] S5. Synchronize the gestures and lip movements output by the multimodal generation module with the original voice signal and generate a 2D digital human video. The synchronization operation is achieved through a dynamic time warping algorithm to align the voice signal and action time.

[0050] In the third aspect, the invention provides the following technical solution: a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multimodal model emotion-driven 2D supernatural digital human video generation method when executing the computer program.

[0051] In a fourth aspect, the present invention provides the following technical solution: a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned multimodal model emotion-driven 2D supernatural digital human video generation method.

[0052] The present invention has the following beneficial effects:

[0053] 1. In the present invention, the speech emotion features are accurately mapped into the digital human's gestures and lip movements through a multimodal generation model, so that the generated 2D digital human video has real and natural emotional expression. The system can generate actions that are highly matched with the emotion category, emotion intensity and tone characteristics, thereby visually presenting an emotionally rich expression effect and making the user experience more realistic.

[0054] 2. In the present invention, motion consistency optimization is introduced in the multimodal generation module. Through time-series frame interpolation and consistency constraints, the abrupt changes of motion between adjacent time steps are effectively reduced. This consistency processing makes the generated gestures and lip movements smoother in time sequence, ensuring the continuity of the video and improving the fluency of the viewing experience.

[0055] 3. In the present invention, by adopting the multi-head attention mechanism of the Transformer structure, the multimodal fusion of emotional features and motion features is realized. This multimodal fusion method enables the system to effectively capture subtle changes in emotional features, and then generate more delicate motion performances, so that the digital human can convey richer emotional details and enhance the expressiveness of emotion-driven videos.

[0056] 4. In the present invention, gestures and lip movements are automatically generated based on emotional features, without the need for a large amount of manual configuration and adjustment, thereby reducing the need for human intervention. The system reduces the cost and time investment of video production through an automated emotion-driven generation mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a system architecture diagram of the emotion-driven 2D supernatural digital human video generation system based on the multimodal model proposed in the present invention;

[0058] Figure 2 This is a diagram showing the architecture of the speech preprocessing module of the emotion-driven 2D supernatural digital human video generation system of the multimodal model proposed in the present invention;

[0059] Figure 3 This is a diagram showing the architecture of the speech emotion analysis module of the emotion-driven 2D supernatural digital human video generation system of the multimodal model proposed in the present invention;

[0060] Figure 4 This is a multimodal generation module architecture diagram of the emotion-driven 2D supernatural digital human video generation system of the multimodal model proposed in the present invention;

[0061] Figure 5 This is a video generation module architecture diagram of the emotion-driven 2D supernatural digital human video generation system of the multimodal model proposed in the present invention;

[0062] Figure 6 This is a method step diagram for the emotion-driven 2D supernatural digital human video generation method based on the multimodal model proposed in the present invention. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Embodiment 1

[0065] Reference Figure 1-Figure 5 In a first embodiment of the present invention, the present invention provides a multimodal model emotion-driven 2D supernatural digital human video generation system, comprising:

[0066] The voice input module is used to receive the user's voice signal and convert it into a digital signal, which includes:

[0067] A sampling and quantization unit for converting analog voice signals into discrete digital signals;

[0068] A short-time Fourier transform unit is used to convert discrete digital signals into frequency domain features and generate a time-frequency feature matrix;

[0069] A noise reduction unit, for performing noise elimination using a Wiener filter;

[0070] The normalization unit is used to normalize the processed signal to the interval [-1,1].

[0071] Specifically, for the voice input module, the present invention aims to receive the user's voice signal and convert the analog signal into a high-quality digital signal through a series of steps, laying the foundation for subsequent emotional feature extraction and multimodal generation. The voice input module includes a sampling and quantization unit, a short-time Fourier transform unit, a noise reduction unit and a normalization unit. Each unit works together to ensure that signal data with applicability, stability and high signal-to-noise ratio is obtained. Through steps such as sampling, frequency domain transformation and noise elimination, the voice input module converts the original analog signal into a discrete time-frequency feature matrix, and provides a standardized numerical range to improve the robustness of the system.

[0072] In this embodiment, the voice input module first receives the analog voice signal input by the user, and converts the analog signal into a discrete digital signal through the sampling and quantization unit. Specifically, the sampling rate can be selected as 16kHz or 44.1kHz to balance the sound quality and processing efficiency.

[0073] The sampling process discretizes the continuous speech signal into data at a finite number of time points. The quantization step rounds the amplitude of the discrete data to a certain range, such as the 16-bit PCM format, to achieve standardization of the audio data.

[0074] In this embodiment, the sampled and quantized discrete signal is input to a short-time Fourier transform unit (STFT) for conversion to the frequency domain to extract the time-frequency feature matrix. The short-time Fourier transform converts the speech signal into a two-dimensional feature matrix reflecting the frequency variation over time by segmenting the signal on the time axis and performing Fourier transform on each segment.

[0075] The formula for STFT is as follows:

[0076]

[0077] Among them, S[m] is a discrete speech signal, w[nm] is a window function (such as a Hamming window or a Hanning window), M is the window length, and f is the frequency.

[0078] This feature matrix can show the distribution characteristics of the signal in the frequency domain and provide effective information for the extraction of emotional features in subsequent sentiment analysis.

[0079] In this embodiment, the frequency domain feature matrix after short-time Fourier transform may contain interference from various environmental noises. In order to further improve the clarity and stability of the signal, the noise reduction unit uses a Wiener filter to perform noise reduction processing on the signal.

[0080] The Wiener filter optimizes the frequency response of the filter by estimating the power spectral density of the signal and noise, thereby reducing the noise portion of the signal.

[0081] Its filter function is expressed as:

[0082]

[0083] Among them, P S (f) represents the power spectral density of the speech signal, P N (f) represents the power spectral density of the noise. Through this filtering operation, the signal-to-noise ratio of the speech signal is improved, providing a relatively pure signal feature for the subsequent modules.

[0084] In this embodiment, the signal after noise reduction needs to be normalized by a normalization unit to normalize the signal amplitude to the interval [-1, 1].

[0085] Normalization can reduce the amplitude differences between different speech signals and make the data more stable in the model. The normalization process is as follows:

[0086]

[0087] Among them, S min and S max Represent the minimum and maximum amplitudes of the signal, respectively. Through normalization, the dynamic range of the signal is normalized, eliminating unnecessary amplitude differences, and providing a highly consistent input for subsequent emotion feature extraction.

[0088] A speech preprocessing module is used to perform noise elimination and normalization processing on speech signals;

[0089] Specifically, the speech preprocessing module aims to further clean and standardize the original digital signal collected by the speech input module to ensure the accuracy and consistency of subsequent emotional feature extraction. The speech preprocessing module mainly includes two parts: noise elimination and normalization processing. Noise elimination is used to remove background noise and improve signal clarity;

[0090] Normalization is used to standardize the signal amplitude to reduce the amplitude difference of the input data. Through the processing of this module, a purer and more stable signal can be obtained, providing high-quality input data for the subsequent sentiment analysis module.

[0091] In this embodiment, the speech preprocessing module first performs noise elimination processing on the speech signal. In order to eliminate the environmental noise in the speech signal, the present invention adopts a Wiener filter to perform noise suppression. The Wiener filter is a linear filtering method based on minimum mean square error, which optimizes the frequency response of the filter by estimating the power spectral density of the signal and the noise.

[0092] Specifically, the transfer function H(f) of the Wiener filter is expressed as follows:

[0093]

[0094] Among them, P S (f) represents the power spectral density of the speech signal, P N (f) represents the power spectral density of the noise. This formula describes how the filter uses the power spectral density of the signal and the noise to adjust its frequency response so that the main components of the speech signal are retained as much as possible while reducing the noise. In practical applications, P S (f) and P N (f) can be obtained through noise estimation and signal segmentation processing, thereby realizing dynamic filtering.

[0095] In this embodiment, the signal after noise elimination may still have inconsistent amplitudes. To solve this problem, the present invention performs normalization processing on the signal after noise reduction to standardize the signal amplitude to a fixed value range.

[0096] The signal amplitude is normalized to the interval [-1, 1] so that different speech signals have a consistent distribution in amplitude and the amplitude difference between different signals is reduced.

[0097] The normalized calculation formula is as follows:

[0098]

[0099] Among them, S min and S max Respectively represent the minimum and maximum amplitude of the signal, S[n] represents the current signal value, S norm [n] is the normalized signal value. The main purpose of the normalization operation is to make all signals within the same amplitude range, thereby avoiding feature extraction bias caused by signal amplitude differences and providing stable input data for subsequent sentiment analysis.

[0100] The speech emotion analysis module extracts emotion features based on the digital signal of the speech preprocessing module; the emotion features include but are not limited to emotion category, emotion intensity and tone features.

[0101] include:

[0102] A convolutional neural network feature extraction unit is used to receive the spectrum feature matrix from the speech preprocessing module and extract discrete digital signals therefrom. The convolutional neural network feature extraction unit extracts a feature map H by a convolution operation. i ;

[0103] A bidirectional long short-term memory network modeling unit is used to receive the feature map H output by the convolutional neural network feature extraction unit and capture the emotional changes in the speech signal based on time series modeling. The bidirectional long short-term memory network modeling unit obtains the hidden state h by forward and backward time series processing. t ;

[0104] Self-attention mechanism unit is used to fuse the sentiment features extracted by convolutional neural network and bidirectional long short-term memory network.

[0105] Specifically, in this embodiment, the speech emotion analysis module first receives the spectrum feature matrix from the speech preprocessing module through the convolutional neural network feature extraction unit, and extracts discrete digital signals from it. The convolutional neural network feature extraction unit obtains local features through convolution operations to capture short-term emotional information in the spectrum feature matrix. The formula for the convolution operation is as follows:

[0106]

[0107] Among them, F represents the input spectrum feature matrix, W k is the convolution kernel, b is the bias term, and f is the activation function (such as ReLU), which is used to nonlinearly map the input signal to the feature space to obtain the feature map H i This feature map contains the local emotional information in the spectrum and is the basis for subsequent time series modeling.

[0108] In this embodiment, the feature map H output by the convolutional neural network feature extraction unit is input into the bidirectional long short-term memory network (Bi-LSTM) modeling unit to further capture the time series emotional changes in the speech signal. Bi-LSTM can globally capture the characteristics of emotional changes over time by transmitting information in the forward and backward time series.

[0109] The calculation formula of Bi-LSTM is as follows:

[0110] h t =σ(W f ·[h t-1 ,x t ]+b f )

[0111] Among them, W f and b f are the weight matrix and bias term respectively, x t is the input feature vector of the current time step, h t is the hidden state of the current time step, and σ is the activation function (usually sigmoid or tanh). Bidirectional LSTM retains the context information in the time dimension, which can better reflect the dynamic change characteristics of emotions and generate temporal emotion representation.

[0112] In this embodiment, in order to further enhance the significance of emotional features, the output of the bidirectional long short-term memory network modeling unit will enter the self-attention mechanism unit for weighted processing. The self-attention mechanism calculates the attention weights, enabling the model to identify and enhance the emotional features of key time steps, thereby enhancing the expression effect of emotional information. The weight calculation formula of the self-attention mechanism is as follows:

[0113]

[0114] Among them, h t represents the hidden state of Bi-LSTM at time step i, W a and b a are the weight matrix and bias term of the self-attention mechanism, v is the trainable weight vector, α i is the attention weight of time step i, and tanh is the activation function.

[0115] Through the self-attention mechanism, different weights can be given to sentiment features at different time steps, thereby focusing on time segments with significant sentiment information.

[0116] A multimodal generation module is used to map the emotional features into digital human gestures and digital human body movements; the multimodal generation module includes:

[0117] Generative adversarial network generator, used to generate digital human gestures and digital human-type movements based on emotional features;

[0118] The discriminator is used to evaluate the generated actions;

[0119] Multi-head attention mechanism for combining emotional features with action generation;

[0120] The motion consistency optimization unit is used to minimize the motion differences of adjacent time steps through temporal interpolation and consistency constraints;

[0121] The multimodal fusion unit uses the multi-head attention mechanism of the Transformer structure to complete the multimodal fusion of emotional features and action features.

[0122] Specifically, in this embodiment, the generative adversarial network generator first receives the emotion feature matrix from the speech emotion analysis module and maps it into gestures and lip movements that match the emotion. The generative adversarial network (GAN) includes a generator G and a discriminator D.

[0123] The generator generates the action features A of the digital human based on the input emotional feature matrix E to ensure that the action can accurately reflect the input emotional features.

[0124] The loss function of GAN is used to optimize the adversarial relationship between the generator and the discriminator, and its expression is as follows:

[0125]

[0126] Among them, A real represents the real action data, G(E) is the generated action data, and the discriminator D is used to evaluate the authenticity of the generated data. Through the adversarial training of the generator and the discriminator, the generator can gradually generate high-quality action features that match the emotional features.

[0127] In this embodiment, the multi-head attention mechanism is used to combine the emotional features with the action generation process to ensure the temporal consistency of the generated gestures and lip movements. The multi-head attention mechanism can focus on the key information in the emotional feature matrix to ensure that the emotional features can be effectively used when generating actions. The calculation formula of the multi-head attention mechanism is as follows:

[0128] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0129] Among them, each attention head i The calculation is W O is the output transformation matrix, Q, K, and V represent query, key, and value matrices, respectively.

[0130] The multi-head attention mechanism uses multiple attention heads to focus on different emotional features, making the generated actions consistent with subtle changes in the input emotions.

[0131] In this embodiment, the motion consistency optimization unit optimizes the generated motion features through time sequence complementation and consistency constraints to minimize the motion differences between adjacent time steps. Ensure the consistency of the generated motion in time sequence. The unit calculates the difference between the motion frames of adjacent time steps and compensates for it to make the motion transition smoother. The loss function of motion consistency optimization is as follows:

[0132]

[0133] Among them, A t and A t+1 are the action features of adjacent time steps respectively.

[0134] In this embodiment, the multimodal fusion unit uses the multi-head attention mechanism of the Transformer structure to complete the multimodal fusion of emotional features and action features. The Transformer model can further improve the matching degree of emotional features and action features through the self-attention mechanism, ensuring that the emotion-driven action generation is more accurate. The attention calculation formula for multimodal fusion is:

[0135]

[0136] Among them, d k is the scaling factor to ensure the numerical stability of the attention score. Through multimodal fusion, emotional features can be efficiently mapped to action features, so that the final generated gestures and lip shapes can accurately reflect the user's emotional input.

[0137] The video generation module is used to synchronize gestures and lip movements with speech signals and generate 2D digital human videos, including:

[0138] Dynamic Time Warping Synchronization Unit, which is used to align the speech signal with gestures and lip movements through a dynamic time warping algorithm;

[0139] The video encoding and format conversion unit is used to encode, compress and convert the format of the generated 2D digital human video.

[0140] Specifically, in this embodiment, the video generation module first time-aligns the generated gestures and lip movements with the voice signal through a dynamic time warping synchronization unit to ensure the synchronous performance of voice and movement in the video.

[0141] Dynamic Time Warping (DTW) is an algorithm for processing time series data alignment, which can effectively solve the problem of asynchrony between speech signals and action sequences. The DTW algorithm minimizes the cumulative distance between speech signals and action symbols by finding the optimal matching path. The calculation process is as follows:

[0142] D(ij)=d(i,j)+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}

[0143] Among them, D(i,j) represents the cumulative distance, and d(i,j) represents the local distance between time steps i and j. DTW forms the best alignment path by gradually comparing and selecting the minimum cumulative distance to ensure the synchronization of speech and action in the time dimension. Through this process, the system can make the digital human's lip shape and gesture movements consistent with the changes in the speech signal, ensuring the coherence and naturalness of the video.

[0144] In this embodiment, the 2D digital human video aligned by the DTW algorithm needs to be further subjected to video encoding and format conversion to adapt to different playback requirements and device compatibility. The video encoding and format conversion unit uses an efficient video encoding format (such as H.264 or H.265) to compress the video data, control the video data within a reasonable range, and ensure video quality and transmission efficiency. The encoded data can adapt to the playback requirements of multiple resolutions and devices, and meet the usage requirements of different platforms.

[0145] The video encoding process includes the following steps:

[0146] 1. Coding compression: compress and encode the generated video data to reduce the size of the video file while ensuring the clarity and smoothness of the image.

[0147] 2. Format conversion: Convert the encoded video data into a standard format (such as MP4 or AVI) to achieve cross-platform compatibility.

[0148] Through the above steps, the video generation module can output 2D digital human videos with good synchronization and efficient coding compression to provide users with a smooth and natural emotional expression video experience.

[0149] Embodiment 2:

[0150] Reference Figure 6 In a second embodiment of the present invention, the present invention provides a method for generating a 2D supernatural digital human video driven by a multimodal model, comprising the following steps:

[0151] S1, receiving the user's voice signal, converting the analog signal into a discrete digital signal through sampling and quantization;

[0152] Specifically, the user's voice signal is received, and the analog signal is converted into a discrete digital signal through sampling and quantization. First, the user's voice signal is collected through an audio input device such as a microphone. The system samples the analog signal and decomposes the continuous audio signal into a series of discrete point data. After sampling, the system quantizes these discrete points and converts them into a standard numerical form for subsequent digital processing. This step ensures that the voice signal is captured in a format suitable for calculation, providing a basis for subsequent modules.

[0153] S2, performing noise elimination and normalization processing on the digital signal generated by the voice input module, wherein the noise elimination reduces the influence of environmental noise through the Wiener filter, and the normalization processing standardizes the signal within a specific range;

[0154] Specifically, the digital signal generated by the voice input module is subjected to noise elimination and normalization processing to improve the quality and consistency of the signal. During the noise elimination process, the system applies noise reduction techniques such as the Wiener filter to suppress background noise and environmental interference, thereby obtaining a clearer voice signal. Next, the normalization process standardizes the signal amplitude within a specific numerical range, usually in the interval [-1,1], to eliminate the deviation caused by the difference in signal amplitude, so that different voice inputs can obtain consistent processing effects. This step ensures the clarity and stability of the voice signal.

[0155] S3, based on the preprocessed signal output by the speech preprocessing module, using a convolutional neural network and a bidirectional long short-term memory network to extract emotional features, this step includes extracting local emotional information from spectral features, capturing emotional changes in time series through a bidirectional long short-term memory network, and weighting emotional features through a self-attention mechanism to generate an emotional feature matrix including emotional category, emotional intensity and tone features;

[0156] Specifically, based on the preprocessed signal output by the speech preprocessing module, convolutional neural network (CNN) and bidirectional long short-term memory network (Bi-LSTM) are used to extract emotional features. First, the system uses CNN to extract local emotional information from the preprocessed signal and identifies specific emotional patterns in the spectrum through convolution operations. Then, the extracted emotional features are input into Bi-LSTM for time series modeling to capture the changes and trends of emotions over time. Bi-LSTM has the ability to transfer information forward and backward, making the emotional features more comprehensive and accurate. In the last step of processing the emotional features, the emotional features at different time steps are weighted through the self-attention mechanism to further highlight the key emotional information. The emotional feature matrix generated in this step contains emotional categories, emotional intensity and tone features, providing driving input for the generation module.

[0157] S4, inputting the emotion feature matrix generated by the speech emotion analysis module into the multimodal generation module, generating digital human gestures and lip movements matching the emotion features through a generative adversarial network, the multimodal generation module also includes a multi-head attention mechanism and a consistency optimization step, optimizing the timing consistency of the generated gestures and lip movements;

[0158] Specifically, the emotional feature matrix generated by the speech emotion analysis module is input into the multimodal generation module, and the digital human gestures and lip movements that match the emotional features are generated through the generative adversarial network (GAN). The generator generates appropriate actions based on the emotional features, the discriminator evaluates the authenticity of the generated actions, and the generation effect is optimized through adversarial training. In addition, the multimodal generation module contains a multi-head attention mechanism to ensure that the emotional features can be fully utilized when generating actions. It also contains a consistency optimization step to reduce the discontinuity of the generated actions between adjacent time steps through time-series interpolation and constraints. This step ensures that the generated gestures and lip movements are natural and smooth in expression, and highly consistent with the input emotions.

[0159] S5. Synchronize the gestures and lip movements output by the multimodal generation module with the original voice signal and generate a 2D digital human video. The synchronization operation is achieved through a dynamic time warping algorithm to align the voice signal and action time.

[0160] Specifically, the gestures and lip movements output by the multimodal generation module are synchronized with the original voice signal, and the final 2D digital human video is generated. In order to achieve the synchronization of voice and action, the alignment operation is completed through the dynamic time warping (DTW) algorithm. The DTW algorithm matches the action sequence according to the changes in the voice signal, finds the optimal time alignment path, and enables the generated gestures and lip movements to accurately match the voice content. Based on the aligned data, the system further synthesizes the actions and voice into the 2D video. The final generated video contains natural lip and gesture movements, showing a smooth and coordinated emotional expression effect.

[0161] Embodiment three:

[0162] The third embodiment of the present invention is based on the same inventive concept. The present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the emotion-driven 2D supernatural digital human video generation method of the multimodal model of the above-mentioned embodiment.

[0163] Embodiment 4:

[0164] The fourth embodiment of the present invention is based on the same inventive concept. The present invention proposes a computer device, which includes: a processor and a memory; the processor and the memory communicate with each other; the memory is used to store instructions; the processor is used to execute the instructions in the memory to execute the emotion-driven 2D supernatural digital human video generation method of the multimodal model of the above embodiment.

[0165] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0166] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal model-based emotion-driven 2D supernatural digital human video generation system, characterized by: include: A voice input module, used to receive the user's voice signal and convert it into a digital signal; A speech preprocessing module is used to perform noise elimination and normalization processing on speech signals; Speech emotion analysis module, which extracts emotional features based on the digital signal of the speech preprocessing module; A multimodal generation module for mapping emotional features into digital human gestures and digital human-type movements; The video generation module is used to synchronize gestures and lip movements with speech signals and generate 2D digital human videos.

2. The emotion-driven 2D supernatural digital human video generation system of the multimodal model according to claim 1, characterized in that: The emotional features extracted by the speech emotion analysis module include but are not limited to emotion category, emotion intensity and tone features.

3. The emotion-driven 2D supernatural digital human video generation system of the multimodal model according to claim 1, characterized in that: The speech preprocessing module comprises: A sampling and quantization unit for converting analog voice signals into discrete digital signals; A short-time Fourier transform unit is used to convert discrete digital signals into frequency domain features and generate a time-frequency feature matrix; The noise reduction unit is used to eliminate noise using a Wiener filter, and the function of the filter is: Where P S (f) and P N (f) represents the power spectral density of speech and noise respectively; The normalization unit is used to normalize the processed signal to the interval [-1,1].

4. The emotion-driven 2D supernatural digital human video generation system of the multimodal model according to claim 1, characterized in that: The speech emotion analysis module comprises: A convolutional neural network feature extraction unit is used to receive the spectrum feature matrix from the speech preprocessing module and extract discrete digital signals therefrom. The convolutional neural network feature extraction unit extracts a feature map H by a convolution operation. i , and its calculation formula is: Among them, F represents the input spectrum feature matrix, W k represents the convolution kernel weight, b represents the bias term, and f is the activation function; A bidirectional long short-term memory network modeling unit is used to receive the feature map H output by the convolutional neural network feature extraction unit and capture the emotional changes in the speech signal based on time series modeling. The bidirectional long short-term memory network modeling unit obtains the hidden state h by forward and backward time series processing. t , and its calculation formula is: h t =σ(W f ·[h t-1 ,x t ]+b f ) Among them, W f is the time step weight matrix, b f is the bias term, x t is the input feature vector of the current time step, h t represents the hidden state of the current time step, σ represents the activation function, which is used to generate the nonlinear mapping of the emotion sequence; Self-attention mechanism unit is used to fuse the sentiment features extracted by convolutional neural network and bidirectional long short-term memory network.

5. The emotion-driven 2D supernatural digital human video generation system of the multimodal model according to claim 1, characterized in that: The multimodal generation module comprises: Generative adversarial network generator, used to generate digital human gestures and digital human-type movements based on emotional features; The discriminator is used to evaluate the generated actions; Multi-head attention mechanism for combining emotional features with action generation; The motion consistency optimization unit is used to minimize the motion differences of adjacent time steps through temporal interpolation and consistency constraints. Its consistency constraint loss is: Among them A t and A t+1 are action frames at adjacent time steps; The multimodal fusion unit uses the multi-head attention mechanism of the Transformer structure to complete the multimodal fusion of emotional features and action features.

6. The emotion-driven 2D supernatural digital human video generation system of the multimodal model according to claim 1, characterized in that: The video generation module comprises: The dynamic time warping synchronization unit is used to align the speech signal with the gestures and lip movements through the dynamic time warping algorithm. The cumulative distance formula is: D(i,j)=d(i,j)+min{D(i-1,j),D(i,j-1),D(i-1,j-1)} Where D(i,j) is the cumulative distance and d(i,j) is the local distance; The video encoding and format conversion unit is used to encode, compress and convert the format of the generated 2D digital human video.

7. The emotion-driven 2D supernatural digital human video generation system of the multimodal model according to claim 5, characterized in that: The calculation formula of the multi-head attention mechanism adopted by the multimodal fusion unit is: MultiHead(Q,K,V)=Concat(head1,…,head h )W O in, Q, K, V represent query, key and value matrices respectively, and W O is the output transformation matrix.

8. A multimodal model-based emotion-driven 2D supernatural digital human video generation method, characterized in that: An emotion-driven 2D supernatural digital human video generation system for a multimodal model according to any one of claims 1 to 7, the method comprising the following steps: S1, receiving the user's voice signal, converting the analog signal into a discrete digital signal through sampling and quantization; S2, performing noise elimination and normalization processing on the digital signal generated by the voice input module, wherein the noise elimination reduces the influence of environmental noise through the Wiener filter, and the normalization processing standardizes the signal within a specific range; S3, based on the preprocessed signal output by the speech preprocessing module, using a convolutional neural network and a bidirectional long short-term memory network to extract emotional features, this step includes extracting local emotional information from spectral features, capturing emotional changes in time series through a bidirectional long short-term memory network, and weighting emotional features through a self-attention mechanism to generate an emotional feature matrix including emotional category, emotional intensity and tone features; S4, inputting the emotion feature matrix generated by the speech emotion analysis module into the multimodal generation module, generating digital human gestures and lip movements matching the emotion features through a generative adversarial network, the multimodal generation module also includes a multi-head attention mechanism and a consistency optimization step, optimizing the timing consistency of the generated gestures and lip movements; S5. Synchronize the gestures and lip movements output by the multimodal generation module with the original voice signal and generate a 2D digital human video. The synchronization operation is achieved through a dynamic time warping algorithm to align the voice signal and action time.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating emotion-driven 2D supernatural digital human videos based on a multimodal model as described in claim 8 is implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the emotion-driven 2D supernatural digital human video generation method of the multimodal model as described in claim 8.

Images