Digital human construction method and device based on audio and video comparative learning and dual guidance mechanism

Abstract

The invention discloses a digital human construction method and device based on audio and video comparative learning and a dual guidance mechanism. Multi-modal feature extraction and fusion are carried out by constructing an audio and video comparison learning model, combined embedding of audio and video features and emotion dynamic regulation and control are realized, and video frame generation is gradually guided by combining a dual guide mechanism of an emotion guide mechanism and a progressive generation guide mechanism, so that the video frame generation efficiency is improved. And the synchronous loss function ensures that the generated content is highly consistent with the emotion and time sequence of the audio, so that the emotion expression of the digital human is natural and smooth, and the generated video has high resolution and rich details. According to the technology, the emotion consistency and the time sequence synchronism of digital human audio and video data expression can be remarkably improved, and the method has wide application value and practical popularization significance in the fields of virtual anchors, intelligent customer services and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human, and in particular to a method and device for constructing a digital human based on audio and video contrast learning and a dual guidance mechanism. Background Art

[0002] With the rapid development of artificial intelligence technology, Digital Human, as a virtual, human-computer interactive technology product, has gradually been widely used in many fields, especially in entertainment, education, medical care, customer service and other industries. The construction of Digital Human usually involves the processing and combination of audio and video information, and a more interactive and immersive communication experience is achieved by generating a virtual image with natural facial expressions and language synchronization capabilities. However, the current Digital Human technology still faces many technical challenges.

[0003] In the existing digital human construction technology, it is usually necessary to rely on a large amount of audio and video with labeled data for training. This method faces the problems of high data annotation costs and unstable generation quality. The synchronous processing of audio and video data is one of the core challenges in constructing a natural digital human image.

[0004] On the other hand, most methods focus on processing a single modality of audio or video, but lack the deep fusion and comparative learning of multimodal information of audio and video. Although some methods try to introduce comparative learning mechanisms to enhance the relevance of audio and video data, they still fail to effectively consider the timing and emotional synchronization of audio and video content. In terms of multimodal emotional expression, traditional methods often ignore the dynamic changes of emotional information in time series, which leads to the lack of consistency and flexibility of the generated digital human in emotional expression, voice synchronization and dynamic performance, making it difficult to achieve truly "intelligent" interaction.

[0005] Therefore, how to build a digital human technology based on the framework of audio and video comparative learning and combine emotional information to accurately synchronize audio and video data and express emotions naturally has become a key issue in digital human research. Summary of the invention

[0006] Purpose of the invention: In response to the above problems, the present invention provides a method and device for constructing a digital human based on audio-video contrast learning and a dual guidance mechanism. The audio-video contrast learning model is used to effectively extract the joint features of audio and video, enhance the correlation between audio and video data, and further improve the naturalness of the generated results. The emotional embedding is dynamically generated through the emotional guidance mechanism, and the facial expressions of the digital human are adjusted to make the facial expressions consistent with the emotional expression of the speech content, thereby improving the naturalness and consistency of the emotional expression. The progressive generation guidance mechanism makes the generation of digital human facial expressions smoother and more natural, and accurately synchronized with the audio content, ensuring the high quality and emotional consistency of video frame generation, and overcoming the problems of audio and video asynchrony and insufficient emotional expression in traditional methods.

[0007] Technical solution: The present invention proposes a method for constructing a digital human based on audio-video contrast learning and a dual guidance mechanism, comprising the following steps:

[0008] Step 1: Preprocess the audio and video dataset D to obtain the audio dataset A and the video dataset V, and determine the facial key point information V of the first frame of each sample in the video dataset keypoints , the specific method is:

[0009] Step 1.1: Define the audio and video dataset D = {d1, d2, ..., d n}, d i represents the i-th audio and video data sample, where n is the total number of samples, and the variable i∈[1,n];

[0010] Step 1.2: Use the audio processing tool ffmpeg to extract audio and video data i Separate the audio data And save it as WAV format file;

[0011] Step 1.3: Use the Python audio processing library Librosa to process the audio file Perform noise reduction and equalization processing to remove background noise and equalize the volume to obtain noise-reduced audio data

[0012] Step 1.4: The audio data set A is obtained by sampling at a fixed rate of 16 kHz and dividing it into fixed-length audio frames according to a 40 ms time window. 25 audio frames are generated per second, and each frame contains 640 sampling points to ensure that the temporal characteristics of the audio are not lost. n}, a i represents the i-th audio data sample;

[0013] Step 1.5: Use the video processing library OpenCV to process the di Extract frame by frame to get the video frame sequence Indicates that at time step j, i The extracted video sample frames, where the variable j∈[1,T], T represents the length of the time series;

[0014] Step 1.6: Use the Mediapipe model to process each video frame Perform facial key point detection, extract the coordinates of facial key points, align the facial area to the predefined reference frame using the scaling and translation method based on the detected key point coordinates, and standardize the facial area to obtain the standardized video dataset V = {v1,v2,...,v n},v i Represents the i-th preprocessed video data sample, and defines the facial key point information of the first frame of each sample as V keypoints .

[0015] Step 2: For the audio dataset A and the video dataset V, use the audio and video comparison learning model M CL Extract joint feature embedding E; the audio and video contrast learning model M CL Including audio feature extraction network M A , video feature extraction network M V , Multimodal feature fusion module M Fusion and contrastive learning loss module L CL , and train the joint feature embedding generator M Joint , the specific method is:

[0016] Step 2.1: Build the audio feature extraction network M A A convolutional neural network layer is used to extract spectral features of audio dataset A using a 5-layer one-dimensional convolutional network. The convolution kernel size of each layer is 3, the step size is 1, and the number of channels is 32, 64, 128, 256, and 512, respectively. Each convolution layer is connected to a batch normalization layer and a ReLU activation layer to standardize the feature distribution and introduce nonlinearity. The spectral information of the audio signal is captured by convolution layer by layer.

[0017] Step 2.2: Build the audio feature extraction network M A The adaptive emotion weighted multi-head self-attention layer (AEWM-Attention Layer) is used to generate each attention head through linear mapping of query, key and value matrices to capture the long-distance dependencies of audio signals and generate basic attention weights. The generation formulas of query matrix Q, key matrix K and value matrix V are as follows:

[0018]

[0019] Among them, W Q , W K , W V is a learnable weight matrix. Each attention head calculates its own attention weight and multiplies it with the corresponding value matrix V.

[0020] Step 2.3: Output features through the convolutional layer Perform adaptive weight calculation and generate the sentiment weighted vector W through a layer of linear transformation and nonlinear activation function emotion , used to dynamically adjust the sentiment weight and define the sentiment weighted vector W emotion The calculation formula is:

[0021]

[0022] Among them, W e and b e is a learnable parameter, σ is the Sigmoid activation function, which is used to limit the weight value between 0 and 1;

[0023] Step 2.4: Introduce an adaptive sentiment weighting mechanism in the multi-head self-attention layer, using the sentiment weighting vector W from step 2.3 emotion The output of the multi-head self-attention in step 2.2 is weighted, and the calculation formula of weighted attention is defined as:

[0024]

[0025] Among them, d k is the dimension of the key vector, T represents the length of the time series, and the adaptive sentiment weighting is applied to the basic attention weight to generate the adaptive weighted attention representation. This sentiment weighting mechanism can dynamically adjust the attention weight according to the sentiment characteristics of the input, ensuring that the time step with high sentiment intensity occupies a more important position in the output, thereby enhancing the capture of key sentiment information;

[0026] Step 2.5: Define audio feature embedding represents the audio feature vector at the tth time step;

[0027] Step 2.6: Each attention head independently calculates the adaptive emotion weighting, repeating steps 2.3 and 2.4 h times. Different heads can capture different emotion changes and temporal dependencies. Then concatenate the outputs of all attention heads to form an audio feature embedding:

[0028] E A =Concat(AEWM-Attention1,...,AEWM-Attentionh )W O

[0029] Where h is the number of attention heads, W O is the final linear transformation matrix;

[0030] Step 2.7: Build the video feature extraction network M V , use 4 layers of 3D convolutional layers to process the video dataset V. The size of each convolution kernel is 3×3×3, the step size is 1×1×1, and the number of channels is 32, 64, 128 and 256 respectively. The batch normalization and ReLU activation function are connected after the convolution layer to extract the spatial and temporal features between video frames. The 3D convolutional layer slides in the time dimension to capture the dynamic changes of facial expressions and head postures in the video, ensuring the continuity of video features in time sequence, and obtaining the video feature embedding sequence Where T represents the length of the time series, each It is a multidimensional feature vector, representing the spatial features and dynamic information of the video at time step t;

[0031] Step 2.8: Introduce the Temporal Emotional Self-supervised Module (TESS) to capture the emotional changes in the video through contrastive learning without explicit emotional labels and select features at time step t The feature at the adjacent time step t+1 is a positive sample pair, indicating a time step with similar emotions. The feature at time step t is selected The features of random distant time steps t+k (k>5) are used as negative sample pairs to represent time steps with significant emotional differences, and the contrast loss function is defined as:

[0032]

[0033] Where d() represents the Euclidean distance of the feature distance measurement function, i and j represent positive sample pairs between adjacent time steps, and j=i+1, i and k represent negative sample pairs with a long time interval, k≠i and |ki|>Δ, Δ is the preset minimum time interval, N represents the total number of all positive and negative sample pairs, and the contrast loss function is used to minimize the feature distance of time steps with similar emotions and increase the distance between time steps with significant emotional differences, thereby generating video feature embedding containing emotional information. The video feature vector representing the t-th time step, where T represents the length of the time series, is aligned with the audio features on the time axis. This embedding contains the temporal features of facial expressions and postures, and achieves preliminary annotation of emotional information through the emotional self-supervision module;

[0034] Step 2.9: Construct multimodal feature fusion module MFusion , embed audio and video features into E A and E V The input is input into the bidirectional long short-term memory network to capture the bidirectional dependency of the input features in the time dimension and ensure that the audio and video features are synchronized in time; wherein, the hidden layer dimension of the bidirectional long short-term memory network is set to 512, and a preliminary aligned audio and video feature sequence is generated through bidirectional processing, and a multi-head self-attention mechanism is connected to further enhance the interaction and correlation between audio and video cross-modal settings. The self-attention layer calculates the weight of each time step to generate a weighted joint feature embedding to achieve deep fusion of multimodal features. The feature sequence after self-attention fusion is input into the multilayer perceptron, which contains 3 linear layers, and the output dimension of each layer is 1024. The ReLU activation function is used to map the fused features to a high-dimensional space to enhance the expression ability of the joint feature embedding, and finally the multimodal joint feature embedding E={e 1 ,e 2 ,...,e T},e t represents the joint feature vector of the tth time step, where T represents the length of the time series;

[0035] Step 2.10: Define the audio and video feature pairs at the same time step is a positive sample pair, an audio and video feature pair at different time steps is a negative sample pair, where j≠i and satisfies |ji|>Δ, Δ is the preset minimum time interval, and the contrast loss function L is defined. CL The calculation formula is as follows:

[0036]

[0037] Where y is the sample label, y=1 for positive sample pairs, y=0 for negative sample pairs, N is the total number of sample pairs, d is the feature distance metric function, cosine distance is used to calculate the similarity between audio and video feature embeddings, and the margin parameter α is set to ensure that the distance between negative sample pairs is greater than that between positive sample pairs;

[0038] The loss L CL Propagate to the audio feature extraction network M through the back propagation algorithm A Video feature extraction network M V And multimodal feature fusion module M Fusion , and update the parameters of each network, using the stochastic gradient descent optimization algorithm, and perform multiple iterations of training until the contrast loss converges to ensure the optimal correlation of the audio and video feature embedding E;

[0039] Step 2.12: Build the joint feature embedding generator M Joint , receiving only the audio feature embedding EA As input, multiplex M Fusion The network structure generates joint feature embedding To make E pred As close to M as possible Fusion The generated true joint feature embedding E defines the reconstruction loss L cond The calculation formula is as follows:

[0040]

[0041] Among them, T represents the length of the time series, and the mean square error is used to measure the generated embedding E pred The difference from the true embedding E is calculated during training by minimizing the reconstruction loss L cond Optimize the joint feature embedding generator M Joint Learn how to transform audio features E A Transformed into an embedding E with joint representation capability pred , for the inference phase.

[0042] Step 3: Based on the joint feature embedding E, the audio and video performance of the digital human is generated through a dual guidance mechanism, which includes an emotional guidance mechanism M emotion and the progressive generation boot mechanism M progress ; Among them, the emotional guidance mechanism M emotion Dynamically generate emotion embedding Z by joint feature embedding E to adjust the facial expression of digital human; progressive generation guidance mechanism M progress Based on the facial key point information V of the first frame keypoints , Joint feature embedding E and emotion embedding Z, progressively generate video frames V gen , and through the joint loss L total Feedback optimization is performed to ensure that the generated video is in sync with the audio in terms of temporal content and emotion by:

[0043] Step 3.1: Construct the emotional guidance mechanism M emotion , generate sentiment embedding Z from the joint feature embedding E, and dynamically regulate sentiment embedding in time series; first, embed the joint feature e at each time step t Perform mean variance normalization and transform the feature e of each time step t Convert to standardized feature e t norm ; Then, the emotion embedding is generated for each time step through the emotion embedding encoding layer, including 2 fully connected layers, each layer is connected to the ReLU activation function Forming sentiment embedding sequence Finally, the dynamic emotion regulation layer is used to capture the emotion changes in the time series, and the single-layer gated recurrent unit GRU structure is used to embed the emotion Z initModeling on time series, and obtaining the dynamically adjusted sentiment embedding Z = {z 1 ,z 2 ,...,z T};

[0044] Step 3.2: Define the sentiment consistency loss function as:

[0045]

[0046] Where T is the length of the time series, z t and z t+1 represents the dynamically regulated emotion embedding at adjacent time steps t and t+1. During the training process, the back propagation algorithm and optimization algorithm are used to optimize the emotion guidance mechanism M. emotion Update the parameters to reduce the emotional consistency loss L emotion_consistency Until the loss converges, make M emotion Ability to generate Z with smooth transition in time series, ensuring that the subsequently generated video frames have natural fluidity in emotional expression;

[0047] Step 3.3: Construct the progressive generation boot mechanism M progress , the specific construction steps are as follows:

[0048] Step 3.3.1: The initial generation module receives the facial key point information V of the first frame keypoints and the initial feature e of the joint feature embedding E 1 , generate the basic feature map F0; including two layers of convolution operations, the convolution kernel sizes are 5×5 and 3×3 respectively, followed by a layer of ReLU activation function, F0 represents the basic structure of the face, which is used for the gradual refinement of the subsequent generation layer. The calculation formula is as follows:

[0049] F0=σ(W init *V keypoints +W E ·e 1 +b init )

[0050] Among them, W init is the convolution weight matrix, W E The linear transformation matrix of the joint feature embedding E is used to embed the joint feature into e 1 Mapped to a spatial dimension that matches the feature map, b init is the bias term, σ is the ReLU activation function;

[0051] Step 3.3.2: Construct an emotion-driven dynamic feature fusion module. At each time step t(t∈[1,T]), the network recursively accepts the input of the current time step, including the emotion embedding z t , Joint feature embeddingt and the feature map F generated at the previous time step t-1 , use a 3×3 convolution kernel to t-1 Perform convolution processing to extract spatial features and obtain the intermediate feature map F t conv , for sentiment embedding z t and joint feature embedding e t Perform element-by-element dot product operation to obtain the interaction feature z t ⊙e t , through a layer of linear transformation to capture z t and e t The interaction relationship between them is calculated through softmax regularization. t ,Right now

[0052] α t =softmax(W α [z t ⊙e t ]+b α )

[0053] Among them, W α is the linear transformation matrix, b α is a bias term, which takes the attention weight α t Acting on the joint feature embedding e t We obtain the joint features adjusted by attention Where W e is the linear transformation matrix, b t As the bias term, the adjusted emotional feature F t att With F t conv Add them together to get the feature map adjusted by the emotional information. Finally, a layer of ReLU activation function is used to introduce nonlinearity to enhance the influence of emotional information on the generated features, and the feature map F of the current time step is output. t ,Right now

[0054] F t =σ(W conv *F t-1 +α t ·W e e t +b t )

[0055] Among them, W conv is the weight matrix of the convolutional layer, and each time step is processed recursively until the feature graph sequence {F1, F2, ..., F T};

[0056] Step 3.3.3: Construct a detail generation module to generate the feature map F at each time step t t Detail enhancement processing is performed, including multiple emotional attention residual blocks (EARB). Each residual block contains a multi-scale convolution structure and an emotional attention mechanism. The multi-scale convolution structure contains three parallel convolution layers with convolution kernel sizes of 3×3, 5×5, and 7×7, respectively. t Perform multi-scale convolution processing to obtain a multi-scale feature map output F t 3×3 、F t 5×5 、F t 7×7 Then, in order to dynamically adjust the fusion weight of the multi-scale feature map, according to the emotion embedding z at the current time step t Generate sentiment adaptive weight vector The calculation formula is:

[0057]

[0058] Among them, W z is the linear transformation matrix, b z As the bias term, the multi-scale feature map is weightedly fused using the sentiment weight vector to obtain the weighted feature map:

[0059]

[0060] After weighted fusion, the input feature map F t Perform 1×1 convolution and connect the adjusted input feature map F through residual connection t And the fused multi-scale feature map Add together to get the enhanced detail feature map:

[0061]

[0062] Step 3.3.4: Construct a high-resolution generation module, and gradually improve the resolution while retaining details through staged upsampling and step-by-step channel downsampling operations; first, use a convolution layer with a convolution kernel size of 3×3 for convolution operation, then perform an upsampling layer Pixel Shuffle operation; then use a convolution layer with a convolution kernel size of 3×3 for feature extraction, perform an upsampling operation again, and finally use a 3×3 convolution for convolution operation, and finally output a video frame sequence V gen = {v 1 ,v 2 ,...,v T};

[0063] Step 3.3.5: Use the video feature extraction network M V For the generated video frame sequence V gen Perform feature extraction to generate video feature embedding And calculate the feature difference of adjacent time steps, defined as For the audio feature extraction network M A Extracted audio features embedding Calculate the feature difference of its adjacent time steps, defined as Based on the time step difference of audio features and the time step difference of generated video features, the audio and video synchronization loss function L is defined sync_loss The calculation formula is as follows:

[0064]

[0065] Where T represents the length of the time series, by minimizing the audio and video synchronization loss L sync_loss , ensuring that the generated video frame sequence is synchronized with the dynamic changes of the audio signal in time sequence, so that the generated video is consistent with the emotion and rhythm of the audio in visual expression;

[0066] Step 3.4: Define the joint loss function L total =λ1L emotion_consistency +λ2L sync_loss , where λ1 and λ2 are hyperparameters used to adjust the weights of emotional consistency and audio-video synchronization. total Training progressive generation guidance mechanism M progress , gradually adjust the generated video frames to ensure that the generated video frame sequence meets the requirements of the input audio in terms of emotional expression and time synchronization.

[0067] Step 4: For the audio and video data A that needs to be inferred inf and video or image data V inf Perform preprocessing to obtain audio frame sequence A pre and standardized facial key point data The specific method is:

[0068] Step 4.1: Define the single audio data that needs to be inferred as A inf , the single video or single picture data that needs to be inferred is V inf ;

[0069] Step 4.2: Use the Python audio processing library Librosa to process audio A infPerform noise reduction and equalization processing to remove background noise and equalize the volume. Perform fixed sampling at a sampling rate of 16kHz and divide it into a fixed-length audio frame sequence according to a 40ms time window. Generate 25 audio frames per second, each frame contains 640 sampling points, and obtain the preprocessed audio frame sequence A pre ={a 1 ,a 2 ,...,a T}, a t represents the audio frame at the tth time step, where T is the length of the time series;

[0070] Step 4.3: Use the Mediapipe model to detect facial key points on the first frame of the image or video, extract the coordinates of the facial key points, align the facial area to the predefined reference frame using the scaling and translation method based on the detected key point coordinates, standardize the facial area, and generate standardized key point data

[0071] Step 5: Based on the audio frame sequence A pre And the initial facial key point data Using the joint feature embedding generator M Joint , Emotional Guidance Mechanism M emotion and the progressive generation boot mechanism M progress Generate video frame sequence and with the audio frame sequence A pre Merge and finally output synchronized audio and video data The specific method is:

[0072] Step 5.1: Place A pre Enter the audio feature extraction network M A Extract audio feature embedding represents the audio feature vector at the tth time step, and T represents the length of the time series;

[0073] Step 5.2: Input to the trained joint feature embedding generator M Joint , generating joint feature embeddings during inference

[0074] Step 5.3: E pred Input to the emotion guidance mechanism M emotion , generate the corresponding sentiment embedding sequence Z inf ={z 1 ,z 2 ,...,z T};

[0075] Step 5.4: Standardize the key point data Joint feature embedding E pred and sentiment embedding sequence Z inf Input to the progressive generation guidance mechanism M progress , gradually generates a video frame sequence that matches the emotion and audio content; among them, the facial key points of the initial frame Used as the basis for facial geometry layout, the joint feature embedding E pred and sentiment embedding sequence Z inf Guided generation of video frame sequences

[0076] Step 5.5: Generate a continuous video frame sequence With audio frame sequence A pre ={a 1 ,a 2 ,...,a T Merge frames by time step to ensure that the audio and video frames are strictly corresponding on the timeline. The video encoding format is H.264 and the audio encoding format is AAC. Merge to generate a single video file in MP4 format and output video content containing synchronized audio.

[0077] The present invention also discloses a digital human construction device based on audio and video contrast learning and dual-guidance mechanism, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the digital human construction method based on audio and video contrast learning and dual-guidance mechanism is implemented.

[0078] Beneficial effects:

[0079] The present invention extracts the joint features of audio and video through an audio-video contrast learning model, and through a multimodal feature fusion module, the audio and video features are deeply fused, further improving the relevance of audio and video data, ensuring high-precision synchronization of audio and video content, and solving the problem of audio and video asynchrony in the prior art, making the generated digital human more natural and consistent in voice and facial expression. Secondly, the emotion guidance mechanism can dynamically generate emotion embeddings, and adjust the facial expressions of the digital human to be consistent with the emotions in the audio in real time, enhancing the coherence and naturalness of emotional expression. Finally, the progressive generation guidance mechanism makes the facial expression generation process smoother and more natural, ensures the synchronization of emotions and timing, and improves the quality of the generated video. In summary, the present invention provides an efficient, accurate and emotionally consistent digital human generation method, which significantly improves the naturalness and interactivity of digital human technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is the overall flow chart of the present invention;

[0081] Figure 2This is the architecture diagram of the audio and video comparison learning model;

[0082] Figure 3 This is the architecture diagram of the dual-guidance mechanism model;

[0083] Figure 4 It is a specific implementation flow chart of the present invention. DETAILED DESCRIPTION

[0084] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0085] The present invention discloses a method and device for constructing a digital human based on audio-video contrast learning and a dual guidance mechanism, comprising the following steps:

[0086] Step 1: Preprocess the audio and video dataset D to obtain the audio dataset A and the video dataset V. The specific method is as follows:

[0087] Step 1: Preprocess the audio and video dataset D to obtain the audio dataset A and the video dataset V, and determine the facial key point information V of the first frame of each sample in the video dataset keypoints , the specific method is:

[0088] Step 1.1: Define the audio and video dataset D = {d1, d2, ..., d n}, d i Represents the i-th audio and video data sample, where n is the total number of samples, and the variable i∈[1,n].

[0089] Step 1.2: Use the audio processing tool ffmpeg to extract audio and video data i Separate the audio data And save as WAV format file.

[0090] Step 1.3: Use the Python audio processing library Librosa to process the audio file Perform noise reduction and equalization processing to remove background noise and equalize the volume to obtain noise-reduced audio data

[0091] Step 1.4: The audio data set A is obtained by sampling at a fixed rate of 16 kHz and dividing it into fixed-length audio frames according to a 40 ms time window. 25 audio frames are generated per second, and each frame contains 640 sampling points to ensure that the temporal characteristics of the audio are not lost. n}, ai represents the i-th audio data sample.

[0092] Step 1.5: Use the video processing library OpenCV to process the d i Extract frame by frame to get the video frame sequence Indicates that at time step j, from d i Extracted video sample frames, where variable j∈[1,T], T represents the length of the time series.

[0093] Step 1.6: Use the Mediapipe model to process each video frame Perform facial key point detection, extract the coordinates of facial key points, align the facial area to the predefined reference frame using the scaling and translation method based on the detected key point coordinates, and standardize the facial area to obtain the standardized video dataset V = {v1,v2,...,v n},v i Represents the i-th preprocessed video data sample, and defines the facial key point information of the first frame of each sample as V keypoints .

[0094] Step 2: For the audio dataset A and the video dataset V, use the audio and video comparison learning model M CL Extract joint feature embedding E; the audio and video contrast learning model M CL Including audio feature extraction network M A , video feature extraction network M V , Multimodal feature fusion module M Fusion and contrastive learning loss module L CL , and train the joint feature embedding generator M Joint , the specific method is:

[0095] Step 2.1: Build the audio feature extraction network M A A convolutional neural network layer is used to extract spectral features of audio dataset A using a 5-layer one-dimensional convolutional network. The convolution kernel size of each layer is 3, the step size is 1, and the number of channels is 32, 64, 128, 256, and 512, respectively. Each convolution layer is connected to a batch normalization layer and a ReLU activation layer to standardize the feature distribution and introduce nonlinearity. The spectral information of the audio signal is captured by convolution layer by layer.

[0096] Step 2.2: Build the audio feature extraction network M AThe adaptive emotion weighted multi-head self-attention layer (AEWM-AttentionLayer) is used to generate each attention head through linear mapping of query, key and value matrices to capture the long-distance dependencies of audio signals and generate basic attention weights. The generation formulas of query matrix Q, key matrix K and value matrix V are as follows:

[0097]

[0098] Among them, W Q , W K , W V is a learnable weight matrix. Each attention head calculates its own attention weight and multiplies it with the corresponding value matrix V.

[0099] Step 2.3: Output features through the convolutional layer Perform adaptive weight calculation and generate the sentiment weighted vector W through a layer of linear transformation and nonlinear activation function emotion , used to dynamically adjust the sentiment weight and define the sentiment weighted vector W emotion The calculation formula is:

[0100]

[0101] Among them, W e and b e is a learnable parameter, and σ is the Sigmoid activation function, which is used to limit the weight value between 0 and 1.

[0102] Step 2.4: Introduce an adaptive sentiment weighting mechanism in the multi-head self-attention layer, using the sentiment weighting vector W from step 2.3 emotion The output of the multi-head self-attention in step 2.2 is weighted, and the calculation formula of weighted attention is defined as:

[0103]

[0104] Among them, d k is the dimension of the key vector, T represents the length of the time series, and the adaptive sentiment weighting is applied to the basic attention weight to generate an adaptive weighted attention representation. This sentiment weighting mechanism can dynamically adjust the attention weight according to the sentiment characteristics of the input, ensuring that the time step with high sentiment intensity occupies a more important position in the output, thereby enhancing the capture of key sentiment information.

[0105] Step 2.5: Define audio feature embedding Represents the audio feature vector at the t-th time step.

[0106] Step 2.6: Each attention head independently calculates the adaptive emotion weighting, repeating steps 2.3 and 2.4 h times. Different heads can capture different emotion changes and temporal dependencies. Then concatenate the outputs of all attention heads to form an audio feature embedding:

[0107] E A =Concat(AEWM-Attention1,...,AEWM-Attention h )W O

[0108] Where h is the number of attention heads, W O is the final linear transformation matrix.

[0109] Step 2.7: Build the video feature extraction network M V , use 4 layers of 3D convolutional layers to process the video dataset V. The size of each convolution kernel is 3×3×3, the step size is 1×1×1, and the number of channels is 32, 64, 128 and 256 respectively. The batch normalization and ReLU activation function are connected after the convolution layer to extract the spatial and temporal features between video frames. The 3D convolutional layer slides in the time dimension to capture the dynamic changes of facial expressions and head postures in the video, ensuring the continuity of video features in time sequence, and obtaining the video feature embedding sequence Where T represents the length of the time series, each It is a multi-dimensional feature vector, representing the spatial features and dynamic information of the video at time step t.

[0110] Step 2.8: Introduce the Temporal Emotional Self-supervised Module (TESS) to capture the emotional changes in the video through contrastive learning without explicit emotional labels and select features at time step t The feature at the adjacent time step t+1 is a positive sample pair, indicating a time step with similar emotions. The feature at time step t is selected The features of random distant time steps t+k (k>5) are used as negative sample pairs to represent time steps with significant emotional differences, and the contrast loss function is defined as:

[0111]

[0112] Where d() represents the Euclidean distance of the feature distance measurement function, i and j represent positive sample pairs between adjacent time steps, and j=i+1, i and k represent negative sample pairs with a long time interval, k≠i and |ki|>Δ, Δ is the preset minimum time interval, N represents the total number of all positive and negative sample pairs, and the contrast loss function is used to minimize the feature distance of time steps with similar emotions and increase the distance between time steps with significant emotional differences, thereby generating video feature embedding containing emotional information. Represents the video feature vector at the tth time step, where T represents the length of the time series, which is aligned with the audio features on the time axis. This embedding contains the temporal features of facial expressions and postures, and realizes the preliminary annotation of emotional information through the emotional self-supervision module.

[0113] Step 2.9: Construct multimodal feature fusion module M Fusion , embed audio and video features into E A and E V The input is input into the bidirectional long short-term memory network to capture the bidirectional dependency of the input features in the time dimension and ensure that the audio and video features are synchronized in time; wherein, the hidden layer dimension of the bidirectional long short-term memory network is set to 512, and a preliminary aligned audio and video feature sequence is generated through bidirectional processing, and a multi-head self-attention mechanism is connected to further enhance the interaction and correlation between audio and video cross-modal settings. The self-attention layer calculates the weight of each time step to generate a weighted joint feature embedding to achieve deep fusion of multimodal features. The feature sequence after self-attention fusion is input into the multilayer perceptron, which contains 3 linear layers, and the output dimension of each layer is 1024. The ReLU activation function is used to map the fused features to a high-dimensional space to enhance the expression ability of the joint feature embedding, and finally the multimodal joint feature embedding E={e 1 ,e 2 ,...,e T},e t represents the joint feature vector at the t-th time step, where T represents the length of the time series.

[0114] Step 2.10: Define the audio and video feature pairs at the same time step is a positive sample pair, an audio and video feature pair at different time steps is a negative sample pair, where j≠i and satisfies |ji|>Δ, Δ is the preset minimum time interval, and the contrast loss function L is defined. CL The calculation formula is as follows:

[0115]

[0116] Where y is the sample label, y=1 for positive sample pairs, y=0 for negative sample pairs, N is the total number of sample pairs, d is the feature distance metric function, cosine distance is used to calculate the similarity between audio and video feature embeddings, and the margin parameter α is set to ensure that the distance between negative sample pairs is greater than that between positive sample pairs;

[0117] The loss L CL Propagate to the audio feature extraction network M through the back propagation algorithm A Video feature extraction network M V And multimodal feature fusion module M Fusion , and update the parameters of each network, using the stochastic gradient descent optimization algorithm, and perform multiple iterative training until the contrast loss converges to ensure the optimal correlation of the audio and video feature embedding E.

[0118] Step 2.12: Build the joint feature embedding generator M Joint , receiving only the audio feature embedding E A As input, multiplex M Fusion The network structure generates joint feature embedding To make E pred As close to M as possible Fusion The generated true joint feature embedding E defines the reconstruction loss L cond The calculation formula is as follows:

[0119]

[0120] Among them, T represents the length of the time series, and the mean square error is used to measure the generated embedding E pred The difference from the true embedding E is calculated during training by minimizing the reconstruction loss L cond Optimize the joint feature embedding generator M Joint Learn how to transform audio features E A Transformed into an embedding E with joint representation capability pred , for the inference phase.

[0121] Step 3: Based on the joint feature embedding E, the audio and video performance of the digital human is generated through a dual guidance mechanism, which includes an emotional guidance mechanism M emotion and the progressive generation boot mechanism M progress ; Among them, the emotional guidance mechanism M emotion Dynamically generate emotion embedding Z by joint feature embedding E to adjust the facial expression of digital human; progressive generation guidance mechanism M progress Based on the facial key point information V of the first frame keypoints , Joint feature embedding E and emotion embedding Z, progressively generate video frames V gen , and through the joint loss L totalFeedback optimization is performed to ensure that the generated video is in sync with the audio in terms of temporal content and emotion by:

[0122] Step 3.1: Construct the emotional guidance mechanism M emotion , generate sentiment embedding Z from the joint feature embedding E, and dynamically regulate sentiment embedding in time series; first, embed the joint feature e at each time step t Perform mean variance normalization and transform the feature e of each time step t Convert to standardized feature e t norm ; Then, the emotion embedding is generated for each time step through the emotion embedding encoding layer, including 2 fully connected layers, each layer is connected to the ReLU activation function Forming sentiment embedding sequence Finally, the dynamic emotion regulation layer is used to capture the emotion changes in the time series, and the single-layer gated recurrent unit GRU structure is used to embed the emotion Z init Modeling on time series, and obtaining the dynamically adjusted sentiment embedding Z = {z 1 ,z 2 ,...,z T}.

[0123] Step 3.2: Define the sentiment consistency loss function as:

[0124]

[0125] Where T is the length of the time series, z t and z t+1 represents the dynamically regulated emotion embedding at adjacent time steps t and t+1. During the training process, the back propagation algorithm and optimization algorithm are used to optimize the emotion guidance mechanism M. emotion Update the parameters to reduce the emotional consistency loss L emotion_consistency Until the loss converges, make M emotion It is able to generate Z with smooth transitions in time series, ensuring that the subsequently generated video frames have natural fluidity in emotional expression.

[0126] Step 3.3: Construct the progressive generation boot mechanism M progress , the specific construction steps are as follows:

[0127] Step 3.3.1: The initial generation module receives the first frame facial key point information V extracted in step 1.6 keypoints and the initial feature e of the joint feature embedding E 1 , generate the basic feature map F0; including two layers of convolution operations, the convolution kernel sizes are 5×5 and 3×3 respectively, followed by a layer of ReLU activation function, F0 represents the basic structure of the face, which is used for the gradual refinement of the subsequent generation layer. The calculation formula is as follows:

[0128] F0=σ(W init *V keypoints +W E ·e 1 +b init )

[0129] Among them, W init is the convolution weight matrix, W E The linear transformation matrix of the joint feature embedding E is used to embed the joint feature into e 1 Mapped to a spatial dimension that matches the feature map, b init is the bias term, and σ is the ReLU activation function.

[0130] Step 3.3.2: Construct an emotion-driven dynamic feature fusion module. At each time step t(t∈[1,T]), the network recursively accepts the input of the current time step, including the emotion embedding z t , Joint feature embedding t and the feature map F generated at the previous time step t-1 , use a 3×3 convolution kernel to t-1 Perform convolution processing to extract spatial features and obtain the intermediate feature map F t conv , for sentiment embedding z t and joint feature embedding e t Perform element-by-element dot product operation to obtain the interaction feature z t ⊙e t , through a layer of linear transformation to capture z t and e t The interaction relationship between them is calculated through softmax regularization. t ,Right now

[0131] α t =softmax(W α [z t ⊙e t ]+b α )

[0132] Among them, W α is the linear transformation matrix, b α is a bias term, which takes the attention weight α t Acting on the joint feature embedding e t We get the joint feature F after attention adjustment. t att =α t ·W e e t +b t , where W e is the linear transformation matrix, bt As the bias term, the adjusted emotional feature F t att With F t conv Add them together to get the feature map adjusted by the emotional information. Finally, a layer of ReLU activation function is used to introduce nonlinearity to enhance the influence of emotional information on the generated features, and the feature map F of the current time step is output. t ,Right now

[0133] F t =σ(W conv *F t-1 +α t ·W e e t +b t )

[0134] Among them, W conv is the weight matrix of the convolutional layer, and each time step is processed recursively until the feature graph sequence {F1, F2, ..., F T}.

[0135] Step 3.3.3: Construct a detail generation module to generate the feature map F at each time step t t Detail enhancement processing is performed, including multiple emotional attention residual blocks (EARB). Each residual block contains a multi-scale convolution structure and an emotional attention mechanism. The multi-scale convolution structure contains three parallel convolution layers with convolution kernel sizes of 3×3, 5×5, and 7×7, respectively. t Perform multi-scale convolution processing to obtain a multi-scale feature map output F t 3×3 、F t 5×5 、F t 7×7 Then, in order to dynamically adjust the fusion weight of the multi-scale feature map, according to the emotion embedding z at the current time step t Generate sentiment adaptive weight vector The calculation formula is:

[0136]

[0137] Among them, W z is the linear transformation matrix, b z As the bias term, the multi-scale feature map is weightedly fused using the sentiment weight vector to obtain the weighted feature map:

[0138]

[0139] After weighted fusion, the input feature map F t Perform 1×1 convolution and connect the adjusted input feature map F through residual connection t And the fused multi-scale feature map Add together to get the enhanced detail feature map:

[0140]

[0141] Step 3.3.4: Construct a high-resolution generation module, and gradually improve the resolution while retaining details through staged upsampling and step-by-step channel downsampling operations; first, use a convolution layer with a convolution kernel size of 3×3 for convolution operation, then perform an upsampling layer Pixel Shuffle operation; then use a convolution layer with a convolution kernel size of 3×3 for feature extraction, perform an upsampling operation again, and finally use a 3×3 convolution for convolution operation, and finally output a video frame sequence V gen = {v 1 ,v 2 ,...,v T}.

[0142] Step 3.3.5: Use the video feature extraction network M V For the generated video frame sequence V gen Perform feature extraction to generate video feature embedding And calculate the feature difference of adjacent time steps, defined as For the audio feature extraction network M A Extracted audio features embedding Calculate the feature difference of its adjacent time steps, defined as Based on the time step difference of audio features and the time step difference of generated video features, the audio and video synchronization loss function L is defined sync_loss The calculation formula is as follows:

[0143]

[0144] Where T represents the length of the time series, by minimizing the audio and video synchronization loss L sync_loss , ensuring that the generated video frame sequence is synchronized with the dynamic changes of the audio signal in time sequence, so that the generated video is consistent with the emotion and rhythm of the audio in visual expression.

[0145] Step 3.4: Define the joint loss function L total =λ1L emotion_consistency +λ2L sync_loss , where λ1 and λ2 are hyperparameters used to adjust the weights of emotional consistency and audio-video synchronization. total Training progressive generation guidance mechanism M progress, gradually adjust the generated video frames to ensure that the generated video frame sequence meets the requirements of the input audio in terms of emotional expression and time synchronization.

[0146] Step 4: For the audio and video data A that needs to be inferred inf and video or image data V inf Perform preprocessing to obtain audio frame sequence A pre and standardized facial key point data The specific method is:

[0147] Step 4.1: Define the single audio data that needs to be inferred as A inf , the single video or single picture data that needs to be inferred is V inf .

[0148] Step 4.2: Use the Python audio processing library Librosa to process audio A inf Perform noise reduction and equalization processing to remove background noise and equalize the volume. Perform fixed sampling at a sampling rate of 16kHz and divide it into a fixed-length audio frame sequence according to a 40ms time window. Generate 25 audio frames per second, each frame contains 640 sampling points, and obtain the preprocessed audio frame sequence A pre ={a 1 ,a 2 ,...,a T}, a t Represents the audio frame at the tth time step, where T is the length of the time series.

[0149] Step 4.3: Use the Mediapipe model to detect facial key points on the first frame of the image or video, extract the coordinates of the facial key points, align the facial area to the predefined reference frame using the scaling and translation method based on the detected key point coordinates, standardize the facial area, and generate standardized key point data

[0150] Step 5: Based on the audio frame sequence A pre And the initial facial key point data Using the joint feature embedding generator M Joint , Emotional Guidance Mechanism M emotion and the progressive generation boot mechanism M progress Generate video frame sequence and with the audio frame sequence A pre Merge and finally output synchronized audio and video data The specific method is:

[0151] Step 5.1: Place A pre Enter the audio feature extraction network M A Extract audio feature embedding represents the audio feature vector at the tth time step, and T represents the length of the time series.

[0152] Step 5.2: Input to the trained joint feature embedding generator M Joint , generating joint feature embeddings during inference

[0153] Step 5.3: E pred Input to the emotion guidance mechanism M emotion , generate the corresponding sentiment embedding sequence Z inf ={z 1 ,z 2 ,...,z T}.

[0154] Step 5.4: Standardize the key point data Joint feature embedding E pred and sentiment embedding sequence Z inf Input to the progressive generation guidance mechanism M progress , gradually generates a video frame sequence that matches the emotion and audio content; among them, the facial key points of the initial frame Used as the basis for facial geometry layout, the joint feature embedding E pred and sentiment embedding sequence Z inf Guided generation of video frame sequences

[0155] Step 5.5: Generate a continuous video frame sequence With audio frame sequence A pre ={a 1 ,a 2 ,...,a T Merge frames by time step to ensure that the audio and video frames are strictly corresponding on the timeline. The video encoding format is H.264 and the audio encoding format is AAC. Merge to generate a single video file in MP4 format and output video content containing synchronized audio.

[0156] The present invention also discloses a digital human construction device based on audio and video contrast learning and dual-guidance mechanism, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. The above-mentioned digital human construction method based on audio and video contrast learning and dual-guidance mechanism is implemented when the computer program is loaded into the processor.

[0157] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism, characterized in that: The steps include: Step 1: Preprocess the audio and video dataset D to obtain the audio dataset A and the video dataset V, and determine the facial key point information V of the first frame of each sample in the video dataset keypoints ; Step 2: For the audio dataset A and the video dataset V, use the audio and video comparison learning model M CL Extract joint feature embedding E; the audio and video contrast learning model M CL Including audio feature extraction network M A , video feature extraction network M V , Multimodal feature fusion module M Fusion and contrastive learning loss module L CL , and train the joint feature embedding generator M Joint ; Step 3: Based on the joint feature embedding E, the audio and video performance of the digital human is generated through a dual guidance mechanism, which includes an emotional guidance mechanism M emotion and the progressive generation boot mechanism M progress ; Among them, the emotional guidance mechanism M emotion Dynamically generate emotion embedding Z by joint feature embedding E to adjust the facial expression of digital human; progressive generation guidance mechanism M progress Based on the facial key point information V of the first frame keypoints , Joint feature embedding E and emotion embedding Z, progressively generate video frames V gen , and through the joint loss L total Feedback optimization is performed to ensure that the generated video is synchronized with the audio in terms of timing content and emotion; Step 4: For the audio and video data A that needs to be inferred inf and video or image data V inf Perform preprocessing to obtain audio frame sequence A pre and standardized facial key point data Step 5: Based on the audio frame sequence A pre And the initial facial key point data Using the joint feature embedding generator M Joint , Emotional Guidance Mechanism M emotion and the progressive generation boot mechanism M progress Generate video frame sequence and with the audio frame sequence A pre Merge and finally output synchronized audio and video data 2. The method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism according to claim 1 is characterized in that: The specific method of step 1 is: Step 1.1: Define the audio and video dataset D = {d1, d2, ..., d n }, d i represents the i-th audio and video data sample, where n is the total number of samples, and the variable i∈[1,n]; Step 1.2: Use the audio processing tool ffmpeg to extract audio and video data i Separate the audio data And save it as WAV format file; Step 1.3: Use the Python audio processing library Librosa to process the audio file Perform noise reduction and equalization processing to remove background noise and equalize the volume to obtain noise-reduced audio data Step 1.4: The audio data set A is obtained by sampling at a fixed rate of 16 kHz and dividing it into fixed-length audio frames according to a 40 ms time window. 25 audio frames are generated per second, and each frame contains 640 sampling points to ensure that the temporal characteristics of the audio are not lost. n }, a i represents the i-th audio data sample; Step 1.5: Use the video processing library OpenCV to process the d i Extract frame by frame to get the video frame sequence Indicates that at time step j, i The extracted video sample frames, where the variable j∈[1,T], T represents the length of the time series; Step 1.6: Use the Mediapipe model to process each video frame Perform facial key point detection, extract the coordinates of facial key points, align the facial area to the predefined reference frame using the scaling and translation method based on the detected key point coordinates, and standardize the facial area to obtain the standardized video dataset V = {v1,v2,...,v n },v i Represents the i-th preprocessed video data sample, and defines the facial key point information of the first frame of each sample as V keypoints .

3. The method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism according to claim 1 is characterized in that: The audio feature extraction network M of step 2 A , as follows: Step 2.1: Build the audio feature extraction network M A A convolutional neural network layer is used to extract spectral features of audio dataset A using a 5-layer one-dimensional convolutional network. The convolution kernel size of each layer is 3, the step size is 1, and the number of channels is 32, 64, 128, 256, and 512, respectively. Each convolution layer is connected to a batch normalization layer and a ReLU activation layer to standardize the feature distribution and introduce nonlinearity. The spectral information of the audio signal is captured by convolution layer by layer. Step 2.2: Build the audio feature extraction network M A The adaptive emotion weighted multi-head self-attention layer is used to set the number of attention heads to 4, the embedding dimension of each head to 128, and the total embedding dimension to 512. Each attention head is generated by linear mapping of query, key, and value matrices to capture the long-distance dependencies of audio signals and generate basic attention weights. The generation formulas of query matrix Q, key matrix K, and value matrix V are as follows: Among them, W Q , W K , W V is a learnable weight matrix. Each attention head calculates its own attention weight and multiplies it with the corresponding value matrix V. Step 2.3: Output features through the convolutional layer Perform adaptive weight calculation and generate the sentiment weighted vector W through a layer of linear transformation and nonlinear activation function emotion , used to dynamically adjust the sentiment weight and define the sentiment weighted vector W emotion The calculation formula is: Among them, W e and b e is a learnable parameter, σ is the Sigmoid activation function, which is used to limit the weight value between 0 and 1; Step 2.4: Introduce an adaptive sentiment weighting mechanism in the multi-head self-attention layer, using the sentiment weighting vector W from step 2.3 emotion The output of the multi-head self-attention in step 2.2 is weighted, and the calculation formula of weighted attention is defined as: Among them, d k is the dimension of the key vector, T represents the length of the time series, and the adaptive sentiment weighting is applied to the basic attention weight to generate the adaptive weighted attention representation. This sentiment weighting mechanism can dynamically adjust the attention weight according to the sentiment characteristics of the input, ensuring that the time step with high sentiment intensity occupies a more important position in the output, thereby enhancing the capture of key sentiment information; Step 2.5: Define audio feature embedding represents the audio feature vector at the tth time step; Step 2.6: Each attention head independently calculates the adaptive emotion weighting, repeating steps 2.3 and 2.4 h times. Different heads can capture different emotion changes and temporal dependencies. Then concatenate the outputs of all attention heads to form an audio feature embedding: E A =Concat(AEWM-Attention1,…,AEWM-Attention h )W O Where h is the number of attention heads, W O is the final linear transformation matrix.

4. The method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism according to claim 1 is characterized in that: The video feature extraction network M in step 2 V , as follows: Step 2.7: Construct a video feature extraction network MV, and use 4 stacked 3D convolution layers to process the video dataset V. The size of each convolution kernel is 3×3×3, the step size is 1×1×1, and the number of channels is 32, 64, 128, and 256, respectively. The batch normalization and ReLU activation function are connected after the convolution layer. The 3D convolution layer slides in both the spatial and temporal dimensions to capture the spatial and temporal features between video frames, extract the dynamic changes of facial expressions and head postures, ensure the continuity of video features in time sequence, and obtain the video feature embedding sequence. Where T represents the length of the time series, each It is a multidimensional feature vector, representing the spatial features and dynamic information of the video at time step t; Step 2.8: Introduce the temporal emotion self-supervision module to capture the emotion changes in the video through contrastive learning without explicit emotion labels and select features at time step t The feature at the adjacent time step t+1 is a positive sample pair, indicating a time step with similar emotions. The feature at time step t is selected The features of random distant time step t+k are used as negative sample pairs, k>5, to represent the time step with significant emotional difference, and the contrast loss function is defined as: Where d() represents the Euclidean distance of the feature distance measurement function, i and j represent positive sample pairs between adjacent time steps, and j=i+1, i and k represent negative sample pairs with a long time interval, k≠i and |ki|>Δ, Δ is the preset minimum time interval, N represents the total number of all positive and negative sample pairs, and the contrast loss function is used to minimize the feature distance of time steps with similar emotions and increase the distance between time steps with significant emotional differences, thereby generating video feature embedding containing emotional information. Represents the video feature vector at the tth time step, which is aligned with the audio feature on the time axis. The video feature is embedded in E V It includes the temporal features of facial expressions and head postures, and realizes the preliminary labeling of emotional information through the emotion self-supervision module.

5. The method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism according to claim 1 is characterized in that: The multimodal feature fusion module M in step 2 Fusion Embed audio and video features into E A and E V The input is input into the bidirectional long short-term memory network to capture the bidirectional dependency of the input features in the time dimension and ensure that the audio and video features are synchronized in time; wherein, the hidden layer dimension of the bidirectional long short-term memory network is set to 512, and a preliminary aligned audio and video feature sequence is generated through bidirectional processing, and a multi-head self-attention mechanism is connected to further enhance the interaction and correlation between audio and video cross-modal settings. The self-attention layer calculates the weight of each time step to generate a weighted joint feature embedding to achieve deep fusion of multimodal features. The feature sequence after self-attention fusion is input into the multilayer perceptron, which contains 3 linear layers, and the output dimension of each layer is 1024. The ReLU activation function is used to map the fused features to a high-dimensional space to enhance the expression ability of the joint feature embedding, and finally the multimodal joint feature embedding E={e 1 ,e 2 ,...,e T },e t represents the joint feature vector at the t-th time step, where T represents the length of the time series.

6. The method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism according to claim 1 is characterized in that: The contrastive learning loss module L in step 2 CL and the joint feature embedding generator M Joint The details are as follows: Define the audio and video feature pairs at the same time step is a positive sample pair, an audio and video feature pair at different time steps is a negative sample pair, where j≠i and satisfies |ji|>Δ, Δ is the preset minimum time interval, and the contrast loss function L is defined. CL The calculation formula is as follows: Where y is the sample label, y=1 for positive sample pairs, y=0 for negative sample pairs, N is the total number of sample pairs, d is the feature distance metric function, cosine distance is used to calculate the similarity between audio and video feature embeddings, and the margin parameter α is set to ensure that the distance between negative sample pairs is greater than that between positive sample pairs; The loss L CL Propagate to the audio feature extraction network M through the back propagation algorithm A Video feature extraction network M V And multimodal feature fusion module M Fusion , and update the parameters of each network, using the stochastic gradient descent optimization algorithm, and perform multiple iterations of training until the contrast loss converges to ensure the optimal correlation of the audio and video feature embedding E; Constructing the joint feature embedding generator M Joint , receiving only the audio feature embedding E A As input, multiplex M Fusion The network structure generates joint feature embedding To make E pred As close to M as possible Fusion The generated true joint feature embedding E defines the reconstruction loss L cond The calculation formula is as follows: Among them, T represents the length of the time series, and the mean square error is used to measure the generated embedding E pred The difference from the true embedding E is calculated during training by minimizing the reconstruction loss L cond Optimize the joint feature embedding generator M Joint Learn how to transform audio features E A Transformed into an embedding E with joint representation capability pred , for the inference phase.

7. The method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism according to claim 1 is characterized in that: The specific method of step 3 is: Step 3.1: Construct the emotional guidance mechanism M emotion , generate sentiment embedding Z from the joint feature embedding E, and dynamically regulate sentiment embedding in time series; first, embed the joint feature e at each time step t Perform mean variance normalization and transform the feature e of each time step t Convert to standardized feature e t norm ; Then, the emotion embedding is generated for each time step through the emotion embedding encoding layer, including 2 fully connected layers, each layer is connected to the ReLU activation function Forming sentiment embedding sequence Finally, the dynamic emotion regulation layer is used to capture the emotion changes in the time series, and the single-layer gated recurrent unit GRU structure is used to embed the emotion Z init Modeling on time series, and obtaining the dynamically adjusted sentiment embedding Z = {z 1 ,z 2 ,...,z T }; Step 3.2: Define the sentiment consistency loss function as: Where T is the length of the time series, z t and z t+1 represents the dynamically regulated emotion embedding at adjacent time steps t and t+1. During the training process, the back propagation algorithm and optimization algorithm are used to optimize the emotion guidance mechanism M. emotion Update the parameters to reduce the emotional consistency loss L emotion_consistency Until the loss converges, make M emotion Ability to generate Z with smooth transition in time series, ensuring that the subsequently generated video frames have natural fluidity in emotional expression; Step 3.3: Construct the progressive generation boot mechanism M progress , the specific construction steps are as follows: Step 3.3.1: The initial generation module receives the facial key point information V of the first frame keypoints and the initial feature e of the joint feature embedding E 1 , generate the basic feature map F0; including two layers of convolution operations, the convolution kernel sizes are 5×5 and 3×3 respectively, followed by a layer of ReLU activation function, F0 represents the basic structure of the face, which is used for the gradual refinement of the subsequent generation layer. The calculation formula is as follows: F0=σ(W init *V keypoints +W E ·he 1 +b init ) Among them, W init is the convolution weight matrix, W E The linear transformation matrix of the joint feature embedding E is used to embed the joint feature into e 1 Mapped to a spatial dimension that matches the feature map, b init is the bias term, σ is the ReLU activation function; Step 3.3.2: Construct an emotion-driven dynamic feature fusion module. At each time step t(t∈[1,T]), the network recursively accepts the input of the current time step, including the emotion embedding z t , Joint feature embedding t and the feature map F generated at the previous time step t-1 , use a 3×3 convolution kernel to t-1 Perform convolution processing to extract spatial features and obtain the intermediate feature map F t conv , for sentiment embedding z t and joint feature embedding e t Perform element-by-element dot product operation to obtain the interaction feature z t ⊙e t , through a layer of linear transformation to capture z t and e t The interaction relationship between them is calculated through softmax regularization. t ,Right now α t =softmax(W α [z t ⊙e t ]+b α ) Among them, W α is the linear transformation matrix, b α is a bias term, which takes the attention weight α t Acting on the joint feature embedding e t We get the joint feature F after attention adjustment. t att =α t ·W e e t +b t , where W e is the linear transformation matrix, b t As the bias term, the adjusted emotional feature F t att With F t conv Add them together to get the feature map adjusted by the emotional information. Finally, a layer of ReLU activation function is used to introduce nonlinearity to enhance the influence of emotional information on the generated features, and the feature map F of the current time step is output. t ,Right now F t =σ(W conv *F t-1 +a t ·W e e t +b t ) Among them, W conv is the weight matrix of the convolutional layer, and each time step is processed recursively until the feature graph sequence {F1, F2, ..., F T }; Step 3.3.3: Construct a detail generation module to generate the feature map F at each time step t t Detail enhancement processing is performed, including multiple emotional attention residual blocks. Each residual block contains a multi-scale convolution structure and an emotional attention mechanism. The multi-scale convolution structure contains three parallel convolution layers with convolution kernel sizes of 3×3, 5×5, and 7×7, respectively. t Perform multi-scale convolution processing to obtain a multi-scale feature map output F t 3×3 、F t 5×5 、F t 7×7 Then, in order to dynamically adjust the fusion weight of the multi-scale feature map, according to the emotion embedding z at the current time step t Generate sentiment adaptive weight vector The calculation formula is: Among them, W z is the linear transformation matrix, b z As the bias term, the multi-scale feature map is weightedly fused using the sentiment weight vector to obtain the weighted feature map: After weighted fusion, the input feature map F t Perform 1×1 convolution and connect the adjusted input feature map F through residual connection t And the fused multi-scale feature map Add together to get the enhanced detail feature map: Step 3.3.4: Construct a high-resolution generation module, and gradually improve the resolution while retaining details through staged upsampling and step-by-step channel downsampling operations; first, use a convolution layer with a convolution kernel size of 3×3 for convolution operation, then perform an upsampling layer Pixel Shuffle operation; then use a convolution layer with a convolution kernel size of 3×3 for feature extraction, perform an upsampling operation again, and finally use a 3×3 convolution for convolution operation, and finally output a video frame sequence V gen = {v 1 ,v 2 ,...,v T }; Step 3.3.5: Use the video feature extraction network M V For the generated video frame sequence V gen Perform feature extraction to generate video feature embedding And calculate the feature difference of adjacent time steps, defined as For the audio feature extraction network M A Extracted audio features embedding Calculate the feature difference of its adjacent time steps, defined as Based on the time step difference of audio features and the time step difference of generated video features, the audio and video synchronization loss function L is defined sync_loss The calculation formula is as follows: Where T represents the length of the time series, by minimizing the audio and video synchronization loss L sync_loss , ensuring that the generated video frame sequence is synchronized with the dynamic changes of the audio signal in time sequence, so that the generated video is consistent with the emotion and rhythm of the audio in visual expression; Step 3.4: Define the joint loss function L total =λ1L emotion_consistency +λ2L sync_loss , where λ1 and λ2 are hyperparameters used to adjust the weights of emotional consistency and audio-video synchronization. total Training progressive generation guidance mechanism M progress , gradually adjust the generated video frames to ensure that the generated video frame sequence meets the requirements of the input audio in terms of emotional expression and time synchronization.

8. The method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism according to claim 1 is characterized in that: The specific method of step 4 is: Step 4.1: Define the single audio data that needs to be inferred as A inf , the single video or single picture data that needs to be inferred is V inf ; Step 4.2: Use the Python audio processing library Librosa to process audio A inf Perform noise reduction and equalization processing to remove background noise and equalize the volume. Perform fixed sampling at a sampling rate of 16kHz and divide it into a fixed-length audio frame sequence according to a 40ms time window. Generate 25 audio frames per second, each frame contains 640 sampling points, and obtain the preprocessed audio frame sequence A pre ={a 1 ,a 2 ,...,a T }, a t represents the audio frame at the tth time step, where T is the length of the time series; Step 4.3: Use the Mediapipe model to detect facial key points on the first frame of the image or video, extract the coordinates of the facial key points, align the facial area to the predefined reference frame using the scaling and translation method based on the detected key point coordinates, standardize the facial area, and generate standardized key point data 9. The method for constructing a digital human based on audio-video contrast learning and dual guidance mechanism according to claim 1 is characterized in that: The specific method of step 5 is: Step 5.1: Place A pre Enter the audio feature extraction network M A Extract audio feature embedding represents the audio feature vector at the tth time step, and T represents the length of the time series; Step 5.2: Input to the trained joint feature embedding generator M Joint , generating joint feature embeddings during inference Step 5.3: E pred Input to the emotion guidance mechanism M emotion , generate the corresponding sentiment embedding sequence Z inf ={z 1 ,z 2 ,...,z T }; Step 5.4: Standardize the key point data Joint feature embedding E pred and sentiment embedding sequence Z inf Input to the progressive generation guidance mechanism M progress , gradually generates a video frame sequence that matches the emotion and audio content; among them, the facial key points of the initial frame Used as the basis for facial geometry layout, the joint feature embedding E pred and sentiment embedding sequence Z inf Guided generation of video frame sequences Step 5.5: Generate a continuous video frame sequence With audio frame sequence A pre ={a 1 ,a 2 ,...,a T Merge frames by time step to ensure that the audio and video frames are strictly corresponding on the timeline. The video encoding format is H.264 and the audio encoding format is AAC. Merge to generate a single video file in MP4 format and output video content containing synchronized audio.

10. A digital human construction device based on audio-video contrast learning and dual-guidance mechanism, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into the processor, the digital human construction method based on audio-video contrast learning and dual guidance mechanism according to any one of claims 1 to 9 is implemented.

Images