Digital human construction method based on three-dimensional perception diffusion and semantic graph reasoning

Abstract

The invention discloses a digital human construction method based on three-dimensional perception diffusion and semantic graph reasoning. According to the method, firstly, input audio and video data are preprocessed, audio features, video frames and face key point information are extracted, and three-dimensional prior information is generated through depth estimation, attitude analysis and normal calculation; then, a semantic graph containing text, emotion, action and context nodes is constructed, multiple rounds of message passing are completed through a graph neural network, and an emotion coding vector and an action intention label are obtained; and inputting a three-dimensional U-Net diffusion generation model by taking the semantic control signal and the three-dimensional prior information as conditions, and generating an expression or action frame sequence conforming to the consistency of semantic logic and a three-dimensional structure. And finally, performing time domain alignment synthesis on the video frame sequence and the audio, and outputting synchronous digital human audio and video contents. Compared with the prior art, the method has the advantages that semantic-driven, natural-expression and coherent-action digital human generation can be realized, and the method has high reality sense and interactivity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and computer vision, and particularly relates to a digital human construction method based on three-dimensional perception diffusion and semantic graph reasoning. BACKGROUND

[0002] With the development of artificial intelligence and human-computer interaction technology, digital humans gradually become an important form of virtual anchors, education and training, intelligent customer service, and digital entertainment. Existing digital human generation methods mainly rely on rule-based animation driving or two-dimensional deep learning models. Although these methods can achieve basic expression transfer and lip synthesis, they have obvious shortcomings in semantic understanding, emotional expression, and three-dimensional consistency. On the one hand, video-driven generation methods lack semantic modeling and are difficult to generate logical actions and expressions according to dialogue content or external context. On the other hand, traditional generation models lack three-dimensional structure constraints and are prone to problems such as expression distortion, unnatural motion, or inconsistency with head posture, resulting in insufficient realism of the synthesis results.

[0003] In recent years, diffusion models have shown superior distribution learning and detail synthesis capabilities in image and video generation, but their application in multi-modal fusion and three-dimensional constraints is still limited. Meanwhile, semantic graph reasoning methods can model text, emotion, action, and context information uniformly and perform multiple rounds of message passing through graph neural networks, thereby better capturing the dependency between multi-modalities. However, existing technologies have not combined semantic graph reasoning with three-dimensional perception diffusion models, and there is a lack of a digital human generation scheme that can guarantee semantic-driven rationality while maintaining three-dimensional structure consistency and temporal alignment.

[0004] Therefore, there is an urgent need for a new method that can integrate semantic information, emotional state, and three-dimensional perception features of audio and video data, utilize the generation capabilities of diffusion models and the logical modeling advantages of semantic graph reasoning, and generate digital human content with natural expressions, coherent actions, semantic consistency, and audio synchronization to improve the realism and immersion of digital humans in interactive scenarios. SUMMARY

[0005] The present application relates to the technical field of artificial intelligence and computer vision, and particularly relates to a digital human construction method based on three-dimensional perception diffusion and semantic graph reasoning.

[0006] Technical solution: The present application proposes a digital human construction method based on three-dimensional perception diffusion and semantic graph reasoning, comprising the following steps:

[0007] Step 1: Obtain the input audio and video dataset to be processed, perform preprocessing, extract audio data and video data respectively, and use the face key point detection model to extract the first video frame of each sample and its corresponding key point set;

[0008] Step 2: Based on the first video frame, extract the depth map using the 3D perception extraction module. Attitude vector and facial normal map The three-dimensional prior information is obtained, and the frame-level comprehensive quality score is calculated through the feature quality evaluation module. Based on the frame-level comprehensive quality score, a feature quality judgment and backoff mechanism is introduced.

[0009] Step 3: Construct a semantic graph containing text nodes, emotion nodes, action nodes, trigger nodes, and context nodes using audio data, video data, 3D prior information, frame-level comprehensive quality scores, and external inputs, and introduce semantic dependency edges, emotion-driven edges, temporal edges, and synchronization edges;

[0010] Step 4: Perform multiple rounds of message passing and embedding updates on the semantic graph based on the graph neural network model to generate semantic control signals for controlling the digital human's expressions and actions. The semantic control signals include emotion encoding vectors and action intention labels.

[0011] Step 5: Using random noise z as input, the semantic control signal and three-dimensional prior information are input into the three-dimensional U-Net diffusion generation model to generate a sequence of facial expressions or actions consistent with the semantic logic and three-dimensional structure.

[0012] Step 6: Synthesize the action frame sequence with the audio data in the temporal domain to output the corresponding digital human audio-visual content.

[0013] Furthermore, the preprocessing in step 1 specifically includes:

[0014] Step 1.1: Combine audio and video datasets Decoded into audio data and video data Audio data For single-channel 16kHz audio signal and video data A sequence of images arranged in frames;

[0015] Step 1.2: Process the audio data Endpoint detection, noise cancellation, and Mel spectrogram feature extraction are performed to construct a set of speech feature vectors. ;

[0016] Step 1.3: Process video data Perform uniform frame rate resampling, resolution normalization, and image grayscale standardization processing;

[0017] Step 1.4: Based on the MediaPipe Face Mesh facial landmark detection model, process the video data. The first frame of each sample in the video frame Perform face detection and key point localization, and extract a set of two-dimensional key points. .

[0018] Furthermore, the specific method for step 2 is as follows:

[0019] Step 2.1: From the input video sequence Extracting image frames The MediaPipe Face Mesh facial landmark detection model was used to locate facial regions and crop them to obtain a standardized facial image sequence. ;

[0020] Step 2.2: Standardize the face image sequence The input is a 3D perception extraction module, which includes a depth estimation submodule, a pose estimation submodule, and a normal estimation submodule. The depth estimation submodule predicts image frames. Spatial depth map The module extracts two-dimensional key points and outputs a depth confidence map; the attitude estimation submodule extracts two-dimensional key points and calculates a three-dimensional attitude vector. Used to characterize the rotational attitude of the head in three-dimensional space; the normal estimation submodule calculates the unit normal vector based on the depth gradient or the cross product of the point cloud neighborhood. ;

[0021] Step 2.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Input the feature quality assessment module to calculate the depth quality score. Posture quality score Normal mass fraction And synthesize into a comprehensive quality score. ;

[0022] Step 2.4: Based on the overall quality score When the value falls below the preset threshold τ, a rollback and repair mechanism is activated for further repair.

[0023] Step 2.5: Integrate and construct the repaired 3D features.

[0024] Furthermore, the comprehensive quality score in step 2.3 Specifically as follows:

[0025] (1) Depth quality score The calculation formula is: ,in, This represents the effective pixel ratio of the depth map. Indicates depth smoothness. As weight, The gradient threshold is used when holes or noise appear in the depth map. Decrease It decreases accordingly;

[0026] (2) Posture mass fraction The calculation formula is: ,in, The keypoint reprojection error is calculated. The upper limit of error, Indicates the proportion of key points detected. This indicates truncation to the [0,1] interval. This is useful when the attitude estimation error is large or key points are missing. Decrease;

[0027] (3) Normal mass fraction The calculation formula is: ,in, The average cosine similarity of the normal directions of adjacent pixels. This represents the abrupt change in the normal direction. As the weight, when the normal direction is discontinuous, Enlarge reduce;

[0028] The three scores are all normalized to the interval [0,1], and the overall quality score is calculated according to the weights:

[0029] ;

[0030] in, .

[0031] Furthermore, the rollback and repair mechanism is as follows:

[0032] When the current frame features are determined to be a low-quality frame, the frame is preferentially backtracked and replaced with the valid feature set of the previous frame.

[0033] If multiple consecutive frames fail, an exponential moving average strategy is used to smoothly repair the features:

[0034] ;

[0035] in, This represents the depth, pose, or normal features of the current frame, i.e. , As a smoothing coefficient, if the repair result still does not reach the quality threshold, the next template rollback is executed;

[0036] If the number of consecutive failed frames exceeds the set limit, the mean template feature preset in the template library will be used as a replacement.

[0037] Furthermore, the specific method of step 3 is as follows:

[0038] Step 3.1: Input speech feature vector Video data Specific frames and sets of facial key points 3D prior information and external input text ;

[0039] Step 3.2: Define the semantic graph ,in For a set of nodes, Let it be the set of edges;

[0040] Step 3.3: Perform sliding window processing on the input stream along the time axis, and construct the following five types of nodes in each time window:

[0041] Text node : Extracted from the text sequence using the semantic coding model BERT, used to represent semantic content; sentiment nodes : Based on speech features With video frame sequence Generated using a fusion network CNN-LSTM to represent emotional states; action nodes : From three-dimensional prior structure The pose and normal changes are calculated and used to represent facial and upper body movements; context nodes Generated from the semantic and sentiment nodes of adjacent windows using the EMA / GRU aggregation algorithm to maintain semantic continuity: Here, EMA (Exponential Moving Average) is used to smooth semantic changes; GRU (Gated Recurrent Unit) is used to preserve semantic-emotional memory, and its output is the contextual features of the current window; trigger node. : Input external trigger signal to trigger or control the digital human's response; the node set is represented as: Each node's features are standardized into a fixed-dimensional vector and used as input for subsequent semantic graph construction steps; the feature inputs for the emotion nodes and action nodes come from a three-dimensional prior structure. and its gating factors ;

[0042] Step 3.4: Based on the frame-level composite quality score, calculate the window-level gating factor within the time window. Window-level metrics are defined as: ,in, The overall quality score is at the frame level. The number of frames within the window; based on The value determines the gating factor: ;

[0043] when At the same time, all structural consistency constraints are retained;

[0044] when At that time, the constraints were partially weakened;

[0045] when At this time, three-dimensional constraints are turned off, and only semantic and emotional connections are retained;

[0046] Step 3.5: In the window Internal calculation of candidate edges:

[0047] Semantic Dependency Edge Based on semantic similarity and prototype matching scores, ;

[0048] Cross-modal synchronization edge : Weighted by modal confidence and time overlap ratio ;

[0049] Time Adjacency Similar nodes in the window and The time-order joins, with weights related to the time interval / confidence level. ,in ;

[0050] Step 3.6: Calculate the basic weights For each candidate edge The basic weight is calculated by comprehensively considering semantic, emotional, and temporal factors: , , The basic weights are obtained by performing linear mixing. The calculation method is as follows:

[0051] ;

[0052] in, ;

[0053] Step 3.7: Apply the window-level gating factor obtained in Step 3.4 As an adjustment coefficient, the basic weights of the candidate edges are weighted and corrected: ;

[0054] Step 3.8: Filtering 3D Gating Weights The nodes and edges are merged to output a semantic graph that meets the requirements. .

[0055] Furthermore, the specific method of step 4 is as follows:

[0056] Using a multi-layer graph neural network (GNN) structure to analyze semantic graphs The system models the node features and edge relationships in the semantic graph. After multiple rounds of message passing, it aggregates the features of action nodes and emotion nodes in the semantic graph to generate emotion encoding vectors. With action intention tag The emotion encoding vector is used to guide the digital human's facial expressions, tone of voice, and interaction context control; the action intention label represents the action type label sequence obtained through reasoning.

[0057] Furthermore, semantic control signals and the three-dimensional perceptual features of the three-dimensional prior structure are used together as conditional inputs. With noise tensor Combining to form diffusion model input pairs The three-dimensional U-Net diffusion generation model, wherein each U-Net module consists of an encoder layer. Cross-layer residual connections Decoder layer The structure employs 3D convolution and multimodal conditional fusion mechanisms to satisfy: .

[0058] Furthermore, the specific method for step 6 is as follows:

[0059] Step 6.1: Perform speech rhythm analysis on audio data A and use ASR to extract phoneme boundary sequences. And use the duration corresponding to each phoneme as the alignment unit;

[0060] Step 6.2: Process the video frame sequence Perform frame rate normalization to ensure it corresponds to the phoneme beat;

[0061] Step 6.3: Based on time alignment function Each frame is mapped to a corresponding phoneme time segment;

[0062] Step 6.4: Use an image compositing engine to merge the aligned video frame sequence with audio A to generate an MP4 video stream with an audio track, forming the output result. .

[0063] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0064] This invention overcomes the shortcomings of existing digital human generation technologies, such as insufficient semantic modeling, inconsistent 3D structures, and audio-video asynchrony, by introducing a joint method of 3D perceptual diffusion and semantic graph reasoning. Specifically, this invention first obtains high-quality input features through audio-video preprocessing and facial keypoint detection, and constructs 3D prior information by combining depth maps, pose vectors, and normal maps, thereby improving the spatial consistency and geometric rationality of the generated results. Second, by constructing a semantic graph containing text, emotion, action, and context nodes, and using graph neural networks to complete multi-round message passing, it can effectively capture the logical relationships between multimodalities, generate context-appropriate emotion codes and action intention labels, and achieve semantically driven expression and action control. Third, by combining semantic signals with 3D prior information through a 3D U-Net diffusion generation model, it ensures that the generated expressions and actions satisfy semantic logic while maintaining 3D structural stability. Finally, through temporal alignment and synthesis of audio and video, it achieves high-fidelity, temporally coherent digital human audio and video output. In summary, this invention can significantly improve the naturalness, realism, and immersion of digital humans in interactive scenarios, and has broad application value. Attached Figure Description

[0065] Figure 1 A flowchart outlining the overall process for constructing a digital human based on 3D perceptual diffusion and semantic graph reasoning. Detailed Implementation

[0066] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0067] This invention discloses a method for constructing a digital human based on three-dimensional perceptual diffusion and semantic graph reasoning, comprising the following steps:

[0068] Step 1: Decode and preprocess the input audio and video dataset, extract audio features, video frame sequences and facial key point information, and store them in a standard format as the input basis for subsequent semantic graph reasoning and 3D perception.

[0069] Step 1.1: Combine audio and video datasets Decoded into audio data and video data Audio data For single-channel 16kHz audio signal and video data It is a sequence of images arranged in frames.

[0070] Step 1.2: Process the audio data Endpoint detection, noise cancellation, and Mel spectrogram feature extraction are performed to construct a set of speech feature vectors.

[0071]

[0072] in For frame number, For frequency band dimension.

[0073] Step 1.3: Process video data Perform uniform frame rate resampling, resolution normalization, and image grayscale standardization.

[0074] Step 1.4: Based on the MediaPipe Face Mesh facial landmark detection model, process the video data. Face detection and key point localization are performed on the first frame image K of each sample, and a two-dimensional key point set is extracted. ,satisfy:

[0075]

[0076] in, This represents the total number of key points, typically 68. For the first Image coordinates of key points.

[0077] Step 1.5: Extract audio features Video frame sequence Key information It is stored in a standard format as input for subsequent semantic graph reasoning and 3D structure perception.

[0078] Step 2: Extract depth map, pose vector, and normal map from the first frame image to construct 3D prior information (T3D), providing 3D structural constraints for subsequent diffusion generation.

[0079] Step 2.1: Video Sequence Extract the first frame image Image cropping and adaptive adjustment strategies are used to obtain standard input image frames. .

[0080] An adaptive adjustment strategy is used to perform multidimensional normalization processing on the input frame, the strategy including:

[0081] (1) Scale adaptation: Automatically scales and aligns with the center based on the detected face bounding box size to ensure consistent facial proportions.

[0082] (2) Brightness Adaptive: Based on the Adaptive Histogram Equalization (CLAHE) algorithm, the brightness and contrast of the image are automatically corrected.

[0083] (3) Resolution Adaptive: Scale the image to the standard size of the model input (e.g., 256×256) and perform edge patching or interpolation while maintaining the aspect ratio.

[0084] Step 2.2: Depth Map This represents the spatial depth of each pixel relative to the camera, calculated from image frames using the monocular depth estimation model MiDaS. The depth map is obtained from prediction and its validity is determined by the following formula:

[0085]

[0086] in, This is a convolutional neural network model that takes image frames as input. Its parameter set;

[0087] Step 2.3: Use the MediaPipe Face Mesh facial landmark detection module to extract 2D key points, and combine it with the EPnP algorithm to calculate the 3D pose vector. Attitude vector Used to characterize the rotational attitude of the head in three-dimensional space, including pitch angle. Yaw angle and roll angle The three-dimensional attitude vector is represented as:

[0088]

[0089] Step 2.4: Normal Diagram For each facial pixel in the video, determine the surface unit normal vector. If the normal is discontinuous, perform filtering optimization. Calculate the unit normal vector based on the depth gradient or the cross product of the point cloud neighborhood. The calculation method is as follows:

[0090]

[0091] in, , Let x be the edge vector of the adjacent triangular facet at that point, and × is the cross product operation. This represents the vector magnitude, used to normalize it to a unit vector.

[0092] Step 2.5: Integrate and construct a 3D prior structure to provide data and 3D prior information for subsequent steps. Represented as the following set:

[0093]

[0094] 3D prior information It is used as a structural condition input in the subsequent 3D diffusion model to improve the spatial consistency and geometric rationality of facial expression generation.

[0095] Step 2.6: The output of Step 2.5 Input the feature quality assessment module to calculate the depth quality score. Posture quality score Normal mass fraction And synthesize into a comprehensive quality score. .

[0096] (1) Depth quality score The calculation formula is: ,in This represents the effective pixel ratio of the depth map. Indicates depth smoothness. As weight, This is the gradient threshold. When holes or noise appear in the depth map, Decrease It then decreases.

[0097] (2) Posture mass fraction The calculation formula is: ,in The keypoint reprojection error is calculated using the EPnP algorithm. The upper limit of error, Indicates the proportion of key points detected. This indicates truncation to the [0,1] interval. When the attitude estimation error is large or key points are missing, Decrease.

[0098] (3) Normal mass fraction The calculation formula is: ,in The average cosine similarity of the normal directions of adjacent pixels. This represents the abrupt change in the normal direction. As the weight. When the normal direction is discontinuous, Enlarge reduce.

[0099] The three scores are all normalized to the interval [0,1], and the overall quality score is calculated according to the weights:

[0100]

[0101] in, When the overall quality score Below the threshold When the current frame is determined to be a low-quality feature frame, the feature quality judgment and rollback mechanism is triggered in step 2.7.

[0102] Step 2.7: When the overall quality score When the value falls below a preset threshold τ, a rollback and repair mechanism is activated to ensure the temporal continuity and geometric stability of the feature sequence.

[0103] When the current frame features are determined to be low-quality, priority is given to backtracking and replacing them with the valid feature set of the previous frame: .

[0104] If multiple consecutive frames fail, an exponential moving average strategy is used to smoothly repair the features: ,in Represents the depth, pose, or normal features of the current frame (i.e., ), This is the smoothing coefficient. If the repair result still does not reach the quality threshold, proceed to the next step of template rollback.

[0105] If the number of consecutive failed frames exceeds the set limit, the mean template feature preset in the template library will be used as a replacement:

[0106] ,in These are standard facial template features obtained by averaging multiple samples, used to maintain the stability of the model's structural input.

[0107] Step 2.5: Integrate and construct the repaired 3D features to build a 3D prior structure: and will Input the 3D diffusion inference module.

[0108] Step 3: Construct a semantic graph containing text, emotion, action, and context nodes using audio features, video frames and keypoint information, 3D prior information, and external input. Establish multimodal dependencies by combining edge weight calculation and gating mechanisms.

[0109] Step 3.1: Input speech feature vector Video frame sequence Specific frames and sets of facial key points 3D prior information and external input text .

[0110] Step 3.2: Define the semantic graph ,in For a set of nodes, Let it be the set of edges.

[0111] Step 3.3: Perform sliding window processing on the input stream along the time axis. Set the window length. Sliding step size Within each time window, the following five types of nodes are constructed sequentially:

[0112] Text node : Extracted from the text sequence using the semantic coding model BERT, and used to represent semantic content.

[0113] Emotional nodes : Based on speech features With video frame sequence Generated by a fusion network CNN-LSTM, used to represent emotional states.

[0114] Action Node : From three-dimensional prior structure The changes in posture and normals are calculated and used to characterize facial and upper body movements.

[0115] Context node Generated by the semantic and sentiment nodes of adjacent windows using the EMA / GRU aggregation algorithm to maintain semantic continuity. .

[0116] EMA: Exponential moving average, used to smooth semantic changes;

[0117] GRU: Gated Recurrent Unit, used to preserve semantic-emotional memories.

[0118] The output is the context features of the current window.

[0119] Trigger node Input external trigger signals (such as user intent, action commands, etc.) to trigger or control the digital human's response.

[0120] The node set is represented as: Each node feature is standardized into a fixed-dimensional vector and used as input for subsequent semantic graph construction steps.

[0121] The feature inputs for emotion nodes and action nodes come from a three-dimensional prior structure. and its gating factors This ensures the consistency between 3D information and speech semantics during the semantic graph construction process.

[0122] Step 3.4: Based on the previously calculated frame-level overall quality score Calculate the window-level gating factor within the time window. It is used to dynamically adjust the three-dimensional consistency constraints in the semantic graph.

[0123] Window-level metrics are defined as: ,in The frame-level overall quality score output in step 2.3. This represents the number of frames within the window.

[0124] according to The value determines the gating factor: .

[0125] when When , it indicates that the three-dimensional features are stable during that time period, preserving all structural consistency constraints. When this occurs, it indicates that the characteristic fluctuation is large, and the constraint is partially weakened. When... At this time, three-dimensional constraints are not used; only semantic and emotional connections are relied upon.

[0126] Step 3.5: In the window Internal calculation of candidate edges:

[0127] Semantic Dependency Edge Based on semantic similarity and prototype matching scores, .

[0128] Cross-modal synchronization edge : Weighted by modal confidence and time overlap ratio .

[0129] Time Adjacency Similar nodes in the window and The time-order joins, with weights related to the time interval / confidence level. ,in .

[0130] Step 3.6: Calculate the basic weights For each candidate edge The basic weight is calculated by comprehensively considering semantic, emotional, and temporal factors: , , The basic weights are obtained by performing linear mixing. The calculation method is as follows:

[0131]

[0132] in The basic weights reflect the initial importance of edges in terms of semantic relevance.

[0133] Step 3.7: Apply the window-level gating factor obtained in Step 3.4 As an adjustment coefficient, the basic weights of the candidate edges are weighted and corrected: .

[0134] Step 3.8: Filtering 3D Gating Weights The nodes and edges are merged to output a semantic graph that meets the requirements. .

[0135] Step 4: Perform multiple rounds of message passing and embedding updates on the semantic graph using a graph neural network to generate semantic control signals C={Eemo, Vact} for controlling the digital human's expressions and movements.

[0136] Step 4.1: Use a multi-layer graph neural network (GNN) structure to process the semantic graph. The node features and edge relationships are modeled, and the updated node features are represented as follows: ,in, Represents the node at level l. Feature representation, For the set of neighboring nodes, For learnable weight matrix, This is the activation function.

[0137] Step 4.2: After multiple rounds of message passing are completed, feature aggregation is performed on action nodes and emotion nodes in the semantic graph to generate emotion encoding vectors. With action intention tag .

[0138] Step 4.3: Emotion Encoding Vector Used to guide the facial expressions, tone of voice, and interactive context control of digital humans.

[0139] Step 4.4: Action Intent Labeling This represents the sequence of action type labels obtained through inference, which is used to constrain the action trajectory generation conditions of the subsequent diffusion model.

[0140] Step 4.5: Semantic Control Signals The conditional information is fed into the 3D diffusion generation module to achieve semantically guided digital human motion generation.

[0141] Step 5: Starting with random noise, the semantic control signal and 3D prior information are used as conditional inputs to the 3D U-Net diffusion generation model to generate a sequence of facial expressions or actions that conform to semantic logic and 3D structure.

[0142] Step 5.1: Initialize the input tensor with random Gaussian noise , as the initial state of the diffusion model, where the dimension is , representing the number of channels, time steps, height, and width, respectively.

[0143] Step 5.2: Transfer semantic control signals Three-dimensional perception information is used as a conditional input. With noise tensor Combining to form diffusion model input pairs .

[0144] Step 5.3: Input the above input pairs into the 3D U-Net diffusion generation model, where each U-Net module consists of an encoder layer. Cross-layer residual connections Decoder layer Composition, satisfying: .

[0145] Step 5.4: Generate Sequence To ensure that the facial expression animation sequence or body movement keyframe sequence conforms to the semantic control logic and the consistency of the 3D structure, each frame... .

[0146] Step 5.5: The diffusion process supports conditional cross-modal alignment, i.e., semantic embedding, 3D features and temporal noise processes are jointly modeled at multiple scales.

[0147] Step 6: Perform temporal alignment and synthesis on the generated video frame sequence and audio data to output digital human audio-visual content that conforms to semantics and 3D structure.

[0148] Step 6.1: Perform speech rhythm analysis on audio data A and use ASR to extract phoneme boundary sequences. And the duration corresponding to each phoneme is used as the alignment unit.

[0149] Step 6.2: Process the video frame sequence Frame rate normalization is performed to ensure it corresponds to the phoneme beat, i.e.: ,in For video frame rate, This represents the total duration of the audio.

[0150] Step 6.3: Based on time alignment function Map each frame Fi to its corresponding phoneme time segment:

[0151]

[0152] in, For frames The corresponding timestamp, For the first The start time of each phoneme.

[0153] Step 6.4: Use an image compositing engine to merge the aligned video frame sequence with audio A to generate an MP4 video stream with an audio track, forming the output result:

[0154]

[0155] Step 6.5: The output result It supports exporting in multiple resolutions and formats for use in digital human interactive displays or storage and distribution.

[0156] Experimental environment:

[0157] This invention is based on an NVIDIA RTX 4090 GPU server (CUDA 12.3, PyTorch 2.1).

[0158] The CPU is an Intel Core i9-13900K, and the memory is 64GB.

[0159] The learning rate during the training phase is 1×10⁻ 4 β1=0.9, β2=0.999,

[0160] Diffusion steps 1000, batch size 4.

[0161] The 3D perception diffusion module borrows the 3D-aware Transformer architecture from DiffTF++, while the semantic graph reasoning module uses the same semantic mask separation method as NeRFFaceEditing. Training data comes from a self-built speech-driven face video dataset (approximately 25 hours of speech and synchronized video), with publicly available datasets such as FFHQ, CelebV-HQ, and VoxCeleb2 used for pre-training and validation.

[0162] Experimental procedure:

[0163] 1. Baseline Model Comparison

[0164] Four existing methods were selected as baselines: ER-NeRF, GeneFace, SadTalker, and TalkingGaussian. Speaker videos were generated under the same speech-driven input and initial posture conditions, and image sharpness, consistency, and lip-sync were compared.

[0165] 2. Ablation test

[0166] The following modules were removed: (1) feature quality judgment and backoff mechanism, (2) semantic graph gating module, and (3) three-dimensional perception extraction module. The impact of each module on temporal consistency and facial expression naturalness was analyzed.

[0167] 3. Evaluation Indicators

[0168] PSNR (Peak Signal-to-Noise Ratio, the higher the better)

[0169] SSIM (structural similarity, up)

[0170] FID (Fréchet Inception Distance, ↓)

[0171] SyncScore (voice-lip synchronization, up)

[0172] T-Var (Time-series variation variance, ↓)

[0173] Experimental Results and Analysis

[0174] 1. Comparison of quantitative results

[0175] Method PSNR SSIM FID SyncScore T-Var ER-NeRF 26.4 0.912 47.2 0.72 0.136 GeneFace 27.8 0.925 43.8 0.76 0.127 SadTalker 28.6 0.931 41.0 0.79 0.122 TalkingGaussian 29.2 0.937 39.2 0.81 0.115 Invention 29.9 0.944 37.6 0.84 0.110

[0176] The method of this invention achieves a robust improvement in sharpness and consistency, with PSNR increased by approximately 0.7 dB, FID decreased by approximately 1.6, and SyncScore increased by 3%.

[0177] 2. Ablation Experiment Results

[0178] Model configuration PSNR SSIM FID SyncScore T-Var Remove feature quality judgment 29.1 0.938 39.1 0.82 0.118 Remove semantic graph gating 29.0 0.937 38.9 0.81 0.120 Remove three-dimensional perception extraction 28.4 0.930 41.5 0.78 0.125 Complete model 29.9 0.944 37.6 0.84 0.110

[0179] The ablation results showed that:

[0180] The 3D perception extraction module mainly improves geometric and facial expression details; the feature quality judgment module significantly reduces jitter; and the semantic graph gating mechanism can improve speech-expression synchronization.

[0181] Experimental conclusion:

[0182] Comprehensive experimental results demonstrate that the proposed digital human construction method based on 3D perceptual diffusion and semantic graph reasoning achieves verifiable improvements in clarity, consistency, and naturalness compared to existing methods, with stable performance enhancements. These results validate the practical feasibility and scientific rationality of this method in voice-driven digital human generation.

Claims

1. A method for constructing a digital human based on three-dimensional perceptual diffusion and semantic graph reasoning, characterized in that, Includes the following steps: Step 1: Obtain the input audio and video dataset to be processed, perform preprocessing, extract audio data and video data respectively, and use the face key point detection model to extract the first video frame of each sample and its corresponding key point set; Step 2: Based on the first video frame, extract the depth map using the 3D perception extraction module. Attitude vector and facial normal map The three-dimensional prior information is obtained, and the frame-level comprehensive quality score is calculated through the feature quality evaluation module. Based on the frame-level comprehensive quality score, a feature quality judgment and backoff mechanism is introduced. Step 3: Construct a semantic graph containing text nodes, emotion nodes, action nodes, trigger nodes, and context nodes using audio data, video data, 3D prior information, frame-level comprehensive quality scores, and external inputs, and introduce semantic dependency edges, emotion-driven edges, temporal edges, and synchronization edges; Step 4: Perform multiple rounds of message passing and embedding updates on the semantic graph based on the graph neural network model to generate semantic control signals for controlling the digital human's expressions and actions. The semantic control signals include emotion encoding vectors and action intention labels. Step 5: Using random noise z as input, the semantic control signal and three-dimensional prior information are input into the three-dimensional U-Net diffusion generation model to generate a sequence of facial expressions or actions consistent with the semantic logic and three-dimensional structure. Step 6: Synthesize the action frame sequence with the audio data in the temporal domain to output the corresponding digital human audio-visual content.

2. The digital human construction method based on three-dimensional perceptual diffusion and semantic graph reasoning according to claim 1, characterized in that, The preprocessing in step 1 specifically includes: Step 1.1: Combine audio and video datasets Decoded into audio data and video data Audio data For single-channel 16kHz audio signal and video data A sequence of images arranged in frames; Step 1.2: Process the audio data Endpoint detection, noise cancellation, and Mel spectrogram feature extraction are performed to construct a set of speech feature vectors. ; Step 1.3: Process video data Perform uniform frame rate resampling, resolution normalization, and image grayscale standardization processing; Step 1.4: Based on the MediaPipe Face Mesh facial landmark detection model, process the video data. The first frame of each sample in the video frame Perform face detection and key point localization, and extract a set of two-dimensional key points. .

3. The digital human construction method based on three-dimensional perceptual diffusion and semantic graph reasoning according to claim 1, characterized in that, The specific method for step 2 is as follows: Step 2.1: From the input video sequence Extracting image frames The MediaPipeFace Mesh facial landmark detection model was used to locate facial regions and crop them to obtain a standardized facial image sequence. ; Step 2.2: Standardize the face image sequence The input is a 3D perception extraction module, which includes a depth estimation submodule, a pose estimation submodule, and a normal estimation submodule. The depth estimation submodule predicts image frames. Spatial depth map The module extracts two-dimensional key points and outputs a depth confidence map; the attitude estimation submodule extracts two-dimensional key points and calculates a three-dimensional attitude vector. Used to characterize the rotational posture of the head in three-dimensional space; The normal estimation submodule calculates the unit normal vector based on the depth gradient or the cross product of the point cloud neighborhood. ; Step 2.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Input the feature quality assessment module to calculate the depth quality score. Posture quality score Normal mass fraction And synthesize into a comprehensive quality score. ; Step 2.4: Based on the overall quality score When the value falls below the preset threshold τ, a rollback and repair mechanism is activated for further repair. Step 2.5: Integrate and construct the repaired 3D features.

4. The digital human construction method based on three-dimensional perceptual diffusion and semantic graph reasoning according to claim 3, characterized in that, The comprehensive quality score in step 2.3 Specifically as follows: (1) Depth quality score The calculation formula is: ,in, This represents the effective pixel ratio of the depth map. Indicates depth smoothness. As weight, The gradient threshold is used when holes or noise appear in the depth map. Decrease It decreases accordingly; (2) Posture mass fraction The calculation formula is: ,in, The keypoint reprojection error is calculated. The upper limit of error, Indicates the proportion of key points detected. This indicates truncation to the [0,1] interval. This is useful when the attitude estimation error is large or key points are missing. Decrease; (3) Normal mass fraction The calculation formula is: ,in, The average cosine similarity of the normal directions of adjacent pixels. This represents the abrupt change in the normal direction. As the weight, when the normal direction is discontinuous, Enlarge reduce; The three scores are all normalized to the interval [0,1], and the overall quality score is calculated according to the weights: ； in, .

5. The digital human construction method based on three-dimensional perceptual diffusion and semantic graph reasoning according to claim 3, characterized in that, The rollback and repair mechanism is as follows: When the current frame features are determined to be a low-quality frame, the frame is preferentially backtracked and replaced with the valid feature set of the previous frame. If multiple consecutive frames fail, an exponential moving average strategy is used to smoothly repair the features: ； in, This represents the depth, pose, or normal features of the current frame, i.e. , As a smoothing coefficient, if the repair result still does not reach the quality threshold, the next template rollback is executed; If the number of consecutive failed frames exceeds the set limit, the mean template feature preset in the template library will be used as a replacement.

6. The digital human construction method based on three-dimensional perceptual diffusion and semantic graph reasoning according to claim 3, characterized in that, The specific method for step 3 is as follows: Step 3.1: Input speech feature vector Video data Specific frames and sets of facial key points 3D prior information and external input text ; Step 3.2: Define the semantic graph ,in For a set of nodes, Let it be the set of edges; Step 3.3: Perform sliding window processing on the input stream along the time axis, and construct the following five types of nodes in each time window: Text node : Extracted from the text sequence using the semantic coding model BERT, used to represent semantic content; sentiment nodes : Based on speech features With video frame sequence Generated using a fusion network CNN-LSTM to represent emotional states; action nodes : From three-dimensional prior structure The pose and normal changes are calculated and used to represent facial and upper body movements; context nodes Generated from the semantic and sentiment nodes of adjacent windows using the EMA / GRU aggregation algorithm to maintain semantic continuity: Here, EMA (Exponential Moving Average) is used to smooth semantic changes; GRU (Gated Recurrent Unit) is used to preserve semantic-emotional memory, and its output is the contextual features of the current window; trigger node. : Input external trigger signal to trigger or control the digital human's response; the node set is represented as: Each node's features are standardized into a fixed-dimensional vector and used as input for subsequent semantic graph construction steps; the feature inputs for the emotion nodes and action nodes come from a three-dimensional prior structure. and its gating factors ; Step 3.4: Based on the frame-level composite quality score, calculate the window-level gating factor within the time window. Window-level metrics are defined as: ,in, The overall quality score is at the frame level. The number of frames within the window; based on The value determines the gating factor: ; when At the same time, all structural consistency constraints are retained; when At that time, the constraints were partially weakened; when At this time, three-dimensional constraints are turned off, and only semantic and emotional connections are retained; Step 3.5: In the window Internal calculation of candidate edges: Semantic Dependency Edge Based on semantic similarity and prototype matching scores, ; Cross-modal synchronization edge : Weighted by modal confidence and time overlap ratio ; Time Adjacency Similar nodes in the window and The time-order joins, with weights related to the time interval / confidence level. ,in ; Step 3.6: Calculate the basic weights For each candidate edge The basic weight is calculated by comprehensively considering semantic, emotional, and temporal factors: , , The basic weights are obtained by performing linear mixing. The calculation method is as follows: ； in, ; Step 3.7: Apply the window-level gating factor obtained in Step 3.4 As an adjustment coefficient, the basic weights of the candidate edges are weighted and corrected: ; Step 3.8: Filtering 3D Gating Weights The nodes and edges are merged to output a semantic graph that meets the requirements. .

7. The digital human construction method based on three-dimensional perceptual diffusion and semantic graph reasoning according to claim 1, characterized in that, The specific method for step 4 is as follows: Using a multi-layer graph neural network (GNN) structure to analyze semantic graphs The system models the node features and edge relationships in the semantic graph. After multiple rounds of message passing, it aggregates the features of action nodes and emotion nodes in the semantic graph to generate emotion encoding vectors. With action intention tag The emotion encoding vector is used to guide the digital human's facial expressions, tone of voice, and interaction context control; the action intention label represents the action type label sequence obtained through reasoning.

8. The digital human construction method based on three-dimensional perceptual diffusion and semantic graph reasoning according to claim 1, characterized in that, Semantic control signals and the three-dimensional perceptual features of the three-dimensional prior structure are used together as conditional inputs. With noise tensor Combining to form diffusion model input pairs The three-dimensional U-Net diffusion generation model, wherein each U-Net module consists of an encoder layer. Cross-layer residual connections Decoder layer The structure employs 3D convolution and multimodal conditional fusion mechanisms to satisfy: .

9. The digital human construction method based on three-dimensional perceptual diffusion and semantic graph reasoning according to claim 1, characterized in that, The specific method for step 6 is as follows: Step 6.1: Perform speech rhythm analysis on audio data A and use ASR to extract phoneme boundary sequences. And use the duration corresponding to each phoneme as the alignment unit; Step 6.2: Process the video frame sequence Perform frame rate normalization to ensure it corresponds to the phoneme beat; Step 6.3: Based on time alignment function Each frame is mapped to a corresponding phoneme time segment; Step 6.4: Use an image compositing engine to merge the aligned video frame sequence with audio A to generate an MP4 video stream with an audio track, forming the output result. .

Images