A method and related equipment for real-time construction of digital humans

Abstract

The application discloses a digital human real-time construction method and related equipment, which first aligns a face area and calculates an affine transformation matrix to generate noise, a mask, image and other latent representations; extracts real-time audio deep temporal features, combines them with various latent representations to form a combined latent tensor; inputs an audio condition diffusion model, generates synchronous face latent representations through multi-step denoising, and finally decodes and inversely transforms to synthesize output animation video streams. The application deeply fuses audio features through a cross-attention mechanism to improve mouth shape and audio synchronization accuracy; multiple types of latent representations provide rich initial information, and cooperate with denoising iteration to make mouth movements more delicate and natural; low-dimensional mapping and efficient matrix operation guarantee real-time performance and meet frame rate requirements; the latent representations are highly adaptive and enhance the generalization of different audio inputs, and comprehensively optimize digital human performance.

Description

Technical Field

[0001] This application relates to the field of digital human technology, and more specifically, to a method and related equipment for real-time construction of digital humans. Background Technology

[0002] Real-time digital humans, as virtual entities integrating computer technology, artificial intelligence, and graphics rendering, have been widely used in online customer service, virtual anchors, and gaming entertainment. Their core requirement is to achieve highly realistic, low-latency facial animation generation to enhance the naturalness of interaction and user experience. However, existing technologies struggle to meet these needs and exhibit significant shortcomings.

[0003] Current mainstream technologies can be mainly divided into four categories: parametric model-driven technologies, which rely on fixed mapping rules between audio features and facial parameters, making it difficult to capture subtle changes in lip shape, resulting in stiff lip movements and poor synchronization, and limited generalization ability for different speakers and emotions; image / video sequence stitching technologies, which are limited by the coverage of the material library, cannot present all combinations of lip shapes and expressions, with obvious stitching marks and insufficient coherence and realism; generative adversarial network (GAN)-based technologies, which have a contradiction between high fidelity and real-time performance, with slow inference speed of complex models, unstable training, and weak generalization ability; and neural radiation field (NeRF)-based technologies, which have high computational requirements for 3D reconstruction and rendering, poor real-time performance, and are difficult to deploy efficiently on general-purpose hardware.

[0004] These shortcomings have led to problems such as low lip-sync accuracy, unnatural movements, insufficient frame rate, and poor generalization to audio input in existing digital humans. Therefore, there is an urgent need for a new technology solution that can balance realism, real-time performance, and generalization. Summary of the Invention

[0005] This application provides a method and related equipment for real-time construction of digital humans. Through steps such as face alignment, multi-type latent representation generation, audio feature fusion, diffusion model denoising, and inverse transform synthesis, high-quality digital human animation synchronous with audio is generated.

[0006] A method for real-time construction of digital humans, comprising:

[0007] Perform face detection and key point localization on sample face video frames, determine face bounding boxes, and calculate the affine transformation matrix to align the face region to a standard size;

[0008] Based on the standardized face image obtained by the target image size and alignment, various latent representations are generated after low-dimensional space mapping and reparameterization adjustment. The various latent representations include at least noise latent representation, mask latent representation, mask image latent representation and image latent representation.

[0009] Extract deep temporal features from real-time audio streams and generate audio feature embedding vectors;

[0010] Using the audio feature embedding vector as conditional input, the various latent representations are scaled according to a preset ratio and concatenated along the channel dimension to form a combined latent tensor.

[0011] The combined latent tensor is input into the audio conditional diffusion model, and audio features are fused through a cross-attention mechanism to perform multi-step denoising iterations to generate a face latent representation synchronized with the audio.

[0012] The latent facial representation is decoded and restored to a pixel-space facial image. An inverse affine transformation is then performed on the pixel-space facial image based on the affine transformation matrix to generate the target facial image.

[0013] A digital human facial animation video stream is obtained by compositing the target face image onto the target background. The digital human body animation video stream corresponding to the real-time audio stream is then matched and combined to construct a real-time digital human animation.

[0014] Optionally, the standardized face image obtained based on the target image size and alignment is used to generate various latent representations after low-dimensional space mapping and reparameterization adjustment, including:

[0015] Based on the target image size and downsampling factor, a multi-frame noise tensor conforming to a standard normal distribution is constructed, and after scaling, a latent noise representation is generated.

[0016] The original mask tensor is scaled by interpolation to generate a latent space mask tensor and then rearranged in dimensions. The latent space mask tensor is then copied and spliced ​​according to the classifier’s free guidance requirements to output the mask latent representation.

[0017] The standardized face image obtained by alignment is multiplied pixel-level with a preset binary mask to obtain a masked image. The masked image is encoded into a low-dimensional latent vector, and after reparameterization sampling and parameter adjustment, a latent representation of the masked image is generated.

[0018] The standardized face image without masking is encoded into a low-dimensional latent vector, and then reparameterized and adjusted to generate an image latent representation.

[0019] Optionally, the process of generating the potential representation of the noise includes:

[0020] The shape of the latent spatial tensor is determined based on the target image size and downsampling factor;

[0021] Generate a single-frame standard normal distribution random tensor, and copy the single-frame random tensor along the time dimension to form a multi-frame noise tensor;

[0022] The multi-frame noise tensor is multiplied by the initial noise standard deviation of the diffusion scheduler to generate a scaled latent noise representation.

[0023] Optionally, the process of generating the latent representation of the mask includes:

[0024] The original mask tensor is scaled to the latent space resolution through interpolation to generate a latent space mask tensor;

[0025] The dimensions of the latent space mask tensor are rearranged to match the input format of the diffusion model.

[0026] Based on the classifier's free guidance requirements, the latent space mask tensor is copied and spliced ​​to output the mask latent representation.

[0027] Optionally, the process of generating the latent representation of the mask image includes:

[0028] The aligned, standardized face image is multiplied pixel-by-pixel with a preset binary mask to generate a masked image.

[0029] The masked images are processed in batches to obtain a first batch set of images;

[0030] Encoding operations are performed on each batch of images in the first batch set, and the mean vector and standard deviation vector representing their distribution are output. The first encoding latent vector of each batch is obtained through reparameterized sampling.

[0031] The first encoded latent vector is offset and scaled to generate a first adjusted latent vector;

[0032] Concatenate all batches of the first adjusted latent vector and adjust the dimensional order to output the masked latent representation.

[0033] Optionally, the process of generating the latent representation of the image includes:

[0034] The standardized face images without masking are processed in batches to obtain a second set of image batches;

[0035] Encoding operations are performed on each batch of images in the second batch set, and the mean vector and standard deviation vector representing their distribution are output. The second encoding latent vector of each batch is obtained through reparameterized sampling.

[0036] The second encoded latent vector is offset and scaled to generate a second adjusted latent vector;

[0037] The second adjusted latent vectors after all batch adjustments are concatenated and their dimensional order is adjusted to output the latent representation of the image.

[0038] Optionally, the audio conditional diffusion model includes a cascaded convolutional coding network, a spatiotemporal feature fusion network, a conditional decoding network, and a timing control module;

[0039] The convolutional coding network is used to perform a downsampling operation on the input combined latent tensor to extract multi-scale spatial features;

[0040] The spatiotemporal feature fusion network is used to inject the audio feature embedding vector into the multi-scale spatial features through a cross-attention mechanism to generate conditional enhancement features.

[0041] The conditional decoding network is used to perform upsampling operations on the conditional enhancement features and fuse the features of the encoding network at each scale to output a single-step denoising result;

[0042] The timing control module is used to schedule multi-step denoising iterations. By cyclically calling the convolutional coding network, the spatiotemporal feature fusion network, and the conditional decoding network during the diffusion process until the iteration termination condition is met, a latent face representation synchronized with the audio is obtained.

[0043] Optionally, the cross-attention mechanism can be implemented in the following ways:

[0044] The audio feature embedding vector is projected into attention key vector and value vector, and the multi-scale spatial features are projected into attention query vector;

[0045] Calculate the similarity matrix between the query vector and the key vector, and generate attention weights;

[0046] The value vector is aggregated based on the attention weights to output conditional enhancement features.

[0047] Optionally, the timing control module performs a multi-step denoising iteration process, including:

[0048] Initialize the current latent representation as the noise latent representation;

[0049] Generate time-step embedding vectors based on the number of diffusion steps;

[0050] The noise latent representation, time-step embedding vector, and audio features are input into the audio conditional diffusion model, and then sequentially passed through the convolutional coding network, the spatiotemporal feature fusion network, and the conditional decoding network to output the predicted noise.

[0051] Based on classifier-free guidance technology, conditional prediction noise and unconditional prediction noise are jointly calculated, and the conditional prediction and unconditional prediction results are fused according to a preset guidance ratio.

[0052] The current latent representation is updated using fused noise according to the diffusion scheduler update rules;

[0053] Return to the process of generating time-step embedding vectors based on the diffusion step count until the preset number of iterations is reached, and output the latent face representation.

[0054] Optionally, the process may also include accelerating the multi-step denoising iteration using the TensorRT acceleration engine.

[0055] Optionally, the latent facial representation is decoded and restored to a pixel-space facial image, and an inverse affine transformation is performed on the pixel-space facial image based on the affine transformation matrix to generate a target facial image, including:

[0056] The latent facial representation is decoded and restored using VAE to obtain a pixel-space facial image.

[0057] Calculate and use the inverse matrix of the affine transformation matrix to perform an inverse affine transformation on the pixel space face image, mapping it back to the face pose and position in the original video frame;

[0058] The face image after inverse affine transformation is upsampled using bilinear interpolation to match the resolution of the original video frame, thereby generating the target face image.

[0059] Optionally, obtaining a digital facial animation video stream by compositing the target face image onto a target background includes:

[0060] Interpolation and magnification are performed on the target background to match the resolution of the target face image.

[0061] The inverse of the affine transformation matrix is ​​called to map and locate the target face image to the position in the target background corresponding to the original face;

[0062] A shrinkage mask matching the contour of the target face image is constructed, a morphological erosion operation is performed on the shrinkage mask to shrink the boundary region, and a soft-edge mask is generated by Gaussian blurring.

[0063] The target face image and the interpolated and enlarged target background are weighted and fused using the soft edge mask to generate a digital facial animation video frame with natural transition edges.

[0064] The continuously generated digital human facial animation video frames are serialized in chronological order to form a digital human facial animation video stream.

[0065] Optionally, the process of combining the digital human facial animation video stream with the digital human body animation video stream corresponding to the real-time audio stream to construct a real-time digital human animation includes:

[0066] Based on the speaking state of the audio frames in the real-time audio stream, the display index is dynamically adjusted using a dual-state and pointer bounce mechanism;

[0067] Based on the display index, drive frame selection is performed from the digital human body animation video stream to achieve synchronous combination with the digital human face animation video stream, thereby generating the digital human real-time animation;

[0068] The execution process of the dual-state machine and pointer bounce mechanism includes:

[0069] Define a dual state including a static state and an active state, set an index direction variable and a boundary threshold, wherein the upper bound of the static region is a preset motion threshold, and the upper bound of the active region is the actual sequence length of the digital human body animation video stream;

[0070] When consecutive audio frames contain a speaking event, the state switches from the static state to the active state, and the index direction is forced to point to the active region;

[0071] When all consecutive audio frames are silent, the state falls back from the action state to the still state, and the index direction is forced to point to the starting point of the still area.

[0072] When the displayed index reaches the boundary, it bounces back to avoid going out of bounds;

[0073] Output the current display index after each update.

[0074] A real-time digital human construction system, comprising:

[0075] The face localization and detection module is used to perform face detection and key point localization on sample face video frames, determine the face bounding box, and calculate the affine transformation matrix that aligns the face region to a standard size.

[0076] The latent representation analysis module is used to generate various latent representations based on the standardized face image obtained by the target image size and alignment, after low-dimensional space mapping and reparameterization adjustment. The various latent representations include at least noise latent representation, mask latent representation, masked image latent representation and image latent representation.

[0077] The audio feature analysis module is used to extract deep temporal features from real-time audio streams and generate audio feature embedding vectors.

[0078] The combined latent tensor module is used to scale and concatenate the various latent representations along the channel dimension using the audio feature embedding vector as conditional input to form a combined latent tensor.

[0079] The iterative diffusion denoising module is used to input the combined latent tensor into the audio conditional diffusion model, fuse audio features through a cross-attention mechanism, and perform multi-step denoising iterations to generate a face latent representation synchronized with the audio.

[0080] The facial animation generation module is used to decode and restore the latent facial representation into a pixel-space facial image, and perform an inverse affine transformation on the pixel-space facial image based on the affine transformation matrix to generate a target facial image.

[0081] The digital human combination construction module is used to obtain a digital human facial animation video stream by compositing the target face image onto the target background, and to match and combine the digital human body animation video stream corresponding to the real-time audio stream to construct a real-time digital human animation.

[0082] Optionally, the real-time digital human construction system adopts a multi-process asynchronous architecture to achieve real-time processing, wherein:

[0083] The audio feature analysis module runs in the main process and asynchronously transmits the audio feature embedding vector to the child process through the first security queue.

[0084] The iterative diffusion denoising module runs in the first subprocess and processes the combined latent tensors through the TensorRT acceleration engine.

[0085] The facial animation generation module runs in a second sub-process and receives the potential facial representation through a second security queue.

[0086] The main process and the child process exchange affine transformation matrices and potential representation data through a shared memory pool.

[0087] A device for real-time construction of a digital human, including a memory and a processor;

[0088] The memory is used to store programs;

[0089] The processor is configured to execute the program to implement the various steps of the real-time digital human construction method as described in any of the preceding claims.

[0090] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the real-time digital human construction method as described in any of the preceding claims.

[0091] A computer program product includes a computer program that, when run by a processor, performs the steps of the real-time digital human construction method as described in any of the preceding claims.

[0092] As can be seen from the above technical solutions, the digital human real-time construction method and related equipment provided in this application embodiment achieve high-quality digital human animation generation synchronized with audio through steps such as face alignment, multi-type latent representation generation, audio feature fusion, diffusion model denoising and inverse transform synthesis.

[0093] This solution addresses the shortcomings of existing technologies at the technical level: First, in audio feature processing, it extracts deep temporal features from real-time audio streams and generates embedding vectors. Leveraging the cross-attention mechanism of the audio conditional diffusion model, audio features are deeply integrated into the generation of facial latent representations. This mechanism allows the model to accurately capture subtle changes in intonation and rhythm in the audio, thereby driving corresponding adjustments in lip movements. Compared to the fixed mapping rules of parameterized models, synchronization accuracy is significantly improved, effectively solving the problem of insufficient synchronization. Second, in the latent representation generation stage, based on the target image size and standardized face image, various latent representations, including noise, masks, and images, are generated through low-dimensional space mapping and reparameterization. These representations describe facial features from different dimensions: noise latent representation increases generation diversity, mask latent representation is used to identify areas that need to remain unchanged, masked image latent representation can highlight key facial regions, and image latent representation preserves basic image information. After scaling and stitching together multiple types of latent representations, rich initial information is provided for the diffusion model. Combined with multi-step denoising iterations, it can generate delicate and natural lip movements, overcoming the shortcomings of insufficient coherence and realism caused by the limitations of the material library in the stitching technology. Thirdly, in terms of real-time performance, high-dimensional image information is transformed into low-dimensional latent representations through low-dimensional spatial mapping, which significantly reduces the amount of data processing. At the same time, affine transformation and inverse transformation use matrix operations, which are computationally efficient and enable the entire generation process to respond quickly, solving the problem of insufficient frame rate caused by the large amount of computation in NeRF technology and the slow inference of GAN models, thus meeting the requirements for real-time output. Fourthly, regarding the generalization of audio input, since various latent representations have strong adaptability and can be effectively combined with different audio feature embedding vectors, and the diffusion model can learn multiple audio patterns during training, it can generate appropriate lip movements regardless of different speech rates and accents, breaking through the limitation of poor generalization of existing technologies for specific audio inputs. Attached Figure Description

[0094] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0095] Figure 1This is a flowchart of a real-time digital human construction method disclosed in an embodiment of this application;

[0096] Figure 2 This is a schematic diagram of a real-time digital human construction method disclosed in an embodiment of this application;

[0097] Figure 3 This is a schematic diagram of a real-time digital human construction system disclosed in an embodiment of this application;

[0098] Figure 4 This is a hardware structure block diagram of a digital human real-time construction device disclosed in an embodiment of this application. Detailed Implementation

[0099] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0100] This application can be used in a wide variety of general-purpose or special-purpose computing device environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, distributed computing environments including any of the above devices, etc.

[0101] The following section introduces the solution proposed in this application. The technical solution is as follows, and details are provided below.

[0102] Figure 1 This is a flowchart of a real-time digital human construction method disclosed in an embodiment of this application.

[0103] Figure 2 This is a schematic diagram of a real-time digital human construction method disclosed in an embodiment of this application.

[0104] like Figure 1 and Figure 2 As shown, the method may include:

[0105] Step S1: Perform face detection and key point localization on the sample face video frames, determine the face bounding box, and calculate the affine transformation matrix to align the face region to the standard size.

[0106] Specifically, for the input sample face video frames, face detection algorithms such as MTCNN or RetinaFace are used for frame-by-frame processing to locate the face region and output the coordinates of the face bounding box. Simultaneously, key feature points are identified (including but not limited to the corners of the eyes, the tip of the nose, the corners of the mouth, and the jawline contour points; the number of key points can be selected according to accuracy requirements). Based on these key point coordinates, an affine transformation matrix is ​​calculated: by matching the detected facial key points with preset standard facial key points (corresponding to standard sizes, such as 256×256 pixels; the standard size can be adjusted according to the application scenario), the optimal affine transformation matrix that can translate, rotate, and scale the original face region to the standard size is solved. This matrix is ​​not only used for face alignment in the current frame but is also stored for pose reconstruction of subsequent generated images, ensuring that the face region maintains geometric consistency during standardization processing.

[0107] Step S2: Based on the standardized face image obtained by the target image size and alignment, various potential representations are generated after low-dimensional space mapping and reparameterization adjustment.

[0108] Specifically, based on the size of the target output image and the standardized face image obtained by alignment according to the affine transformation matrix, various latent representations are generated through low-dimensional space mapping and reparameterization adjustment. These latent representations include at least noise latent representation, mask latent representation, masked image latent representation, and image latent representation.

[0109] Noise latent representation: The shape of the latent spatial tensor is determined based on the target size and downsampling factor, generating single-frame random noise that conforms to a standard normal distribution. This noise is then copied along the time dimension to match the number of video frames and scaled using the initial noise standard deviation of the diffusion scheduler to form a noise latent representation, providing a basis for randomness in the generation process.

[0110] Masked latent representation: Resized binary masks (highlighting dynamic regions such as the mouth) are used to identify regions that need to remain unchanged. These masks are resized to fit the latent space and are provided directly as conditional information to the diffusion model, guiding the model to keep specific regions invariant during generation.

[0111] Masked Image Latent Representation: A standardized face image is multiplied pixel-level with a preset fixed binary mask (highlighting key dynamic areas such as the mouth) to generate a masked image. Then, the masked image is mapped to a low-dimensional latent space by a VAE encoder, which outputs the mean and standard deviation of the latent distribution. A latent vector is generated using reparameterization techniques. After being adjusted by scaling and offset factors configured by the VAE, the masked image latent representation is formed, which preserves the feature information of the masked area (mainly around the mouth) and provides reference information for the diffusion model of the areas that need to remain unchanged in the current frame.

[0112] Image latent representation: For a standardized face image that has not undergone masking, the same encoding and reparameterization process as that of masked latent representation is used to generate a low-dimensional latent vector that preserves complete facial features. After scaling and offset adjustment, the image latent representation is formed, providing basic facial feature information.

[0113] Step S3: Extract deep temporal features from the real-time audio stream and generate audio feature embedding vectors.

[0114] Specifically, for temporal feature extraction of real-time input audio streams (such as speech signals with a sampling rate of 16kHz), a model such as the Whisper model can be used: First, the audio stream is segmented into segments of fixed duration (such as 30ms / frame). The Whisper encoder extracts deep temporal features, including speech content, intonation, rhythm, and emotion-related acoustic features. These features are mapped through a fully connected layer into fixed-dimensional (such as 768-dimensional) audio feature embedding vectors. These vectors not only contain semantic information of the speech but also retain dynamic changes over time, which can be directly used as conditional input for subsequent diffusion models to achieve accurate association between audio and facial animation.

[0115] Step S4: Using the audio feature embedding vector as a conditional input, scale the various latent representations according to a preset ratio and concatenate them along the channel dimension to form a combined latent tensor.

[0116] Specifically, using the audio feature embedding vector generated in step S3 as conditional input, the noise latent representation, mask latent representation, masked image latent representation, and image latent representation obtained in step S2 are first scaled by a preset ratio (the scaling factor is determined through model training and is used to balance the contribution weights of different latent representations). Then, the scaled latent representations are concatenated with the audio feature embedding vector along the channel dimension: for example, the noise latent representation has 4 channels, the mask and image latent representations each have 4 channels, and the audio feature embedding vector has 4 channels after dimensional expansion. After concatenation, a combined latent tensor with 16 channels is formed. This tensor integrates noise randomness, key region features, overall face features, and audio conditional information, providing complete input data for the diffusion model.

[0117] Step S5: Input the combined latent tensor into the audio conditional diffusion model, fuse audio features through a cross-attention mechanism, and perform multi-step denoising iterations to generate a face latent representation synchronized with the audio.

[0118] Specifically, the combined latent tensor is input into the audio conditional diffusion model, and the latent representation of the face is generated through the following process:

[0119] Cross-attention fusion: Audio features are embedded into vectors and projected as attention keys and values, while multi-scale spatial features extracted by the model backbone network are projected as queries; the similarity matrix between the query and the key is calculated to obtain attention weights, and value vectors are aggregated based on the weights, so that audio features are deeply integrated into visual features at all levels, guiding the generation of facial dynamics that match the audio.

[0120] Multi-step denoising iteration: Initialize the current latent representation as a noisy latent representation; Iterate according to a preset number of steps: Generate a time-step embedding vector (identifying the current denoising stage); Input the current latent representation, time-step vector, and audio features into the model, and output predicted noise; Optimize the result using classifier-free guidance (fusing conditional and unconditional predicted noise, weighted according to a preset ratio); Update the current latent representation using fused noise according to scheduler rules (such as the DDIM update formula); After the iteration, obtain a face latent representation synchronized with the audio.

[0121] Furthermore, it also includes using the TensorRT acceleration engine to accelerate and optimize the multi-step denoising iteration process. The specific optimization methods are as follows:

[0122] The core computational units of the audio conditional diffusion model are converted into a TensorRT-compatible intermediate representation (IR). TensorRT's optimizer performs operator fusion (e.g., merging convolution, batch normalization, and activation functions into a single computational unit), inter-layer connection optimization, and computation graph simplification, reducing memory accesses and computational redundancy. FP16 (half-precision) or INT8 (integer precision) quantization techniques are employed to reduce model parameter storage and computational complexity while maintaining generation quality. FP16 quantization reduces GPU memory usage and computation by 50%, while INT8 quantization further compresses it to 25%. Simultaneously, quantization errors are corrected using a calibration dataset to avoid excessive precision loss. Kernel automatic tuning is performed based on the architectural characteristics of NVIDIA GPUs (e.g., TensorCore, CUDA cores), selecting the optimal thread block partitioning, memory access mode, and computation scheduling strategy to maximize hardware utilization. For example, TensorCore is used to accelerate matrix multiplication operations, improving the computational efficiency of cross-attention mechanisms and convolutional layers. The computational task of multi-step denoising iteration is decomposed into stages such as preprocessing, model inference, and postprocessing. Parallel pipeline processing of each stage is realized through TensorRT's asynchronous execution interface, reducing data transmission latency between CPU and GPU and ensuring that the time consumption of each denoising iteration is controlled within the real-time threshold.

[0123] Step S6: Decode the latent face representation to restore it to a pixel space face image, and perform an inverse affine transformation on the pixel space face image based on the affine transformation matrix to generate the target face image.

[0124] Specifically, the latent facial representation is decoded using a variational autoencoder (VAE) decoding module to map it from the feature space to the pixel space, generating a corresponding pixel-space face image. Next, based on the affine transformation matrix, its inverse matrix is ​​obtained through matrix inversion, and this inverse matrix is ​​used to perform an inverse affine transformation on the pixel-space face image, restoring the image to the state of the face in the original video frame in terms of geometric pose and spatial position. Finally, to ensure the image resolution matches the original video frame, the face image after the inverse affine transformation is upsampled using a bilinear interpolation algorithm. By constructing a continuous grayscale mapping relationship between pixels, the image resolution is increased to match the original video frame, ultimately generating the target face image.

[0125] The specific execution process is as follows:

[0126] ① The latent facial representation is decoded and restored using VAE to obtain a pixel-space facial image.

[0127] Specifically, the latent representation of the face is an abstract face feature in the form of a feature vector. This is input into the decoder module of the variational autoencoder (VAE). The decoder performs feature mapping and dimensional expansion on the latent representation through a multi-layer neural network structure, gradually transforming the abstract features into pixel information with spatial dimensions. Finally, it outputs a pixel space face image with a preset size. This image contains visual features such as the texture and contour of the face, but it has not yet been aligned with the face pose and position of the original video frame.

[0128] ② Calculate and use the inverse matrix of the affine transformation matrix to perform an inverse affine transformation on the pixel space face image, mapping it back to the face pose and position in the original video frame.

[0129] Specifically, the affine transformation matrix is ​​inverted based on the rules of linear algebra to obtain the corresponding affine inverse matrix. Then, the affine inverse matrix is ​​applied to each pixel coordinate of the face image in pixel space. Geometric correction operations such as rotation, translation, scaling, and cropping are performed on the image through coordinate transformation formulas to ensure that the pose (such as pitch angle and yaw angle) and spatial position (such as coordinates in the video frame) of the face image in pixel space are consistent with the initial state of the face in the original video frame, thereby achieving precise alignment of the face image at the geometric level.

[0130] ③ Upsample the face image after inverse affine transformation using bilinear interpolation to match the resolution of the original video frame and generate the target face image.

[0131] Specifically, for face images that may have reduced resolution after inverse affine transformation, the pixel value of each pixel at the target resolution is calculated using a bilinear interpolation algorithm: for the target pixel to be generated, its corresponding floating-point coordinates in the original image are determined, and four neighboring pixels around these coordinates are selected. The grayscale value or RGB value of the target pixel is calculated based on the distance weight of each neighboring pixel. The upsampling process is completed by traversing all pixels at the target resolution, so that the number of pixels in the width and height of the final generated target face image is completely consistent with the original video frame, and the image details remain continuous and smooth.

[0132] Step S7: By synthesizing the target face image onto the target background to obtain a digital human facial animation video stream, match and combine the digital human body animation video stream corresponding to the real-time audio stream to construct a real-time digital human animation.

[0133] Specifically, morphological erosion and Gaussian blur are used to generate a soft mask, which is then weighted and fused with the original background. A binary mask for the face region is created, and the mask boundary is shrunk through morphological erosion to avoid edge imperfections. Gaussian blur parameters are adaptively calculated based on the face region area, and the eroded mask is Gaussian blurred to generate a soft mask with smooth edges. A weighted fusion operation is then performed to achieve a natural blending of the face region with the original video background, generating the final digital human animation frame.

[0134] As can be seen from the above technical solutions, the digital human real-time construction method and related equipment provided in this application embodiment achieve high-quality digital human facial animation generation synchronized with audio through steps such as face alignment, multi-type latent representation generation, audio feature fusion, diffusion model denoising and inverse transform synthesis.

[0135] This solution addresses the shortcomings of existing technologies at the technical level: First, in audio feature processing, it extracts deep temporal features from real-time audio streams and generates embedding vectors. Leveraging the cross-attention mechanism of the audio conditional diffusion model, audio features are deeply integrated into the generation of facial latent representations. This mechanism allows the model to accurately capture subtle changes in intonation and rhythm in the audio, thereby driving corresponding adjustments in lip movements. Compared to the fixed mapping rules of parameterized models, synchronization accuracy is significantly improved, effectively solving the problem of insufficient synchronization. Second, in the latent representation generation stage, based on the target image size and standardized face image, various latent representations, including noise, masks, and images, are generated through low-dimensional space mapping and reparameterization. These representations describe facial features from different dimensions: noise latent representation increases generation diversity, mask latent representation is used to identify areas that need to remain unchanged, masked image latent representation can highlight key facial regions, and image latent representation preserves basic image information. After scaling and stitching together multiple types of latent representations, rich initial information is provided for the diffusion model. Combined with multi-step denoising iterations, it can generate delicate and natural lip movements, overcoming the shortcomings of insufficient coherence and realism caused by the limitations of the material library in the stitching technology. Thirdly, in terms of real-time performance, high-dimensional image information is transformed into low-dimensional latent representations through low-dimensional spatial mapping, which significantly reduces the amount of data processing. At the same time, affine transformation and inverse transformation use matrix operations, which are computationally efficient and enable the entire generation process to respond quickly, solving the problem of insufficient frame rate caused by the large amount of computation in NeRF technology and the slow inference of GAN models, thus meeting the requirements for real-time output. Fourthly, regarding the generalization of audio input, since various latent representations have strong adaptability and can be effectively combined with different audio feature embedding vectors, and the diffusion model can learn multiple audio patterns during training, it can generate appropriate lip movements regardless of different speech rates and accents, breaking through the limitation of poor generalization of existing technologies for specific audio inputs.

[0136] In some embodiments of this application, the process of generating various latent representations after low-dimensional space mapping and reparameterization adjustment of the standardized face image obtained based on the target image size and alignment in step S2 is described, which may specifically include:

[0137] Step S21: Based on the target image size and downsampling factor, construct a multi-frame noise tensor that conforms to the standard normal distribution, and generate a latent noise representation after scaling;

[0138] Step S22: Scale the original mask tensor to generate a latent space mask tensor through interpolation and rearrange its dimensions. Perform copy and splice operations on the latent space mask tensor according to the classifier's free guidance requirements to output the mask latent representation.

[0139] Step S23: Multiply the aligned standardized face image with a preset binary mask at the pixel level to obtain a masked image. Encode the masked image into a low-dimensional latent vector. After reparameterization sampling and parameter adjustment, generate a latent representation of the masked image.

[0140] Step S24: Encode the unmasked standardized face image into a low-dimensional latent vector, and generate an image latent representation after reparameterization sampling and parameter adjustment.

[0141] Specifically, based on the target image size and a preset downsampling factor, the shape of the latent spatial tensor is determined (e.g., when the target size is 256×256, the downsampled latent tensor size is 64×64, with 4 channels). A single-frame random noise tensor conforming to a standard normal distribution (N(0,1)) is generated, and this tensor is copied along the time dimension to match the number of frames in the video sequence (e.g., if generating a 30-frame animation, it is copied 30 times), forming a multi-frame noise tensor. The multi-frame noise tensor is multiplied by the initial noise standard deviation of the diffusion scheduler (e.g., DDIMScheduler) to complete the scaling adjustment of the noise amplitude, ultimately generating a latent noise representation, providing initial random input for the subsequent diffusion denoising process. The downsampling factor can be adjusted according to the model accuracy requirements (e.g., 8x downsampling is suitable for low-computing-power scenarios); the distribution of random noise can be replaced with a truncated normal distribution to limit extreme values; the generation of multi-frame noise can use inter-frame correlation constraints to avoid animation jitter caused by sudden noise changes.

[0142] The original mask tensor is scaled using interpolation to match its size with the dimensions of the latent space, thus generating a latent space mask tensor. Next, the latent space mask tensor is rearranged in dimensions to adjust its data dimension order and adapt to the dimensionality requirements of subsequent processing. Then, based on the actual needs of the classifier's free guidance (e.g., guidance strength, number of guidances), the rearranged latent space mask tensor is copied and concatenated to expand its scale in the guidance dimension, ultimately outputting a mask latent representation. This representation can be used to constrain specific regions during the generation process.

[0143] The aligned, standardized face image is multiplied at the pixel level with a preset binary mask (e.g., a mask image where only the mouth region is set to 1 and the rest to 0) to obtain a masked image (retaining only pixel information of dynamic key regions such as the mouth). The masked image is input into the encoder in a preset batch size (e.g., 8 images per batch), and the encoder outputs a mean vector and a standard deviation vector representing its probability distribution. A low-dimensional latent vector is generated through reparameterized sampling (sampling noise vectors from a standard normal distribution and calculating it using the formula "latent vector = mean + standard deviation × noise"). This vector is then adjusted using preset offset factors (e.g., vae.config.shift_factor) and scaling factors (e.g., vae.config.scaling_factor) (adjustment formula: latent vector = (encoded vector - offset factor) × scaling factor). The adjusted vectors from all batches are concatenated, and the dimensional order is adjusted (e.g., from "batch × channel × height × width" to the model input format) to finally generate the latent representation of the masked image, used for feature learning focusing on dynamic facial regions. Binary masks can be replaced with grayscale masks to achieve gradient weights; the batch size of the encoder can be dynamically adjusted according to the video memory; reparameterized sampling can optimize the distribution fitting accuracy through KL divergence constraints.

[0144] For the complete, standardized face image without masking, the same encoding, reparameterization sampling, and parameter adjustment process as in step S23 is performed: the complete face image is input into the encoder in batches, and the mean and standard deviation vectors are output. Low-dimensional latent vectors are generated through reparameterization sampling. After offset and scaling adjustments, all batch vectors are concatenated and their dimensional order is adjusted to generate the image latent representation. This representation preserves the complete static features of the face (such as face shape, skin color, and eyebrow shape), providing basic facial feature information for the diffusion model and ensuring consistency between the generated digital face and the reference image. The encoding of the complete face image can share encoder weights with the masked image to reduce parameters; the offset and scaling factors for parameter adjustment can be dynamically learned through model training, rather than being fixed values; the image latent representation can incorporate identity feature encoding to enhance adaptability to different personalities.

[0145] Based on this, the specific generation processes of noise latent representation, mask latent representation, masked image latent representation, and image latent representation are further explained.

[0146] The process of generating the latent representation of the noise includes:

[0147] ① Determine the shape of the latent spatial tensor based on the target image size and downsampling factor;

[0148] ② Generate a single-frame standard normal distribution random tensor, and copy the single-frame random tensor along the time dimension to form a multi-frame noise tensor;

[0149] ③ Multiply the multi-frame noise tensor by the initial noise standard deviation of the diffusion scheduler to generate a scaled latent noise representation.

[0150] Specifically, the shape of the latent spatial tensor is calculated based on the target image size (e.g., 256×256 pixels) and a preset downsampling factor (e.g., 4 times). For example, when the target image size is 256×256, after 4 times downsampling, the spatial dimension of the latent spatial tensor is 64×64. Combined with the number of channels (e.g., 4 channels), a tensor shape of "batch × channel × height × width" is formed (e.g., 1×4×64×64). A single-frame random tensor conforming to a standard normal distribution (N(0,1)) is generated, and its shape is consistent with the determined latent spatial tensor shape. This single-frame random tensor is copied along the time dimension (i.e., the direction of the video frame sequence), and the number of copies is consistent with the number of frames in the target video (e.g., if a 30-frame animation is generated, it is copied 30 times), forming a multi-frame noise tensor (with a shape of "number of frames × channel × height × width"). The initial noise standard deviation parameter of the diffusion scheduler (such as DDIMScheduler) is called, and the generated multi-frame noise tensor is multiplied element-wise with the standard deviation to complete the scaling adjustment of the noise amplitude. Finally, the scaled noise latent representation is obtained, which provides the initial random input for the multi-step denoising process of the diffusion model.

[0151] The process of generating the latent representation of the mask includes:

[0152] ① Scale the original mask tensor to the latent spatial resolution through interpolation to generate a latent spatial mask tensor;

[0153] ② The dimensions of the latent space mask tensor are rearranged to match the input format of the diffusion model;

[0154] ③ Based on the classifier's free guidance requirements, the latent space mask tensor is copied and spliced ​​to output the mask latent representation.

[0155] Specifically, a preset binary mask image (e.g., a mask where only the mouth region is 1 and other regions are 0) is loaded to obtain a mask tensor, whose data type is binary. The original mask tensor is directly scaled from the image resolution (e.g., 256×256) to the latent spatial resolution (e.g., 32×32) using bilinear interpolation, with the scaling ratio determined by the VAE's downsampling factor, generating a latent spatial mask tensor. Dimensional manipulation tools such as einops are used to rearrange the dimensions of the latent spatial mask tensor, changing the dimensions from "frames × channels × height × width" to "1 × channel × frames × height × width" to match the diffusion model's requirements for the input tensor's dimensionality format. It is determined whether to enable Classifier Free Guidance (CFG). If enabled, the latent spatial mask tensor is copied and concatenated along the batch dimension to form conditional / unconditional input pairs; if not enabled, the original tensor remains unchanged, and the final output is the mask latent representation. This representation directly indicates the spatial regions that the diffusion model needs to focus on during denoising, serving as spatial conditional information to guide the generation process.

[0156] The process of generating the latent representation of the mask image includes:

[0157] ① Perform pixel-by-pixel multiplication between the aligned standardized face image and the preset binary mask to generate a masked image;

[0158] ② The masked images are processed in batches to obtain a first set of image batches;

[0159] ③ Perform encoding operations on each batch of images in the first batch set, output the mean vector and standard deviation vector representing their distribution, and obtain the first encoding latent vector of each batch through reparameterized sampling;

[0160] ④ Perform offset and scaling operations on the first encoded latent vector to generate a first adjusted latent vector;

[0161] ⑤ Concatenate all batches of the first adjusted latent vector and adjust the dimension order to output the mask latent representation.

[0162] Specifically, a preset fixed binary mask image (e.g., a mask where only the mouth area is 1 and other areas are 0) is loaded. The aligned, standardized face image is then multiplied pixel-by-pixel by this mask; that is, each pixel value of the standardized face image is multiplied by the corresponding binary value in the mask, generating a masked image (only retaining pixel information of key areas such as the mouth, setting non-masked areas to zero). The masked images are batched according to a preset batch size (e.g., 4 images per batch), forming a masked image batch set. The batch size can be dynamically adjusted according to the hardware memory capacity. For each batch of images in the masked image batch set, the image is input to a VAE encoder for encoding, outputting the mean vector and standard deviation vector representing the probability distribution of the batch of masked images in the latent space. Through reparameterized sampling (randomly sampling from this latent distribution and calculating according to the formula "mask-encoded latent vector = mean vector + standard deviation vector × noise vector"), the mask-encoded latent vector for each batch is obtained. The masked encoded latent vectors are adjusted using the offset and scaling factors configured in the VAE model. The adjustment formula is: Masked Adjusted Latent Vector = (Mask-Encoded Latent Vector - Offset Factor) × Scaling Factor, where the offset and scaling factors are pre-trained parameters of the VAE model used to normalize the latent vectors to a suitable distribution space. All batches of masked adjusted latent vectors are concatenated along the batch dimension to form a complete latent tensor, and the dimensional order is adjusted (e.g., from "frame number × channel × height × width" to "1 × channel × frame number × height × width") to match the input format of the subsequent diffusion model. The final output is a masked image latent representation that preserves the visual feature information of the masked region, providing the diffusion model with reference conditions for maintaining consistent regions in the current frame.

[0163] The process of generating the latent representation of the image includes:

[0164] ① The standardized face images without masking are processed in batches to obtain a second set of image batches;

[0165] ② Perform encoding operations on each batch of images in the second image batch set, output the mean vector and standard deviation vector representing their distribution, and obtain the second encoding latent vector of each batch through reparameterized sampling;

[0166] ③ Perform offset and scaling operations on the second encoded latent vector to generate a second adjusted latent vector;

[0167] ④ Concatenate all batches of adjusted second latent vectors and adjust the dimensional order to output the latent representation of the image.

[0168] Specifically, for complete, standardized face images that have not undergone masking (retaining full-face pixel information), batch processing is performed using the same batch size as in the masked image latent representation processing, forming a second image batch set. For each batch of images in the second image batch set, the same encoding operation as in the masked image latent representation processing is performed: the input encoder outputs a mean vector and a standard deviation vector representing its probability distribution. The second encoded latent vector for each batch is obtained through reparameterized sampling. The second encoded latent vector is adjusted using the same offset and scaling factors as in the masked image latent representation processing, with the adjustment formula: Second Adjusted Latent Vector = (Second Encoded Latent Vector - Offset Factor) × Scaling Factor, ensuring that the distribution characteristics of the image latent representation and the masked latent representation in the latent space are consistent. All batches of second adjusted latent vectors are concatenated along the batch dimension to form a complete latent tensor. The dimensional order is then adjusted in the same way as in the masked image latent representation processing, outputting the image latent representation, which retains the static features of the complete face (such as face shape, skin color, eyebrow shape, etc.).

[0169] In some embodiments of this application, an audio conditional diffusion model is described, which includes a cascaded convolutional coding network, a spatiotemporal feature fusion network, a conditional decoding network, and a timing control module. Specifically:

[0170] The convolutional coding network is used to perform a downsampling operation on the input combined latent tensor to extract multi-scale spatial features;

[0171] The spatiotemporal feature fusion network is used to inject the audio feature embedding vector into the multi-scale spatial features through a cross-attention mechanism to generate conditional enhancement features.

[0172] The conditional decoding network is used to perform upsampling operations on the conditional enhancement features and fuse the features of the encoding network at each scale to output a single-step denoising result;

[0173] The timing control module is used to schedule multi-step denoising iterations. By cyclically calling the convolutional coding network, the spatiotemporal feature fusion network, and the conditional decoding network during the diffusion process until the iteration termination condition is met, a latent face representation synchronized with the audio is obtained.

[0174] Specifically, the convolutional coding network consists of multiple cascaded convolutional blocks, each containing a convolutional layer, a batch normalization layer, and an activation function (such as ReLU). The input combined latent tensor is downsampled through layer-by-layer convolutional operations: the initial layer uses a 3×3 convolutional kernel with a stride of 2 to reduce the feature map size to half its original size while increasing the number of channels (e.g., from 16 channels to 32 channels); subsequent layers repeat similar operations, generating multi-scale spatial features with different spatial resolutions through multiple downsampling operations. These features not only contain local details of the face (such as lip texture) but also encompass global structural information (such as facial contours), laying the foundation for subsequent fusion of audio features.

[0175] The spatiotemporal feature fusion network achieves deep fusion of audio and visual features through a cross-attention mechanism. First, the audio feature embedding vector is transformed into attention "key vectors" and "value vectors" through a linear projection layer. Then, the multi-scale spatial features output by the convolutional coding network (processed separately for each scale) are transformed into attention "query vectors" through another linear projection layer. The similarity matrix between the query vector and the key vector is calculated (e.g., through dot product), normalized using the Softmax function to obtain attention weights, and then weighted and aggregated based on these weights to generate conditionally enhanced features that fuse audio information. This process is performed separately on spatial features at multiple scales, ensuring that visual features at different resolutions are accurately guided by audio features.

[0176] The conditional decoding network adopts a structure symmetrical to the convolutional coding network, consisting of multiple cascaded deconvolutional blocks. Each deconvolutional block contains a transposed convolutional layer, a batch normalization layer, and an activation function. The input conditional enhancement features are upsampled through layer-by-layer deconvolution operations: the initial layer uses a 3×3 transposed convolutional kernel with a stride of 2 to double the feature map size while reducing the number of channels; at each upsampling stage, spatial features of the corresponding scale in the convolutional coding network are fused through skip connections (e.g., when decoding to a 64×64 scale, features at the 64×64 scale in the encoding stage are fused), supplementing detailed information. Finally, the output convolutional layer outputs the single-step denoising result (i.e., the prediction of noise in the current latent representation).

[0177] The timing control module is responsible for coordinating the entire process of multi-step denoising iterations: First, the current latent representation is initialized as a noisy latent representation, and the total number of denoising steps is set. In each iteration, the embedding vector for the current time step (encoding time information through sine and cosine functions) is generated and input into the convolutional coding network along with the combined latent tensor and audio feature embedding vector. The convolutional coding network, spatiotemporal feature fusion network, and conditional decoding network are called iteratively to obtain the noise prediction result for the current step. According to the update rules of the diffusion scheduler (such as DDIM), the current latent representation is updated using the noise prediction. The above process is repeated until the preset number of denoising steps is completed, and the face latent representation synchronized with the audio is output. At the same time, the module integrates a TensorRT acceleration interface, converting the inference process of the convolutional coding network, fusion network, and decoding network into execution by the TensorRT optimization engine, ensuring the efficiency of each iteration.

[0178] The implementation methods of the cross-attention mechanism include:

[0179] The audio feature embedding vector is projected into attention key vector and value vector, and the multi-scale spatial features are projected into attention query vector;

[0180] Calculate the similarity matrix between the query vector and the key vector, and generate attention weights;

[0181] The value vector is aggregated based on the attention weights to output conditional enhancement features.

[0182] Specifically, the generated audio feature embedding vector is projected into an attention key vector and a value vector through two independent linear projection layers (learnable weight matrices), respectively. The key vector is used to calculate similarity, and the value vector is used for feature aggregation. At the same time, the multi-scale spatial features output by the convolutional coding network are projected into an attention query vector through another linear projection layer. The query vector has the same dimension as the key vector and the value vector.

[0183] The similarity matrix between the query vector and the key vector is calculated using vector dot product, where division by the square root of the dimension helps mitigate the vanishing gradient problem. The similarity matrix is ​​then normalized using the Softmax function to generate an attention weight matrix. The weights are in the range [0,1] and sum to 1 in each row, representing the correlation strength between audio features and spatial features at each location.

[0184] Feature Aggregation and Output: A weighted aggregation operation is performed on the value vector based on the attention weight matrix, i.e., Conditional Enhancement Feature = Attention(Weight) × Value. This weighted aggregation is achieved through matrix multiplication, giving higher weights to regions in the spatial features that are highly correlated with the audio features. The aggregated features are then residually concatenated with the original multi-scale spatial features (e.g., feature addition), ultimately outputting a conditional enhancement feature that fuses audio information. This feature preserves spatial details while carrying audio-driven dynamic guidance information.

[0185] Furthermore, the timing control module performs a multi-step denoising iteration process, including:

[0186] ① Initialize the current latent representation as the noise latent representation;

[0187] ② Generate time-step embedding vectors based on the number of diffusion steps;

[0188] ③ Input the noise latent representation, time step embedding vector and audio features into the audio conditional diffusion model, and output the predicted noise by passing it sequentially through the convolutional coding network, the spatiotemporal feature fusion network and the conditional decoding network;

[0189] ④ Based on classifier-free guidance technology, conditional prediction noise and unconditional prediction noise are jointly calculated, and the conditional prediction and unconditional prediction results are fused according to a preset guidance ratio;

[0190] ⑤ Update the current latent representation using fused noise according to the diffusion scheduler update rules;

[0191] ⑥ Return to the process of generating time-step embedding vectors based on the diffusion step count until the preset number of iterations is reached, and output the latent face representation.

[0192] Specifically, the current latent representation is initialized with the noisy latent representation generated in step S21, serving as the starting point for the diffusion denoising process. This representation contains random noise information conforming to a standard normal distribution, providing initial input for subsequent denoising iterations. Based on the preset total number of diffusion steps and the current iteration's time step (e.g., step t, t∈[0, total number of steps]), the time step is encoded using sine and cosine functions to generate a time step embedding vector. This vector is used to transmit temporal information of the current denoising stage to the model, helping the model distinguish noise characteristics at different iteration stages. The current latent representation, the time step embedding vector, and the audio feature embedding vector are jointly input into the audio conditional diffusion model. The model sequentially extracts multi-scale spatial features through a convolutional coding network, fuses audio features through a spatiotemporal feature fusion network using a cross-attention mechanism, and performs upsampling and outputs single-step prediction noise through a conditional decoding network. This noise represents the noise components that need to be removed from the current latent representation.

[0193] Based on classifier-free guided noise (CFG) technology, two types of noise prediction results are calculated: conditional prediction noise (noise prediction when audio features are input) and unconditional prediction noise (noise prediction when no audio features are input). The two prediction results are weighted and fused according to a preset guidance ratio (fused noise = conditional prediction noise × guidance ratio + unconditional prediction noise × (1 - guidance ratio)), enhancing the guiding role of audio features in the denoising process and improving the synchronization between the generated result and the audio. According to the update rules of a diffusion scheduler (such as DDIMScheduler), the generated fused noise is used to update the current latent representation, specifically through the formula "current latent representation = current latent representation - fused noise × scheduler parameters," gradually reducing the noise ratio in the latent representation to approximate the target face features.

[0194] The process returns to generate the embedding vector for the next time step, repeating the denoising and update process until the preset total number of iterations is reached. The final output is a latent facial representation synchronized with the audio, which has undergone multi-step denoising to remove most noise and contains clear facial dynamic features. The total number of iterations can be dynamically adjusted according to real-time requirements; the time-step embeddings can be replaced with learnable parameterized vectors; and the guidance ratio can be adaptively optimized through model training to avoid overfitting the audio features while ensuring synchronization.

[0195] In some embodiments of this application, the process of step S7, which involves synthesizing the target face image onto the target background to obtain a digital human facial animation video stream, matching and combining the digital human body animation video stream corresponding to the real-time audio stream, and constructing a real-time digital human animation, is described.

[0196] A digital facial animation video stream is obtained by compositing the target face image onto the target background, which may specifically include:

[0197] ① Perform interpolation magnification on the target background to match the resolution of the target face image.

[0198] Specifically, a preset interpolation algorithm (such as bilinear interpolation, bicubic interpolation, etc.) is used to scale the pixel matrix of the target background. By calculating the gray value mapping relationship between the pixel to be interpolated and the surrounding known pixels, the number of pixels in the target background is increased or decreased until its width and height pixel values ​​are completely consistent with the target face image, thus providing a resolution benchmark for subsequent image synthesis.

[0199] ② Call the inverse of the affine transformation matrix to map and locate the target face image to the position in the target background corresponding to the original face.

[0200] Specifically, based on the inverse of the aforementioned affine transformation matrix, coordinate transformation operations are performed on the coordinates of each pixel in the target face image. Through geometric mapping such as rotation, translation, and scaling, the spatial position and angle of the target face image in the target background are kept consistent with the positional relationship of the face relative to the background in the original video frame, ensuring accurate positioning.

[0201] ③ Construct a shrinkage mask that matches the contour of the target face image, perform morphological erosion on the shrinkage mask to shrink the boundary region, and generate a soft-edge mask through Gaussian blur processing.

[0202] Specifically, firstly, the contour boundary of the target face image is extracted using an edge detection algorithm, and a binary mask is generated based on this contour (the area inside the mask is the target face, and the area outside the mask is the background). Then, a morphological erosion operation is performed on the mask using a preset structuring element, causing the mask boundary to shrink inward by a preset pixel distance, forming a shrunken mask. Next, Gaussian blur processing is performed on the boundary area of ​​the shrunken mask, so that the mask edge smoothly transitions from completely opaque to completely transparent, generating a soft-edge mask with gradient transition characteristics, preparing for edge fusion.

[0203] ④ The target face image and the interpolated and enlarged target background are weighted and fused using the soft edge mask to generate a digital facial animation video frame with natural transition edges.

[0204] Specifically, a weighted calculation is performed on the corresponding pixels of the target face image and the target background based on the pixel values ​​of the soft edge mask (ranging from 0 to 255): for areas with a mask value of 255, the pixels of the target face image are completely preserved; for areas with a mask value of 0, the pixels of the target background are completely preserved; for transition areas with mask values ​​between 0 and 255, the pixels of the two are fused according to the mask value ratio, so that the edge transition between the target face and the background is natural, eliminating compositing traces and generating a single-frame digital facial animation image.

[0205] ⑤ The continuously generated digital human facial animation video frames are serialized in chronological order to form a digital human facial animation video stream.

[0206] Specifically, according to the timestamps corresponding to each video frame (which are consistent with the timestamps of the original video frames), multiple frames of digital facial animation images are arranged sequentially to form a video stream with a continuous time dimension. Its frame rate is synchronized with the original video frame sequence to ensure smooth animation.

[0207] The process of combining the digital human facial animation video stream with the digital human body animation video stream corresponding to the real-time audio stream to construct a real-time digital human animation may specifically include:

[0208] ① Based on the speaking state of the audio frames in the real-time audio stream, the display index is dynamically adjusted using a dual-state and pointer bounce mechanism.

[0209] The execution process of the dual-state machine and pointer bounce mechanism includes:

[0210] Define a dual state including a static state and an active state, set an index direction variable and a boundary threshold, wherein the upper bound of the static region is a preset motion threshold, and the upper bound of the active region is the actual sequence length of the digital human body animation video stream;

[0211] When consecutive audio frames contain a speaking event, the state switches from the static state to the active state, and the index direction is forced to point to the active region;

[0212] When all consecutive audio frames are silent, the state falls back from the action state to the still state, and the index direction is forced to point to the starting point of the still area.

[0213] When the displayed index reaches the boundary, it bounces back to avoid going out of bounds;

[0214] Output the current display index after each update.

[0215] Specifically, the initialization of the state and core parameters is performed first. Two basic states are defined: the static state and the dynamic state. The static state corresponds to the digital human body state when there is no speaking event in the real-time audio stream, and only contains a preset sequence of small micro-movement frames (such as slight head shaking, slight shoulder tremors, etc., used to simulate body movements in a natural static state). The dynamic state corresponds to the digital human body state when there is a speaking event in the real-time audio stream, and contains a complete sequence of action frames that match the speech content (such as changes in gestures accompanying speech, adjustments in body posture, etc.). Two core parameters are set: one is the index direction variable, which has two values: positive increment and negative decrement, used to control the increase and decrease direction of the display index, respectively; the other is the boundary threshold, where the upper limit of the static area is the preset motion threshold, which is set according to the amplitude requirements of the small micro-movements of the digital human body, and usually ranges from 5 to 15 frames, and the lower limit of the static area is fixed at 0; the upper limit of the motion area is the actual sequence length of the digital human body animation video stream, that is, the total number of complete motion frames contained in the video stream, and this total number must be greater than or equal to the aforementioned preset motion threshold, and the lower limit of the motion area is the upper limit of the static area plus 1.

[0216] Next, the state switching logic based on the speaking state of the audio frames is executed. A speech activity detection algorithm is used to determine the speaking state of consecutive audio frames in the real-time audio stream: if the algorithm determines that 3 to 5 consecutive audio frames contain speaking events (the determination criteria include, but are not limited to, the speech energy of the audio frame being higher than a preset threshold, the detection of speech feature spectrum, etc.), the trigger state switches from the static state to the active state, and the index direction variable is forcibly set to positive increment, causing the display index to move towards the active region containing the complete active frame; if the algorithm determines that 5 to 8 consecutive audio frames are silent (i.e., no speaking events), the trigger state falls back from the active state to the static state, and the index direction variable is forcibly set to negative decrement, causing the display index to move towards the starting point of the static region (i.e., the position with index 0).

[0217] Subsequently, boundary bounce processing of the display index is performed to prevent the index from going out of bounds. When the digital human is currently stationary, if the current display index reaches the upper limit of the stationary region (i.e., the aforementioned preset motion threshold), the index direction variable is switched from positive increment to negative decrement, causing the display index to decrease in the opposite direction starting from the upper limit. If the display index reaches the lower limit of the stationary region (i.e., 0), the index direction variable is switched from negative decrement to positive increment, causing the display index to increase in the positive direction starting from 0. This ensures that the display index always moves cyclically within the index range corresponding to the stationary region and does not go out of bounds. When the digital human is currently in motion, if the current display index reaches the upper limit of the motion region (i.e., the actual sequence length of the body animation video stream), the index direction variable is switched from positive increment to negative decrement, causing the display index to decrease in the opposite direction starting from the upper limit. If the display index reaches the lower limit of the motion region (i.e., the value of the upper limit of the stationary region plus 1), the index direction variable is switched from negative decrement to positive increment, causing the display index to increase in the positive direction starting from the lower limit. This ensures that the display index moves cyclically within the index range corresponding to the motion region and avoids going out of bounds.

[0218] Finally, the display index is updated and output. Every time an audio frame period synchronized with the video frame rate is reached (typically 33 milliseconds / frame, but can be adjusted according to the actual application's frame rate requirements), the display index is updated based on the current index direction variable. This is achieved by adding the current display index to the increment / decrement value corresponding to the index direction variable (adding 1 for forward increments and subtracting 1 for reverse decrements). After each update, the updated display index is immediately output. This display index serves as a unique identifier for selecting the corresponding body animation frame from the digital human body animation video stream. This identifier ensures that the selected body animation frame maintains temporal order with the concurrent digital human facial animation frames and real-time audio stream, achieving temporal synchronization among the three.

[0219] ② Drive frame selection from the digital human body animation video stream according to the display index to achieve synchronous combination with the digital human face animation video stream, and generate the digital human real-time animation;

[0220] Specifically, based on the display index, the corresponding body animation frame is selected from the digital human body animation video stream; through the timestamp alignment mechanism, the body animation frame is spatially stitched with the digital human facial animation video frame of the same period (such as positioning the facial animation frame to the head area of ​​the body animation frame), so that facial expressions and body movements are consistent in time and space; the continuously stitched frame sequence is combined in chronological order to finally form a real-time digital human animation that includes facial and body movements and matches the real-time audio stream.

[0221] The following describes a real-time digital human construction system provided by an embodiment of this application. The real-time digital human construction system described below can be referred to in correspondence with the real-time digital human construction method described above.

[0222] See Figure 3 , Figure 3 This is a schematic diagram of a real-time digital human construction system disclosed in an embodiment of this application.

[0223] like Figure 3 As shown, the real-time digital human construction system may include:

[0224] The face localization and detection module 110 is used to perform face detection and key point localization on sample face video frames, determine the face bounding box, and calculate the affine transformation matrix that aligns the face region to a standard size.

[0225] The latent representation analysis module 120 is used to generate various latent representations based on the standardized face image obtained by the target image size and alignment, after low-dimensional space mapping and reparameterization adjustment. The various latent representations include at least noise latent representation, mask latent representation, masked image latent representation and image latent representation.

[0226] The audio feature analysis module 130 is used to extract deep temporal features of real-time audio streams and generate audio feature embedding vectors.

[0227] The combined latent tensor module 140 is used to scale and concatenate the various latent representations along the channel dimension using the audio feature embedding vector as a conditional input to form a combined latent tensor.

[0228] The iterative diffusion denoising module 150 is used to input the combined latent tensor into the audio conditional diffusion model, fuse audio features through a cross-attention mechanism, and perform multi-step denoising iteration to generate a face latent representation synchronized with the audio.

[0229] The facial animation generation module 160 is used to decode and restore the latent facial representation into a pixel-space facial image, and perform an inverse affine transformation on the pixel-space facial image based on the affine transformation matrix to generate a target facial image.

[0230] The digital human combination construction module 170 is used to obtain a digital human facial animation video stream by compositing the target face image onto the target background, and to match and combine the digital human body animation video stream corresponding to the real-time audio stream to construct a real-time digital human animation.

[0231] As can be seen from the above technical solutions, the digital human real-time construction method and related equipment provided in this application embodiment achieve high-quality digital human facial animation generation synchronized with audio through steps such as face alignment, multi-type latent representation generation, audio feature fusion, diffusion model denoising and inverse transform synthesis.

[0232] This solution addresses the shortcomings of existing technologies at the technical level: First, in audio feature processing, it extracts deep temporal features from real-time audio streams and generates embedding vectors. Leveraging the cross-attention mechanism of the audio conditional diffusion model, audio features are deeply integrated into the generation of facial latent representations. This mechanism allows the model to accurately capture subtle changes in intonation and rhythm in the audio, thereby driving corresponding adjustments in lip movements. Compared to the fixed mapping rules of parameterized models, synchronization accuracy is significantly improved, effectively solving the problem of insufficient synchronization. Second, in the latent representation generation stage, based on the target image size and standardized face image, various latent representations, including noise, masks, and images, are generated through low-dimensional space mapping and reparameterization. These representations describe facial features from different dimensions: noise latent representation increases generation diversity, mask latent representation is used to identify areas that need to remain unchanged, masked image latent representation can highlight key facial regions, and image latent representation preserves basic image information. After scaling and stitching together multiple types of latent representations, rich initial information is provided for the diffusion model. Combined with multi-step denoising iterations, it can generate delicate and natural lip movements, overcoming the shortcomings of insufficient coherence and realism caused by the limitations of the material library in the stitching technology. Thirdly, in terms of real-time performance, high-dimensional image information is transformed into low-dimensional latent representations through low-dimensional spatial mapping, which significantly reduces the amount of data processing. At the same time, affine transformation and inverse transformation use matrix operations, which are computationally efficient and enable the entire generation process to respond quickly, solving the problem of insufficient frame rate caused by the large amount of computation in NeRF technology and the slow inference of GAN models, thus meeting the requirements for real-time output. Fourthly, regarding the generalization of audio input, since various latent representations have strong adaptability and can be effectively combined with different audio feature embedding vectors, and the diffusion model can learn multiple audio patterns during training, it can generate appropriate lip movements regardless of different speech rates and accents, breaking through the limitation of poor generalization of existing technologies for specific audio inputs.

[0233] Optionally, the real-time digital human construction system adopts a multi-process asynchronous architecture to achieve real-time processing, wherein:

[0234] The audio feature analysis module runs in the main process and asynchronously transmits the audio feature embedding vector to the child process through the first security queue.

[0235] The iterative diffusion denoising module runs in the first subprocess and processes the combined latent tensors through the TensorRT acceleration engine.

[0236] The facial animation generation module runs in a second sub-process and receives the potential facial representation through a second security queue.

[0237] The main process and the child process exchange affine transformation matrices and potential representation data through a shared memory pool.

[0238] Specifically, the main process is responsible for system initialization, resource management, and overall process scheduling. The audio feature analysis module, as the core component of the main process, receives external audio streams in real time and performs feature extraction to generate audio feature embedding vectors. To avoid audio processing blocking subsequent calculations, the module writes the audio feature embedding vectors into a first safe queue and transmits the data to child processes asynchronously, ensuring a continuous supply of audio features and lightweight operation of the main process.

[0239] The first subprocess independently runs the iterative diffusion denoising module, focusing on computationally intensive diffusion model inference tasks. This module acquires audio feature embedding vectors by listening to the first secure queue and reads pre-cached latent representations (noise, masks, image latent representations) and affine transformation matrices from the shared memory pool (the shared memory pool is implemented using memory-mapped files or inter-process shared memory technology to avoid copying large amounts of data across processes). The module combines the audio features and latent representations to form a combined latent tensor, calls the TensorRT acceleration engine to perform multi-step denoising iterations, generates a face latent representation, and writes it to a designated area in the shared memory pool for the next module to read.

[0240] The second subprocess runs the facial animation generation module, which obtains the latent facial representation generated by the first subprocess by listening to a second security queue (independent of the first security queue to avoid data races). The module reads the corresponding affine transformation matrix from the shared memory pool, performs decoding, inverse affine transformation, and background compositing operations to generate the target facial image. The synthesized images are sorted by timestamp, assembled into a digital facial animation video stream, and output (e.g., directly pushed to a display device or network transmission interface via the GPU rendering pipeline).

[0241] The main process and child processes achieve efficient interaction of large data (such as latent representations and affine matrices) through a shared memory pool, and only pass lightweight control information (such as data indexes and timestamps) through a safe queue, significantly reducing inter-process communication latency. At the same time, each process is bound to an independent CPU core and GPU computing unit, and the batch size and inference priority are dynamically adjusted through a load balancing algorithm.

[0242] The digital human real-time construction system provided in this application embodiment can be applied to digital human real-time construction devices. Figure 4 The hardware structure block diagram of the digital human real-time construction device is shown, with reference to Figure 4 The hardware structure of a real-time digital human construction device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0243] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0244] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0245] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0246] The memory stores a program, which the processor can call. The program is used for:

[0247] Perform face detection and key point localization on sample face video frames, determine face bounding boxes, and calculate the affine transformation matrix to align the face region to a standard size;

[0248] Based on the standardized face image obtained by the target image size and alignment, various latent representations are generated after low-dimensional space mapping and reparameterization adjustment. The various latent representations include at least noise latent representation, mask latent representation, mask image latent representation and image latent representation.

[0249] Extract deep temporal features from real-time audio streams and generate audio feature embedding vectors;

[0250] Using the audio feature embedding vector as conditional input, the various latent representations are scaled according to a preset ratio and concatenated along the channel dimension to form a combined latent tensor.

[0251] The combined latent tensor is input into the audio conditional diffusion model, and audio features are fused through a cross-attention mechanism to perform multi-step denoising iterations to generate a face latent representation synchronized with the audio.

[0252] The latent facial representation is decoded and restored to a pixel-space facial image. An inverse affine transformation is then performed on the pixel-space facial image based on the affine transformation matrix to generate the target facial image.

[0253] A digital human facial animation video stream is obtained by compositing the target face image onto the target background. The digital human body animation video stream corresponding to the real-time audio stream is then matched and combined to construct a real-time digital human animation.

[0254] Optionally, the refined and extended functions of the program can be referred to the above description.

[0255] This application embodiment also provides a readable storage medium that can store a program suitable for execution by a processor, the program being used for:

[0256] Perform face detection and key point localization on sample face video frames, determine face bounding boxes, and calculate the affine transformation matrix to align the face region to a standard size;

[0257] Based on the standardized face image obtained by the target image size and alignment, various latent representations are generated after low-dimensional space mapping and reparameterization adjustment. The various latent representations include at least noise latent representation, mask latent representation, mask image latent representation and image latent representation.

[0258] Extract deep temporal features from real-time audio streams and generate audio feature embedding vectors;

[0259] Using the audio feature embedding vector as conditional input, the various latent representations are scaled according to a preset ratio and concatenated along the channel dimension to form a combined latent tensor.

[0260] The combined latent tensor is input into the audio conditional diffusion model, and audio features are fused through a cross-attention mechanism to perform multi-step denoising iterations to generate a face latent representation synchronized with the audio.

[0261] The latent facial representation is decoded and restored to a pixel-space facial image. An inverse affine transformation is then performed on the pixel-space facial image based on the affine transformation matrix to generate the target facial image.

[0262] A digital human facial animation video stream is obtained by compositing the target face image onto the target background. The digital human body animation video stream corresponding to the real-time audio stream is then matched and combined to construct a real-time digital human animation.

[0263] Optionally, the refined and extended functions of the program can be referred to the above description.

[0264] This application also provides a computer program product, including a computer program, wherein the computer program is executed by a processor using the following method:

[0265] Perform face detection and key point localization on sample face video frames, determine face bounding boxes, and calculate the affine transformation matrix to align the face region to a standard size;

[0266] Based on the standardized face image obtained by the target image size and alignment, various latent representations are generated after low-dimensional space mapping and reparameterization adjustment. The various latent representations include at least noise latent representation, mask latent representation, mask image latent representation and image latent representation.

[0267] Extract deep temporal features from real-time audio streams and generate audio feature embedding vectors;

[0268] Using the audio feature embedding vector as conditional input, the various latent representations are scaled according to a preset ratio and concatenated along the channel dimension to form a combined latent tensor.

[0269] The combined latent tensor is input into the audio conditional diffusion model, and audio features are fused through a cross-attention mechanism to perform multi-step denoising iterations to generate a face latent representation synchronized with the audio.

[0270] The latent facial representation is decoded and restored to a pixel-space facial image. An inverse affine transformation is then performed on the pixel-space facial image based on the affine transformation matrix to generate the target facial image.

[0271] A digital human facial animation video stream is obtained by compositing the target face image onto the target background. The digital human body animation video stream corresponding to the real-time audio stream is then matched and combined to construct a real-time digital human animation.

[0272] Optionally, the refined and extended functions of the program can be referred to the above description.

[0273] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0274] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0275] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A digital human real-time construction method, characterized in that, The method comprises the following steps: Performing face detection and key point positioning on sample face video frames, determining a face bounding box, and calculating an affine transformation matrix for aligning the face region to a standard size; Based on the target image size and the standardized face image obtained by alignment, generate various latent representations after low-dimensional space mapping and reparameterization adjustment, wherein the various latent representations at least include noise latent representation, mask latent representation, mask image latent representation and image latent representation; Extracting deep temporal features of real-time audio stream to generate audio feature embedding vector; Taking the audio feature embedding vector as the conditional input, scaling the various latent representations by a preset ratio and concatenating them along the channel dimension to form a combined latent tensor; Input the combined latent tensor into the audio-conditioned diffusion model, fuse the audio features through cross-attention mechanism, and generate face latent representation synchronized with audio through multi-step denoising iteration; Decoding the face latent representation to restore it to a pixel space face image, performing affine inverse transformation on the pixel space face image based on the affine transformation matrix to generate a target face image; Obtaining a digital human facial animation video stream by synthesizing the target face image to a target background, matching and combining a digital human body animation video stream corresponding to the real-time audio stream to construct a digital human real-time animation.

2. The method of claim 1, wherein, The method for generating various latent representations based on the target image size and the standardized face image obtained by alignment comprises: Based on the target image size and the down-sampling factor, construct a multi-frame noise tensor conforming to the standard normal distribution, and generate a noise latent representation after scaling; Scale the original mask tensor to generate a latent space mask tensor through interpolation operation and perform dimension rearrangement, copy and concatenate the latent space mask tensor according to the free guidance requirements of the classifier to output a mask latent representation; Multiply the standardized face image obtained by alignment with a preset binary mask to obtain a mask processing image, encode the mask processing image into a low-dimensional latent vector, and generate a mask image latent representation after reparameterization sampling and parameter adjustment; Encode the standardized face image without mask processing into a low-dimensional latent vector, and generate an image latent representation after reparameterization sampling and parameter adjustment.

3. The method of claim 2, wherein, The generation process of the noise latent representation comprises: Determine the latent space tensor shape according to the target image size and the down-sampling factor; Generate a single-frame random tensor conforming to the standard normal distribution, copy the single-frame random tensor along the time dimension to form a multi-frame noise tensor; Multiply the multi-frame noise tensor by the initial noise standard deviation of the diffusion scheduler to generate a scaled noise latent representation.

4. The method of claim 2, wherein, The generation process of the mask latent representation comprises: Scale the original mask tensor to the latent space resolution through interpolation operation to generate a latent space mask tensor; Rearrange the dimensions of the latent space mask tensor to adjust the tensor dimension order to match the input format of the diffusion model; According to the free guidance requirements of the classifier, copy and concatenate the latent space mask tensor to output a mask latent representation.

5. The method of claim 2, wherein, The generation process of the mask image latent representation comprises: Performing pixel-by-pixel multiplication on the obtained standardized face image and the preset binary mask to generate a mask processing image; Batch processing the mask processing image to obtain a first image batch set; Performing encoding operation on each batch of images in the first image batch set to output mean vector and standard deviation vector representing the distribution thereof, and obtaining first encoding latent vector of each batch through reparameterization sampling; Performing offset and scaling operation on the first encoding latent vector to generate first adjusted latent vector; Splicing the first adjusted latent vectors of all batches and adjusting the dimension order to output mask latent representation.

6. The method of claim 2, wherein, The generation process of the image latent representation comprises: Batch processing the unmasked standardized face image to obtain a second image batch set; Performing encoding operation on each batch of images in the second image batch set to output mean vector and standard deviation vector representing the distribution thereof, and obtaining second encoding latent vector of each batch through reparameterization sampling; Performing offset and scaling operation on the second encoding latent vector to generate second adjusted latent vector; Splicing the second adjusted latent vectors of all batches and adjusting the dimension order to output image latent representation.

7. The method of claim 1, wherein, The audio-conditioned diffusion model comprises a cascaded convolutional encoding network, a spatio-temporal feature fusion network, a conditional decoding network and a timing control module; The convolutional encoding network is configured to perform down-sampling operation on the input combined latent tensor to extract multi-scale spatial features; The spatio-temporal feature fusion network is configured to inject the audio feature embedding vector into the multi-scale spatial features through cross-attention mechanism to generate conditionally enhanced features; The conditional decoding network is configured to perform up-sampling operation on the conditionally enhanced features and fuse the features of each scale to output single-step denoising result; The timing control module is configured to schedule multi-step denoising iteration, and obtain the face latent representation synchronized with the audio by cyclically calling the convolutional encoding network, the spatio-temporal feature fusion network and the conditional decoding network in the diffusion process until the iteration termination condition is met.

8. The method of claim 7, wherein, The implementation of the cross-attention mechanism comprises: Projecting the audio feature embedding vector into attention key vector and value vector, and projecting the multi-scale spatial features into attention query vector; Calculating the similarity matrix of the query vector and the key vector to generate attention weight; Weighted aggregation of the value vector based on the attention weight outputs the conditionally enhanced features.

9. The method of claim 7, wherein, The process of the timing control module performing multi-step denoising iteration comprises: Initializing the current latent representation as the noise latent representation; Generating time step embedding vector according to the number of diffusion steps; Inputting the noise latent representation, time step embedding vector and audio feature into the audio-conditioned diffusion model, and sequentially passing through the convolutional encoding network, the spatio-temporal feature fusion network and the conditional decoding network to output predicted noise; Based on the classifier-free guidance technique, jointly calculating the conditionally predicted noise and the unconditionally predicted noise, and fusing the conditionally predicted and the unconditionally predicted results according to the preset guidance ratio; updating the current latent representation with fused noise according to a diffusion scheduler update rule; returning to performing the process of generating time step embedding vectors according to diffusion step number coding until a preset iteration step number is reached, and outputting a face latent representation.

10. The method of claim 1, wherein, Further comprising accelerating and optimizing the multi-step denoising iteration process using a TensorRT acceleration engine.

11. The method of claim 1, wherein, decoding the face latent representation to restore it to a pixel space face image, performing affine inverse transformation on the pixel space face image based on the affine transformation matrix, and generating a target face image, including: decoding the face latent representation to restore it to a pixel space face image using a VAE; calculating and using the inverse matrix of the affine transformation matrix to perform affine inverse transformation on the pixel space face image, and mapping back to the face pose and position in the original video frame; performing bilinear interpolation upsampling on the face image after affine inverse transformation to match the resolution of the original video frame, and generating a target face image.

12. The method of claim 1, wherein, by synthesizing the target face image into a target background to obtain a digital human facial animation video stream, including: performing interpolation enlargement processing on the target background to match the resolution of the target face image; calling the inverse matrix of the affine transformation matrix to map and position the target face image to the position corresponding to the original face in the target background; constructing a shrinkage mask matching the contour of the target face image, performing a morphological erosion operation on the shrinkage mask to shrink the boundary region, and generating a soft edge mask through Gaussian blur processing; using the soft edge mask to perform weighted fusion on the target face image and the target background after interpolation enlargement, generating a digital human facial animation video frame with a natural transition edge; sequencing the continuously generated digital human facial animation video frames in chronological order to form a digital human facial animation video stream.

13. The method of claim 1, wherein, combining the digital human facial animation video stream with the corresponding digital human body animation video stream of the real-time audio stream to construct a digital human real-time animation process, including: based on the speaking state of the audio frames of the real-time audio stream, using a double-state and pointer bounce mechanism to dynamically adjust the display index; according to the display index, driving frame selection from the digital human body animation video stream to realize synchronous combination with the digital human facial animation video stream, and generating the digital human real-time animation; wherein the execution process of the double-state and pointer bounce mechanism includes: defining a double-state including a static state and a motion state, setting an index direction variable and a boundary threshold, wherein the upper limit of the static region is a preset motion threshold, and the upper limit of the motion region is the real sequence length of the digital human body animation video stream; when consecutive audio frames contain speaking events, the state switches from the static state to the motion state, and the index direction is forcibly directed to the motion region; when consecutive audio frames are all silent, the state falls back from the motion state to the static state, and the index direction is forcibly directed to the starting point of the static region; when the display index reaches the boundary, perform reverse bounce to avoid out-of-bound; output the current display index after each update.

14. A digital human real-time construction system, characterized by, includes: The face positioning detection module is configured to perform face detection and key point positioning on a sample face video frame, determine a face bounding box, and calculate an affine transformation matrix for aligning the face region to a standard size. The latent representation analysis module is configured to generate various latent representations based on a target image size and a standardized face image obtained by alignment, low-dimensional space mapping, and reparameterization adjustment, wherein the various latent representations at least include a noise latent representation, a mask latent representation, a mask image latent representation, and an image latent representation. The audio feature analysis module is configured to extract deep temporal features of a real-time audio stream and generate an audio feature embedding vector. The combined latent tensor module is configured to input the audio feature embedding vector as a condition, scale the various latent representations by a preset ratio, and concatenate them along the channel dimension to form a combined latent tensor. The iterative diffusion denoising module is configured to input the combined latent tensor into an audio-conditioned diffusion model, fuse audio features through a cross-attention mechanism, and perform multi-step denoising iteration to generate a face latent representation synchronized with the audio. The face animation generation module is configured to decode the face latent representation into a pixel space face image, perform affine inverse transformation on the pixel space face image based on the affine transformation matrix, and generate a target face image. The digital human combined construction module is configured to synthesize the target face image into a target background to obtain a digital human face animation video stream, match and combine a digital human body animation video stream corresponding to the real-time audio stream, and construct a digital human real-time animation.

15. The system of claim 14, wherein, The digital human real-time construction system adopts a multi-process asynchronous architecture to realize real-time processing, wherein: The audio feature analysis module runs in the main process and asynchronously transmits the audio feature embedding vector to the sub-process through a first secure queue. The iterative diffusion denoising module runs in the first sub-process and processes the combined latent tensor through a TensorRT acceleration engine. The face animation generation module runs in the second sub-process and receives the face latent representation through a second secure queue. The main process and the sub-process exchange affine transformation matrices and latent representation data through a shared memory pool.

16. A digital human real-time construction device, characterized by, The computer program is executed by the processor to implement the steps of the digital human real-time construction method according to any one of claims 1-13. The computer program is executed by the processor to implement the steps of the digital human real-time construction method according to any one of claims 1-13. The computer program is executed by the processor to implement the steps of the digital human real-time construction method according to any one of claims 1-13.

17. A readable storage medium, having stored thereon a computer program, characterized in that, ​ 18. A computer program product comprising a computer program, characterized in that, ​

Images