Digital human live video synthesis method and system

Abstract

The invention belongs to the field of image processing, and provides a digital human live video synthesis method and system, and the method comprises the steps: receiving the multi-modal data input of a user, and the multi-modal data comprises voice, text and image; key features are extracted from the multi-modal data; fusing the key features to form a fusion condition; generating a corresponding radiation field representation according to the fusion condition by using a neural network model; finely adjusting the radiation field parameters according to real-time input; and converting the generated radiation field into video output.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method and system for synthesizing digital human live video. Background Art

[0002] In recent years, with the rapid development of artificial intelligence, computer vision and natural language processing technologies, digital human technology has been widely used in virtual anchors, intelligent customer service, education and training, live e-commerce and other fields. Digital human live video synthesis technology generates anthropomorphic virtual images through computers, enabling it to generate synchronized expressions, lip shapes and movements in real time based on user voice, text and image input. However, the existing digital human live video synthesis methods still have many technical challenges in terms of real-time performance, natural interaction, personalized customization, system security and scalability, and urgently need to be further optimized.

[0003] The video synthesis process involves complex calculations, such as 3D rendering, voice-driven lip synchronization, and expression animation generation. The computational overhead is large, resulting in a long system response time, which is difficult to meet the needs of real-time interactive scenarios such as live broadcasts. Multimodal data (voice, text, and images) processing usually uses independent computing processes, which do not fully optimize data flow and computing efficiency, resulting in accumulated delays.

[0004] Existing technologies still have a lag in voice and video synchronization, especially in low-bandwidth or high-latency network environments, where the lip shape and voice are often out of sync, affecting the viewing experience. Current expression and action synthesis mostly relies on preset animations, rather than dynamic adjustments based on the emotional information input by the user, resulting in a lack of naturalness in the generated video, making it difficult to effectively express user emotions.

[0005] Most existing technologies use single-modal or weak multimodal fusion methods, which fail to fully utilize the associated information of multimodal data such as voice, text and images, resulting in a low match between the digital human's expression, action and semantics, affecting the coordination of interaction. The lack of efficient cross-modal feature association methods leads to inaccurate information transmission between different input modes, affecting the quality of the synthesized video.

[0006] Existing systems have weak support for personalized customization of digital humans. It is difficult for users to freely adjust the appearance, voice and behavior patterns of digital humans, and they cannot meet the specific needs of different industries or brands. Traditional customization methods often require a lot of manual intervention or in-depth editing, and have a high threshold for use, which is not conducive to popularization and application.

[0007] The current digital human live broadcast system usually requires users to have a certain technical background, and the configuration and operation are relatively complex, which is not suitable for ordinary users. The interaction method is single, mainly relying on mouse and keyboard input, and lacks voice command support, which limits the convenience of operation.

[0008] The system architecture is not flexible enough and its scalability is limited. Most existing systems adopt a non-modular design, which makes it difficult to expand functions and maintain them, and it is difficult to adapt to diverse application needs. Many digital human systems can only run on specific platforms or devices, lack cross-platform compatibility, and limit application scenarios.

[0009] Since digital human technology involves sensitive data such as user voice and images, existing systems often fail to provide a complete data encryption and access control mechanism, which poses a risk of user privacy leakage. The existing permission management mechanism is relatively extensive and cannot achieve fine-grained role management and operation permission allocation, affecting the security and reliability of the system.

[0010] The existing digital human live broadcast system lacks intelligent scheduling in terms of resource optimization, and it is difficult to dynamically adjust computing resources according to system load conditions, resulting in performance bottlenecks or resource waste. Traditional operation and maintenance methods rely on manual monitoring and manual adjustments, lacking automated fault detection and repair mechanisms, affecting system stability.

[0011] The existing system has quality problems in rendering, such as insufficient picture details, motion blur, and unrealistic expressions, which affect the user experience. The video output format support is limited and cannot meet the compatibility requirements of different platforms and devices, limiting the scope of application.

[0012] Existing technologies still have lags or deviations in the synchronization of voice and video, resulting in the problem of lip shape and voice not matching, which reduces the realism of interaction. There is a lack of intelligent synchronization adjustment mechanism, and it is impossible to dynamically optimize the delay and frame rate of video output according to the network environment and hardware performance, which affects the fluency. Summary of the invention

[0013] In order to solve the problems in the prior art, the present invention provides a method for synthesizing a digital human live video, which is characterized by comprising the following steps:

[0014] Receiving multimodal data input from a user, wherein the multimodal data includes: voice, text, and image;

[0015] extracting key features from the multimodal data;

[0016] Fusion of the key features to form fusion conditions;

[0017] Using a neural network model, generating a corresponding radiation field representation according to the fusion condition;

[0018] fine-tuning the radiation field parameters based on real-time input;

[0019] Convert the generated radiation field to video output.

[0020] Further, converting the generated radiation field into a video output includes:

[0021] Adjust the digital human’s facial expressions and movements based on the extracted speech and text features;

[0022] Encode the rendered video and transmit it to the user end in real time.

[0023] Furthermore, low-latency processing is performed on speech feature extraction and video synthesis;

[0024] Monitor the synchronization status of voice and video in real time and automatically correct possible time deviations.

[0025] Furthermore, the following method is used:

[0026] Users can configure and control the live video of the digital human through simple clicks and inputs;

[0027] Users control the digital human’s movements, expressions, and content display through voice commands;

[0028] Allows users to adjust the appearance, voice and behavior of digital humans according to their needs.

[0029] Furthermore, the system is managed as follows:

[0030] Manage the permissions and roles of system users to ensure that different users have different access rights;

[0031] Monitor and manage the usage of system resources to ensure efficient and stable operation of the system;

[0032] Record system operation logs and user operation records, and provide real-time monitoring and troubleshooting functions;

[0033] Implement data encryption and access control to ensure the security and privacy of system and user data.

[0034] The present invention also provides a digital human live video synthesis system, which is characterized by comprising the following modules:

[0035] An input processing module, used to receive multimodal data input from a user, wherein the multimodal data includes: voice, text, and image;

[0036] A feature extraction module, used to extract key features from the multimodal data;

[0037] A conditional fusion module, used to fuse the key features to form a fusion condition;

[0038] A radiation field calculation module, used to generate a corresponding radiation field representation according to the fusion condition using a neural network model;

[0039] A feature fine-tuning module, used for fine-tuning the radiation field parameters according to real-time input;

[0040] A video synthesis module is used to convert the generated radiation field into a video output.

[0041] Furthermore, the video synthesis module is also used for:

[0042] Adjust the digital human’s facial expressions and movements based on the extracted speech and text features;

[0043] Encode the rendered video and transmit it to the user end in real time.

[0044] Furthermore, the system also includes a real-time synchronization module, which is used to:

[0045] Low-latency processing for speech feature extraction and video synthesis;

[0046] Monitor the synchronization status of voice and video in real time and automatically correct possible time deviations.

[0047] Furthermore, the system further comprises a user interaction module, wherein the user interaction module is used to:

[0048] Users can configure and control the live video of the digital human through simple clicks and inputs;

[0049] Users control the digital human’s movements, expressions, and content display through voice commands;

[0050] Allows users to adjust the appearance, voice and behavior of digital humans according to their needs.

[0051] Furthermore, the system further comprises a system management module, and the system management module is used for:

[0052] Manage the permissions and roles of system users to ensure that different users have different access rights;

[0053] Monitor and manage the usage of system resources to ensure efficient and stable operation of the system;

[0054] Record system operation logs and user operation records, and provide real-time monitoring and troubleshooting functions;

[0055] Implement data encryption and access control to ensure the security and privacy of system and user data.

[0056] The digital human live video synthesis method and system of the present invention have the following advantages:

[0057] 1. High real-time performance

[0058] Low-latency processing: Adopts conditional radiance field (CRF) and direct fine-tuning technology to achieve efficient video synthesis and low-latency processing, ensuring real-time user interaction.

[0059] Fast response: The system can quickly respond to user voice, text and image input, generate synthetic video in real time, and enhance the interactive experience.

[0060] 2. Natural interactive experience

[0061] Precise synchronization: Through precise voice feature extraction and facial expression synchronization technology, natural and smooth digital human videos are generated to improve user immersion and satisfaction.

[0062] Emotion recognition and expression: The system can recognize the user's emotions and adjust the digital human's expression and tone in real time to enhance interactive emotional communication.

[0063] 3. Multimodal Data Fusion

[0064] Comprehensive feature extraction: Efficiently integrate multimodal data such as voice, text, and images to ensure that the digital human's behavior and expressions can accurately reflect user input.

[0065] Cross-modal association: The system can understand and associate information between different modalities to achieve more intelligent and coordinated interaction effects.

[0066] 4. Flexible customization

[0067] Appearance and voice customization: Provides a variety of options, allowing users to adjust the appearance, voice and behavior of digital humans as needed to meet the personalized needs of different application scenarios.

[0068] Behavior mode configuration: Users can customize the digital human’s movements and expressions to suit specific interaction needs or brand image.

[0069] 5. Easy-to-use user interface

[0070] Intuitive Graphical User Interface (GUI): Provides a friendly operation interface that allows users to easily configure and control the digital human without professional knowledge.

[0071] Voice command support: Supports control via voice commands to improve the convenience and interactivity of operations.

[0072] 6. Modularity and Scalability

[0073] Modular design: The system adopts a modular structure to facilitate functional expansion and maintenance, support compatibility with multiple cloud platforms and multiple devices, and improve the scalability and adaptability of the system.

[0074] Flexible integration: Ability to seamlessly integrate with other systems and services to meet the needs of different business and technical environments.

[0075] 7. Safety and Reliability

[0076] Fine-grained permission control: Through meticulous permission management, ensure that different users and roles can only access and operate authorized functions and data.

[0077] Data encryption and protection: Use advanced data encryption technology to ensure the security and privacy of user data and system information.

[0078] Comprehensive logging and monitoring: The system records detailed operation logs and operating status, supports real-time monitoring and rapid troubleshooting, and improves system stability and reliability.

[0079] 8. Intelligent Management and Optimization

[0080] AI-driven resource optimization: Use artificial intelligence technology to intelligently optimize system resources and predict faults, improve system operating efficiency and stability, and reduce maintenance costs.

[0081] Automated operation and maintenance: The system can automatically detect and optimize resource usage, reduce manual intervention, and improve operation and maintenance efficiency.

[0082] 9. High-quality video output

[0083] Realistic rendering effect: Using advanced rendering engine to generate high-quality, realistic digital human videos, enhancing visual effects and user experience.

[0084] Multi-format support: Supports multiple video format output to meet the playback requirements of different platforms and devices, ensuring the wide compatibility of the video.

[0085] 10. Intelligent real-time synchronization

[0086] Delay optimization and synchronization correction: Through optimization algorithms and real-time monitoring, accurate synchronization of voice and video is ensured to improve the naturalness and smoothness of interaction.

[0087] Dynamic adjustment mechanism: The system can automatically adjust synchronization parameters based on real-time data, adapt to different network environments and usage scenarios, and maintain high-quality synchronization effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0089] Figure 1 is a flow chart of the method of the present invention;

[0090] Figure 2 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0091] Below, the invention is preferably described in conjunction with the accompanying drawings and specific implementation methods.

[0092] In one embodiment of the present invention, Figure 1 As shown, a method for synthesizing a digital human live video is provided, which is used to generate high-simulation virtual human video content with natural voice, expression and coordinated movements in real time. The method comprises the following specific steps:

[0093] A multimodal data input from a user is received, wherein the multimodal data includes: voice, text, and image.

[0094] Receive multimodal data input from a user, where the multimodal data includes voice data, text data, and image data, specifically including:

[0095] Voice data input, obtaining the user's voice signal through a microphone or audio acquisition device, parsing the voice content to extract voice features, and using them to drive the digital human's voice and lip matching;

[0096] Text data input, receiving text content input by users through text input devices (such as keyboards, touch screens, and voice-to-text systems), and performing semantic analysis on the input text to guide the digital human's language expression and emotional regulation;

[0097] Image data input, obtaining the digital human facial image or video stream provided by the user. The image data can be used to construct or adjust the digital human image and generate a high-simulation video output in combination with other input data.

[0098] Extract key features from the multimodal data. The key features include voice features, video features and text features, specifically including:

[0099] Speech feature extraction: pre-processing the input speech data, including noise reduction, normalization and feature enhancement; using convolutional neural networks, long short-term memory networks or transformer models to analyze speech data and extract pitch, intonation, energy distribution, rhythm and emotional features to drive the lip synchronization and emotional expression of digital humans;

[0100] Video feature extraction: Use face detection and key point recognition algorithms such as MTCNN, FACS, and OpenPose to analyze the input facial images or videos, extract facial feature points, expression changes, muscle movement trajectories, and head posture information to guide the expression changes and dynamic adjustments of digital humans;

[0101] Text feature extraction uses natural language processing technology to perform word segmentation, part-of-speech tagging and semantic analysis on the text data input by the user, and extracts the semantic information, emotional tendency and intention of the text based on pre-trained language models such as BERT, GPT or Transformer to guide the language expression and emotional regulation of digital humans.

[0102] The key features are integrated to form integration conditions. The speech features, text features and video features are integrated to form a comprehensive condition input for digital human video synthesis, which specifically includes the following steps:

[0103] Feature standardization and alignment: normalize the extracted speech features, text features, and video features respectively, and use time alignment algorithms (such as dynamic time warping or attention mechanism) to synchronize the different modal data to ensure the temporal consistency of feature matching;

[0104] Feature encoding and mapping, using multimodal neural networks, such as multimodal transformers, cross-modal attention mechanisms, or long short-term memory networks, to encode the features of each modality and map them to a unified feature space to eliminate the dimensionality differences between modalities;

[0105] Feature fusion, using weighted fusion, gated recurrent unit fusion, attention mechanism fusion or multimodal Bayesian reasoning methods to interactively fuse speech features, text features and video features to form comprehensive conditional input;

[0106] Fusion feature optimization, by training deep neural networks, optimizes the weights of fusion features based on supervised learning or self-supervised learning, so that the fused features have higher semantic consistency and information completeness, which can be used to drive subsequent digital human video synthesis.

[0107] The neural network model is used to generate the corresponding radiation field representation according to the fusion conditions. This step maps and optimizes the multimodal features through the neural network model to generate a high-precision radiation field, thereby achieving a high-simulation, low-latency and natural and smooth synthesis effect of the digital human live video. The specific implementation process is as follows:

[0108] First, obtain the fusion condition input:

[0109] (1) Multimodal feature fusion

[0110] Voice features: pitch, tone, speaking speed, rhythm, emotional tendency

[0111] Text features: semantic structure, keywords, sentiment tags, contextual information

[0112] Video features: facial key points, expression changes, head posture, lighting environment

[0113] (2) Unified Representation

[0114] Multimodal encoding techniques (such as Transformer-based fusion networks) are used to map data from different modalities into the same latent space to eliminate the dimensionality differences between different modalities.

[0115] Weighted fusion and self-attention mechanisms are used to improve the accuracy of feature matching, so that the final fusion condition can accurately describe the user's current interaction status.

[0116] In this step, the speech, text and video features are efficiently integrated to ensure the complementarity of information from different data sources and enhance the ability of semantic understanding. Information loss is avoided, and a complete fusion feature is constructed through a deep neural network to improve the quality of video synthesis.

[0117] The neural radiation field is then reconstructed, and a neural network model based on NeRF or its optimized versions, such as Dynamic NeRF, DeformableNeRF, and FastNeRF, is used to construct a radiation field representation in three-dimensional space.

[0118] Why choose NeRF?

[0119] Traditional 3D animation requires manual modeling and animation design. NeRF uses neural networks to automatically learn scene information such as lighting, color, density, etc. to achieve high-precision 3D reconstruction. Compared with image generation methods such as GAN, NeRF can achieve more detailed volume rendering and avoid the limitations of surface modeling.

[0120] In the implementation, MLP is used to map the fusion condition input to a high-dimensional latent space to learn richer features such as lighting, color, density, etc. Combined with position encoding, the ability to retain high-frequency information is improved, making the rendering result clearer.

[0121] The color and transparency of each voxel are calculated based on the input perspective, lighting and motion status. Layered sampling technology is used to improve rendering efficiency and reduce computing overhead.

[0122] Combined with time-series neural networks, such as RNN, LSTM, or autoregressive models, it ensures that the radiation field parameters change dynamically over time to match the real-time requirements of live broadcast scenarios.

[0123] In this step, MLP high-dimensional coding is used to retain rich lighting and color details and improve visual quality. Layered sampling is used to improve rendering efficiency, reduce computing costs, and ensure live broadcast fluency. Timing modeling allows digital humans to perform natural continuous movements without frame skipping or freezes.

[0124] Then the key parameters of the radiation field are predicted. Through the hidden variable learning mechanism of the neural network, the key parameters of the radiation field are predicted according to the fusion conditions, including:

[0125] (1) Density distribution

[0126] Control the density value of different points in the scene and determine the light propagation path;

[0127] Enhance density adjustment in high dynamic areas such as mouth shape and eyes to adapt to real-time expression changes.

[0128] (2) Color field

[0129] Generate color information for each point in three-dimensional space to simulate real light reflection effects;

[0130] Combined with ambient light compensation technology, digital humans can maintain consistent visual performance under different lighting conditions.

[0131] In this step, the lip synchronization is optimized through density adjustment to make the speech-driven mouth movements more accurate. Color optimization ensures lighting consistency so that the digital human can maintain natural light and shadow effects in different environments.

[0132] Finally, the generated radiation field is converted into video output using a volume rendering method, and an optimization strategy is combined to improve the clarity, naturalness, and temporal coherence of the synthesized video, including:

[0133] (1) Rendering loss optimization

[0134] Calculate the error between the synthesized frame and the target frame to reduce color deviation and improve rendering accuracy.

[0135] (2) Adversarial training uses the GAN training method to make the generated video more realistic and avoid distortion or blurring of the digital human image.

[0136] (3) Timing consistency loss

[0137] Constrain the difference between consecutive frames to ensure smooth transitions between frames and prevent flickering or frame breaks.

[0138] In this step, adversarial training is used to improve video clarity and make the digital human more realistic. Timing consistency optimization eliminates lag and improves the smoothness of live broadcast.

[0139] Fine-tune the radiation field parameters according to real-time input. In this step, the radiation field parameters are dynamically adjusted through real-time input data to optimize the real-time, accuracy, fluency and naturalness of video synthesis, ensure that the digital human live video can synchronize the user's voice, text and expression, and achieve a highly simulated interactive experience. The specific implementation method is as follows:

[0140] First, real-time data acquisition and timing synchronization

[0141] (1) Multimodal data acquisition:

[0142] Collect voice data, including pitch, intonation, rhythm, phoneme boundaries and other features to drive the digital human's mouth movement and emotional expression;

[0143] Collect text data, analyze syntactic structure and emotional tendency, and provide semantic guidance for facial expressions, head movements, etc.

[0144] Collect image data, detect the user's facial expression changes, head posture and other visual features to enhance the naturalness of digital human synthesized videos.

[0145] (2) Multimodal time synchronization:

[0146] Since speech, text, and image data have different sampling rates, a timing alignment algorithm (such as dynamic time warping or attention mechanism) is used to time align the data of each modality;

[0147] Incrementally update the data through a sliding window mechanism or autoregressive models (such as AR, LSTM) to ensure that the latest input data can quickly adjust the behavior of the digital human, reduce computing overhead, and improve response speed.

[0148] This step can ensure the temporal consistency of voice, text and image data, so that the digital human can respond to user input in real time, avoid the asynchrony caused by time lag between different modalities, and improve the naturalness of the live broadcast.

[0149] Then perform adaptive adjustment of radiation field parameters

[0150] (1) Introducing a learnable parameter adjustment module:

[0151] Based on the original radiation field model, a dynamic parameter optimization module is constructed to adjust the following key parameters:

[0152] Density field (: controls the transparency and volume of pixels to enhance the clarity and dynamic responsiveness of the mouth area;

[0153] Color field: adjusts the color information of high dynamic areas such as mouth and eyes to make the synthesized video more realistic;

[0154] Dynamic deformation parameters: Fine-tune facial expressions and head posture to ensure that expressions and movements match the emotions of the speech.

[0155] (2) Use optimization algorithms for efficient adjustment:

[0156] Gradient descent (such as SGD, Adam, etc.): optimize the parameters of the radiation field model to ensure that the generated image can quickly adapt to input changes;

[0157] Low-rank adaptation technology: Only specific high-dynamic areas (such as mouth and eyes) are adjusted without affecting the overall model structure, thereby reducing the computational burden and accelerating convergence;

[0158] Time Series Forecasting:

[0159] Predict the speech-expression-gesture relationship in the next few frames and adjust the radiation field parameters in advance to avoid abrupt visual jumps;

[0160] Autoregressive prediction is used to improve inter-frame smoothness, making the expression changes of digital humans smoother.

[0161] This step performs local optimization on key areas rather than global adjustments to avoid unnecessary calculations and increase rendering speed. It uses time series modeling to improve inter-frame coherence, reduce sudden changes in the screen, and improve viewing quality.

[0162] Finally, video synthesis optimization is performed to reduce artifacts and improve visual quality

[0163] (1) Artifact removal and perceptual loss optimization:

[0164] Use adversarial training and perceptual loss to optimize the texture details of the facial area and prevent blurred boundaries or image jitter;

[0165] Motion compensation technology is used to reduce mouth image tearing or freezing caused by high-speed speech movement.

[0166] (2) Adaptive interpolation and rendering optimization:

[0167] In low-bandwidth or high-load environments, frame insertion algorithms (such as FLAVR and DAIN) are used to smooth transitions and improve video fluency;

[0168] Combined with volume rendering optimization (such as FastNeRF and Instant-NGP), it reduces computational redundancy, improves rendering frame rate, and ensures smooth live broadcasting.

[0169] For example:

[0170] The user enters text during the live broadcast: "Welcome everyone to the live broadcast room!"

[0171] Voice characteristics: high pitch

[0172] Text sentiment: Warm, welcoming

[0173] Image features: Smiling expression

[0174] System optimization for lip sync:

[0175] Adjust the density field and color field to optimize the rendering of the mouth area so that the mouth shape accurately matches the voice input;

[0176] Use phoneme-lip alignment to ensure that the mouth opening and closing movements are consistent with the speech rhythm.

[0177] Fine-tune the expression:

[0178] By extracting emotional features, detecting voice emotions and adjusting the dynamics of eyes and eyebrows, the digital human can show a warm and friendly smile;

[0179] Adjust the weight of expressions (such as increasing eye brightness and raising mouth corners) to enhance the interactivity of live broadcasts.

[0180] Perform head posture optimization:

[0181] Adjust the head angle according to the fluctuation of voice tone to avoid sluggish movements and make it more expressive;

[0182] When the tone of voice changes, the dynamic deformation parameters are adjusted to make the head deflect naturally and enhance the sense of interaction. Through the above fine-tuning, the digital human anchor can accurately match the voice, expression and posture, and generate highly realistic, natural and smooth videos in real time, giving the audience a more immersive interactive experience.

[0183] The generated radiation field is converted into video output. The specific steps include:

[0184] The volume rendering technology is used to perform integral calculations on the density distribution and color distribution in the radiation field to generate the corresponding two-dimensional image frames.

[0185] Through the ray casting method, voxel data in the scene is sampled along the direction of the light from the target perspective, and the color and transparency information are calculated based on the volume accumulation principle to synthesize high-quality frame images.

[0186] Furthermore, image optimization and compensation can be performed:

[0187] Adopt super-resolution reconstruction technology to enhance the clarity of output video and avoid blur or pixel distortion.

[0188] Combined with time consistency optimization, jitter or flickering across frames can be eliminated to ensure video smoothness.

[0189] Finally, frame sequence synthesis and encoding are performed:

[0190] Combine the rendered continuous image frames in time order to form a video stream that meets the target frame rate;

[0191] Use video compression coding (such as H.264, H.265, AV1) to compress video data and reduce storage and transmission overhead.

[0192] The real-time video output and streaming media transmission are then combined with GPU accelerated rendering (such as CUDA, TensorRT) to achieve low-latency video generation and output; streaming media protocols (such as RTMP, WebRTC) are used to push the generated video to the live broadcast platform or terminal device to achieve real-time playback.

[0193] Based on any one of the above embodiments or a combination of the embodiments, this embodiment further implements a real-time synchronization mechanism to ensure the synchronization of voice and facial movements, including delay optimization and synchronization correction.

[0194] In order to improve the real-time performance of digital human live video synthesis, the following optimization strategies are used to reduce the computational delay in speech feature extraction and video generation:

[0195] An efficient feature extraction algorithm is adopted, and lightweight neural networks (such as MobileNet, EfficientNet, DistilBERT) are used to extract speech features, text features, and video features to reduce computational complexity; dimensionality reduction techniques (such as PCA, t-SNE) are combined to reduce the dimensionality of high-dimensional features, reduce computational overhead, and increase the speed of feature extraction.

[0196] A low-latency neural network architecture is adopted. In the process of radiation field generation, NeRF acceleration technology (such as Instant-NGP, FastNeRF, PlenOctrees) is used to reduce the calculation time of light sampling and volume rendering; the knowledge distillation method is used to compress large-scale neural network models into lightweight models to improve the reasoning speed.

[0197] Use hardware acceleration and parallel computing, use GPU parallel computing (such as CUDA, TensorRT, cuDNN), and improve reasoning efficiency through deep learning framework optimization (such as TensorFlow XLA, PyTorch JIT); combine FPGA or TPU acceleration to further reduce computing latency in embedded or cloud environments.

[0198] Adaptive frame rate adjustment: When computing resources are limited, dynamically adjust the frame rate of video generation (such as reducing it from 60FPS to 30FPS) to ensure smoothness; use interpolation algorithms (such as FLAVR and DAIN) to improve visual effects through interpolation while reducing the rendering frame rate, thereby reducing the perception of delay.

[0199] To ensure the synchronization of voice and video, the following technologies are used to monitor and correct possible time deviations in real time:

[0200] The speech-video alignment mechanism is adopted. In the speech input stage, the short-time Fourier transform and phoneme alignment are used to calculate the timestamp of the speech signal. In the video generation process, the lip synchronization detection algorithm is used to match the lip shape with the time frame of the speech.

[0201] Dynamic time warping or CTC (Connectionist Temporal Classification) algorithm is used to automatically adjust the time sequence of voice and video.

[0202] Real-time time deviation monitoring and adaptive correction are adopted, and the optical flow analysis method is used to detect whether the movement trajectory of facial features conforms to the rhythm of voice input; combined with the temporal attention mechanism, the frame synchronization of voice and video is predicted in real time, and dynamic adjustment is made when deviation is detected; feedback control systems (such as PID control) are used to fine-tune the audio and video frame rates to keep them in time synchronization at all times.

[0203] End-to-end self-supervised optimization is used: multimodal alignment neural networks (such as AV-HuBERT and Wav2Lip) are trained to directly optimize speech-video synchronization during the generation phase; temporal consistency loss is used to ensure that the changes in mouth shape and expression between consecutive frames match the rhythm of the speech input; during the training process, synthetic adversarial networks are used to constrain the temporal synchronization between different modalities to improve the fluency of the final output.

[0204] For example:

[0205] During the live broadcast, if the user says "Hello everyone, welcome to my live broadcast room!", but due to network fluctuations or computing delays, the initial output video has a lip lag problem, the system will automatically correct it in the following ways:

[0206] Align the speech and video frames, calculate the phoneme timing of the speech, and adjust the time offset of the mouth frame to accurately match the mouth movement with the phoneme change.

[0207] Dynamic compensation: If it is detected that the mouth shape lags behind by 2 frames (about 33 milliseconds), the system will interpolate through the time series prediction model to advance the mouth movement by 2 frames to ensure visual and auditory synchronization.

[0208] Adaptive adjustment: if the overall delay is detected to be too high (>100ms), the video resolution will be reduced or the calculation priority will be adjusted to reduce the delay and improve the synchronization effect.

[0209] Ultimately, the optimized digital human live video can ensure that the voice and lip shape are highly synchronized, making the user viewing experience smoother and more natural.

[0210] Based on any one of the foregoing embodiments or a combination of the embodiments, this implementation provides multiple user interaction modes to improve the usability and interactivity of the system, specifically including:

[0211] Graphical user interface interaction support provides a visual graphical user interface, which includes an interactive control panel, a parameter adjustment area and a real-time preview window, allowing users to complete the configuration, parameter setting and control of the digital human live video through operations such as clicking, sliding, and inputting. The GUI interface supports different resolutions and operating terminals, including but not limited to computers, tablet devices and mobile terminals.

[0212] Voice command interactive control uses voice recognition technology to support users to control the digital human's movements, expressions and content display in real time through voice commands. The voice commands include but are not limited to operations such as start, pause, expression adjustment, and voice synthesis content switching. Based on natural language processing and voice command parsing modules, the voice content input by the user is parsed in real time and converted into executable command signals to drive the digital human to make corresponding action adjustments.

[0213] The personalized customization function provides multi-dimensional personalized parameter adjustment options, allowing users to adjust the digital human's appearance characteristics (including but not limited to hairstyle, skin color, clothing, facial features, etc.), voice characteristics (including timbre, pitch, speaking speed, etc.) and behavior patterns (including expression style, movement amplitude, interaction method, etc.) according to their needs to adapt to different application scenarios, such as virtual anchors, education and training, intelligent customer service, etc.; the personalized customization supports adaptive optimization based on deep learning models, so that the appearance and behavior of the digital human can be dynamically adjusted based on the user's historical preferences.

[0214] Based on any one of the foregoing embodiments or a combination of the embodiments, this embodiment provides the administrator with comprehensive system configuration and management functions, specifically including:

[0215] User management provides user authority and role management functions, and supports hierarchical management of system users based on identity authentication mechanism. The hierarchical management includes but is not limited to administrator, ordinary user and visitor permissions; access control list-based or role-based access control strategies are adopted to ensure that different users have corresponding access rights and operation permissions to prevent unauthorized operations or data tampering.

[0216] Resource management: real-time monitoring and dynamic allocation of system resource usage, including but not limited to computing resources, storage resources, and bandwidth resources. Load balancing mechanism is used to optimize resource utilization, and efficient task scheduling is achieved based on adaptive scheduling algorithms (such as dynamic resource allocation DRF and minimizing resource waste MRW) to ensure system stability and real-time responsiveness.

[0217] Logging and monitoring provide system operation log and user operation log recording functions, adopt a distributed log storage architecture, perform structured storage and indexing of log data, and improve query efficiency; support real-time monitoring of key system indicators (such as CPU occupancy, memory usage, network traffic, etc.), combine anomaly detection algorithms (such as abnormal behavior detection based on LSTM or Transformer) to analyze abnormal operations, and trigger early warning mechanisms to support system fault diagnosis and fault recovery.

[0218] Security management, adopting multi-level security strategies to ensure the security of the system and user data, including but not limited to data encryption, access control and intrusion detection, specifically including: data encryption, using AES, RSA or elliptic curve encryption technology to encrypt the storage and transmission of user sensitive data to prevent data leakage; access control, combined with the access authentication mechanism based on zero trust architecture, ensures access security through multi-factor authentication; intrusion detection, using behavioral analysis and machine learning methods (such as abnormal traffic detection, malicious code identification) to detect system security threats to prevent malicious attacks and data tampering.

[0219] like Figure 2 As shown, in one embodiment, the present invention further provides a digital human live video synthesis system, the system comprising the following modules:

[0220] An input processing module, used to receive multimodal data input from a user, wherein the multimodal data includes: voice, text, and image;

[0221] A feature extraction module, used to extract key features from the multimodal data;

[0222] A conditional fusion module, used to fuse the key features to form a fusion condition;

[0223] A radiation field calculation module, used to generate a corresponding radiation field representation according to the fusion condition using a neural network model;

[0224] A feature fine-tuning module, used for fine-tuning the radiation field parameters according to real-time input;

[0225] A video synthesis module is used to convert the generated radiation field into a video output.

[0226] In one embodiment, the present invention further improves the aforementioned digital human live video synthesis system, wherein the video synthesis module is further used for:

[0227] Adjust the digital human’s facial expressions and movements based on the extracted speech and text features;

[0228] Encode the rendered video and transmit it to the user end in real time.

[0229] In one embodiment, the present invention further improves the aforementioned digital human live video synthesis system, wherein the system further includes a real-time synchronization module for:

[0230] Low-latency processing for speech feature extraction and video synthesis;

[0231] Monitor the synchronization status of voice and video in real time and automatically correct possible time deviations.

[0232] In one embodiment, the present invention further improves the aforementioned digital human live video synthesis system, wherein the system further comprises a user interaction module, and the user interaction module is used to:

[0233] Users can configure and control the live video of the digital human through simple clicks and inputs;

[0234] Users control the digital human’s movements, expressions, and content display through voice commands;

[0235] Allows users to adjust the appearance, voice and behavior of digital humans according to their needs.

[0236] In one embodiment, the present invention further improves the aforementioned digital human live video synthesis system, wherein the system further comprises a system management module, and the system management module is used to:

[0237] Manage the permissions and roles of system users to ensure that different users have different access rights;

[0238] Monitor and manage the usage of system resources to ensure efficient and stable operation of the system;

[0239] Record system operation logs and user operation records, and provide real-time monitoring and troubleshooting functions;

[0240] Implement data encryption and access control to ensure the security and privacy of system and user data.

[0241] It should be noted that the explanation of the aforementioned embodiment of the digital human live video synthesis method is also applicable to the device of the embodiment of the present application and will not be repeated here.

[0242] In order to implement the above embodiment, the present application also proposes a computer device, which is a schematic diagram of the structure of the computer device. When the instruction processor in the computer device executes, the digital human live video synthesis method in the above embodiment is implemented.

[0243] In order to implement the above-mentioned embodiments, the embodiments of the present application also propose a non-temporary computer-readable storage medium, which stores a computer program. When the computer program is run on a computer, the computer executes the digital human live video synthesis method in the above-mentioned embodiments.

[0244] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0245] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0246] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk and other media that can store program codes.

[0247] The above is only a specific implementation method of the present application. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this application, which should be included in the protection scope of this application. The scope of protection of this application shall be based on the protection scope of the claims. For some module structures that are not particularly clear in the present invention, the contents recorded in the prior art shall prevail. The prior art mentioned in the aforementioned background technology section and the specific embodiment section of the present invention can be used as a part of the present invention to understand the meaning of some technical features or parameters.

Claims

1. A method for synthesizing digital human live video, characterized in that: The method comprises the following steps: Receiving multimodal data input from a user, wherein the multimodal data includes: voice, text, and image; extracting key features from the multimodal data; Fusion of the key features to form fusion conditions; Using a neural network model, generating a corresponding radiation field representation according to the fusion condition; fine-tuning the radiation field parameters based on real-time input; Convert the generated radiation field to video output.

2. The method for synthesizing digital human live video according to claim 1, characterized in that: The conversion of the generated radiation field into a video output includes: Adjust the digital human’s facial expressions and movements based on the extracted speech and text features; Encode the rendered video and transmit it to the user end in real time.

3. The method for synthesizing digital human live video according to claim 1, characterized in that: Low-latency processing for speech feature extraction and video synthesis; Monitor the synchronization status of voice and video in real time and automatically correct possible time deviations.

4. The method for synthesizing digital human live video according to claim 1, characterized in that: Use the following method to operate: Users can configure and control the live video of the digital human through simple clicks and inputs; Users control the digital human’s movements, expressions, and content display through voice commands; Allows users to adjust the appearance, voice and behavior of digital humans according to their needs.

5. The method for synthesizing digital human live video according to claim 1, characterized in that: Use the following methods to manage the system: Manage the permissions and roles of system users to ensure that different users have different access rights; Monitor and manage the usage of system resources to ensure efficient and stable operation of the system; Record system operation logs and user operation records, and provide real-time monitoring and troubleshooting functions; Implement data encryption and access control to ensure the security and privacy of system and user data.

6. A digital human live video synthesis system, characterized in that: The system includes the following modules: An input processing module, used to receive multimodal data input from a user, wherein the multimodal data includes: voice, text, and image; A feature extraction module, used to extract key features from the multimodal data; A conditional fusion module, used to fuse the key features to form a fusion condition; A radiation field calculation module, used to generate a corresponding radiation field representation according to the fusion condition using a neural network model; A feature fine-tuning module, used for fine-tuning the radiation field parameters according to real-time input; A video synthesis module is used to convert the generated radiation field into a video output.

7. The digital human live video synthesis system according to claim 6, characterized in that: The video synthesis module is also used for: Adjust the digital human’s facial expressions and movements based on the extracted speech and text features; Encode the rendered video and transmit it to the user end in real time.

8. The digital human live video synthesis system according to claim 6 is characterized in that The system also includes a real-time synchronization module, which is used to: Low-latency processing for speech feature extraction and video synthesis; Monitor the synchronization status of voice and video in real time and automatically correct possible time deviations.

9. The digital human live video synthesis system according to claim 6, characterized in that: The system further comprises a user interaction module, wherein the user interaction module is used to: Users can configure and control the live video of the digital human through simple clicks and inputs; Users control the digital human’s movements, expressions, and content display through voice commands; Allows users to adjust the appearance, voice and behavior of digital humans according to their needs.

10. The digital human live video synthesis system according to claim 6, characterized in that: The system further comprises a system management module, wherein the system management module is used for: Manage the permissions and roles of system users to ensure that different users have different access rights; Monitor and manage the usage of system resources to ensure efficient and stable operation of the system; Record system operation logs and user operation records, and provide real-time monitoring and troubleshooting functions; Implement data encryption and access control to ensure the security and privacy of system and user data.

Images