Digital human synthesis system based on interactive multimedia editing and cloud rendering technology

Abstract

The invention relates to the technical field of digital human synthesis, in particular to a digital human synthesis system based on an interactive multimedia editing and cloud rendering technology, which comprises a multi-mode interactive editing front end, and the multi-mode interactive editing front end is coupled with a distributed action synthesis engine. The distributed action synthesis engine is coupled with an elastic cloud rendering cluster, and the elastic cloud rendering cluster is coupled with an intelligent synchronous controller. According to the digital human synthesis system based on the interactive multimedia editing and cloud rendering technology, through triple innovation of algorithm-architecture-theory, the end-to-end delay is less than 80ms under film and television level image quality, and a performance benchmark is established for next-generation digital human application; the problem that in the actual use process of a digital man-machine synthesis system in the prior art, when a GPU conducts path tracking under the 4K resolution, the single-frame rendering time exceeds 500 ms, and the 100 ms threshold value of real-time interaction cannot be met is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital human synthesis, in particular to a digital human synthesis system based on interactive multimedia editing and cloud rendering technology. BACKGROUND

[0002] The digital human synthesis system is a virtual digital human generation and interaction platform based on artificial intelligence, computer graphics, speech synthesis and recognition, natural language processing and other multi-field technologies. The system generates highly realistic digital human images by simulating human appearance, voice, behavior and thinking, and realizes natural interaction with users. It is widely used in entertainment, education, medical care, finance, customer service and other fields, supports personalized customization, including facial features (facial features, expressions), body shape, clothing, hairstyle, etc., can generate digital human images of different styles such as hyper-realism, cartoon and animation, and realizes high-precision restoration and dynamic display of digital human appearance through 3D modeling, texture mapping, real-time rendering and other technologies. Using deep learning algorithms, text can be converted into natural and fluent speech output, supporting multiple languages, dialects and tone selection. Integrating speech recognition technology, it can analyze user voice commands in real time and realize two-way voice interaction. Through semantic understanding, emotion analysis and context modeling, the digital human can understand user intentions and generate responses that meet the context, supporting multi-round dialogue, casual conversation, question and answer and other scenarios, and improving the naturalness and intelligence level of interaction. Combined with optical motion capture, inertial sensors or AI algorithms, the digital human can realize natural and smooth body movements and expression changes. Integrating voice, text, image, video and other interactive modes, it realizes cross-modal information fusion and collaborative response, supports emotional interaction, adjusts response strategies according to user emotions, and enhances user experience.

[0003] However, the digital human synthesis system of the prior art has a single-frame rendering time of more than 500ms when the GPU performs path tracking at a 4K resolution in actual use, which cannot meet the 100ms threshold of real-time interaction. To solve the above problems, a digital human synthesis system based on interactive multimedia editing and cloud rendering technology is provided. SUMMARY

[0004] The purpose of the present application is to provide a digital human synthesis system based on interactive multimedia editing and cloud rendering technology to solve the problems raised in the background art. To achieve the above purpose, the present application provides the following technical solution: a digital human synthesis system based on interactive multimedia editing and cloud rendering technology, comprising a multi-modal interactive editing front end, the multi-modal interactive editing front end is coupled with a distributed action synthesis engine, the distributed action synthesis engine is coupled with an elastic cloud rendering cluster, and the elastic cloud rendering cluster is coupled with an intelligent synchronization controller.

[0005] Preferably, the multi-modal interaction editing front end integrates LiDAR, IMU, RGB-D camera multi-source sensing data, transmits only changed parameters, and provides low-precision real-time rendering.

[0006] Preferably, the integration of LiDAR, IMU, RGB-D camera multi-source sensing data uses extended Kalman filter (EKF) to fuse multi-sensor data, and the formula is:

[0007]

[0008] Wherein, H k is the observation model Jacobian matrix.

[0009] Preferably, the transmission of only changed parameters uses priority queue scheduling, and the formula is:

[0010]

[0011] Wherein, Q is the rendering quality evaluation function.

[0012] Preferably, the distributed work synthesis engine converts raw sensing data into realistic limb movement, performs joint restriction, collision avoidance, and mass distribution simulation, and completes calculation at a 120Hz update frequency.

[0013] Preferably, the conversion of raw sensing data into realistic limb movement uses an improved Kalman filter to eliminate noise, and the formula is:

[0014]

[0015] Wherein, W k is an adaptive weight matrix.

[0016] The distribution simulation uses an improved FABRIK algorithm, and the iterative formula is:

[0017] θ n+1 =θ n +J(θ n )(x d -f(θ n ));

[0018] The pseudo-inverse matrix optimization calculation formula is:

[0019] J=J T (JJ T +λI) -1 (λ=0.1)。

[0020] Preferably, the elastic cloud rendering cluster uses distributed execution of ray tracing calculation, allocates computing power based on the importance of the role, and eliminates the inherent noise of path tracing.

[0021] Preferably, the distributed execution light ray tracing calculation dynamically adjusts to adopt a density calculation formula:

[0022]

[0023] Wherein, δ is the volume scattering coefficient;

[0024] The inherent noise of the elimination path tracking adopts an SVGF filtering algorithm, and the formula is:

[0025]

[0026] Wherein, is an adaptive mixing coefficient.

[0027] Preferably, the multi-modal interactive editing front end comprises a data acquisition module, the data acquisition module is coupled with a signal processing module, the signal processing module is coupled with a multi-modal fusion module, and the multi-modal fusion module is coupled with an interactive logic module.

[0028] Preferably, the interactive logic module allocates tasks according to user intentions, including query, control, recommendation, generates reasonable responses in combination with business rules and machine learning models, maintains conversation states, and ensures the coherence of interaction.

[0029] Compared with the prior art, the beneficial effects of the present application are:

[0030] Through the multi-modal interactive editing front end, the distributed action synthesis engine, the elastic cloud rendering cluster and the intelligent synchronization controller, the algorithm-intensive tasks such as light ray tracing are migrated to the cloud, the parameter stream and the video stream are transmitted separately, the network delay is offset through kinematics extrapolation, the solving speed of the FABRIK algorithm is improved, the adaptive calculation amount is reduced, the double-channel transmission bandwidth is saved, the multi-person scene capacity is expanded, the quality-delay decoupling feasibility is strictly proved, the multi-person scene convergence mathematical model is established, through the algorithm-architecture-theory triple innovation, the end-to-end delay is realized under the film-level picture quality, which sets a performance benchmark for the next generation of digital human application, and solves the problem that the single-frame rendering time of the GPU exceeds 500ms when performing path tracking at 4K resolution in the actual use process of the digital human synthesis system of the prior art, which cannot meet the 100ms threshold of real-time interaction. BRIEF DESCRIPTION OF DRAWINGS

[0031] Fig. 1 The system block diagram of the digital human synthesis system based on interactive multimedia editing and cloud rendering technology of the present application is shown in the figure;

[0032] Fig. 2A system block diagram of a multi-modal interactive editing front end of a digital person synthesis system based on interactive multimedia editing and cloud rendering technology of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0034] Please refer to Figs. 1-2The application provides a technical solution: a digital person synthesis system based on interactive multimedia editing and cloud rendering technology, comprising a multi-modal interactive editing front end, which is a core component built on a multi-modal interactive technology system, and its architecture design needs to integrate multi-channel interaction modes such as voice, vision, and touch. The system architecture is usually divided into a perception layer, a processing layer, and a performance layer: the perception layer collects multi-source data through microphone arrays, cameras, force feedback devices, etc.; the processing layer uses a deep learning model with a Transformer architecture to realize cross-modal feature fusion, such as aligning voice features and gesture trajectories in the feature space; the performance layer outputs the interaction results through virtual reality (VR) headsets, holographic projection devices, etc., based on a two-dimensional alignment mechanism of timestamp synchronization and semantic association, using the dynamic time warping (DTW) algorithm to handle the time offset of voice and gestures, and combining the BERT model to realize feature alignment at the semantic level. For example, in a virtual meeting scenario, the system can recognize both the voice command "zoom in" and the pinch action of the gesture, implement interface scaling, and use a federated learning framework to complete preliminary feature extraction on local devices, and then complete feature fusion on the cloud through a secure multi-party computation (SMC) protocol. This architecture not only ensures data privacy but also realizes collaborative optimization among multiple devices, typical applications such as the coordinated control of smartphones, tablets, and smart speakers in smart home scenarios, and the optimization of interactive strategies based on reinforcement learning, with the system dynamically adjusting the combination of interaction modalities based on user behavior data. For example, in a noisy environment, automatically switch to a touch-priority mode, and in a quiet environment, prefer to respond to voice commands to balance interaction efficiency and accuracy. The multi-modal interactive editing front end is coupled with a distributed action synthesis engine, and simulation modules are decoupled into independent services. gRPC protocol is used for inter-service communication, Kubernetes container orchestration is used for dynamic resource scheduling, and after local data training on edge devices, the global model is updated on the cloud through model aggregation algorithms such as FedAvg. This architecture enables the engine to continuously learn user action habits, such as optimizing virtual character walking animations based on different user gait characteristics, and using a hybrid acceleration scheme of CUDA+TensorRT to complete deep learning inference on GPUs and real-time physical simulation on FPGAs. Typical cases include character motion generation in games, which can support real-time rendering of up to 200 virtual characters, service discovery and failover mechanisms based on the Raft protocol, which can complete service migration within 50ms when a computing node fails, ensuring the continuity of action synthesis, and a resilient cloud rendering cluster coupled with the distributed action synthesis engine, with a resource scheduling algorithm based on reinforcement learning dynamically allocating computing resources based on task characteristics such as resolution, frame rate, and light and shadow complexity.For example, when processing 8K resolution video rendering, the system can automatically allocate 8 NVIDIA A100 GPUs, use a block rendering and progressive transmission scheme, divide the rendering task into 256x256 pixel sub-regions, and push the rendering results in real time through WebSocket. This architecture allows designers to preview the effects in real time during rendering, uses a three-copy storage strategy, and combines the data verification function of the ZFS file system to ensure the reliability of the rendering data. In the event of hardware failure, the system can complete data recovery within 10 minutes. The elastic cloud rendering cluster is coupled with an intelligent synchronization controller, uses the Extended Kalman Filter (EKF) algorithm, and combines laser radar point cloud data and IMU inertial data to achieve centimeter-level positioning accuracy. Typical applications such as unmanned forklifts for cargo handling have a positioning error of less than ±5mm, and a real-time communication network based on the CANopen protocol supports 100 nodes at high speed. In the context of industrial robot collaborative work, millisecond-level synchronization control of multiple robotic arms can be achieved, and the Model Predictive Control (MPC) algorithm can be used to adjust control parameters in real time based on load changes. For example, in numerical control machine tool processing, the system can automatically adjust the feed rate based on material hardness.

[0035] In this embodiment, the multi-modal interactive editing front end integrates LiDAR, IMU, and RGB-D camera multi-source sensor data, transmits only the parameters that have changed, provides low-precision real-time rendering, uses the ICP (Iterative Closest Point) algorithm to spatially align the sparse point cloud generated by LiDAR with the dense point cloud of the RGB-D camera, removes outliers through the RANSAC algorithm, achieves millimeter-level spatial accuracy, uses a deep learning-based point cloud segmentation model (such as PointNet++), adds semantic labels (such as "tabletop" and "wall") to the point cloud data, provides a semantic basis for subsequent interactions, uses the acceleration and angular velocity data of the IMU, corrects the drift of the visual odometry of the RGB-D camera through the Extended Kalman Filter (EKF) algorithm, and improves the spatial positioning accuracy. When the IMU detects a sudden movement (such as acceleration > 5m / s 2) is triggered, ensuring complete capture of dynamic scenes, time synchronization of sensors is achieved through PTP (Precision Time Protocol) with an error of within ±10μs, the geometric features of LiDAR, color features of RGB-D and motion features of IMU are aligned in the feature space, a cross-modal attention mechanism based on Transformer architecture is used to realize semantic association of multi-source data, an inter-frame difference algorithm is used to transmit only the parameters that have changed compared with the previous frame (such as point cloud addition / deletion, color change, motion vector), the scene is divided into 16x16 grids, and the change parameters are calculated independently for each grid, reducing the global calculation complexity, the point cloud coordinates of LiDAR are quantized to 8 bits, and the color information of RGB-D is sub-sampled to 4:2:0 chroma, achieving more than 50% data volume compression, Huffman coding is used for lossless compression of change parameters, combined with LZ77 dictionary compression algorithm, further reducing the transmission bandwidth requirement, real-time monitoring of network bandwidth is achieved through TCP congestion control algorithm, dynamically adjusting the parameter compression ratio, giving high priority to key parameters (such as user gesture trajectory) to ensure real-time interaction, dynamically adjusting the point cloud density according to the distance of the object from the camera, for example, using a sparse point cloud of 0.1 points / cm 2 when the distance is >5m, and switching to a high-density point cloud of 1 point / mm 2 when the distance is <1m, converting point cloud data into voxel grid, generating triangle mesh through Marching Cubes algorithm, realizing hardware accelerated rendering, mapping color information of RGB-D camera to geometric model, using Mipmap technology to realize multi-level texture details, improving rendering efficiency, writing GLSL shader program to realize real-time coloring of point cloud on GPU, supporting PBR (Physically Based Rendering) material effect, based on IMU motion data, using Kalman filtering algorithm to predict the motion trajectory of the object, reducing rendering delay, through Temporal Anti-Aliasing (TAA) technology, combining historical frame information to eliminate the aliasing effect caused by low-precision rendering, achieving millimeter-level gesture tracking accuracy by fusing LiDAR and RGB-D data, supporting real-time recognition of complex gestures (such as pinch, swipe), users can directly manipulate virtual objects through gestures, the system renders the operation results in real time with a delay of less than 50ms, combining LiDAR and IMU data to achieve centimeter-level spatial positioning, supporting real-time navigation of AR glasses, scanning the environment through RGB-D camera to generate a low-precision but real-time 3D map for path planning, designers can rotate and scale virtual models through gestures, the system renders model changes in real time, supporting multi-user collaborative review, compared with traditional high-precision rendering, the system bandwidth requirement is reduced by 70%, the rendering delay is reduced by 80%, suitable for mobile devices and cloud rendering scenarios.

[0036] In this embodiment, the multi-source sensor data of LiDAR, IMU, and RGB-D camera are fused by Extended Kalman Filter (EKF) to fuse multi-sensor data, and the formula is:

[0037]

[0038] wherein H k is the Jacobian matrix of the observation model.

[0039] In this embodiment, only the parameters that change are transmitted using priority queue scheduling, and the formula is:

[0040]

[0041] wherein Q is the rendering quality evaluation function.

[0042] In this embodiment, the distributed work synthesis engine converts the original sensor data into realistic limb movement, performs joint restriction, collision avoidance, and mass distribution simulation, and completes the calculation at a 120Hz update frequency. At a 120Hz update frequency, multi-source sensor data (such as IMU, force feedback device, and visual marker point) is converted into limb movement that conforms to physical rules, while joint restriction, collision avoidance, and mass distribution simulation are completed. Multi-modal sensor fusion: IMU (Inertial Measurement Unit), force feedback gloves, and optical motion capture system. IMU data Kalman filter denoising, force feedback data time alignment, and optical marker point space calibration. Only key motion parameters (such as joint angles and contact force vectors) are transmitted, using inter-frame difference + Huffman encoding compression rate of 80%. Real-time transport protocol (such as RTP) based on UDP, network jitter compensation algorithm, bone animation calculation, collision detection, and physical simulation are distributed to multi-GPU / multi-core CPU nodes, parallel computing is realized using CUDA Streams, calculation delay <8ms, real-time bone binding and skinning of Unity / Unreal engine are supported, contact force and friction force are output through force feedback device, joint angles are limited to [-180°, 180°] through Clamp function, the scene is divided into 1m 3 grid, only adjacent grids are subjected to collision detection, AABB bounding box of bone nodes is constructed, objects with no collision possibility are quickly removed, centroid offset is calculated based on human inertia parameters (such as head mass ratio of 8%), Projective Gauss-Seidel (PGS) solver is used, multi-rigid body contact and friction force calculation are supported, task distribution ratio is adjusted in real time according to GPU / CPU load, and L1 / L2 cache is used to accelerate frequently accessed data.

[0043] In this embodiment, the original sensor data is converted into realistic limb movement using an improved Kalman filter to eliminate noise, and the formula is:

[0044]

[0045] where W k is an adaptive weight matrix;

[0046] The distribution simulation uses an improved FABRIK algorithm, and the iterative formula is:

[0047] θ n+1 = θ n + J(θ n )(x d -f(θ n ));

[0048] The pseudo-inverse matrix optimization formula is:

[0049] J = J T (JJ T + λI) -1 (λ = 0.1).

[0050] In this embodiment, the elastic cloud rendering cluster uses distributed execution of ray tracing calculation, allocates computing power based on the importance of the role, eliminates the inherent noise of path tracing, and realizes efficient ray tracing rendering in the elastic cloud rendering cluster through a distributed computing framework., combined with the dynamic allocation of role importance algorithm resources, and targeted to eliminate the inherent noise of the path tracking algorithm, while ensuring the quality of key visual elements, improving the overall rendering efficiency, dividing the scene into grid units (such as 1m 3 ), each unit independently allocates rendering tasks, based on scene semantic analysis (such as role face, hand, key props), assigns priority (P0-P3 level) to different areas, dynamically adjusts the task allocation ratio (such as P0 level task allocation 70% computing power) through real-time monitoring of cluster computing power (GPU utilization, memory bandwidth), using the "master node + computing node" architecture, the master node is responsible for task scheduling and result merging, and the computing node executes the core calculation of ray tracing. Through data prefetching and caching strategy, reduce cross-node data transmission (such as texture map, material parameter), support node fault automatic migration, ensure that the rendering task does not interrupt, combined with deep learning model (such as VGG16) to extract scene features, calculate the visual weight of the role area, according to the user interaction history (such as gaze point, operation frequency), dynamically adjust the role importance score, divide the ray tracing task into P0 (high fidelity), P1 (medium fidelity), P2 (low fidelity) three level queues according to the importance of the role.

[0051] In this embodiment, the density calculation formula for dynamic adjustment of distributed execution of ray tracing calculation is:

[0052]

[0053] wherein, δ is the volume scattering coefficient;

[0054] The inherent noise of the path tracking is eliminated by using an SVGF filtering algorithm, and the formula is:

[0055]

[0056] wherein, is an adaptive mixing coefficient.

[0057] In the embodiment, the multi-modal interactive editing front end comprises a data acquisition module, the data acquisition module is coupled with a signal processing module, the signal processing module is coupled with a multi-modal fusion module, and the multi-modal fusion module is coupled with an interactive logic module.

[0058] In the embodiment, the interactive logic module assigns tasks according to user intentions, including query, control, and recommendation, generates reasonable responses in combination with business rules and machine learning models, maintains the conversation state, and ensures the coherence of the interaction.

[0059] In the above embodiments, an interactive digital human editing framework, cloud distributed rendering and lightweight terminal collaboration, multi-modal data fusion intelligent digital human generation, collaborative cloud editing and version management, and a real-time cloud rendering driven digital human live broadcast system are also used.

[0060] The interactive digital human editing framework proposes an interactive editing interface, supports non-professional users to adjust the background and voice parameters of the digital human in real time through natural interaction, and breaks through the complexity of digital human video editing. A template video one-key generation is provided to reduce the manual debugging time.

[0061] The cloud distributed rendering and lightweight terminal collaboration designs a modular rendering architecture: high-load video synthesis, voice synthesis, and digital human mouth shape generation tasks are split to the cloud GPU cluster, and the terminal only processes low-latency interactive instructions to realize real-time synthesis of 4K digital humans.

[0062] The multi-modal data fusion intelligent digital human generation combines voice driving (ASR+TTS) and text semantic analysis to generate synchronized expressions, mouth shapes, and limb movements, solving the traditional digital human multi-modal separation problem. Based on the GAN cross-modal style transfer technology, users can input a video to one-key match the digital human appearance style (such as cartoon, realistic);

[0063] The collaborative cloud editing and version management supports multi-user remote collaborative editing of the same digital human project, ensures data consistency through operation logs and conflict resolution algorithms. Combined with blockchain technology, the editing history is stored as evidence to ensure traceability of digital human IP copyright;

[0064] The real-time cloud rendering driven digital human live broadcast system directly applies the synthesized digital human to the live broadcast scene through cloud rendering stream (WebRTC low delay push stream), and the audience end can interactively change the behavior of the digital human (such as voting to trigger a specific action).

[0065] The above embodiments show and describe the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A digital human synthesis system based on interactive multimedia editing and cloud rendering technology, characterized by: The system comprises a multimodal interactive editing front end, wherein the multimodal interactive editing front end is coupled to a distributed action synthesis engine, the distributed action synthesis engine is coupled to an elastic cloud rendering cluster, and the elastic cloud rendering cluster is coupled to an intelligent synchronization controller.

2. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 1 is characterized by: The multimodal interactive editing front end integrates multi-source sensor data from LiDAR, IMU, and RGB-D cameras, transmits only the parameters that have changed, and provides low-precision real-time rendering.

3. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 2 is characterized by: The integrated LiDAR, IMU, and RGB-D camera multi-source sensor data is fused using the extended Kalman filter (EKF) formula: Among them, H k is the observation model Jacobian matrix.

4. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 2 is characterized by: The priority queue scheduling is used to schedule only the parameters that have changed, and the formula is: Among them, Q is the rendering quality evaluation function.

5. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 1 is characterized by: The distributed work synthesis engine converts raw sensor data into realistic limb movements, performs joint constraints, collision avoidance, and mass distribution simulation, and completes calculations at a 120Hz update frequency.

6. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 1 is characterized by: The raw sensor data is converted into realistic limb motion using an improved Kalman filter to eliminate noise. The formula is: Among them, W k is the adaptive weight matrix; The distribution simulation adopts the improved FABRIK algorithm, and the iteration formula is: i n+1 =θ n +J(θ n )(x d -f(θ n )); The pseudo-inverse matrix optimization calculation formula is: J=J T (JJ T +λI) -1 (λ=0.1)。 7. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 1 is characterized by: The elastic cloud rendering cluster uses distributed execution of ray tracing calculations, allocates computing power based on role importance, and eliminates the inherent noise of path tracing.

8. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 1 is characterized by: The distributed execution of ray tracing calculations dynamically adjusts the density calculation formula: Where δ is the volume scattering coefficient; The SVGF filtering algorithm is used to eliminate the inherent noise of path tracing. The formula is: in, is the adaptive mixing coefficient.

9. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 1 is characterized in that: The multimodal interactive editing front end includes a data acquisition module, the data acquisition module is coupled to a signal processing module, the signal processing module is coupled to a multimodal fusion module, and the multimodal fusion module is coupled to an interactive logic module.

10. The digital human synthesis system based on interactive multimedia editing and cloud rendering technology according to claim 1 is characterized in that: The interaction logic module assigns tasks based on user intent, including query, control, and recommendation, combines business rules and machine learning models to generate reasonable responses, maintain session status, and ensure the continuity of interaction.

Images