Real-time digital human driving method and system based on multi-modal somatosensory interaction algorithm

Abstract

The invention discloses a real-time digital human driving method and system based on a multi-modal somatosensory interaction algorithm, and relates to the technical field of computer vision and virtual digital humans, and the method comprises the following steps: S1, collecting and processing multi-modal data; s2, end-to-end 3D key point detection is carried out; s3, performing depth optimization and scale calibration; s4, 6D rotation represents a rotation quaternion; s5, performing multi-modal fusion optimization; s6, time sequence filtering and attitude stability processing; and S7, performing skeleton mapping. According to the method, RGB, point cloud and audio three-mode deep fusion is realized, PTP or NTP millisecond-level time synchronization and confidence weighted fusion are combined, the complex scene detection success rate and depth precision are improved, the problem of poor single-mode robustness is solved, a wearing-free pure vision scheme is adopted, a user can deploy a fixed monocular RGB camera without wearing any equipment, and the user experience is improved. And in cooperation with personalized scale calibration, zero wearing burden and no calibration requirement are realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and virtual digital human technology, specifically to a real-time digital human driving method and system based on a multimodal somatosensory interaction algorithm. Background Technology

[0002] With the rapid development of technologies such as virtual reality, augmented reality, and metaverse, real-time digital human-driven technology has a wide range of applications in fields such as remote collaboration, virtual anchors, motion-sensing interaction, and film and television production. The core of digital human-driven technology lies in capturing the movements and expressions of real people and mapping them onto virtual digital human models in real time to achieve natural and smooth virtual avatar performance.

[0003] Existing marker-based optical motion capture systems can achieve millimeter-level accuracy through multiple infrared cameras and reflective markers, but the hardware cost is as high as hundreds of thousands to millions of yuan. They require deployment in a controlled environment (occupying a large space and complex installation). Marker occlusion will lead to tracking failure, and professional marking equipment must be worn. They are completely unsuitable for consumer-grade and mobile scenarios. IMU-based motion capture systems require 15-20 IMU sensors to be worn on key parts of the human body, which is a heavy burden on users and does not meet the requirements of wearable interaction; they have cumulative error and drift problems, requiring frequent calibration every 30 minutes or so, and cannot capture facial micro-expressions, costing thousands to tens of thousands of yuan, making them unsuitable for fixed camera scenarios. Visual motion capture systems based on depth cameras rely on depth devices such as Kinect and RealSense to acquire RGB-D data. The detection range is only 3-5 meters. The depth sensor fails in strong outdoor light or backlight environments. The hardware cost is higher than that of ordinary RGB cameras, and the power consumption is about 15W, making it difficult to adapt to low-power edge devices. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a real-time digital human driving method and system based on a multimodal somatosensory interaction algorithm, which solves the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a real-time digital human driving method based on a multimodal somatosensory interaction algorithm, comprising the following steps: S1. Multimodal data acquisition and processing: The main mode is a fixed-position monocular RGB camera, paired with optional depth cameras or point cloud sensors and microphone arrays. All devices output data streams with millisecond-level hardware timestamps and achieve time synchronization through PTP or NTP protocols. S2, End-to-end 3D key point detection: The architecture of an end-to-end 3D keypoint deep learning model includes: Input layer: Receives the preprocessed RGB image; Backbone feature extraction network: Employs a lightweight CNN or hybrid architecture to extract multi-scale features, reducing model computation while maintaining feature extraction capability; Multi-scale feature pyramid: It integrates feature maps of different resolutions and includes layers P1-P5. By integrating features of different scales, it can effectively capture the global pose and local details of the human body. 3D regression head: includes XY coordinate regression branch, Z coordinate regression branch, 6D rotation regression branch, and confidence prediction branch, used to predict the three-dimensional position and pose information of key points; Output layer: Outputs 3D coordinates (X,Y,Z), 6D rotation representation, and confidence level of 567 key points; S3, Depth Optimization and Scale Calibration: The 3D key points are matched with the aligned point cloud to the nearest neighbor, the depth consistency error is calculated, and the depth of key points with errors exceeding the threshold is corrected. S4 and 6D rotations represent quaternions: First, the Gram-Schmidt orthogonalization algorithm is applied to the 6D rotation representation output by the model to orthogonalize it. This algorithm eliminates the linear dependence between vectors, thereby reconstructing the complete 3D rotation matrix and ensuring that the matrix satisfies the orthogonality condition. The 6D rotation representation is the first two columns of the rotation matrix, which contain 6 values. The orthogonally reconstructed 3D rotation matrix is ​​converted into quaternion form according to the mathematical transformation formula. S5, Multimodal fusion optimization: Based on the RGB frame timestamp, the depth point cloud and audio are time-aligned to compensate for the inherent latency of each modality; Calculate the confidence scores for each of the three modes: RGB confidence score: c_rgb = w1 × network confidence score + w2 × lighting quality + w3 × occlusion detection + w4 × motion blur; Point cloud confidence: c_depth = w1 × point cloud density + w2 × depth consistency + w3 × noise level; Audio confidence score: c_audio = w1 × VAD speech detection + w2 × SNR signal-to-noise ratio + w3 × sound source direction accuracy; The 3D keypoint locations are obtained using a weighted average: p_fused=(c_rgb×p_rgb+c_depth×p_depth)÷(c_rgb+c_depth); Joint rotation is achieved using a quaternion-weighted average. Facial expressions are combined with audio MFCC features to adjust the BlendShape weights. Specifically: First, Mel-frequency cepstral coefficients (MFCC) features of the audio are extracted to analyze the prosody, emotion, and other information of the speech. Then, a mapping relationship between MFCC features and BlendShape weights is established. The model is trained using machine learning algorithms (such as neural networks) to learn the optimal BlendShape weight configuration corresponding to different combinations of MFCC features. During real-time driving, the MFCC features of the input audio are input into the trained model, and the model outputs the corresponding BlendShape weight parameters to dynamically adjust the 52 ARKit standard expressions of the digital human, achieving a natural match between facial expressions and audio. S6. Timing filtering and attitude stabilization processing: The state vector contains 3D position, velocity, and acceleration. The filtering intensity is dynamically adjusted according to the motion velocity to reduce position noise. Spherical linear interpolation is performed on quaternions to avoid Euler angle gimbal lock; instantaneous attitude shocks are detected by the RANSAC algorithm, and historical trajectory prediction and weighted averaging are used to mitigate jumps. This improves attitude timing stability and solves the problem of significant inter-frame jitter in existing technologies; S7, Skeleton Mapping: The camera coordinate system of 3D key points is converted to the UE5 world coordinate system, and the transformation matrix is ​​adapted to the UE5 Mannequin skeleton specification. Based on the hot-loadable JSON / YAML mapping table, 567 3D key points are mapped to the target digital human skeleton, i.e., facial mapping: 478 facial key points are accurately mapped to facial blendshape or expression skeleton through the mapping table to drive the digital human's rich facial expressions.

[0006] Furthermore, in step S1, the RGB image undergoes image denoising, white balance correction, and exposure correction. The human body region is detected using YOLO and SSD algorithms, and the region of interest (ROI) is extracted and then normalized. The point cloud is subjected to statistical filtering to remove noise points, radius filtering to remove isolated points, and downsampling to reduce the amount of data. The point cloud coordinates are aligned to the RGB camera coordinate system to ensure spatial consistency. The audio is pre-emphasized, denoised, and processed by short-time Fourier transform (STFT) to finally extract MFCC features.

[0007] Furthermore, the specific process of the YOLO and SSD algorithms for detecting human body regions and extracting regions of interest is as follows: Load a pre-trained YOLO or SSD model into the runtime environment. This model has been trained on a large amount of human image data and can identify the features and location of the human body in the image. The processed image data is input into the loaded algorithm model, which analyzes the entire image. YOLO and SSD algorithms extract features from the image through convolutional neural networks, quickly scan the image at different scales, and detect the presence of human targets. Once a human is detected, a detection result containing human location information (boundary box coordinates) and category information (human category) is generated. Based on the bounding box coordinates of the human target in the detection results, the region containing the human body is extracted from the original image, i.e., the ROI.

[0008] Furthermore, in step S2, model training is implemented as follows: training is conducted based on 100,000+ proprietary labeled images, using a labeling method that combines multi-view camera triangulation, depth sensor assistance, and SMPL-X model fitting to ensure 3D position labeling error <5mm and consistency >98%. Employing a multi-task joint loss function: At the same time, several key metrics were optimized to improve the overall performance of the model, including... Used to measure the difference between the 3D model or pose predicted by the model and the real 3D data, to ensure that the form and movement of the digital human in three-dimensional space are consistent with reality; Focus on depth information in images or scenes to ensure accurate representation of digital humans at different distances and levels, enhancing the sense of three-dimensionality and realism; Reprojection loss: After projecting the 3D model back onto the 2D image plane, the difference between the predicted image and the original 2D image is calculated, so that the 3D reconstruction result of the digital human can accurately match the input data in the 2D view. : To ensure the rationality and accuracy of skeletal movement in relation to the skeletal structure of digital humans, making their movements natural and smooth, and in accordance with the laws of human movement; Considering the temporal continuity of digital human movements, avoid unnatural phenomena such as pauses and jumps in movements, and ensure smooth transitions in the time dimension of movements; : Constrain the rotation angle and direction of each part of the digital human to make the rotation of limbs and joints conform to physical laws and actual movement requirements; Accelerated through TensorRT+DLA collaboration, with DLA handling feature extraction and the GPU performing 3D regression, achieving pipeline parallelism and reaching an inference speed of 30fps with FP32 accuracy on Jetson Nano; directly outputting 567 3D keypoints, including 478 facial keypoints, 49 body keypoints, and 40 hand keypoints, with the data format conforming to the UE5 Mannequin skeleton topology, and each keypoint containing 3D coordinates (X,Y,Z), 6D rotation representation, and confidence level (0-1).

[0009] Furthermore, in step S3, the specific process for calculating the depth consistency error is as follows: For each matched 3D keypoint, obtain its actual depth value in the point cloud. and the depth value predicted by the model. The depth consistency error is calculated using the following formula. : The depth correction process is as follows: Set a depth error threshold. ,when Exceed At times, the depth predicted by the model deviates significantly from the actual depth of the point cloud, requiring correction of the depth of key points; The correction method is to use point cloud depth values. Direct substitution model predicts depth values Alternatively, a weighted average method can be used, i.e.: in These are weighting coefficients, which are adjusted according to the actual situation to balance the impact of point cloud data and model predictions. Based on anthropometric standard bone length ratios, such as height:armspan = 1:1 and upper arm:forearm = 1:1.1, bone length constraints are established. Combined with user-specific T-pose / A-pose calibration data, a scale system (scale = Σw) is calculated. i × ), correct the scale of 3D key points to ensure that it conforms to the proportions of the real human body or the scale of the target digital human skeleton.

[0010] Furthermore, in step S4, the 3D rotation matrix after orthogonal reconstruction... Convert it to a quaternion using the following mathematical conversion formula. form: when In this case, an alternative formula must be used: like and ,but: like and ,but: like and ,but: After conversion, combined with biomechanical joint angle limitations, unreasonable rotation values ​​are eliminated to ensure that joint movement conforms to the laws of human anatomy.

[0011] Furthermore, in step S5, the Bayesian fusion and extended Kalman filtering further optimize the fusion result, specifically: First, the raw sensor data is denoised and normalized to provide high-quality input for subsequent fusion; then, a probability model for each sensor data is defined, and an initial probability distribution is set based on prior knowledge. Using Bayes' theorem: By incorporating new observational data and updating the posterior probability distribution, the initial fusion of multi-source data is achieved, resulting in a fused state estimate. in, Indicates in the event Under the conditions that the event occurs The probability of something occurring is called the posterior probability, which reflects the probability of something occurring in the known future. In the event of A reassessment of the likelihood of occurrence; Indicates in the event Under the conditions that the event occurs The probability of something occurring is called the likelihood, which reflects... When it happens The probability of it occurring; It is an event The prior probability of an event occurring, that is, the probability of an event occurring without any other information. An initial estimate of the probability of occurrence; For the event The probability of occurrence is a normalization constant used to ensure It is a reasonable probability value, and its role is to adjust the joint probability. Make adjustments to make Between 0 and 1; Extended Kalman Filter (EKF): Based on the system's state transition equations, it predicts the state at the next time step, obtaining a priori estimates. in, It is a nonlinear state transition function; It is the optimal estimate from the previous moment; For input; simultaneously calculate the prior estimated covariance. , It is the Jacobian matrix of the state transition function with respect to the state; For process noise covariance; The predicted state is transformed into the observation space through the observation equation to obtain the predicted observation values. , Given a nonlinear observation function; calculate the Kalman gain. , It is the Jacobian matrix of the observation function with respect to the state; To measure the noise covariance; finally, based on the actual observed values Update state estimation and update the estimated covariance ; The state estimate updated by EKF is used as the prior information for the next Bayesian fusion. The above steps are repeated, and the accuracy and stability of the fusion results are gradually improved through continuous iteration. In step S6, the procedure for detecting instantaneous attitude impact is as follows: Using sensors such as inertial measurement units, real-time attitude data is collected at a fixed frequency, including but not limited to angular velocity and acceleration information; the collected raw data is filtered to remove noise interference. Set an angular velocity threshold, which needs to be determined based on the specific application scenario and equipment characteristics, as the standard for judging instantaneous attitude impact; determine the number of iterations for the RANSAC algorithm. Related to the amount of sample data and the noise ratio, it can be obtained through empirical formulas. Calculate, where, The expected confidence level; This is an estimate of the noise ratio; The minimum number of samples required to fit the model; set an interior point determination threshold to determine whether the collected angular velocity data points belong to interior points, which is determined based on the sensor measurement error; Randomly select from the preprocessed data For each data point, construct an attitude angular velocity model. Calculate the error between the remaining data points and this model. Data points with errors less than the inlier threshold are classified as inliers; otherwise, they are classified as outliers. Repeat this process, recording the model with the most inliers in each iteration as the current optimal model. Continue this process until a set number of iterations is reached. Stop iterating when the time comes; The angular velocity values ​​of the interior points in the final optimal model are statistically analyzed. If the angular velocity of any interior point exceeds the preset angular velocity threshold, it is determined that an instantaneous attitude impact has occurred.

[0012] Furthermore, the real-time digital human driving method based on the multimodal somatosensory interaction algorithm also includes the following steps: S8 and LiveLink source data processing: Following the UE5.6 LiveLink protocol, a FLiveLinkFrameData structure containing timestamps, subject names, skeleton transformation arrays, BlendShape weight arrays, and custom attributes is constructed; binary serialization, fixed-point encoding, quaternion compression, and differential encoding are used to reduce the amount of data. Using binary serialization: By converting data into binary format, Protocol Buffers takes up less space when storing and transmitting compared to JSON's text format. It uses a compact binary representation, avoiding the large number of delimiters and redundant key-value pairs in JSON, which can significantly reduce the data size. Fixed-point coding: Position information is represented by 16-bit fixed-point numbers. While ensuring an accuracy of 0.1mm, it reduces the space occupied by data. Compared with floating-point numbers, fixed-point numbers do not need to store the exponent part when representing values ​​within a fixed range, making more efficient use of binary bits to store data. Quaternion compression: Quaternions are used to represent rotations in three-dimensional space. By utilizing the mathematical properties of quaternions, only the three smallest components are stored. During recovery, the fourth component is derived based on mathematical relationships, achieving a 25% reduction in data volume without affecting the accurate representation of rotation information. Differential coding: only records the difference between the current frame and the previous frame, rather than the complete data content; since the changes between adjacent frames during the movement of a digital human are usually small, this method can significantly reduce the amount of data, especially when transmitting and storing continuous motion data, it can greatly reduce data redundancy; S9, Multi-protocol network transmission: Based on network type, for local area networks (LANs): UDP protocol is preferred due to its extremely low transmission latency. While maintaining high-speed transmission, it effectively solves the unreliability problem of the UDP protocol itself, ensuring stable data transmission. For example, in closed campus or office network environments, communication between devices can be completed quickly and accurately. Internet: Using TCP or WebSocket protocols, transmission latency can be controlled to <20ms; combined with LZ4 compression algorithm, data transmission volume can be significantly reduced and transmission efficiency improved; for example, when digital humans interact with users in real time in an internet environment, smooth visuals and data accuracy can be guaranteed. Local communication: It adopts a shared memory approach to achieve near-zero latency transmission. By using zero-copy technology, it avoids repeated copying of data in memory, greatly improving data read and write speeds. It is suitable for high-frequency data interaction between different processes within the same device. It also sets an adaptive transmission strategy and dynamically adjusts parameters: real-time monitoring of key indicators such as network RTT, packet loss rate and available bandwidth, sending frame rate based on 30fps, and automatically reducing the frame rate when insufficient bandwidth is detected to ensure transmission continuity; at the same time, it dynamically adjusts the data compression ratio according to network conditions, reducing the compression ratio to ensure data quality when bandwidth is sufficient, and increasing the compression ratio to reduce the amount of data transmitted when bandwidth is tight. Delay compensation mechanism: Introducing a prediction algorithm based on kinematic models, predicting the action state of the next moment based on the historical data and movement patterns of the digital human's actions, and using the predicted data to render the screen in advance when network delay occurs, compensating for the timing deviation caused by network delay, ensuring that the digital human's actions displayed on the receiving end are coherent and natural, and maintaining data timing consistency. S10, UE5.6 rendering driver: The UE5.6 LiveLinkSource plugin receives data packets and performs decompression, deserialization, CRC check and frame sequence number check to verify the validity of the data. The parsed skeletal transformation and BlendShape weights are mapped to MetaHuman or a custom digital human skeleton. If the frame rate does not match, temporal interpolation is performed. The skeletal transformation and facial weights are applied, and the final rendering result is output at 30, 60 or 120 fps.

[0013] A system that applies the above-mentioned real-time digital human driving method based on a multimodal somatosensory interaction algorithm.

[0014] This invention provides a real-time digital human driving method and system based on a multimodal somatosensory interaction algorithm, which has the following beneficial effects: 1. This real-time digital human driving method and system based on multimodal somatosensory interaction algorithm achieves deep fusion of RGB, point cloud and audio three modalities for the first time. Combined with millisecond-level time synchronization of PTP or NTP and confidence-weighted fusion, it is beneficial to improve the detection success rate and depth accuracy in complex scenes, solve the problem of poor robustness of single modality, and adopts a wearless pure vision solution. Users do not need to wear any devices. A fixed monocular RGB camera can be deployed. With personalized scale calibration, there is zero burden of wearables and no calibration requirements.

[0015] 2. The real-time digital human driving method and system based on multimodal somatosensory interaction algorithm adopts an end-to-end 3D key point detection architecture, directly outputting 567 3D key points, namely 478 points for the face + 49 points for the body + 40 points for the hands, avoiding the accumulation of 2D to 3D conversion errors. The number of key points is 17 times that of MediaPipe, meeting the requirements of high-fidelity driving.

[0016] 3. This real-time digital human driving method and system based on multimodal motion-sensing interaction algorithm constructs a complete technology chain from data acquisition, multimodal fusion, temporal filtering to LiveLink transmission and UE5 rendering. The data compression rate is significantly reduced compared to JSON, and it supports UDP, TCP, and shared memory multi-protocol transmission. It can be directly used in production environments, comprehensively covering scenarios such as motion-sensing interaction, virtual digital humans, and film and television production, and significantly reducing the hardware threshold and application cost of digital human driving technology. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall architecture of a real-time digital human driving method and system based on a multimodal somatosensory interaction algorithm according to the present invention. Figure 2 This is a schematic diagram showing the distribution of 567 3D key points in a real-time digital human driving method and system based on a multimodal somatosensory interaction algorithm according to the present invention. Figure 3 This is a schematic diagram of the multimodal data time alignment process of a real-time digital human driving method and system based on a multimodal somatosensory interaction algorithm according to the present invention. Figure 4 This is a schematic diagram of the end-to-end 3D keypoint detection deep learning model architecture of a real-time digital human driving method and system based on a multimodal somatosensory interaction algorithm according to the present invention. Detailed Implementation

[0018] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0019] like Figures 1-4 As shown, the present invention provides a technical solution: a real-time digital human driving method based on a multimodal somatosensory interaction algorithm, comprising the following steps: S1. Multimodal data acquisition and processing: The main mode is a fixed-position monocular RGB camera, paired with optional depth cameras or point cloud sensors and microphone arrays. All devices output data streams with millisecond-level hardware timestamps and achieve time synchronization through PTP or NTP protocols. By acquiring raw data in three modalities—RGB image, depth point cloud, and audio—and ensuring data timestamp consistency, it breaks through the limitations of existing single-modal data acquisition. The RGB image is subjected to image denoising, white balance correction, and exposure correction. The human body region is detected by YOLO and SSD algorithms and the region of interest (ROI) is extracted. Then the ROI is normalized. Statistical filtering is applied to the point cloud to remove noise points, radius filtering is applied to remove isolated points, and downsampling is applied to reduce the amount of data. The point cloud coordinates are aligned to the RGB camera coordinate system to ensure spatial consistency. The audio is pre-emphasized, denoised, and processed by Short Time Fourier Transform (STFT) to finally extract MFCC features. The specific process of YOLO and SSD algorithms detecting human body regions and extracting regions of interest is as follows: Load a pre-trained YOLO or SSD model into the runtime environment. This model has been trained on a large amount of human image data and can identify the features and location of the human body in the image. The processed image data is input into the loaded algorithm model, which analyzes the entire image. YOLO and SSD algorithms extract features from the image through convolutional neural networks, quickly scan the image at different scales, and detect the presence of human targets. Once a human is detected, a detection result containing human location information (boundary box coordinates) and category information (human category) is generated. Based on the bounding box coordinates of the human target in the detection results, the region containing the human body, i.e., the ROI, is extracted from the original image; This optimizes the quality of the raw data, unifies the data format and coordinate system, and provides clean input for subsequent algorithm processing. S2, End-to-end 3D key point detection: The architecture of an end-to-end 3D keypoint deep learning model includes: Input layer: Receives preprocessed RGB images (256×256 or 512×512); Backbone feature extraction network: Employs a lightweight CNN or hybrid architecture (CNN+Transformer) to extract multi-scale features, reducing model computation while maintaining feature extraction capability; Multi-scale feature pyramid: It integrates feature maps of different resolutions and includes layers P1-P5. By integrating features of different scales, it can effectively capture the global pose and local details of the human body. 3D regression head: includes XY coordinate regression branch, Z coordinate (depth) regression branch, 6D rotation regression branch, and confidence prediction branch, used to predict the three-dimensional position and pose information of key points; Output layer: Outputs 3D coordinates (X,Y,Z), 6D rotation representation, and confidence level of 567 key points; Model training implementation: Based on training with over 100,000 proprietary labeled images, a labeling method combining multi-view camera triangulation, depth sensor assistance, and SMPL-X model fitting was adopted to ensure 3D position labeling error <5mm and consistency >98%. Employing a multi-task joint loss function: At the same time, several key metrics were optimized to improve the overall performance of the model, including... (3D spatial loss) is used to measure the difference between the 3D model or pose predicted by the model and the real 3D data, to ensure that the form and movement of the digital human in three-dimensional space are consistent with reality; (Depth loss): Focus on depth information in images or scenes to ensure accurate presentation of digital humans at different distances and levels, and enhance the sense of three-dimensionality and realism; Reprojection loss: After projecting the 3D model back onto the 2D image plane, the difference between the predicted image and the original 2D image is calculated, so that the 3D reconstruction result of the digital human can accurately match the input data in the 2D view. (Skeletal loss): For the skeletal structure of digital humans, ensure the rationality and accuracy of skeletal movement, so that the movements of digital humans are natural and smooth, and conform to the laws of human movement; (Time series loss): Consider the temporal continuity of digital human actions to avoid unnatural phenomena such as stuttering and jumping, and ensure smooth transition of actions in the time dimension. (Rotational Loss): Constraining the rotation angles and directions of each part of the digital human to ensure that the rotation of limbs and joints conforms to physical laws and actual movement requirements; Accelerated through TensorRT+DLA collaboration, with DLA handling feature extraction and the GPU performing 3D regression, a pipelined parallelism is achieved, reaching an inference speed of 30fps with FP32 precision on a Jetson Nano (10W power consumption). It directly outputs 567 3D keypoints (478 for the face, 49 for the body, and 40 for the hands), with the data format conforming to the UE5 Mannequin skeleton topology. Each keypoint includes three-dimensional coordinates (X, Y, Z), a 6D rotation representation, and a confidence level (0-1). This avoids the accumulation of errors in the existing 2D to 3D conversion, and directly outputs high-density, high-precision 3D key points, solving the problem of a small number of key points (such as MediaPipe with only 33 points); S3, Depth Optimization and Scale Calibration: The 3D keypoints are matched with the aligned point cloud using nearest neighbor matching. The depth consistency error is calculated, and keypoints with errors exceeding the threshold are depth-corrected. The specific process for calculating the depth consistency error is as follows: For each matched 3D keypoint, obtain its actual depth value in the point cloud. and the depth value predicted by the model. The depth consistency error is calculated using the following formula. : The depth correction process is as follows: Set a depth error threshold. ,when Exceed At times, the depth predicted by the model deviates significantly from the actual depth of the point cloud, requiring correction of the depth of key points; The correction method is to use point cloud depth values. Direct substitution model predicts depth values Alternatively, a weighted average method can be used, i.e.: in These are weighting coefficients, which are adjusted according to the actual situation to balance the impact of point cloud data and model predictions. Based on anthropometric standard bone length ratios (e.g., height:armspan = 1:1, upper arm:forearm = 1:1.1), skeletal length constraints are established. Combined with user-specific T-pose / A-pose calibration data, a scale system (scale = Σw) is calculated. i × ), correct the scale of 3D key points to ensure that they conform to the proportions of the real human body or the scale of the target digital human skeleton; This improves the depth accuracy of 3D key points, thus solving the problem of inaccurate scale in existing technologies; S4 and 6D rotations represent quaternions: First, the Gram-Schmidt orthogonalization algorithm is applied to the 6D rotation representation output by the model to orthogonalize it. This algorithm eliminates the linear dependence between vectors, thereby reconstructing the complete 3D rotation matrix and ensuring that the matrix satisfies the orthogonality condition. The 6D rotation representation is the first two columns of the rotation matrix, which contain 6 values. The orthogonally reconstructed 3D rotation matrix is ​​then converted into quaternion form using the following mathematical transformation formula: 3D rotation matrix after orthogonal reconstruction Convert it to a quaternion using the following mathematical conversion formula. form: when In this case, an alternative formula must be used: like and ,but: like and ,but: like and ,but: After conversion, combined with biomechanical joint angle limitations, such as 0-150° for the elbow joint and 0-135° for the knee joint, unreasonable rotation values ​​are eliminated to ensure that joint movement conforms to the laws of human anatomy. Therefore, 6D representation is used to reduce rotation errors due to its advantages of no manifold constraints and stable training. At the same time, quaternions are used to achieve compact storage and efficient interpolation to adapt to subsequent processing and transmission. S5, Multimodal fusion optimization: Based on the RGB frame timestamp, the depth point cloud and audio are time-aligned to compensate for the inherent latency of each modality; Calculate the confidence scores for each of the three modes: RGB confidence score: c_rgb = w1 × network confidence score + w2 × lighting quality + w3 × occlusion detection + w4 × motion blur; Point cloud confidence: c_depth = w1 × point cloud density + w2 × depth consistency + w3 × noise level; Audio confidence score: c_audio = w1 × VAD speech detection + w2 × SNR signal-to-noise ratio + w3 × sound source direction accuracy; The 3D keypoint locations are obtained using a weighted average: p_fused=(c_rgb×p_rgb+c_depth×p_depth)÷(c_rgb+c_depth); Joint rotation is achieved using a quaternion-weighted average. Facial expressions are combined with audio MFCC features to adjust the BlendShape weights. Specifically: First, Mel-frequency cepstral coefficients (MFCC) features of the audio are extracted to analyze the prosody, emotion, and other information of the speech. Then, a mapping relationship between MFCC features and BlendShape weights is established. The model is trained using machine learning algorithms (such as neural networks) to learn the optimal BlendShape weight configuration corresponding to different combinations of MFCC features. During real-time driving, the MFCC features of the input audio are input into the trained model, and the model outputs the corresponding BlendShape weight parameters to dynamically adjust the 52 ARKit standard expressions of the digital human, achieving a natural match between facial expressions and audio. The fusion results are further optimized using Bayesian fusion and Extended Kalman Filter (EKF), specifically: First, the raw sensor data is denoised and normalized to provide high-quality input for subsequent fusion; then, a probability model for each sensor data is defined, and an initial probability distribution is set based on prior knowledge. Using Bayes' theorem: By incorporating new observational data and updating the posterior probability distribution, the initial fusion of multi-source data is achieved, resulting in a fused state estimate. in, Indicates in the event Under the conditions that the event occurs The probability of something occurring is called the posterior probability, which reflects the probability of something occurring in the known future. In the event of A reassessment of the likelihood of occurrence; Indicates in the event Under the conditions that the event occurs The probability of something occurring is called the likelihood, which reflects... When it happens The probability of it occurring; It is an event The prior probability of an event occurring, that is, the probability of an event occurring without any other information. An initial estimate of the probability of occurrence; For the event The probability of occurrence is a normalization constant used to ensure It is a reasonable probability value, and its role is to adjust the joint probability. Make adjustments to make Between 0 and 1; Extended Kalman Filter (EKF): Based on the system's state transition equations, it predicts the state at the next time step, obtaining a priori estimates. in, It is a nonlinear state transition function; It is the optimal estimate from the previous moment; For input; simultaneously calculate the prior estimated covariance. , It is the Jacobian matrix of the state transition function with respect to the state; For process noise covariance; The predicted state is transformed into the observation space through the observation equation to obtain the predicted observation values. , Given a nonlinear observation function; calculate the Kalman gain. , It is the Jacobian matrix of the observation function with respect to the state; To measure the noise covariance; finally, based on the actual observed values Update state estimation and update the estimated covariance ; The state estimate updated by EKF is used as the prior information for the next Bayesian fusion. The above steps are repeated, and the accuracy and stability of the fusion results are gradually improved through continuous iteration. This improves the success rate of detection in complex scenes and solves the problem of poor robustness of existing single-modal detection. S6. Timing filtering and attitude stabilization processing: The state vector contains 3D position, velocity, and acceleration. The filtering intensity is dynamically adjusted according to the motion speed (intensity is reduced to maintain response during fast motion and intensity is increased to suppress jitter during slow motion) to reduce position noise. Spherical linear interpolation is performed on quaternions to avoid Euler angle gimbal lock; instantaneous attitude shocks (angular velocity exceeding the threshold) are detected using the RANSAC algorithm, and historical trajectory prediction and weighted averaging are used to mitigate attitude jumps. The process for detecting instantaneous attitude shocks is as follows: Using sensors such as inertial measurement units (IMUs), real-time attitude data, including angular velocity and acceleration, is collected at a fixed frequency. The collected raw data is then filtered to remove noise interference, for example, by using Kalman filtering or complementary filtering, to improve the accuracy and stability of the data. Set an angular velocity threshold, which needs to be determined based on the specific application scenario and equipment characteristics, as the standard for judging instantaneous attitude impact; determine the number of iterations for the RANSAC algorithm. Related to the amount of sample data and the noise ratio, it can be obtained through empirical formulas. Calculate, where, The expected confidence level; This is an estimate of the noise ratio; The minimum number of samples required to fit the model; set an interior point determination threshold to determine whether the collected angular velocity data points belong to interior points, which is determined based on the sensor measurement error; Randomly select from the preprocessed data For each data point, construct an attitude angular velocity model. Calculate the error between the remaining data points and this model. Data points with errors less than the inlier threshold are classified as inliers; otherwise, they are classified as outliers. Repeat this process, recording the model with the most inliers in each iteration as the current optimal model. Continue this process until a set number of iterations is reached. Stop iterating when the time comes; The angular velocity values ​​of the interior points in the final optimal model are statistically analyzed. If the angular velocity of an interior point exceeds the preset angular velocity threshold, it is determined that an instantaneous attitude impact has occurred. Based on the 3σ criterion and motion trajectory model (RANSAC fitting), abnormal key points are identified and replaced to ensure that the inter-frame position jitter is <3mm and the rotation jitter is <1°. This improves attitude timing stability and solves the problem of significant inter-frame jitter in existing technologies; S7, Skeleton Mapping: The camera coordinate system of 3D key points is converted to the UE5 world coordinate system, and the transformation matrix is ​​adapted to the UE5 Mannequin skeleton specification. Based on the hot-loadable JSON / YAML mapping table, 567 3D key points are mapped to the target digital human skeleton, i.e., facial mapping: 478 facial key points are accurately mapped to facial blendshape or expression bones through the mapping table to drive the rich facial expressions of the digital human. Body mapping: Map 49 key body points to the UE5 body skeleton according to predetermined rules to ensure natural and smooth body movements of the digital human; Hand mapping: 40 key hand points are mapped to the bones of both hands to simulate fine hand movements; For distal joints such as the wrist and ankle, the FABRIK or CCD inverse kinematics algorithm is used for solving the problem. Input the desired position of the end joint; The joint angles are iteratively calculated using the FABRIK or CCD algorithm to adjust the joint chain posture. Continuous optimization is performed until the position of the end joint is precisely matched with the target position, ensuring the realism and accuracy of the digital human's limb movements; This establishes a connection between the source key points and the target digital human skeleton, solving the problem that existing technologies lack skeleton mapping and cannot directly drive digital humans; S8 and LiveLink source data processing: Following the UE5.6 LiveLink protocol, a FLiveLinkFrameData structure is constructed, containing timestamps (WorldTime + Timecode), subject name, skeleton transformation array (position + rotation + scaling), blendShape weight array, and custom attributes (global confidence, detection quality). Binary serialization (ProtocolBuffers), fixed-point encoding (position using 16-bit fixed-point numbers with 0.1mm precision), quaternion compression, and differential encoding (changes relative to the previous frame) are used to reduce the amount of data. Using binary serialization (ProtocolBuffers): By converting data into binary format, ProtocolBuffers takes up less space when storing and transmitting compared to JSON's text format. It uses a compact binary representation, avoiding a large number of delimiters and redundant key-value pairs in JSON, which can significantly reduce the data size. Fixed-point coding (position using 16-bit fixed-point numbers with a precision of 0.1mm): Position information is represented by 16-bit fixed-point numbers. While ensuring a precision of 0.1mm, the data space occupied is reduced. Compared with floating-point numbers, fixed-point numbers do not require additional storage of the exponent when representing values ​​within a fixed range, making more efficient use of binary bits to store data. Quaternion compression: Quaternions are used to represent rotations in three-dimensional space. By utilizing the mathematical properties of quaternions, only the three smallest components are stored. During recovery, the fourth component can be derived based on mathematical relationships, achieving a 25% reduction in data volume without affecting the accurate expression of rotation information. Differential coding (change relative to the previous frame): only records the difference between the current frame and the previous frame, rather than the complete data content; since the changes between adjacent frames during the movement of a digital human are usually small, this method can significantly reduce the amount of data, especially when transmitting and storing continuous motion data, it can greatly reduce data redundancy; This generates transmission data that conforms to the real-time rendering engine standard, reducing the amount of data to adapt to network transmission. S9, Multi-protocol network transmission: Based on network type, for local area networks (LANs): UDP protocol is preferred due to its extremely low transmission latency (<5ms). While maintaining high-speed transmission, it effectively solves the unreliability problem of the UDP protocol itself, ensuring stable data transmission. For example, in closed campus or office network environments, communication between devices can be completed quickly and accurately. Internet: Using TCP or WebSocket protocols, transmission latency can be controlled to <20ms; combined with LZ4 compression algorithm, data transmission volume can be significantly reduced and transmission efficiency improved; for example, when digital humans interact with users in real time in an internet environment, smooth visuals and data accuracy can be guaranteed. Local communication: It adopts a shared memory approach to achieve near-zero latency (<1ms) transmission. Zero-copy technology avoids repeated copying of data in memory, greatly improving data read and write speeds. It is suitable for high-frequency data interaction between different processes within the same device. It also sets an adaptive transmission strategy and dynamically adjusts parameters: real-time monitoring of key indicators such as network RTT (round-trip time), packet loss rate, and available bandwidth; sending frame rate based on 30fps; automatically reducing the frame rate when insufficient bandwidth is detected to ensure transmission continuity; and dynamically adjusting the data compression ratio according to network conditions, reducing the compression ratio to ensure data quality when bandwidth is sufficient, and increasing the compression ratio to reduce the amount of data transmitted when bandwidth is tight. Delay compensation mechanism: Introduce a prediction algorithm based on kinematic model (an existing algorithm, which will not be described in detail here). Based on the historical data and motion patterns of the digital human's movements, predict the movement state at the next moment. When network delay occurs, use the predicted data to render the screen in advance to compensate for the timing deviation caused by network delay, ensuring that the digital human's movements displayed on the receiving end are coherent and natural, and maintaining the consistency of data timing. This enables low-latency, highly reliable data transmission, solving the problems of existing technologies lacking compatible transmission protocols and having high latency. S10, UE5.6 rendering driver: The UE5.6 LiveLinkSource plugin receives data packets and performs decompression, deserialization, CRC check and frame sequence number check to verify the validity of the data. The parsed skeletal transformation and BlendShape weights are mapped to MetaHuman or a custom digital human skeleton. If the frame rate does not match, temporal interpolation is performed. The skeletal transformation and facial weights are applied, and the final rendering result is output at 30, 60 or 120 fps. This completes the closed loop from algorithm output to real-time rendering of the digital human, achieving an end-to-end latency of <60ms, meeting the requirements for real-time driving.

[0020] In summary, this real-time digital human driving method and system based on multimodal somatosensory interaction algorithm should be used firstly.

[0021] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A real-time digital human driving method based on a multimodal somatosensory interaction algorithm, characterized in that: Includes the following steps: S1. Multimodal data acquisition and processing: The main mode is a fixed-position monocular RGB camera, paired with optional depth cameras or point cloud sensors and microphone arrays. All devices output data streams with millisecond-level hardware timestamps and achieve time synchronization through PTP or NTP protocols. S2, End-to-end 3D key point detection: The architecture of an end-to-end 3D keypoint deep learning model includes: Input layer: Receives the preprocessed RGB image; Backbone feature extraction network: Employs a lightweight CNN or hybrid architecture to extract multi-scale features, reducing model computation while maintaining feature extraction capability; Multi-scale feature pyramid: It integrates feature maps of different resolutions and includes layers P1-P5. By integrating features of different scales, it can effectively capture the global pose and local details of the human body. 3D regression head: includes XY coordinate regression branch, Z coordinate regression branch, 6D rotation regression branch, and confidence prediction branch, used to predict the three-dimensional position and pose information of key points; Output layer: Outputs 3D coordinates (X,Y,Z), 6D rotation representation, and confidence level of 567 key points; S3, Depth Optimization and Scale Calibration: The 3D key points are matched with the aligned point cloud to the nearest neighbor, the depth consistency error is calculated, and the depth of key points with errors exceeding the threshold is corrected. S4 and 6D rotations represent quaternions: First, the Gram-Schmidt orthogonalization algorithm is applied to the 6D rotation representation output by the model to orthogonalize it. This algorithm eliminates the linear dependence between vectors, thereby reconstructing the complete 3D rotation matrix and ensuring that the matrix satisfies the orthogonality condition. The 6D rotation representation is the first two columns of the rotation matrix, which contain 6 values. The orthogonally reconstructed 3D rotation matrix is ​​converted into quaternion form according to the mathematical transformation formula. S5, Multimodal fusion optimization: Based on the RGB frame timestamp, the depth point cloud and audio are time-aligned to compensate for the inherent latency of each modality; The 3D keypoint locations are calculated using a weighted average: p_fused=(c_rgb×p_rgb+c_depth×p_depth)÷(c_rgb+c_depth); Joint rotation is calculated using a quaternion-weighted average, and facial expression is combined with audio MFCC features to adjust the BlendShape weights. S6. Timing filtering and attitude stabilization processing: The state vector contains 3D position, velocity, and acceleration. The filtering intensity is dynamically adjusted according to the motion velocity to reduce position noise. Spherical linear interpolation is performed on quaternions to avoid Euler angle gimbal lock; instantaneous attitude shocks are detected by the RANSAC algorithm, and historical trajectory prediction and weighted averaging are used to mitigate jumps. This improves attitude timing stability and solves the problem of significant inter-frame jitter in existing technologies; S7, Skeleton Mapping: The camera coordinate system of 3D key points is converted to the UE5 world coordinate system, and the transformation matrix is ​​adapted to the UE5 Mannequin skeleton specification. Based on the hot-loadable JSON / YAML mapping table, 567 3D key points are mapped to the target digital human skeleton, i.e., facial mapping: 478 facial key points are accurately mapped to facial blendshape or expression skeleton through the mapping table to drive the digital human's rich facial expressions.

2. The real-time digital human driving method based on a multimodal somatosensory interaction algorithm according to claim 1, characterized in that: In step S1, the RGB image undergoes image denoising, white balance correction, and exposure correction. The human body region is detected by YOLO and SSD[1] algorithms, and the region of interest (ROI) is extracted. Then, the ROI is normalized. The point cloud is subjected to statistical filtering to remove noise points, radius filtering to remove isolated points, and downsampling to reduce the amount of data. The point cloud coordinates are aligned to the RGB camera coordinate system to ensure spatial consistency. The audio is pre-emphasized, denoised, and processed by short-time Fourier transform (STFT) to finally extract MFCC features.

3. The real-time digital human driving method based on a multimodal somatosensory interaction algorithm according to claim 2, characterized in that: The specific process of using the YOLO and SSD algorithms to detect human body regions and extract regions of interest is as follows: Load a pre-trained YOLO or SSD model into the runtime environment. This model has been trained on a large amount of human image data and can identify the features and location of the human body in the image. The processed image data is input into the loaded algorithm model, which analyzes the entire image. YOLO and SSD algorithms extract features from the image through convolutional neural networks, quickly scan the image at different scales, and detect the presence of human targets. Once a human is detected, a detection result containing human location information (boundary box coordinates) and category information (human category) is generated. Based on the bounding box coordinates of the human target in the detection results, the region containing the human body is extracted from the original image, i.e., the ROI.

4. The real-time digital human driving method based on a multimodal somatosensory interaction algorithm according to claim 1, characterized in that: In step S2, model training is implemented as follows: training is conducted based on 100,000+ proprietary labeled images, using a labeling method of multi-view camera triangulation + depth sensor assistance + SMPL-X model fitting to ensure 3D position labeling error <5mm and consistency >98%; Employing a multi-task joint loss function: At the same time, several key metrics were optimized to improve the overall performance of the model, including... Used to measure the difference between the 3D model or pose predicted by the model and the real 3D data, to ensure that the form and movement of the digital human in three-dimensional space are consistent with reality; Focus on depth information in images or scenes to ensure accurate representation of digital humans at different distances and levels, enhancing the sense of three-dimensionality and realism; Reprojection loss: After projecting the 3D model back onto the 2D image plane, the difference between the predicted image and the original 2D image is calculated, so that the 3D reconstruction result of the digital human can accurately match the input data in the 2D view. : To ensure the rationality and accuracy of skeletal movement in relation to the skeletal structure of digital humans, making their movements natural and smooth, and in accordance with the laws of human movement; Considering the temporal continuity of digital human movements, avoid unnatural phenomena such as pauses and jumps in movements, and ensure smooth transitions in the time dimension of movements; : Constrain the rotation angle and direction of each part of the digital human to make the rotation of limbs and joints conform to physical laws and actual movement requirements; Accelerated through TensorRT+DLA collaboration, with DLA handling feature extraction and the GPU performing 3D regression, achieving pipeline parallelism and reaching an inference speed of 30fps with FP32 accuracy on Jetson Nano; directly outputting 567 3D keypoints, including 478 facial keypoints, 49 body keypoints, and 40 hand keypoints, with the data format conforming to the UE5 Mannequin skeleton topology, and each keypoint containing 3D coordinates (X,Y,Z), 6D rotation representation, and confidence level (0-1).

5. The real-time digital human driving method based on a multimodal somatosensory interaction algorithm according to claim 1, characterized in that: In step S3, the specific process for calculating the depth consistency error is as follows: For each matched 3D keypoint, obtain its actual depth value in the point cloud. and the depth value predicted by the model. The depth consistency error is calculated using the following formula. : The depth correction process is as follows: Set a depth error threshold. ,when Exceed At times, the depth predicted by the model deviates significantly from the actual depth of the point cloud, requiring correction of the depth of key points; The correction method uses point cloud depth values. Direct substitution model predicts depth values Alternatively, a weighted average method can be used, i.e.: in These are weighting coefficients, which are adjusted according to the actual situation to balance the impact of point cloud data and model predictions. Based on anthropometric standard bone length ratios, such as height:armspan = 1:1 and upper arm:forearm = 1:1.1, skeletal length constraints are established. Combined with user-specific T-pose / A-pose calibration data, a scale system (scale = Σw) is calculated. i × ), correct the scale of 3D key points to ensure that it conforms to the proportions of the real human body or the scale of the target digital human skeleton.

6. The real-time digital human driving method based on a multimodal somatosensory interaction algorithm according to claim 1, characterized in that: In step S4, the 3D rotation matrix after orthogonal reconstruction Convert it to a quaternion using the following mathematical conversion formula. form: when In this case, an alternative formula must be used: like and ,but: like and ,but: like and ,but: After conversion, combined with biomechanical joint angle limitations, unreasonable rotation values ​​are eliminated to ensure that joint movement conforms to the laws of human anatomy.

7. The real-time digital human driving method based on a multimodal somatosensory interaction algorithm according to claim 1, characterized in that: In step S5, Bayesian fusion and extended Kalman filtering further optimize the fusion result, specifically: First, the raw sensor data is denoised and normalized to provide high-quality input for subsequent fusion; then, a probability model for each sensor data is defined, and an initial probability distribution is set based on prior knowledge. Using Bayes' theorem: By incorporating new observational data and updating the posterior probability distribution, the initial fusion of multi-source data is achieved, resulting in a fused state estimate. in, Indicates in the event Under the conditions that the event occurs The probability of something occurring is called the posterior probability, which reflects the probability of something occurring in the known future. In the event of A reassessment of the likelihood of occurrence; Indicates in the event Under the conditions that the event occurs The probability of something occurring is called the likelihood, which reflects... When it happens The probability of it occurring; It is an event The prior probability of an event occurring, that is, the probability of an event occurring without any other information. An initial estimate of the probability of occurrence; For the event The probability of occurrence is a normalization constant used to ensure It is a reasonable probability value, and its role is to adjust the joint probability. Make adjustments to make Between 0 and 1; Extended Kalman Filter (EKF): Based on the system's state transition equations, it predicts the state at the next time step, obtaining a priori estimates. in, It is a nonlinear state transition function; It is the optimal estimate from the previous moment; For input; simultaneously calculate the prior estimated covariance. , It is the Jacobian matrix of the state transition function with respect to the state; For process noise covariance; The predicted state is transformed into the observation space through the observation equation to obtain the predicted observation values. , Given a nonlinear observation function; calculate the Kalman gain. , It is the Jacobian matrix of the observation function with respect to the state; To measure the noise covariance; finally, based on the actual observed values Update state estimation and update the estimated covariance ; The state estimate updated by EKF is used as the prior information for the next Bayesian fusion. The above steps are repeated, and the accuracy and stability of the fusion results are gradually improved through continuous iteration. In step S6, the procedure for detecting instantaneous attitude impact is as follows: Using sensors such as inertial measurement units, real-time attitude data is collected at a fixed frequency, including but not limited to angular velocity and acceleration information; the collected raw data is filtered to remove noise interference. Set an angular velocity threshold, which needs to be determined based on the specific application scenario and equipment characteristics, as the standard for judging instantaneous attitude impact; determine the number of iterations for the RANSAC algorithm. Related to the amount of sample data and the noise ratio, it can be obtained through empirical formulas. Calculate, where, The expected confidence level; This is an estimate of the noise ratio; The minimum number of samples required to fit the model; set an interior point determination threshold to determine whether the collected angular velocity data points belong to interior points, which is determined based on the sensor measurement error; Randomly select from the preprocessed data For each data point, construct an attitude angular velocity model. Calculate the error between the remaining data points and this model. Data points with errors less than the inlier threshold are classified as inliers; otherwise, they are classified as outliers. Repeat this process, recording the model with the most inliers in each iteration as the current optimal model. Continue this process until a set number of iterations is reached. Stop iterating when the time comes; The angular velocity values ​​of the interior points in the final optimal model are statistically analyzed. If the angular velocity of any interior point exceeds the preset angular velocity threshold, it is determined that an instantaneous attitude impact has occurred.

8. The real-time digital human driving method based on a multimodal somatosensory interaction algorithm according to claim 1, characterized in that: The real-time digital human driving method based on multimodal somatosensory interaction algorithm further includes the following steps: S8 and LiveLink source data processing: Following the UE5.6 LiveLink protocol, a FLiveLinkFrameData structure containing timestamps, subject names, skeleton transformation arrays, BlendShape weight arrays, and custom attributes is constructed; binary serialization, fixed-point encoding, quaternion compression, and differential encoding are used to reduce the amount of data. Using binary serialization: By converting data into binary format, Protocol Buffers takes up less space when storing and transmitting compared to JSON's text format. It uses a compact binary representation, avoiding the large number of delimiters and redundant key-value pairs in JSON, which can significantly reduce the data size. Fixed-point coding: Position information is represented by 16-bit fixed-point numbers. While ensuring an accuracy of 0.1mm, it reduces the space occupied by data. Compared with floating-point numbers, fixed-point numbers do not need to store the exponent part when representing values ​​within a fixed range, making more efficient use of binary bits to store data. Quaternion compression: Quaternions are used to represent rotations in three-dimensional space. By utilizing the mathematical properties of quaternions, only the three smallest components are stored. During recovery, the fourth component is derived based on mathematical relationships, achieving a 25% reduction in data volume without affecting the accurate representation of rotation information. Differential coding: only records the difference between the current frame and the previous frame, rather than the complete data content; since the changes between adjacent frames during the movement of a digital human are usually small, this method can significantly reduce the amount of data, especially when transmitting and storing continuous motion data, it can greatly reduce data redundancy; S9, Multi-protocol network transmission: Based on network type, for local area networks (LANs): UDP protocol is preferred due to its extremely low transmission latency. While maintaining high-speed transmission, it effectively solves the unreliability problem of the UDP protocol itself, ensuring stable data transmission. For example, in closed campus or office network environments, communication between devices can be completed quickly and accurately. Internet: Using TCP or WebSocket protocols, transmission latency can be controlled to <20ms; combined with LZ4 compression algorithm, data transmission volume can be significantly reduced and transmission efficiency improved; for example, when digital humans interact with users in real time in an internet environment, smooth visuals and data accuracy can be guaranteed. Local communication: It adopts a shared memory approach to achieve near-zero latency transmission. By using zero-copy technology, it avoids repeated copying of data in memory, greatly improving data read and write speeds. It is suitable for high-frequency data interaction between different processes within the same device. It also sets an adaptive transmission strategy and dynamically adjusts parameters: real-time monitoring of key indicators such as network RTT, packet loss rate and available bandwidth, sending frame rate based on 30fps, and automatically reducing the frame rate when insufficient bandwidth is detected to ensure transmission continuity; at the same time, it dynamically adjusts the data compression ratio according to network conditions, reducing the compression ratio to ensure data quality when bandwidth is sufficient, and increasing the compression ratio to reduce the amount of data transmitted when bandwidth is tight. Delay compensation mechanism: Introducing a prediction algorithm based on kinematic models, predicting the action state of the next moment based on the historical data and movement patterns of the digital human's actions, and using the predicted data to render the screen in advance when network delay occurs, compensating for the timing deviation caused by network delay, ensuring that the digital human's actions displayed on the receiving end are coherent and natural, and maintaining data timing consistency. S10, UE5.6 rendering driver: The UE5.6 LiveLinkSource plugin receives data packets and performs decompression, deserialization, CRC check and frame sequence number check to verify the validity of the data. The parsed skeletal transformation and BlendShape weights are mapped to MetaHuman or a custom digital human skeleton. If the frame rate does not match, temporal interpolation is performed. The skeletal transformation and facial weights are applied, and the final rendering result is output at 30, 60 or 120 fps.

9. A system, characterized in that: The system uses a real-time digital human driving method based on a multimodal somatosensory interaction algorithm as described in any one of claims 1-8.

Images