Simulation virtual image generation method based on motion capture

Abstract

The invention relates to the technical field of motion capture, and discloses a simulation virtual image generation method based on motion capture, and the method comprises the following steps: S1, motion data collection and preprocessing: employing a hybrid capture system to collect human body motion data; s2, action feature extraction and model construction: generating a clothes grid based on a basic model topology and learning appearance features; s3, generating and calibrating an action driving sequence: establishing a mapping relation between a driving signal and the basic model and optimizing an action matching degree; s4, dynamic response calculation and rendering; and S5, virtual image generation optimization. According to the technical scheme of data preprocessing based on hybrid capture and multi-algorithm fusion, the collaborative improvement effect of action data noise reduction, error correction and feature optimization is achieved, and compared with the technical scheme of single sensor acquisition or single algorithm processing in the prior art, the technical scheme of data preprocessing based on hybrid capture and multi-algorithm fusion is adopted. The problems that data are prone to interference, accumulated drift errors exist, and feature extraction is not comprehensive are solved.

Description

Technical Field

[0001] This invention relates to the field of motion capture technology, specifically to a method for generating simulated virtual images based on motion capture. Background Technology

[0002] Motion capture-based virtual avatar generation technology is a comprehensive technology that integrates sensor data acquisition, algorithm processing, 3D modeling, and rendering. It is widely used in various fields such as virtual live streaming, film and television production, rehabilitation training simulation, and game development. This technology captures real human motion data to drive virtual avatars to achieve natural and accurate motion replication, breaking down the barriers between virtual and reality. It provides users with an immersive experience and can assist in fulfilling core needs such as motion analysis and skill simulation in professional scenarios, becoming a key supporting technology in digital content production and intelligent interaction.

[0003] In the current virtual avatar generation technology, most of the motion data acquisition and preprocessing stages use a single sensor for data acquisition or only use a single type of algorithm to perform simple data processing, making it difficult to balance the comprehensiveness of data acquisition and the accuracy of processing.

[0004] Data collected by a single sensor is susceptible to environmental interference and noise from the device itself, resulting in insufficient data accuracy. At the same time, a single algorithm cannot effectively solve the cumulative drift error caused by long-term operation of the sensor, and it is difficult to take into account both local details and global trends during feature extraction, resulting in missing feature information. Ultimately, this affects the realism and stability of virtual image motion reconstruction, restricting the application of the technology in high-precision and high-real-time scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for generating simulated virtual images based on motion capture, which solves the problems of data being easily interfered with, having cumulative drift errors, and having incomplete feature extraction caused by single sensor acquisition or single algorithm processing in existing technologies.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for generating simulated virtual avatars based on motion capture, comprising the following steps:

[0007] S1. Motion data acquisition and preprocessing: A hybrid capture system is used to acquire human motion data. The raw data is filtered and denoised, and coordinate system and sensor cumulative error correction are performed. When the preset equipment synchronization conditions and data accuracy conditions are met, the next step is executed.

[0008] S2. Action Feature Extraction and Model Building: Extract the temporal sequence and dynamic features of human key points through a deep convolutional network, build a parameterized human basic model and bind skeletal features, generate clothing mesh based on the topology of the basic model and learn appearance characteristics, and proceed to the next step when the preset feature integrity condition, skeletal binding accuracy condition and mesh adaptation condition are met.

[0009] S3. Action-driven sequence generation and calibration: Input the feature sequence into the dynamic occlusion frame to complete the occlusion frame action data, generate virtual image driving signals, establish the mapping relationship between the driving signals and the basic model and optimize the action matching degree. When the preset driving signal timing conditions and action synchronization conditions are met, proceed to the next step.

[0010] S4. Dynamic Response Calculation and Rendering: Calculate the collision response and deformation data between clothing and human body based on the driving signal, fuse motion data and deformation data for neural rendering, and proceed to the next step when the preset deformation calculation frame rate and rendering image quality conditions are met.

[0011] S5. Virtual Image Generation Optimization: Perform inter-frame smoothing on the rendering sequence and output a virtual image. Generation is completed when the preset generation sequence delay condition is met.

[0012] Preferably, in S1, the hybrid capture system collects human motion data by: collecting human joint angle and acceleration data through an inertial measurement unit, and simultaneously collecting human surface contour and spatial coordinate data through an infrared camera. The two types of data are aligned by timestamps to form an original motion data set.

[0013] Preferably, in step S1, the preprocessing includes the following sub-steps:

[0014] S101. First, Kalman filtering is used to reduce noise in the original data to eliminate environmental noise and equipment noise interference.

[0015] S102. Then, the cumulative error of the inertial measurement unit is corrected by complementary filtering to compensate for the measurement deviation of a single sensor.

[0016] S103. Finally, the image coordinates acquired by the infrared camera and the local coordinates of the inertial measurement unit are unified to the preset world coordinate system to complete coordinate system one.

[0017] Preferably, in S1, the action feature extraction includes: inputting the preprocessed action data into a deep convolutional network, first extracting the low-level spatial features through the convolutional layer, then reducing the dimensionality through the pooling layer, and gradually aggregating local and global features through a 6-stage feature fusion module, and finally outputting the temporal sequence and dynamic features of human body key points;

[0018] Among them, the temporal sequence of human body key points is output through the key point detection head, and the dynamic features are output through the dynamic analysis head.

[0019] Preferably, in step S2, the model construction includes the following sub-steps:

[0020] S201. Construct a parametric basic human model based on human anatomy, and define the degrees of freedom of skeletal joints and muscle attachment points.

[0021] S202. Map the extracted human body key point time sequence one by one with the skeletal nodes of the basic model to complete the skeletal feature binding.

[0022] S203. Based on the surface topology of the basic model, a physics engine is used to generate clothing meshes that fit the human body contour, divide the mesh into units and set material parameters.

[0023] S204. Input real clothing image samples through a neural appearance network, learn the texture, gloss and light reflection characteristics of clothing, and assign the learning results to the clothing mesh.

[0024] Preferably, in step S3, the action-driven sequence generation includes: inputting the extracted feature sequence into a dynamic occlusion framework, first masking the feature data corresponding to the occlusion frame, then constraining the feature reconstruction process through a diffusion loss function, and supplementing the missing action data of the occlusion frame based on the correlation between the action trend and dynamic features of adjacent frames to form a complete virtual image driving signal sequence.

[0025] Preferably, in step S3, the motion calibration includes the following sub-steps:

[0026] S301. Establish the mapping relationship between the driving signals and the basic model skeletal joints, and define the joint range of motion corresponding to each driving parameter;

[0027] S302. Construct a generator network G and a discriminant network D. The generator network G receives driving signals and outputs the model action prediction results. The discriminant network D is used to distinguish between predicted action and real action data.

[0028] S303. By iteratively training the loss function composed of reconstruction loss and KL divergence term, the matching degree between driving signal and model action is optimized, and action synchronization calibration is completed.

[0029] Preferably, in S4, the specific method for dynamic response calculation is as follows: the driving signal is input into the physics engine, the human joint motion trajectory corresponding to the driving signal is first analyzed, and then the collision detection result between the clothing and the human body surface is calculated based on the physical properties of the clothing mesh. The deformation displacement of the clothing is solved through finite element analysis to generate frame-by-frame deformation data. The specific method for neural rendering is as follows: the motion data and deformation data are used as input, the shooting perspective information is integrated through the perspective encoding module, and the image frame containing dynamic details is generated through the rendering network.

[0030] Preferably, in step S4, the neural rendering process supports viewpoint-dependent adjustment, dynamically adjusts the lighting and shadow effects of the image according to preset viewpoint parameters, and calculates dynamic shadows in real time during the rendering process to ensure the consistency between the virtual image and the ambient lighting and shadows, and the resolution of the rendered image is not lower than a preset threshold.

[0031] Preferably, in S5, the optimization method for inter-frame smoothing processing includes: dynamically adjusting the weight allocation strategy of the bilateral filter by combining the human dynamics features of joint torque, muscle contraction degree and motion acceleration extracted in S2.

[0032] When a fast-movement scene with joint motion acceleration exceeding a preset threshold is detected in the dynamic features, the spatial distance weight coefficient is increased to 0.6-0.8, and the grayscale similarity weight coefficient is decreased to 0.2-0.4.

[0033] When the joint motion acceleration is less than or equal to the preset threshold for slow motion or static scenes, the weighting coefficient is adjusted in the opposite direction.

[0034] The timing information of the action after S3 calibration is referenced synchronously during the filtering process.

[0035] This invention provides a method for generating simulated virtual avatars based on motion capture. It has the following beneficial effects:

[0036] 1. This invention achieves a synergistic improvement in motion data noise reduction, error correction, and feature optimization by employing a hybrid capture and multi-algorithm fusion data preprocessing technology. Compared with existing technologies that use single sensor acquisition or single algorithm processing, this invention solves the problems of data being susceptible to interference, cumulative drift error, and incomplete feature extraction.

[0037] 2. This invention adopts an action-driven technology solution of anatomical adaptation modeling and adversarial training calibration, which achieves a high degree of consistency between virtual image movements and real human movements and precise skeletal binding. Compared with the existing technology of generalized model construction and lack of targeted calibration, it solves the problems of low motion reproduction and poor model-human adaptability.

[0038] 3. The present invention adopts an optimized technical solution of dynamic weighted filtering and multimodal fusion rendering, which achieves the effect of balancing motion detail preservation and screen smoothness, and adapting to different scene requirements. Compared with the existing fixed parameter filtering or single-dimensional rendering technical solutions, it solves the problems of blurry fast motion, jittery slow motion, and insufficient scene adaptability. Attached Figure Description

[0039] Figure 1 This is a schematic diagram illustrating the steps of a motion capture-based method for generating a simulated virtual avatar according to the present invention.

[0040] Figure 2 This is a schematic diagram of the preprocessing steps in a motion capture-based simulation virtual image generation method of the present invention.

[0041] Figure 3 This is a schematic diagram illustrating the model construction steps of a motion capture-based simulation virtual avatar generation method according to the present invention.

[0042] Figure 4 This is a schematic diagram of the motion calibration steps in a motion capture-based virtual character generation method according to the present invention. Detailed Implementation

[0043] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Please see the appendix Figure 1 - Appendix Figure 4 This invention provides a method for generating simulated virtual avatars based on motion capture, comprising the following steps:

[0045] S1. Motion data acquisition and preprocessing: A hybrid capture system is used to acquire human motion data. The raw data is filtered and denoised, and coordinate system and sensor cumulative error correction are performed. When the preset equipment synchronization conditions and data accuracy conditions are met, the next step is executed.

[0046] In S1, the hybrid capture system collects human motion data including: collecting human joint angle and acceleration data through an inertial measurement unit, and simultaneously collecting human surface contour and spatial coordinate data through an infrared camera. The two types of data are aligned by timestamp to form a set of raw motion data.

[0047] In S1, the preprocessing includes the following sub-steps:

[0048] S101. First, Kalman filtering is used to reduce noise in the original data to eliminate environmental noise and equipment noise interference.

[0049] S102. Then, the cumulative error of the inertial measurement unit is corrected by complementary filtering to compensate for the measurement deviation of a single sensor.

[0050] S103. Finally, the image coordinates acquired by the infrared camera and the local coordinates of the inertial measurement unit are unified to the preset world coordinate system to complete coordinate system one.

[0051] In S1, action feature extraction includes: inputting preprocessed action data into a deep convolutional network, first extracting low-level spatial features through convolutional layers, then reducing dimensionality through pooling layers, and gradually aggregating local and global features through a 6-stage feature fusion module, finally outputting the temporal sequence and dynamic features of human key points;

[0052] Among them, the temporal sequence of human key points is output through the key point detection head, and the dynamic features are output through the dynamic analysis head;

[0053] Specifically, considering that the raw data collected by the hybrid capture system is susceptible to interference from ambient light and electromagnetic noise from the equipment, a linear Kalman filter is used to achieve real-time noise reduction. Data quality is optimized through a two-stage iterative process of prediction and update. The core formulas include:

[0054] State prediction equation: ;

[0055] Covariance prediction equation: ;

[0056] Kalman gain equation: ;

[0057] State update equation: ;

[0058] Covariance update equation: ;

[0059] in, The predicted state vector for the k-th frame. State transition matrix, The filtered state vector of the (k-1)th frame. To control the input matrix, For the control input vector of the k-th frame, For process noise, The prediction covariance matrix of the k-th frame, State transition matrix transpose matrix, For the process noise covariance matrix, The Kalman gain for the k-th frame, Observation matrix transpose matrix, For matrix inverse matrix, The filtered state vector of the k-th frame. For the raw observation data of the k-th frame, For identity matrix, Let the state covariance matrix of the k-th frame be... To observe the noise covariance matrix, This is the observation matrix.

[0060] To address the issue of accumulated drift error in inertial measurement units (IMUs), a first-order complementary filter is used to fuse calibration data from the infrared camera to correct the error.

[0061] ;

[0062] in, For the merged corrected data, For filter weight coefficients, Raw data collected by the inertial measurement unit, Calibration data collected by an infrared camera.

[0063] In the action feature extraction process, to balance local detail features with global action trends, a 6-stage weighted fusion strategy is adopted. The core formula includes:

[0064] ;

[0065] in, The final output includes the temporal sequence and dynamic characteristics of key human body points. The weight coefficients of the features in the i-th stage, For the local or global features output by the i-th stage convolutional layer, This is the sequence number of the feature fusion stage, with values ​​ranging from 1 to 6.

[0066] S2. Action Feature Extraction and Model Building: Extract the temporal sequence and dynamic features of human key points through a deep convolutional network, build a parameterized human basic model and bind skeletal features, generate clothing mesh based on the topology of the basic model and learn appearance characteristics, and proceed to the next step when the preset feature integrity condition, skeletal binding accuracy condition and mesh adaptation condition are met.

[0067] In S2, model construction includes the following sub-steps:

[0068] S201. Construct a parametric basic human model based on human anatomy, and define the degrees of freedom of skeletal joints and muscle attachment points.

[0069] S202. Map the extracted human body key point time sequence one by one with the skeletal nodes of the basic model to complete the skeletal feature binding.

[0070] S203. Based on the surface topology of the basic model, a physics engine is used to generate clothing meshes that fit the human body contour, divide the mesh into units and set material parameters.

[0071] S204. Input real clothing image samples through a neural appearance network, learn the texture, gloss and light reflection characteristics of clothing, and assign the learning results to the clothing mesh.

[0072] Specifically, to transform high-dimensional fused features into concrete parameters that can be directly used for model-driven operations, feature mapping is achieved through fully connected layers. The core formulas include:

[0073] Key point time series: ;

[0074] Dynamic characteristics: ;

[0075] in, Let be the coordinate vector of the human key points in frame t. Map weight matrices to key points The fusion features output by S1 Keypoint offset vector, Let t be the dynamic feature vector of the t-th frame. For dynamic feature mapping weight matrix, For dynamic characteristic offset vector, This is the frame number.

[0076] To ensure that the skeletal motion of the parametric human baseline model is consistent with real motion data, it is necessary to calculate the skeletal rigging error and verify the accuracy. The core formulas include:

[0077] ;

[0078] in, For skeleton binding error, For the number of skeletal nodes, The angle of the i-th skeletal node in the basic model For the extracted joint angle data corresponding to the i-th key point, This represents the skeletal node number, with values ​​ranging from 1 to N.

[0079] To ensure that the generated clothing mesh fits the human body model and adapts to the range of motion, it is necessary to comprehensively evaluate the contour overlap and size deviation. The core formulas include:

[0080] ;

[0081] in, For mesh adaptation, For weighting coefficients, The overlapping area between the clothing mesh and the human body model's outline, For the surface area of ​​the human body model, Key dimensions for clothing mesh, These are the corresponding key dimensions of the human body model.

[0082] S3. Action-driven sequence generation and calibration: Input the feature sequence into the dynamic occlusion frame to complete the occlusion frame action data, generate virtual image driving signals, establish the mapping relationship between the driving signals and the basic model and optimize the action matching degree. When the preset driving signal timing conditions and action synchronization conditions are met, proceed to the next step.

[0083] In S3, the action-driven sequence generation includes: inputting the extracted feature sequence into the dynamic occlusion framework, first masking the feature data corresponding to the occlusion frame, then constraining the feature reconstruction process through the diffusion loss function, and supplementing the missing action data of the occlusion frame based on the correlation between the action trend and dynamic features of adjacent frames to form a complete virtual image driving signal sequence.

[0084] In S3, motion calibration includes the following sub-steps:

[0085] S301. Establish the mapping relationship between the driving signals and the basic model skeletal joints, and define the joint range of motion corresponding to each driving parameter;

[0086] S302. Construct a generator network G and a discriminant network D. The generator network G receives driving signals and outputs the model action prediction results. The discriminant network D is used to distinguish between predicted action and real action data.

[0087] S303. By iteratively training the loss function composed of reconstruction loss and KL divergence term, the matching degree between driving signal and model action is optimized, and action synchronization calibration is completed.

[0088] Specifically, to address the feature loss problem caused by the susceptibility of infrared cameras to occlusion, the feature reconstruction process is constrained by a diffusion loss function. The core formula includes:

[0089] ;

[0090] in, For the diffusion loss function value, For mean square error loss, For the reconstructed feature sequences, For the true feature sequence that is not obscured, Regularization coefficient For KL divergence loss, For reconstruction of distribution based on occlusion features, For the prior distribution of true features, The masking feature sequence is marked by the mask.

[0091] To optimize the matching degree between driving signals and model actions, a generative adversarial network is used for training. The core formulas include:

[0092] ;

[0093] in, For the total loss function value, For weighting coefficients, To rebuild the losses, For KL divergence loss, To counter the losses caused by network attacks.

[0094] S4. Dynamic Response Calculation and Rendering: Calculate the collision response and deformation data between clothing and human body based on the driving signal, fuse motion data and deformation data for neural rendering, and proceed to the next step when the preset deformation calculation frame rate and rendering image quality conditions are met.

[0095] In S4, the specific method for dynamic response calculation is as follows: the driving signal is input into the physics engine, the motion trajectory of the human joint corresponding to the driving signal is first analyzed, and then the collision detection results between the clothing and the human body surface are calculated based on the physical properties of the clothing mesh. The deformation displacement of the clothing is solved through finite element analysis to generate frame-by-frame deformation data. The specific method for neural rendering is as follows: the motion data and deformation data are used as input, the shooting perspective information is integrated through the perspective encoding module, and the rendering network generates image frames containing dynamic details.

[0096] In S4, the neural rendering process supports viewpoint-dependent adjustment, dynamically adjusting the lighting and shadow effects of the image according to preset viewpoint parameters, and calculating dynamic shadows in real time during the rendering process to ensure the consistency of virtual image with ambient lighting and shadow, and the resolution of the rendered image is not lower than the preset threshold.

[0097] Specifically, to realistically simulate the deformation effect of clothing during human movement, the linear elastic finite element method is used for solution. The core formulas include:

[0098] ;

[0099] in, The stiffness matrix of the clothing mesh, For the deformation displacement vector of the mesh node, This is the vector of the contact force between the human body and the clothing;

[0100] To fuse motion data and deformation data to generate high-quality image frames, a neural rendering network is used. The core formula includes:

[0101] ;

[0102] in, For the rendered output image pixel value matrix, For the Sigmoid activation function, For the feature mapping function of the neural rendering network, For action data feature vectors, For clothing deformation data feature vectors, Encoding vectors for viewpoints For rendering offset vector;

[0103] To evaluate the quality of rendered images, peak signal-to-noise ratio (PSNR) is used as a quantification metric. The core formula includes:

[0104] ;

[0105] in, For peak signal-to-noise ratio, Maximum pixel value Mean square error, For the generated image for rendering, For reference only.

[0106] S5. Virtual Image Generation Optimization: Perform inter-frame smoothing on the rendering sequence and output a virtual image. Generation is completed when the preset generation sequence delay condition is met.

[0107] In S5, the optimization method for inter-frame smoothing includes: combining the human dynamics features extracted in S2, such as joint torque, muscle contraction, and motion acceleration, to dynamically adjust the weight allocation strategy of the bilateral filter.

[0108] When a fast-movement scene with joint motion acceleration exceeding a preset threshold is detected in the dynamic features, the spatial distance weight coefficient is increased to 0.6-0.8, and the grayscale similarity weight coefficient is decreased to 0.2-0.4.

[0109] When the joint motion acceleration is less than or equal to the preset threshold for slow motion or static scenes, the weighting coefficient is adjusted in the opposite direction.

[0110] The timing information of the action after S3 calibration is referenced synchronously during the filtering process;

[0111] Specifically, to avoid blurring of fast motion or jitter of slow motion caused by inter-frame smoothing, the filter weights are dynamically adjusted based on dynamic characteristics. The core formulas include:

[0112] ;

[0113] ;

[0114] in, Spatial distance weighting coefficient, For the joint motion acceleration of the current frame, For acceleration threshold, This represents the grayscale similarity weighting coefficient.

[0115] Based on dynamically adjusted weights, inter-frame pixel fusion is achieved. The core formula includes:

[0116] ;

[0117] in, For the fusion Pixel values ​​of coordinates for Neighborhood window, For the pixel coordinates within the neighborhood window, For pixels and Euclidean distance, For pixels and grayscale difference, For neighboring pixels The original value, For pixels The original value.

[0118] To ensure that the output of the virtual avatar meets the scene latency requirements, the generation sequence latency needs to be accurately calculated. The core formulas include:

[0119] ;

[0120] in, For the delay of generating the sequence, For the collection timestamp of the original action data, This is the output timestamp for the virtual avatar file.

[0121] The following is a description with reference to specific embodiments:

[0122] Example 1: This example is applied to the real-time virtual anchor interaction scenario on a live streaming platform. The core requirements are low latency, high motion synchronization, and smooth video.

[0123] S1. Motion Data Acquisition and Preprocessing: A lightweight hybrid capture system is adopted, including 12 wearable inertial measurement units (IMUs) of model XsensMTwAwinda and 4 high-definition infrared cameras with a resolution of 1920×1080 and a frame rate of 60fps. These are worn on the anchor's head, torso, and key joints of the limbs. The IMUs acquire the anchor's joint angles in real time with an accuracy of ±0.5° and acceleration data sampling rate of 120Hz. The infrared cameras simultaneously acquire the anchor's surface contour and spatial coordinate data, which are transmitted to the processing terminal via Bluetooth 5.2 and aligned according to millisecond-level timestamps. The original motion data set is generated. During preprocessing, Kalman filtering with Q matrix = 0.01 and R matrix = 0.1 is first used to reduce noise in the original data to eliminate lighting interference and electromagnetic noise from the equipment in the live broadcast environment. Then, complementary filtering is used to correct the accumulated error of the IMU. Error calibration is performed every 10 frames to compensate for the drift deviation of a single sensor. Finally, the image coordinates of the infrared camera and the local coordinates of the IMU are unified to the Unreal Engine 5 world coordinate system to complete coordinate system one. When the device synchronization rate is ≥98% and the data error after preprocessing is ≤0.08mm, proceed to the next step.

[0124] S2. Action Feature Extraction and Model Building: Preprocessed action data is input into a lightweight deep convolutional network, MobileNetV3, and a 6-stage feature fusion module. The convolutional layers use 3×3 kernels, and the pooling layers have a maximum pooling stride of 2. A keypoint detection head outputs a temporal sequence of 118 human keypoints, and a dynamic analysis head outputs joint torque, muscle contraction, and motion acceleration with a sampling interval of 5ms. During model building, a parametric human body model is first constructed based on the anchor's body parameters: height 175cm, weight 65kg. 23 degrees of freedom for the skeletal joints are defined, with a focus on optimizing the flexibility of the neck, shoulder, and hand joints, and 18 muscle attachment points. Next, the extracted key point time sequence is mapped one by one to the basic model skeletal nodes, with a skeletal binding error ≤0.4°. Then, the NVIDIA PhysX 5.1 physics engine is used to generate a customized clothing mesh for the anchor, with a mesh unit size of 0.5mm, an elastic coefficient of 0.3, and a friction coefficient of 0.2 to adapt to the daily movement range of the live streaming scene. Finally, 300 high-definition images of real clothing are input through the neural appearance network to learn the texture of cotton shirts and the characteristics of light and shadow reflection. The learning results are assigned to the clothing mesh. When the feature sequence integrity is ≥99%, the skeletal binding error is ≤0.4°, and the mesh fit is ≥92%, the next step is performed.

[0125] S3. Action-Driven Sequence Generation and Calibration: The feature sequence is input into a dynamic occlusion frame with an occlusion rate of 30%. The hand and shoulder frame data that are occluded by the infrared camera are masked and marked. The feature reconstruction is constrained by the diffusion loss function with a weight coefficient of 0.8. Based on the correlation between the action trend of adjacent frames and the muscle contraction degree, the missing data of the occluded frames is filled in to generate a virtual image driving signal sequence with a frame rate of 60fps. During action calibration, the mapping relationship between the driving signal and the skeletal joints of the basic model is first established. The range of motion of the neck joint is defined as pitch ±45°, rotation ±30°, and the range of motion of the hand joint is finger flexion ±90°. Then, a lightweight generative network GU-Net architecture and a discriminative network DCNN architecture are constructed. The generative network G receives the driving signal and outputs the action prediction result. The discriminative network D distinguishes between the predicted action and the real action data. Finally, the loss function consisting of a reconstruction loss weight of 0.6 and a KL divergence term weight of 0.4 is iteratively trained for 200 rounds to optimize the action matching degree. When the timing error of the driving signal is ≤30ms and the action synchronization similarity is ≥98%, the next step is performed.

[0126] S4. Dynamic Response Calculation and Rendering: The driving signal is input into the PhysX5.1 physics engine to analyze the joint motion trajectories of the anchor's head rotation, hand gestures, etc. The collision detection results between clothing and human body surface are calculated at a detection frequency of 120Hz. The deformation displacement of clothing is solved through finite element analysis to generate frame-by-frame deformation data. Neural rendering uses a TensorRT-accelerated neural rendering network, taking motion data and deformation data as input. The view encoding module incorporates the fixed view angle of 15° for live streaming. The rendering network generates 1080P resolution image frames, supporting real-time lighting and shadow adjustment to adapt to the warm light background of the live streaming scene. Dynamic shadows are calculated in real time during the rendering process, with a shadow resolution of 2048×2048 to ensure the consistency of lighting and shadow between the virtual anchor and the live streaming background. The rendering frame rate is ≥60fps. When the deformation calculation frame rate is ≥80fps and the PSNR of the rendered image is ≥38dB, proceed to the next step.

[0127] S5. Virtual Avatar Generation Optimization: Inter-frame smoothing is combined with motion acceleration features extracted in S2, and bilateral filter weights are dynamically adjusted. When the anchor's rapid hand-waving joint motion acceleration is detected to be greater than 5m / s², the virtual avatar generation optimization is applied. 2 The spatial distance weighting coefficient was set to 0.7, the grayscale similarity weighting coefficient was set to 0.3, and the edge details of hand movements were preserved. When the anchor is speaking statically, the joint motion acceleration is ≤5m / s². 2The grayscale similarity weight coefficient was set to 0.7, and the spatial distance weight coefficient was set to 0.3 to eliminate minor image jitter. During the filtering process, S3-calibrated motion timing information was used to avoid disrupting the synchronization between lip movements and speech. Subsequently, the H.265 low-latency coding algorithm was employed to encapsulate the image sequence in MP4 format, embedding timestamps to synchronize live audio. The output virtual avatar file underwent latency verification to ensure a latency ≤15ms, meeting the requirements of real-time live interaction, thus completing the generation process.

[0128] Example 2: A scenario for assessing patient motor function in the field of medical rehabilitation, with core requirements being high movement accuracy, anatomical consistency, and traceability;

[0129] S1. Motion Data Acquisition and Preprocessing: A high-precision hybrid capture system is employed, comprising 17 medical-grade inertial measurement units (IMUs) of the Delsys Trigno model and 8 infrared cameras with a resolution of 3840×2160 and a frame rate of 100fps. The IMUs are fixed to the walls and ceiling of the rehabilitation training room, while the IMUs are worn on the patient's limb joints, spine, and pelvis. The IMUs acquire joint angle data for rehabilitation training exercises such as knee flexion and extension, and shoulder abduction with an accuracy of ±0.1°, and acceleration data with a sampling rate of 200Hz. Simultaneously, the infrared cameras acquire the patient's body surface contour and spatial coordinate data, which are transmitted via wired Ethernet. The data is input to the processing terminal and aligned with microsecond-level timestamps to form a raw motion data set. During preprocessing, Kalman filtering with Q matrix = 0.001 and R matrix = 0.05 is used to reduce noise in the raw data and eliminate interference from the metal reflections of rehabilitation equipment in the rehabilitation room. Then, complementary filtering is used to correct the accumulated error of the IMU. Error calibration is performed every 5 frames to compensate for sensor drift during long-term training. Finally, the image coordinates of the infrared camera and the local coordinates of the IMU are unified to the DICOM medical imaging coordinate system to complete coordinate system one. When the device synchronization rate is ≥99% and the data error after preprocessing is ≤0.05mm, proceed to the next step.

[0130] S2. Motion Feature Extraction and Model Construction: Preprocessed motion data is input into a high-precision deep convolutional network VGG-19 and a 6-stage feature fusion module. Convolutional layers use 3×3 kernels, and pooling layers have an average pooling stride of 2. A keypoint detection head outputs a temporal sequence of 122 human keypoints, focusing on optimizing keypoints in joint spaces. A dynamic analysis head outputs joint torque, muscle contraction, and motion acceleration with a sampling interval of 2ms. During model construction, a parametric human baseline model is first built based on the patient's CT image data, strictly adhering to human anatomical structures. 32 degrees of freedom for skeletal joints are defined, with key rehabilitation joints such as the knee and hip joints highlighted, along with 25 muscle attachment points including core muscle groups such as the quadriceps and deltoid. The extracted key point time sequence is then mapped one by one to the basic model skeletal nodes, with a skeletal binding error ≤0.3°. Subsequently, the ANSYSLS-DYNA physics engine is used to generate a mesh elastic fabric for the patient's rehabilitation clothing. The mesh unit size is 0.2mm, and the elasticity coefficient is set to 0.5 and the friction coefficient to 0.3, adapting to the large-amplitude movements and equipment contact scenarios of rehabilitation training. Finally, 500 high-definition image samples of medical rehabilitation clothing are input through a neural appearance network to learn the breathability texture and low reflectivity of the fabric. The learning results are assigned to the clothing mesh, while retaining the visualization effect of the model's skeletal marker points. When the feature sequence integrity is ≥99.5%, the skeletal binding error is ≤0.3°, and the mesh fit is ≥95%, proceed to the next step.

[0131] S3. Motion-Driven Sequence Generation and Calibration: The feature sequence is input into a dynamic occlusion frame with an occlusion rate of 50%. The leg and waist frame data occluded by the rehabilitation equipment are masked and marked. The feature reconstruction is constrained by the diffusion loss function with a weight coefficient of 0.9. Based on the correlation between the movement trend of adjacent frames and joint torque, the missing data of the occluded frames is filled in, and a virtual image driving signal sequence with a frame rate of 100fps is generated. During motion calibration, the mapping relationship between the driving signal and the skeletal joints of the basic model is first established. Key rehabilitation assessment parameters such as the range of motion of the knee joint (flexion and extension 0-120°) and the range of motion of the shoulder (abduction 0-180°) are defined. Then, the generative network GResNet and Transformer architecture and the discriminative network DVGG-16 architecture are constructed. The generative network G receives the driving signal and outputs the motion prediction result. The discriminative network D distinguishes the predicted motion from the standard rehabilitation motion data. Finally, the loss function consisting of a reconstruction loss weight of 0.7 and a KL divergence term weight of 0.3 is iteratively trained for 300 rounds to optimize the motion matching degree. When the timing error of the driving signal is ≤20ms and the motion synchronization similarity is ≥99%, the next step is performed.

[0132] S4. Dynamic Response Calculation and Rendering: The driving signal is input into the ANSYSLS-DYNA physics engine to analyze the joint motion trajectory of the patient's rehabilitation movements. Based on the physical properties of the clothing mesh and the contact parameters of rehabilitation equipment such as elastic bands, the collision detection result is calculated at a detection frequency of 200Hz. The deformation displacement of clothing and skin is solved through finite element analysis to generate frame-by-frame deformation data. The neural rendering uses a high-precision neural rendering network, taking the motion data and deformation data as input. The view encoding module incorporates multi-angle evaluation view information, including front view, side view, and top view. The rendering network generates 4K resolution image frames. The rendering process supports view-dependent adjustment and switching of any evaluation view. Dynamic shadows are calculated in real time with a shadow resolution of 4096×4096 to ensure that the joint motion trajectory and deformation details of the virtual image can be accurately traced. When the deformation calculation frame rate is ≥100fps and the PSNR of the rendered image is ≥40dB, the next step is performed.

[0133] S5. Virtual Image Generation Optimization: Inter-frame smoothing combined with the joint torque and motion acceleration features extracted in S2, dynamically adjusting the bilateral filter weights, when the patient's joint motion acceleration is detected to be greater than 3 m / s² during rapid rehabilitation movements. 2 The spatial distance weighting coefficient was set to 0.8, and the grayscale similarity weighting coefficient was set to 0.2. To preserve the precise details of the joint motion trajectory, the joint motion acceleration was ≤3 m / s² when the patient slowly performed the movement. 2 The grayscale similarity weight coefficient was set to 0.8, and the spatial distance weight coefficient was set to 0.2 to eliminate image jitter caused by muscle tremors. During the filtering process, the motion timing information after S3 calibration was used to ensure that the motion data was consistent with the rehabilitation assessment timeline. Subsequently, a lossless encoding algorithm was used to encapsulate the image sequence in GLB format, embedding traceability information such as patient ID and training time. The output virtual image file was delayed to ensure that the delay was ≤10ms, which met the accuracy analysis requirements of offline assessment and completed the generation.

[0134] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating simulated virtual avatars based on motion capture, characterized in that, Includes the following steps: S1. Motion data acquisition and preprocessing: A hybrid capture system is used to acquire human motion data. The raw data is filtered and denoised, and coordinate system and sensor cumulative error correction are performed. When the preset equipment synchronization conditions and data accuracy conditions are met, the next step is executed. S2. Action Feature Extraction and Model Building: Extract the temporal sequence and dynamic features of human key points through a deep convolutional network, build a parameterized human basic model and bind skeletal features, generate clothing mesh based on the topology of the basic model and learn appearance characteristics, and proceed to the next step when the preset feature integrity condition, skeletal binding accuracy condition and mesh adaptation condition are met. S3. Action-driven sequence generation and calibration: Input the feature sequence into the dynamic occlusion frame to complete the occlusion frame action data, generate virtual image driving signals, establish the mapping relationship between the driving signals and the basic model and optimize the action matching degree. When the preset driving signal timing conditions and action synchronization conditions are met, proceed to the next step. S4. Dynamic Response Calculation and Rendering: Calculate the collision response and deformation data between clothing and human body based on the driving signal, fuse motion data and deformation data for neural rendering, and proceed to the next step when the preset deformation calculation frame rate and rendering image quality conditions are met. S5. Virtual Image Generation Optimization: Perform inter-frame smoothing on the rendering sequence and output a virtual image. Generation is completed when the preset generation sequence delay condition is met.

2. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In S1, the hybrid capture system collects human motion data by: collecting human joint angle and acceleration data through an inertial measurement unit, and simultaneously collecting human surface contour and spatial coordinate data through an infrared camera. The two types of data are aligned by timestamps to form a set of original motion data.

3. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In step S1, the preprocessing includes the following sub-steps: S101. First, Kalman filtering is used to reduce noise in the original data to eliminate environmental noise and equipment noise interference. S102. Then, the cumulative error of the inertial measurement unit is corrected by complementary filtering to compensate for the measurement deviation of a single sensor. S103. Finally, the image coordinates acquired by the infrared camera and the local coordinates of the inertial measurement unit are unified to the preset world coordinate system to complete coordinate system one.

4. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In S1, the action feature extraction includes: inputting the preprocessed action data into a deep convolutional network, first extracting the low-level spatial features through the convolutional layer, then reducing the dimensionality through the pooling layer, and gradually aggregating local and global features through a 6-stage feature fusion module, and finally outputting the temporal sequence and dynamic features of human body key points. Among them, the temporal sequence of human body key points is output through the key point detection head, and the dynamic features are output through the dynamic analysis head.

5. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In step S2, model construction includes the following sub-steps: S201. Construct a parametric basic human model based on human anatomy, and define the degrees of freedom of skeletal joints and muscle attachment points. S202. Map the extracted human body key point time sequence one by one with the skeletal nodes of the basic model to complete the skeletal feature binding. S203. Based on the surface topology of the basic model, a physics engine is used to generate clothing meshes that fit the human body contour, divide the mesh into units and set material parameters. S204. Input real clothing image samples through a neural appearance network, learn the texture, gloss and light reflection characteristics of clothing, and assign the learning results to the clothing mesh.

6. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In S3, the action-driven sequence generation includes: inputting the extracted feature sequence into the dynamic occlusion framework, first masking the feature data corresponding to the occlusion frame, then constraining the feature reconstruction process through the diffusion loss function, and supplementing the missing action data of the occlusion frame based on the correlation between the action trend and dynamic features of adjacent frames to form a complete virtual image driving signal sequence.

7. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In step S3, the motion calibration includes the following sub-steps: S301. Establish the mapping relationship between the driving signals and the basic model skeletal joints, and define the joint range of motion corresponding to each driving parameter; S302. Construct a generator network G and a discriminant network D. The generator network G receives driving signals and outputs the model action prediction results. The discriminant network D is used to distinguish between predicted action and real action data. S303. By iteratively training the loss function composed of reconstruction loss and KL divergence term, the matching degree between driving signal and model action is optimized, and action synchronization calibration is completed.

8. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In S4, the specific method for dynamic response calculation is as follows: the driving signal is input into the physics engine, the human joint motion trajectory corresponding to the driving signal is first analyzed, and then the collision detection result between the clothing and the human body surface is calculated based on the physical properties of the clothing mesh. The clothing deformation displacement is solved through finite element analysis to generate frame-by-frame deformation data. The specific method of neural rendering is as follows: motion data and deformation data are used as input, and the shooting perspective information is incorporated through the perspective encoding module. The rendering network then generates image frames containing dynamic details.

9. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In S4, the neural rendering process supports viewpoint-dependent adjustment, dynamically adjusts the lighting and shadow effects of the image according to preset viewpoint parameters, and calculates dynamic shadows in real time during the rendering process to ensure the consistency between the virtual image and the ambient lighting and shadows, and the resolution of the rendered image is not lower than a preset threshold.

10. The method for generating a simulated virtual image based on motion capture according to claim 1, characterized in that: In S5, the optimization method for inter-frame smoothing includes: dynamically adjusting the weight allocation strategy of the bilateral filter by combining the human dynamics features of joint torque, muscle contraction and motion acceleration extracted in S2. When a fast-movement scene with joint motion acceleration exceeding a preset threshold is detected in the dynamic features, the spatial distance weight coefficient is increased to 0.6-0.8, and the grayscale similarity weight coefficient is decreased to 0.2-0.

4. When the joint motion acceleration is less than or equal to the preset threshold for slow motion or static scenes, the weighting coefficient is adjusted in the opposite direction. The timing information of the action after S3 calibration is referenced synchronously during the filtering process.

Images