An immersive, no-wearable motion capture digital human sensory interaction system

Abstract

The application discloses an immersive no-wearing motion capture digital human body feeling interaction system and relates to the technical field of human-computer interaction.The system comprises a scene simulation layer, a character simulation layer and a motion simulation layer.The scene simulation layer is responsible for building and simulating the whole simulation environment or background, and provides a lifelike background for characters and motions.The character simulation layer is responsible for simulating the character roles in the simulation, and realizes a virtual human model with a film-level real sense.The motion simulation layer is responsible for simulating the specific motions of the character roles, and is responsible for processing the interaction between the characters and the environment.The immersive no-wearing motion capture digital human body feeling interaction system is based on the double-computing mainboard hardware architecture created by the application, and realizes the output of better interaction effects under the demand of lower hardware computing power through the full-stack algorithm optimization from the code bottom layer to the application layer, so that the low-delay and high-natural digital human body feeling interaction is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, in particular to an immersive no-wearing motion capture digital human body interaction system. BACKGROUND

[0002] In the current wave of digitization, virtual human interaction experience is gradually becoming the new darling of various fields. For a long time, virtual human driving mainly relies on wearable motion capture devices, and users need to realize motion capture by wearing sensors or marker points. However, with the continuous breakthrough of technology, a new no-wearing motion capture digital human real-time driving scheme has emerged. It is based on artificial intelligence image recognition technology, and motion capture is realized through RGB camera monocular camera. The motion data is transmitted to the 3D virtual human real-time interaction platform in real time, and the virtual human is rendered in real time to ensure that the virtual human can present synchronously with the real human motion, realizing smooth real-time driving interaction.

[0003] The no-wearing motion capture digital human scheme is not only a technological innovation, but also a new milestone in user virtual-real interaction experience. With the continuous maturity of technology and the continuous expansion of application scenarios, the no-wearing motion capture digital human scheme will shine in more fields, whether it is immersive experience of cultural tourism, interesting exercise of community health, or shocking performance of commercial brands, or extreme entertainment of body sensing games. It will bring users unprecedented surprises and pleasures in a more natural, smooth and personalized interaction way.

[0004] However, in actual use, due to the influence of factors such as hardware performance, software setting, network stability, etc. on motion capture data, once the hardware performance is insufficient, the software setting is improper, or the network is unstable, the motion capture data may drop frame, resulting in problems such as lag, incoherent and unnatural motion of digital human body interaction, affecting the effect of digital human body interaction. In addition, for some special human postures, actions, etc., the existing technology often cannot accurately identify and respond, resulting in delay, error, etc. in interaction. SUMMARY

[0005] The purpose of the present application is to provide an immersive no-wearing motion capture digital human body interaction system to solve the problems raised in the background art.

[0006] To achieve the above purpose, the present application provides the following technical scheme: an immersive no-wearing motion capture digital human body interaction system, comprising a scene simulation layer, a character simulation layer and a motion simulation layer.

[0007] The scene simulation layer is responsible for building and simulating the entire simulation environment or background, providing a realistic background for subsequent characters and motions, making the simulation more realistic and immersive, including the following modules:

[0008] Terrain Simulation Module: Responsible for real terrain restoration, using oblique photography technology to obtain real terrain and surrounding environment, and obtaining map data (such as vector slices and terrain elevation) through Mapbox API, combining UE5 Landscape system to build terrain;

[0009] Terrain Processing Module: Through Quixel Megascans asset library, integrate millions of scanned real materials (such as rocks, soil), directly applied to the terrain, and combined with custom masks (such as moss distribution, weathering effects), achieve high-precision surface details, specifically through nodal tools to define rules (such as vegetation distribution, rock placement), automatically generate large-scale terrain;

[0010] Shadow Processing Module: Provides delicate shadow transitions through virtual shadow map technology, achieving consistency in light and shadow at near and far distances;

[0011] Dynamic Weather Module: Combining UE5's Niagara particle system and volume fog, simulates the impact of weather such as rain, snow, and sandstorms on the terrain, enhancing immersion;

[0012] The character simulation layer is responsible for simulating the characters in the simulation, realizing a virtual human model with film-level realism, including the following modules:

[0013] Character Simulation Module: Using MetaHumans to provide preset general character templates as a starting point for creation, supporting export to DCC software such as Maya for deep customization;

[0014] Hair Simulation Module: Realize complex hairstyles through MayaXGenGroom tools, and use linear wire deformer to drive dynamic hair simulation;

[0015] The action simulation layer is responsible for simulating the specific actions of the character, including action capture, editing, synthesis and rendering, ensuring that the character's actions are smooth, natural, and match the environment and tasks, as well as handling the interaction between the character and the environment, such as collision detection and physical feedback, including the following modules:

[0016] Motion Capture Module: Responsible for human posture and gesture key point detection, providing a flexible framework to process and transmit human posture and gesture key point data, and supporting flexible deployment of multiple devices (including CPU, GPU, NPU, BPU, etc.);

[0017] Receiving processing module: using UDP protocol to receive motion capture data from external devices, including 33 3D body key points and 21 3D key points of both hands, as well as 478 3D face key points, and realizing data analysis, coordinate conversion and quaternion calculation in UDPSubsystem, using adaptive interpolation mechanism to smooth the data and effectively suppress jitter;

[0018] Posture analysis module: responsible for real-time detection of specific action patterns;

[0019] Timeout detection module: responsible for data validity detection and timeout processing, using hand data timeout reset mechanism to solve the data loss in motion capture;

[0020] Skeleton control module: through calculating the relationship between palm orientation and forearm axis, realizing natural twisting effect, and based on the change of pelvic position, realizing the overall character movement, supporting smooth transition and abnormality detection, through intelligent application and automatic reset of hand data, ensuring smooth transition to the default posture when data loss occurs.

[0021] Further, the terrain simulation module uses Landscaping plug-in to import GIS data to create Landscape, WorldComposition or WorldPartition, the Landscaping plug-in supports programmatic generation of roads and generation of vegetation according to landuse data, and online map texture is realized in combination with Mapbox (relevant authorization will comply with Unreal Engine's own end user license agreement (EULA), Autodesk's official Maya XGen Groom commercial authorization, Mapbox's relevant service terms, etc. authorization terms).

[0022] Further, the specific operation of the deep customization is that the user superimposes custom geometry based on the mesh and rig data of MetaHuman, and realizes 1:1 matching through Wrap3 tool.

[0023] Further, the hair simulation module reduces the number of CVs through techniques such as curve template construction and cylinder winding, avoids the crash of Unreal Engine due to high-density hair, balances performance and details, and realizes cross-platform dynamic effect import through Alembic cache. And through the use of Maya's XGen combined with GS CurveTools to make layered hair strands, FiberShop to generate hairpins and bake textures, using Physion plug-in combined with DER physical simulation, NiagaraGPU simulation, using Epic's Metahuman shader, etc. Adjust the thickness, density and simulation parameters (such as bending damping) of the hair to ensure the compatibility of complex hairstyles on different hardware devices.

[0024] Further, the motion capture module is separately deployed on a computing mainboard, calls the camera to collect video, obtains motion capture data, and sends the motion capture data to another computing mainboard through UDP for subsequent processing and application, and the two computing mainboards are connected through a network cable.

[0025] Further, the motion capture module uses a cascade detection method to improve precision and efficiency, directly transmits key information between modules, and the core process is as follows:

[0026] 1) The camera collects video frames and pre-processes them;

[0027] 2) Key facial feature detection: the model outputs the estimated value of 478 three-dimensional facial feature points;

[0028] 3) Pose detection: the whole image is detected to obtain the human body region;

[0029] 4) Pose key point extraction: based on the human body region, the image is cropped, and 33 3D human key points are inferred;

[0030] 5) Hand region detection: the wrist position is determined using the pose result, and a high-resolution hand region is cropped;

[0031] 6) Hand key point extraction: 21 3D key points of each hand are detected on the ROI;

[0032] 7) Key point smoothing processing: the detection result is filtered and smoothed;

[0033] 8) Result formatting and UDP transmission: after formatting the data, it is sent to the target address.

[0034] Further, the human body region cropped image is a 2D image, and its 3D coordinate inference is specifically implemented in the following way:

[0035] I. 3D coordinate inference implementation:

[0036] Realize the mapping from monocular camera input to 3D motion capture;

[0037] Add depth estimation during model training, and encode relative depth into z coordinate;

[0038] x, y coordinates are normalized to 0-1, and z coordinate represents relative depth;

[0039] II. Model training data source and optimization:

[0040] Use an open source dataset containing depth information;

[0041] Use a custom dataset collected by a depth camera;

[0042] Ensure data accuracy through artificial calibration;

[0043] Align and unify the depth data of multiple datasets, establish a unified depth reference system;

[0044] Data augmentation and cleaning are performed;

[0045] III. Model architecture upgrade:

[0046] Upgrade the original 2D key point definition to a 3D key point containing depth estimation;

[0047] Optimize the loss function design to balance the 2D and 3D prediction accuracy.

[0048] Further, the motion capture module uses the human body detection result of the previous frame to predict the position of the human body in the current frame, to reduce the frequent calling of full-image detection and reduce the computational overhead. The specific strategies include:

[0049] I. Predict the human body displacement in the current frame based on the detection result of the previous frame, and use speed and acceleration information to make reasonable prediction;

[0050] II. In most cases, use the prediction result to skip the expensive full-frame detection, and only when the prediction deviates greatly, trigger the detection again.

[0051] Further, the motion capture module adopts different strategies for human key points and hand tracking to ensure the smoothness and stability of the detection result:

[0052] I. Human key point stability: use One-Euro filtering algorithm to filter high-frequency noise while maintaining low latency, reducing detection jitter;

[0053] II. Hand key point stability: use the ROI information of the previous frame of the hand to adjust the hand ROI through a more stable detection framework and real-time smoothing algorithm to ensure the continuous stability of the hand details and position;

[0054] The performance optimization of the motion capture module includes:

[0055] I. The model is quantized for edge computing devices, supporting NPU or BPU acceleration, and reducing CPU load;

[0056] II. Use data pipeline processing, and each processing stage is executed in parallel;

[0057] III. Implement an asynchronous processing framework to eliminate the blocking between IO and computation;

[0058] IV. Optimize memory allocation and data transmission to reduce unnecessary copying.

[0059] Further, the posture analysis module specifically includes:

[0060] T-pose correction: By analyzing the angle of the arm relative to the spine, it determines whether the user is in a T-pose, and is used for automatic calibration and posture reset;

[0061] Body rotation analysis: Analyze the difference in rotation between the upper and lower body to determine whether it is a full-body rotation or a partial rotation, so as to apply digital human motion performance more naturally.

[0062] This invention provides an immersive, wearable-free motion capture digital human body sensory interaction system, which has the following beneficial effects:

[0063] This invention employs dual data verification, combining static posture detection and a dynamic timeout mechanism to ensure the reliability of motion capture data. Furthermore, it achieves smooth processing throughout the entire process from data reception to skeletal application, resulting in fluid and natural movements. Combined with intelligent forearm twisting, it automatically calculates the forearm twisting angle based on palm orientation, resolving the unnatural elbow rotation issue in traditional IK systems. Simultaneously, the system enables adaptive displacement detection, intelligently determining overall displacement based on pelvic position changes and supporting multiple coordinate systems. This ensures smooth detection and application of overall body movement without stuttering, especially in accurately recognizing and responding to special postures (such as the T-pose). In addition, pre-caching of skeletal indexes and transformation matrices significantly improves runtime performance.

[0064] Based on Cooler Technology's proprietary dual-computing motherboard hardware architecture, this system achieves superior interactive effects with lower hardware computing power requirements through full-stack algorithm optimization from the code layer to the application layer, thus realizing low-latency and highly natural digital human-like interaction. Attached Figure Description

[0065] Figure 1 This is a logic block diagram of an immersive, wear-free motion capture digital human body sensory interaction system according to the present invention. Detailed Implementation

[0066] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0067] like Figure 1 As shown, an immersive, wearable-free motion capture digital human body sensory interaction system includes:

[0068] Scene simulation layer

[0069] The simulation layer constructs and simulates the entire simulation environment or background, providing a realistic backdrop for characters and actions, making the simulation more realistic and immersive. In this embodiment, the scene simulation layer includes the following modules:

[0070] Terrain simulation module: Real terrain and surrounding environment are obtained by using oblique photography technology, and map data (such as vector slices and terrain elevation) are obtained through Mapbox API, and the terrain is constructed by combining the Landscape system of UE5. Specifically, the terrain simulation module uses the Landscaping plug-in to import GIS data to create Landscape, WorldComposition or WorldPartition, and the Landscaping plug-in supports procedural generation of roads and vegetation generation according to landuse data, and online map textures are realized by combining Mapbox;

[0071] Terrain processing module: Through Quixel Megascans asset library, millions of scanned real materials (such as rocks and soil) are integrated and directly applied to the terrain, and combined with custom masks (such as moss distribution and weathering effects), high-precision surface details are realized. Specifically, by defining rules (such as vegetation distribution and rock placement) through a node-based tool, large-scale terrain is automatically generated. For example, after setting parameters, UE5 can dynamically generate forests, deserts and other diversified landforms, and realize randomization of level layout;

[0072] Shadow processing module: Fine shadow transitions are provided through virtual shadow map technology to achieve consistency of light and shadow at near and far distances;

[0073] Dynamic weather module: Niagara particle system and volume fog of UE5 are combined to simulate the influence of weather such as rain, snow and sandstorm on the terrain, and enhance the sense of immersion;

[0074] Character simulation layer

[0075] The character role in simulation is realized to achieve a virtual human model with a film-level realistic feeling. In this embodiment, the character simulation layer includes the following modules:

[0076] Character simulation module: MetaHumans are used to provide preset general character templates as a starting point for creation, and support for exporting to DCC software such as Maya for deep customization. That is, users based on the Mesh and Rig data of MetaHuman, superimpose custom geometry, and realize 1:1 matching through Wrap3 tool. This modular design not only retains the standardization of MetaHuman animation compatibility, but also allows technicians to break through the preset limits.

[0077] Hair simulation module: complex hairstyles are achieved through MayaXGenGroom tools, and linear wire deformer is used to drive hair dynamic simulation. Specifically, the number of CVs is reduced through techniques such as curve template construction and cylinder winding to avoid Unreal Engine crashing due to high-density hair, and to balance performance and details. Cross-platform dynamic effect import is achieved through Alembic caching. Layered hair strands are made using Maya's XGen combined with GS CurveTools, FiberShop is used to generate hairpins and bake textures, Physion plugins are used in combination with DER physical simulation, Niagara GPU simulation, and Epic's Metahuman shader is used. Adjusting hair thickness, density, and simulation parameters (such as bending damping) ensures the compatibility of complex hairstyles on different hardware devices.

[0078] Action simulation layer

[0079] Simulate specific actions of a character role, including action capture, editing, synthesis and rendering, to ensure smooth and natural actions of the character, and match the environment and tasks, as well as handle interactions between the character and the environment, such as collision detection, physical feedback, etc. In this embodiment, the action simulation layer includes the following modules:

[0080] Action capture module: human pose and gesture key point detection, which provides a flexible framework to process and transmit human pose and gesture key point data, and supports flexible deployment of multiple devices (including CPU, GPU, NPU, BPU, etc.).

[0081] The action capture module is deployed on a computing mainboard separately, calls the camera to collect video, and sends the motion capture data to another computing mainboard for subsequent processing and application through UDP after obtaining the motion capture data, and the two computing mainboards are connected through a network cable. This distributed architecture design enables the system to better balance the computing load and ensure the real-time performance of the motion capture system;

[0082] The action capture module uses a cascade detection method to improve accuracy and efficiency, and directly transmits key information between modules, such as the results of pose detection used to intercept the human body area, and subsequent pose key point extraction performed on the key point reasoning in that area. The core process is as follows:

[0083] 1) The camera collects video frames and pre-processes them;

[0084] 2) Key facial feature detection: the model outputs an estimated value of 478 three-dimensional facial feature points;

[0085] 3) Pose detection: detect the human body area in the whole image;

[0086] 4) Pose key point extraction: based on the human body area, the image is cropped and 33 3D human key points are inferred;

[0087] 5) Hand region detection: Determine wrist position using pose results, crop high-resolution hand region;

[0088] 6) Hand key point extraction: Detect 21 3D key points for each hand on ROI;

[0089] 7) Key point smoothing: Filter and smooth the detection results;

[0090] 8) Result formatting and UDP transmission: Format data and send to target address.

[0091] Among them, the human region cropped image is a 2D image, and the 3D coordinate inference is specifically implemented by the following way:

[0092] Ⅰ. 3D coordinate inference implementation:

[0093] Realize the mapping from monocular camera input to 3D motion capture;

[0094] Add depth estimation during model training, encode relative depth into z coordinate;

[0095] x, y coordinates are normalized to 0-1, and z coordinate represents relative depth;

[0096] Ⅱ. Model training data source and optimization:

[0097] Use open source dataset containing depth information;

[0098] Use custom dataset collected by depth camera;

[0099] Ensure data accuracy through manual calibration;

[0100] Align and unify the depth data of multiple datasets to establish a unified depth reference system;

[0101] Data augmentation and cleaning;

[0102] Ⅲ. Model architecture upgrade:

[0103] Upgrade the original 2D key point definition to 3D key point containing depth estimation;

[0104] Optimize loss function design to balance 2D and 3D prediction accuracy.

[0105] The motion capture module uses the human body detection result of the previous frame to predict the position of the human body in the current frame, to reduce the frequent call of full image detection and reduce the computational overhead. The specific strategies include:

[0106] Ⅰ. Predict human displacement in the current frame based on the detection result of the previous frame, and use velocity and acceleration information to make reasonable prediction;

[0107] II. In most cases, the prediction result is used to skip the expensive full-frame detection, and only when the prediction deviates significantly, the detection is re-triggered.

[0108] To ensure the smoothness and stability of the detection results, the motion capture module adopts different strategies for human key points and hand tracking:

[0109] I. Human key point stability: One-Euro filter algorithm is used to filter high-frequency noise while maintaining low latency, reducing detection jitter;

[0110] II. Hand key point stability: Use the hand ROI information of the previous frame to adjust the hand ROI through a more stable detection framework and real-time smoothing algorithm to ensure the continuous stability of hand details and position.

[0111] The performance optimization of the motion capture module includes:

[0112] I. The model is quantized for edge computing devices, supporting NPU or BPU acceleration, reducing CPU load;

[0113] II. Data pipeline processing is used, and each processing stage is executed in parallel;

[0114] III. Asynchronous processing framework is implemented to eliminate the blocking between IO and calculation;

[0115] IV. Optimize memory allocation and data transmission to reduce unnecessary copying.

[0116] Receive processing module: UDP protocol is used to receive motion capture data from external devices, including 33 3D body key points and 21 3D key points for each hand, as well as 478 3D face key points. Data parsing, coordinate conversion, and quaternion calculation are implemented in UDPSubsystem, and adaptive interpolation mechanism is used for smoothing processing to effectively suppress jitter.

[0117] Posture analysis module: real-time detection of specific motion patterns, including:

[0118] T-pose correction: by analyzing the angle of the arm relative to the spine, it is determined whether the user is in T-pose, which is used for automatic calibration and posture reset;

[0119] Body rotation analysis: analyze the difference between upper and lower body rotation to determine whether it is full body rotation or local rotation, which is used for more natural application of digital human motion performance.

[0120] Timeout detection module: data effective detection and timeout processing, using hand data timeout reset mechanism to solve the data loss in motion capture.

[0121] Skeleton control module: By calculating the relationship between the palm orientation and the forearm axial orientation, the natural twisting effect is achieved, solving the unnatural elbow twisting problem in traditional IK system, and based on the change of pelvic position, the overall character movement is realized, supporting smooth transition and anomaly detection, through the intelligent application and automatic reset of hand data, ensuring smooth transition to the default posture when data is lost.

[0122] In this embodiment, the technical implementation of the skeleton-driven motion capture and displacement determination system is as follows:

[0123] 1. Palm and forearm spatial relationship calculation method

[0124] Based on the skeleton point position data, the system uses analytic geometry principle to calculate the accurate spatial relationship between the palm and the forearm, realizing high-precision wrist flexion and forearm rotation control.

[0125] 1.1 Forearm supination / pronation angle calculation

[0126] The forearm supination / pronation angle is calculated by the following steps:

[0127] 1) Define the forearm axial vector vec(v_arm) as the unit vector from the elbow to the wrist;

[0128] 2) Define the palm normal vector vec(n_palm) by calculating the palm key points;

[0129] 3) Define the palm upward direction vec(u_palm) as: vec(u_palm) = vec(v_arm) x vec(n_palm);

[0130] 4) Calculate the ideal upward direction vec(u_ideal) as:

[0131] vec(u_ideal) = vec(u_ref) - (vec(u_ref) cdot vec(v_arm)) cdot vec(v_arm) where vec(u_ref) is the reference upward direction (usually the torso upward direction);

[0132] 5) Calculate the supination / pronation angle θ_sup as:

[0133] θ_sup = cos^(-1)(vec(u_palm) cdot vec(u_ideal));

[0134] 6) Angle direction determination: If (vec(u_palm) x vec(u_ideal)) cdot vec(v_arm) < 0, then θ_sup = -θ_sup;

[0135] 1.2 Wrist flexion / extension angle calculation

[0136] The flexion / extension angle of the wrist is calculated by the following steps:

[0137] 1) Define the palm direction vector vec(d_hand) as the unit vector from the wrist to the base of the middle finger;

[0138] 2) Project the palm direction vector onto the forearm vertical plane: vec(d_proj) = vec(d_hand) - (vec(d_hand) · vec(v_arm)) · vec(v_arm);

[0139] 3) Calculate the flexion / extension axis vec(a_flex): vec(a_flex) = vec(v_arm) x vec(d_proj);

[0140] 4) Calculate the flexion angle θ_flex:

[0141] θ_flex = cos^(-1)(vec(d_proj) · vec(u_palm))

[0142] 5) Angle direction determination: if vec(d_proj) · (vec(a_flex) x vec(u_palm)) < 0, then θ_flex = -θ_flex

[0143] 1.3 Skeleton rotation quaternion composition

[0144] Convert the calculated angles to quaternion rotations and apply them to the corresponding skeleton:

[0145] 1) Forearm rotation quaternion: Q_forearm = Q(vec(v_arm), θ_sup · 0.6) where Q(vec(v), θ) denotes the quaternion of rotating θ radians around the axis vec(v);

[0146] 2) Wrist flexion / extension quaternion: Q_wrist = Q(vec(a_flex), θ_flex · 0.7);

[0147] 3) Update the skeleton pose:

[0148] Q_final_forearm = Q_forearm · Q_current_forearm Q_final_wrist = Q_wrist · Q_current_wrist

[0149] 2. Displacement recognition system based on step frequency analysis

[0150] The system adopts dynamic time window step frequency analysis method, realizes high-precision walking displacement recognition and speed calculation.

[0151] 2.1 Step detection algorithm

[0152] 1) Define ground reference height: Z_ground = Z_pelvis - h_offset, where h_offset is a predefined vertical offset from pelvis to ground (usually 90 cm);

[0153] 2) Foot lift state determination: Left foot lift: (Z_left_foot - Z_ground) > h_threshold, Right foot lift: (Z_right_foot - Z_ground) > h_threshold, where h_threshold is the foot lift height threshold (usually 15 cm);

[0154] 3) Step detection conditions:

[0155] 2.2 Time window step frequency analysis

[0156] 1) Maintain step timestamp queue Q = {t_1, t_2,..., t_n};

[0157] 2) When a new step is detected, add the current time t_current to the queue;

[0158] 3) Remove timestamps in the queue that exceed the time window: if (t_current - t) > T_window then remove t from Q, where T_window is the time window value (usually 0.5 seconds);

[0159] 4) Calculate step frequency: f_step = |Q| / T_window, where |Q| is the number of timestamps in the queue;

[0160] 5) Calculate target movement speed: v_target = f_step · S_step, where S_step is the speed coefficient corresponding to each step (usually 60 cm / s);

[0161] 6) Smooth speed change: v_current = FInterp(v_current, v_target, Δt, α), where α is the smoothing coefficient (usually 5.0), and FInterp is the smoothing interpolation function;

[0162] 2.3 Body orientation calculation

[0163] Body orientation is calculated by the following steps:

[0164] 1) Get the position vectors of the pelvis and the left and right clavicles: vec(v_left) = vec(p_left_clavicle) - vec(p_pelvis)

[0165] vec(v_right) = vec(p_right_clavicle) - vec(p_pelvis)

[0166] 2) Calculate the normal vector: vec(n) = vec(v_right) x vec(v_left)

[0167] 3) Project onto the horizontal plane: vec(n_horizontal) = norm(n_x, n_y, 0)

[0168] 4) Set the movement direction: vec(d_move) = vec(n_horizontal)

[0169] 3. Pelvis rotation accumulation calculation

[0170] The system implements a pelvis rotation accumulation calculation method based on arm posture, effectively solving the problem of insufficient pelvis rotation in motion capture.

[0171] 3.1 Arm posture detection

[0172] 1) Left arm 90 degree posture determination: θ_left_arm = ∠(vec(p_left_shoulder) - vec(p_left_hip), vec(p_left_shoulder) - vec(p_left_elbow)) LeftArm90 = |θ_left_arm - 90°| < θ_threshold;

[0173] 2) Right arm 90 degree posture determination: θ_right_arm = ∠(vec(p_right_shoulder) - vec(p_right_hip), vec(p_right_shoulder) - vec(p_right_elbow)) RightArm90 = |θ_right_arm - 90°| < θ_threshold;

[0174] 3.2 Pelvis rotation accumulation

[0175] 1) When the left arm is in a 90 degree posture, rotate the pelvis to the right: Q_Δ_left = Q(vec(v_forward), +3°) Q_pelvis = Q_Δ_left · Q_pelvis;

[0176] 2) Right arm 90 degree pose, rotate pelvis left: Q_Δ_right = Q(vec(v_forward), -3°) Q_pelvis = Q_Δ_right · Q_pelvis;

[0177] 4. Body plane normal vector calculation

[0178] This system proposes a robust body plane normal vector calculation method based on four-point construction, which can be effectively applied to wrist position adjustment and other scenarios.

[0179] 4.1 Body plane normal vector calculation

[0180] 1) Get four key points: left shoulder vec(p_LS), right shoulder vec(p_RS), left hip vec(p_LH), right hip vec(p_RH)

[0181] 2) Calculate diagonal vectors:

[0182] vec(d_1) = vec(p_LS) - vec(p_RH) vec(d_2) = vec(p_RS) - vec(p_LH);

[0183] 3) Calculate plane normal vector: vec(n_body) = norm(vec(d_1) × vec(d_2));

[0184] 4) Ensure normal vector faces forward: if vec(n_body) · vec(v_forward) < 0, then vec(n_body) = -vec(n_body);

[0185] 4.2 Wrist position adjustment based on visibility

[0186] 1) Get wrist visibility value v_wrist;

[0187] 2) When visibility is below threshold, adjust wrist position in the opposite direction of the body plane normal vector: vec(p_wrist_adjusted) = vec(p_wrist) - d_adjustment · vec(n_body), where d_adjustment is the adjustment distance (usually 15 cm).

[0188] The embodiments of the present application are presented by way of example and description, and are not intended to be exhaustive or to limit the application to the form disclosed. Many modifications and variations will be apparent to those skilled in the art. Embodiments are chosen and described in order to best explain the principles of the application and its practical application, and to thereby enable others skilled in the art to best utilize the application in various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. An immersive, no-wear motion capture digital mannequin haptics interaction system, characterized in that, It includes a scene simulation layer, a character simulation layer, and a motion simulation layer; The character simulation layer is responsible for simulating the character in the simulation, realizing a virtual human model with a film-level realistic sense, and includes the following modules: Role simulation module: MetaHumans provides a preset general role template as the starting point for creation, supporting export to DCC software for deep customization; Hair simulation module: complex hairstyles are realized through MayaXGenGroom tools, and linear wire deformer is used to drive dynamic hair simulation; The motion simulation layer is responsible for simulating the specific actions of the character, and for handling the interaction between the character and the environment, including the following modules: Motion capture module: responsible for human posture and gesture key point detection, providing a flexible framework to process and transmit human posture and gesture key point data, and supporting flexible deployment of multiple devices; Receiving and processing module: uses UDP protocol to receive motion capture data from external devices, and realizes data analysis, coordinate conversion, and quaternion calculation in UDPSubsystem, and uses adaptive interpolation mechanism for smooth data processing; Posture analysis module: responsible for real-time detection of specific action patterns; Timeout detection module: responsible for data validity detection and timeout processing, using hand data timeout reset mechanism to solve the data loss in motion capture; Skeletal control module: through the calculation of the relationship between the palm orientation and the forearm axis, the natural twisting effect is realized, and based on the change of the pelvic position, the overall role movement is realized, supporting smooth transition and abnormality detection; The motion capture module adopts different strategies for human key point and hand tracking: Ⅰ, human key point stability: a euro filter algorithm is used to filter high-frequency noise while maintaining low delay, reducing detection jitter; Ⅱ, hand key point stability: use the hand ROI information of the previous frame, adjust the hand ROI through the detection framework and real-time smoothing algorithm to ensure the continuous stability of the hand details and position; The posture analysis module specifically includes: T posture correction: by analyzing the angle of the arm relative to the spine, it is determined whether the user is in T posture, which is used for automatic calibration and posture reset; Body rotation analysis: analyze the rotation difference between the upper and lower body to determine whether it is a full body rotation or a local rotation, which is used to more naturally apply digital human action performance.

2. The immersive, no-wearable motion capture digital mannequin sensory interaction system of claim 1, wherein, The scene simulation layer is responsible for building and simulating the entire simulation environment or background, providing a realistic background for characters and actions, including the following modules: Terrain simulation module: responsible for real terrain restoration, using oblique photography technology to obtain real terrain and surrounding environment, and obtaining map data through MapboxAPI, combining UE5's Landscape system to build terrain; Terrain processing module: through Quixel Megascans asset library, integrate real materials, directly applied to the terrain, and combined with custom masks to realize high-precision ground details, specifically through the definition of rules by the nodal tool to automatically generate large-scale terrain; Shadow processing module: provides shadow transition through virtual shadow map technology, realizes the consistency of light and shadow in near and far distances; Dynamic Weather Module: Combining UE5's Niagara particle system and volume fog, simulates the impact of weather on terrain, enhancing immersion.

3. The immersive, no-wearable motion capture digital mannequin sensory interaction system of claim 2, wherein, The terrain simulation module uses the Landscaping plugin to import GIS data to create Landscape, WorldComposition, or WorldPartition, which supports procedural road generation and vegetation generation based on landuse data, while integrating Mapbox for online map textures.

4. The immersive, no-wearable motion capture digital mannequin sensory interaction system of claim 1, wherein, The deep customization operation: based on the user's MetaHuman mesh and binding data, superimpose custom geometry, and use Wrap3 tool for 1:1 matching.

5. The immersive, no-wearable motion capture digital mannequin haptics interaction system of claim 1, wherein, The hair simulation module reduces the number of CVs by building a curve template and using cylinder wrapping technology to avoid Unreal Engine crashes due to high-density hair, and uses Alembic caching for cross-platform dynamic effect import.

6. The immersive, no-wearable motion capture digital mannequin sensory interaction system of claim 1, wherein, The motion capture module is deployed on a separate computing motherboard, which calls the camera to collect video and sends the motion capture data to another computing motherboard for subsequent processing and application through UDP, and the two computing motherboards are connected through a network cable. The motion capture module uses a cascade detection method, and directly transmits key information between modules. The core process is as follows: 1) Camera captures video frames and pre-processes them; 2) Key facial feature detection: the model outputs an estimated value of 478 three-dimensional facial feature points; 3) Pose detection: detect the human body region in the whole image; 4) Pose key point extraction: crop the image based on the human body region and infer 33 3D human key points; 5) Hand region detection: determine the wrist position using the pose result and crop the high-resolution hand region; 6) Hand key point extraction: detect 21 3D key points for each hand in the ROI; 7) Key point smoothing: filter and smooth the detection results; 8) Result formatting and UDP transmission: format the data and send it to the target address.

7. The immersive, no-wearable motion capture digital mannequin sensory interaction system of claim 6, wherein, The human region cropped image is a 2D image, and its 3D coordinate inference is achieved by the following methods: Ⅰ, 3D coordinate inference implementation: Realize the mapping from monocular camera input to 3D motion capture; Add depth estimation during model training to encode relative depth into z coordinate; Normalize x, y coordinates to 0-1, and z coordinate represents relative depth; Ⅱ, Model training data source and optimization: Use open source datasets containing depth information; Use custom datasets collected by depth cameras; Ensure data accuracy through manual calibration; Align and unify the depth data of multiple datasets to establish a unified depth reference system; Perform data augmentation and cleaning; Ⅲ, Model architecture upgrade: Upgrade the original 2D key point definition to a 3D key point containing depth estimation; Optimize the loss function design to balance 2D and 3D prediction accuracy.

8. The immersive, no-wearable motion capture digital mannequin sensory interaction system of claim 7, wherein, The motion capture module uses the human body detection result of the previous frame to predict the position of the human body in the current frame to reduce the frequent call of full-image detection and reduce the computational overhead. The specific strategies include: I. Based on the detection results of the last frame, predict the human displacement in the current frame, and make reasonable estimates using speed and acceleration information; II. In most cases, skip the expensive full-frame detection using the prediction results, and only trigger detection again when the prediction deviates significantly.

Images