A method and system for real-time human pose tracking in XR scenes

Abstract

This invention discloses a real-time human pose tracking method and system for XR scenes. This invention relates to the field of computer vision technology and solves the technical problems of being unable to overcome the modal barriers of visual and temporal features and the degradation of diffusion models. This invention uses dynamic Gaussian filtering and ROI cropping on RGB images to suppress noise while focusing on the human body region. Historical pose normalization covers spatial, temporal, and physiological constraints, ensuring the consistency of the physical meaning of the input data. It achieves multi-dimensional, scene-adaptive preprocessing, laying the data foundation for subsequent algorithms. A cross-attention mechanism breaks down the modal boundaries of visual and temporal features, generating multi-modal conditional vectors that simultaneously encode the current limb appearance and historical movement trends, providing accurate reasoning basis for the diffusion model. It achieves dynamic, scene-adaptive feature collaboration, constructing theoretical poses based on the first 5 frames and initially dynamically fusing them with the diffusion model's predictions, preserving details while conforming to motion inertia.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for real-time human pose tracking in XR scenes. Background Technology

[0002] In XR interaction scenarios, human posture tracking is a core supporting technology for achieving natural interaction and building immersive experiences. With the popularization of XR devices, scenarios such as virtual fitness, remote collaboration, and immersive games place higher demands on the accuracy, real-time performance, and robustness of posture tracking.

[0003] According to patent application CN117291951A, a multi-human pose tracking method based on human key points is disclosed. The method detects human bodies in a video frame; it performs target association matching between the detected human bodies and already tracked targets; for initial new targets that fail to match, it performs human key point tracking to filter and delete false targets; and it tracks real human bodies and successfully matched human bodies using human key point tracking. The method expands the human tracking area, obtains the coordinates and confidence scores of human key points, determines whether human bodies are occluded to exclude duplicate tracking, deletes the tracked target, and outputs the position information and key point information of the human tracking target. This process is repeated until human target tracking is complete.

[0004] However, some existing technologies cannot overcome the modal barriers of visual and temporal features, dynamically associate limb contours and joint movement trends, and generate multimodal conditional vectors that accurately guide posture reasoning. At the same time, although diffusion models have powerful generation capabilities, the original architecture has a slow reasoning speed and is not optimized for posture degradation modes in XR scenarios. It also lacks the integration of the physical laws of human movement, and the generated postures may violate physiological common sense and cannot be directly used for interaction. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a real-time human pose tracking method and system for XR scenes, which solves the problems of being unable to overcome the modal barriers of visual and temporal features and the degradation of diffusion models.

[0006] To achieve the above objectives, the present invention provides a real-time human pose tracking method for XR scenes, which specifically includes the following steps:

[0007] S1. Acquire the current frame RGB image, the previous N frames of historical pose sequence and IMU data collected by the XR device, perform noise reduction processing on the RGB image, and normalize the historical pose sequence.

[0008] S2. Visual features of the current frame RGB image are extracted by an image encoder, temporal features of the historical pose sequence are extracted by a temporal encoder, and the visual features and temporal features are dynamically aligned using a cross-attention mechanism to generate a multimodal conditional vector.

[0009] S3. Input the multimodal conditional vector into the lightweight diffusion model engine, perform fast denoising through the inverse diffusion process, and generate the preliminary prediction result of the 3D pose of the current frame. The lightweight diffusion model engine includes a 4-layer U-Net architecture and adopts a 10-step DDIM sampling strategy.

[0010] S4. Perform Kalman filtering to smooth the preliminary prediction results and output the final 3D pose of the current frame.

[0011] As a further embodiment of the present invention, the value of N in S1 is 5, the historical posture sequence contains 6-DOF parameters of 24 joints, and the normalization process is to map the joint coordinates to the range of [-1,1].

[0012] As a further aspect of the present invention, the image encoder uses a MobileNetV3 network to output 128-dimensional visual features; the temporal encoder uses a GRU network to output 64-dimensional temporal features; and the cross-attention mechanism achieves dynamic alignment between visual features and temporal features through query-key-value mapping, outputting a 192-dimensional multimodal conditional vector.

[0013] As a further aspect of the present invention, the reverse diffusion process of the diffusion model engine is specifically implemented as follows:

[0014] Define the noise scheduling parameters for forward diffusion. ∈[1e-4,0.02], T=1000 steps, during the backdiffusion process, noise is predicted using a 4-layer U-Net. U-Net employs depthwise separable convolutions and a DDIM sampling strategy, reducing the 1000-step denoising process to 10 steps. Each step is performed using the formula... Update the hidden state, where P t For the noisy pose at step t, , for The cumulative product, is a noise prediction network, and c is a multimodal vector.

[0015] As a further aspect of the present invention, the lightweight diffusion model engine also includes a temporal consistency constraint module, which analyzes the process from two aspects: hidden state inheritance and loss function optimization, and de-diffused the hidden state h of the (t-1)th frame. t-1 Inject birth noise from frame t, and the hidden state h t-1It includes motion memory of the previous frame's pose, and a joint velocity smoothing term is added to the loss function during model training. Specifically, The first item To ensure the accuracy of noise prediction, the second item... It constrains the joint velocity v of the current frame. t Compared to the joint velocity v in the previous frame t-1 The differences, among which The value of v is 0.3. t v is the joint velocity in the current frame. t-1 This refers to the joint velocity in the previous frame.

[0016] As a further aspect of the present invention, the Kalman filter is combined with IMU data for dynamic correction, and the IMU data includes real-time measurement values ​​from the accelerometer and gyroscope.

[0017] A real-time human pose tracking system for XR scenes includes:

[0018] The data acquisition module is used to receive the current frame RGB image, the previous N frames of historical pose sequence and IMU data acquired by the XR device, and transmit the acquired information to the data preprocessing module.

[0019] The data preprocessing module is used to denoise the obtained RGB image and normalize the historical pose sequence to obtain the corresponding preprocessed data, which is then transmitted to the multimodal conditional fusion module.

[0020] The multimodal conditional fusion module includes an image encoder, a temporal encoder, and a cross-attention submodule. The image encoder is used to perform feature parsing on RGB images to obtain visual features, and the temporal encoder is used to model the motion trend of historical pose sequences to obtain temporal features. A cross-attention mechanism is introduced to associate the two to generate a multimodal conditional vector, which is then transmitted to the lightweight processing module.

[0021] The lightweight processing module is used to input the obtained multimodal conditional vector into the lightweight diffusion model engine and perform fast denoising to generate a preliminary prediction result of the 3D pose of the current frame. At the same time, the obtained preliminary prediction result is transmitted to the post-processing module.

[0022] The post-processing module is used to perform temporal smoothing on the preliminary prediction results using the Kalman filter algorithm to obtain the final 3D pose and transmit it to the information display module.

[0023] Information display module, which is used for tracking based on the final 3D pose obtained.

[0024] As a further embodiment of the present invention, the image encoder is a MobileNetV3 network, the temporal encoder is a GRU network, the output dimension of the cross-attention submodule is 192-dimensional, the lightweight diffusion model engine adopts INT8 quantization processing, the model size is ≤8MB, and the single-frame processing latency is ≤12ms.

[0025] As a further embodiment of the present invention, the XR device is AR glasses, VR headset or MR all-in-one machine, and the XR device is equipped with a monocular RGB camera and a 6-axis IMU sensor.

[0026] This invention provides a method and system for real-time human pose tracking in XR scenes. Compared with existing technologies, it has the following advantages:

[0027] This invention employs dynamic Gaussian filtering and ROI cropping on RGB images to suppress noise while focusing on the human body region. Historical pose normalization covers spatial, temporal, and physiological constraints, ensuring the consistency of the physical meaning of the input data. This enables multi-dimensional, scene-adaptive preprocessing, laying a data foundation for subsequent algorithms. The cross-attention mechanism breaks down the modal boundaries between visual and temporal features, generating multi-modal conditional vectors that simultaneously encode the current limb appearance and historical movement trends, providing accurate reasoning basis for the diffusion model and achieving dynamic, scene-adaptive feature collaboration.

[0028] The present invention utilizes a 4-layer U-Net and depthwise separable convolution to reduce the number of model parameters. The DDIM sampling strategy compresses thousands of inference steps to 10 steps and delays single-frame processing. The temporal consistency module, through hidden state inheritance and joint velocity smoothing loss analysis, forces pose inference to conform to the laws of human movement. The general diffusion model has no physiological constraints and easily generates poses with instantaneous joint shifts and movements that violate mechanics, making the output pose more natural and reducing inter-frame jitter. The theoretical pose is constructed based on the first 5 frames and dynamically fused with the preliminary prediction of the diffusion model. Details are preserved if they match the motion inertia, and the trajectory is corrected if there are abnormal jumps. The final output 3D pose not only resists single-frame noise but also ensures smooth movement. Attached Figure Description

[0029] Figure 1 This is a diagram illustrating the steps and methods of the present invention;

[0030] Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1

[0033] Please see Figure 1 This application provides a method for real-time human pose tracking in XR scenes, which specifically includes the following steps:

[0034] Step 1: Acquire three types of data in real time through the heterogeneous sensor array built into the XR device. Specifically, these include RGB image streams (acquired by a monocular high-definition camera (such as the 12-megapixel RGB camera of MetaQuest 3) at a rate of 30fps, with a resolution of 1920×1080, an FOV of 90°, and covering the main joint areas of the upper body), historical pose sequences (caching the human pose parameters of the last 5 frames (each frame contains the three-dimensional coordinates and quaternion rotation parameters of 24 key joints), for example, storing the user's punching posture in the previous 5 frames in a virtual fitness scene), and IMU data streams (6-axis inertial measurement unit (such as ICM-42688-P) outputs acceleration (±16g) and angular velocity (±2000° / s) data at a sampling rate of 1000Hz). At the same time, the obtained RGB images are denoised to obtain preprocessed RGB images, and the historical pose sequences are normalized to obtain preprocessed historical pose sequences.

[0035] Specifically, denoising RGB images includes temporal filtering and region of interest extraction, where temporal filtering employs an adaptive Gaussian filtering algorithm. =0.5-1.5 (dynamically adjusted) to suppress random noise while preserving edge features;

[0036] Region of Interest (ROI) extraction predicts the human body position in the current frame based on historical poses and dynamically crops 800×800 pixel ROI regions to reduce background interference.

[0037] Normalization of historical pose sequences specifically includes spatial normalization, temporal window alignment, and outlier correction.

[0038] Spatial normalization specifically projects joint coordinates onto a unit spherical space to eliminate the impact of user height differences.

[0039] Time window alignment uses a sliding window mechanism to synchronize RGB frames and IMU data, and resamples the IMU's 1000Hz data to 30Hz through linear interpolation to ensure timestamp consistency;

[0040] Outlier correction means detecting and replacing joint angles that exceed physiological limits (such as elbow flexion exceeding 180°, which can effectively prevent postural changes caused by violent movements in dance training scenarios).

[0041] Step 2: Use a lightweight image encoder (such as the optimized MobileNetV3) to perform feature analysis on the preprocessed RGB image. Specifically, extract 128 core features from the image, which are called visual features. These include edge gradients of limb contours (such as contour changes when the arm swings), texture information around joints (such as the relationship between elbow wrinkles and joint angles), and brightness features that are adaptive to lighting (such as shoulder shadows that can still be recognized under strong light).

[0042] Simultaneously, a temporal encoder (such as a single-layer GRU network) is used to model the motion trend of the previous 5 frames of historical pose sequence, generating 64-dimensional temporal features, which are denoted as temporal features. The specific processing method is as follows:

[0043] Capture the dynamic patterns of limb movements, including joint speed (such as the acceleration change of the wrist from rest to swing), movement continuity (such as the coordinated movement rhythm of shoulder-elbow-wrist in a punching motion), and the inertial characteristics of the direction of movement (such as the tendency of the arm to fall naturally after being raised).

[0044] A cross-attention mechanism is introduced to achieve intelligent association between visual features and temporal features, generating a 192-dimensional multimodal conditional vector. Specifically, this mechanism focuses on the associated region through "dynamic weight allocation": when the visual feature shows "hand close to head", the weight of "shoulder rotation" in the temporal feature will be increased, so that the two form a synergy in the "arm raising" action; conversely, if the temporal feature shows "leg moving quickly", the weight of "knee area" in the visual feature will be enhanced.

[0045] Step 3: Input the obtained multimodal conditional vector into the lightweight diffusion model engine. The core of the diffusion model is to simulate the process of "pose degradation and recovery": In reality, occlusion (such as an arm crossing to block the abdomen) and motion blur (such as image trailing caused by rapid arm swings) can make pose information become blurred, as if "contaminated by noise". The inverse diffusion process is like "intelligent noise reduction" - using the multimodal conditional vector as "clues", it gradually removes noise, restores the hidden true pose, and performs noise reduction through the inverse diffusion process. The inverse diffusion process specifically includes the following:

[0046] Define the noise scheduling parameters for forward diffusion. ∈[1e-4,0.02], T=1000 steps, initial steps (small) Add only weak noise, preserve the basic pose contours (e.g., to simulate a scene with mild motion blur), and then proceed with subsequent steps (large-scale) The noise is gradually increased to make the attitude data approach complete randomness (such as simulating attitude loss caused by severe occlusion).

[0047] The core of the inverse diffusion process is to predict the noise in the current noisy pose using a 4-layer lightweight U-Net. To achieve progressive denoising, depthwise separable convolution is used instead of traditional convolution. The number of channels is dynamically adjusted according to [32, 64, 128, 64]. The shallow layer captures basic features such as joint positions, while the deep layer focuses on limb motion associations (such as shoulder-elbow-wrist coordination constraints). The input is a noisy pose P. t The current time step t, the multimodal conditional vector, and the output prediction noise. By employing the DDIM sampling strategy, the 1000-step denoising process is reduced to 10 steps, with each step using the formula... Update the hidden state, where P t For the noisy pose at step t, , for The cumulative product, For a noise prediction network, c is a multimodal vector;

[0048] The lightweight diffusion model engine introduces a temporal consistency constraint module, analyzing it from two aspects: hidden state inheritance and loss function optimization. This module then de-diversifies the hidden state h in frame t-1. t-1 Inject birth noise from frame t, and the hidden state h t-1 It includes motion memory of the previous frame's posture (such as arm swing direction and joint rotation trend), and a joint velocity smoothing term is added to the loss function during model training. Specifically, The first item To ensure the accuracy of noise prediction, the second item... It constrains the joint velocity v of the current frame. t Compared to the joint velocity v in the previous frame t-1 The differences, among which The value of v is 0.3. t v is the joint velocity in the current frame. t-1 This refers to the joint velocity in the previous frame.

[0049] Step 4: Generate preliminary prediction results of the 3D pose of the current frame based on the diffusion model, and perform temporal smoothing processing using Kalman filtering. The specific processing method is as follows:

[0050] Based on the dynamic posture model built from the first 5 frames, the motion trend is accurately analyzed—for example, extracting the continuous change patterns of velocity and acceleration from arm swing data, and inferring the "theoretical posture" of the current frame. When the diffusion model outputs preliminary results (which may be subject to instantaneous jitter such as joint position jumps due to interference such as image blurring), "dual-state fusion calibration" is initiated:

[0051] If the preliminary results match the historical motion inertia (such as the arm continuing its upward trajectory), a high confidence weight is assigned to preserve the fingertip details, muscle stretching, and other posture features. Once an abnormal jump is detected (such as a joint shifting backward without physical basis), the theoretical posture is used as the anchor point. Through dynamic weight allocation, the jump noise is weakened, the posture trajectory is corrected, and the joint movement returns to a smooth curve that conforms to human biomechanics, thus suppressing inter-frame jitter from the root.

[0052] Example 2

[0053] Please see Figure 2 This application provides a real-time human pose tracking system for XR scenes. The system includes: a data acquisition module, a data preprocessing module, a multimodal conditional fusion module, a lightweight processing module, a post-processing module, and an information display module, and is combined with… Figure 2 It can be seen that the information between the above functional modules is transmitted in one direction only.

[0054] The data acquisition module receives the current frame RGB image, the previous N frames of historical pose sequences, and IMU data acquired by the XR device, and transmits the acquired information to the data preprocessing module.

[0055] The data preprocessing module is used to denoise the obtained RGB image and normalize the historical pose sequence to obtain the corresponding preprocessed data, which is then transmitted to the multimodal conditional fusion module. The specific processing method is the same as that in step one.

[0056] The multimodal conditional fusion module includes an image encoder, a temporal encoder, and a cross-attention submodule. The image encoder is used to perform feature parsing on RGB images to obtain visual features, and the temporal encoder is used to model the motion trend of historical pose sequences to obtain temporal features. A cross-attention mechanism is introduced to associate the two to generate a multimodal conditional vector, which is then transmitted to the lightweight processing module. The specific processing method is the same as that in step two.

[0057] The lightweight processing module is used to input the obtained multimodal conditional vector into the lightweight diffusion model engine and perform fast denoising to generate a preliminary prediction result of the 3D pose of the current frame. At the same time, the obtained preliminary prediction result is transmitted to the post-processing module, and the specific processing method is the same as the processing process in step three.

[0058] The post-processing module is used to perform temporal smoothing on the preliminary prediction results using the Kalman filter algorithm to obtain the final 3D pose, and then transmits it to the information display module. The specific processing method is the same as that in step four.

[0059] Information display module, which is used for tracking based on the final 3D pose obtained.

[0060] The data in the above formulas are all calculated using numerical values, without substituting the units of the parameters. In addition, the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0061] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for real-time human pose tracking in XR scenes, characterized in that, The method specifically includes the following steps: S1. Acquire the current frame RGB image, the previous N frames of historical pose sequence and IMU data collected by the XR device, perform noise reduction processing on the RGB image, and normalize the historical pose sequence. S2. Visual features of the current frame RGB image are extracted by an image encoder, temporal features of the historical pose sequence are extracted by a temporal encoder, and the visual features and temporal features are dynamically aligned using a cross-attention mechanism to generate a multimodal conditional vector. S3. Input the multimodal conditional vector into the lightweight diffusion model engine, and perform fast denoising through the inverse diffusion process to generate the preliminary prediction result of the 3D pose of the current frame. The lightweight diffusion model engine includes a 4-layer U-Net architecture and adopts a 10-step DDIM sampling strategy. The lightweight diffusion model engine also includes a temporal consistency constraint module, which analyzes the process from two aspects: hidden state inheritance and loss function optimization, and de-diversifies the hidden state h in frame t-1. t-1 Inject birth noise from frame t, and the hidden state h t-1 It includes motion memory of the previous frame's pose, and a joint velocity smoothing term is added to the loss function during model training. Specifically, The first item To ensure the accuracy of noise prediction, the second item... It constrains the joint velocity v of the current frame. t Compared to the joint velocity v in the previous frame t-1 The differences, among which The value of v is 0.

3. t v is the joint velocity in the current frame. t-1 The joint velocity of the previous frame; S4. Smooth the preliminary prediction results using Kalman filtering and output the final 3D pose of the current frame.

2. The method for real-time human pose tracking in an XR scene according to claim 1, characterized in that, In S1, N takes the value of 5. The historical pose sequence contains 6-DOF parameters for 24 joints. The normalization process maps the joint coordinates to the range of [-1, 1].

3. The method for real-time human pose tracking in an XR scene according to claim 1, characterized in that, In step S2, the image encoder uses a MobileNetV3 network to output 128-dimensional visual features; the temporal encoder uses a GRU network to output 64-dimensional temporal features; the cross-attention mechanism achieves dynamic alignment of visual features and temporal features through query-key-value mapping, and outputs a 192-dimensional multimodal conditional vector.

4. The method for real-time human pose tracking in an XR scene according to claim 1, characterized in that, In step S3, the specific method of the reverse diffusion process of the diffusion model engine is as follows: Define the noise scheduling parameters for forward diffusion. ∈[1e-4,0.02], T=1000 steps, during the backdiffusion process, noise is predicted using a 4-layer U-Net. U-Net employs depthwise separable convolutions and a DDIM sampling strategy, reducing the 1000-step denoising process to 10 steps. Each step is performed using the formula... Update the hidden state, where P t For the noisy pose at step t, , for The cumulative product, Let c be the noise prediction network function, and c be the multimodal vector.

5. The method for real-time human pose tracking in an XR scene according to claim 1, characterized in that, In step S4, the Kalman filter is used to perform dynamic correction in conjunction with IMU data, which includes real-time measurements from the accelerometer and gyroscope.

6. A real-time human posture tracking system for XR scenes, characterized in that, include: The data acquisition module is used to receive the current frame RGB image, the previous N frames of historical pose sequence and IMU data acquired by the XR device, and transmit the acquired information to the data preprocessing module. The data preprocessing module is used to denoise the obtained RGB image and normalize the historical pose sequence to obtain the corresponding preprocessed data, which is then transmitted to the multimodal conditional fusion module. The multimodal conditional fusion module includes an image encoder, a temporal encoder, and a cross-attention submodule. The image encoder is used to perform feature parsing on RGB images to obtain visual features, and the temporal encoder is used to model the motion trend of historical pose sequences to obtain temporal features. A cross-attention mechanism is introduced to associate the two to generate a multimodal conditional vector, which is then transmitted to the lightweight processing module. The lightweight processing module inputs the obtained multimodal conditional vector into the lightweight diffusion model engine. The lightweight diffusion model engine also includes a temporal consistency constraint module, which analyzes the process from two aspects: hidden state inheritance and loss function optimization, and dedivides the hidden state h of the (t-1)th frame. t-1 Inject birth noise from frame t, and the hidden state h t-1 It includes motion memory of the previous frame's pose, and a joint velocity smoothing term is added to the loss function during model training. Specifically, The first item It constrains the joint velocity v of the current frame. t Compared to the joint velocity v in the previous frame t-1 The differences, among which The value of v is 0.

3. t v is the joint velocity in the current frame. t-1 The joint velocity of the previous frame is used for fast denoising processing to generate a preliminary prediction result of the 3D pose of the current frame, and the obtained preliminary prediction result is transmitted to the post-processing module. The post-processing module is used to perform temporal smoothing on the preliminary prediction results using the Kalman filter algorithm to obtain the final 3D pose and transmit it to the information display module. Information display module, which is used for tracking based on the final 3D pose obtained.

7. The XR scene real-time human posture tracking system according to claim 6, characterized in that, The image encoder is a MobileNetV3 network, the temporal encoder is a GRU network, the output dimension of the cross-attention submodule is 192-dimensional, the lightweight diffusion model engine uses INT8 quantization, the model size is ≤8MB, and the single-frame processing latency is ≤12ms.

8. The XR scene real-time human posture tracking system according to claim 6, characterized in that, The XR device is an AR glasses, VR headset, or MR all-in-one device, and the XR device is equipped with a monocular RGB camera and a 6-axis IMU sensor.

Images