XR scene real-time human body posture tracking method and system

Abstract

The invention discloses a real-time human body posture tracking method and system for an XR scene, relates to the technical field of computer vision, solves the technical problems that a modal barrier of vision and time sequence characteristics cannot be broken through and a diffusion model is degraded, and aims at RGB images, dynamic Gaussian filtering and ROI cutting, noise is restrained, a human body area is focused, and the real-time human body posture tracking effect is achieved. Historical attitude normalization covers space, time and physiological constraints, physical significance consistency of input data is ensured, multi-dimensional and scene-adaptive preprocessing is realized, a data foundation is laid for a subsequent algorithm, a cross attention mechanism breaks through modal boundaries of vision and time sequence features, multi-modal condition vectors are generated, and a multi-modal model is established. And meanwhile, the current limb appearance and the historical motion trend are coded, an accurate reasoning basis is provided for the diffusion model, dynamic and scene-adaptive feature collaboration is realized, a theoretical attitude is constructed based on the first five frames, the theoretical attitude and the diffusion model are preliminarily predicted and dynamically fused, and details are reserved when motion inertia is fit.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for real-time human posture tracking in XR scenarios. Background Art

[0002] In XR interactive scenarios, human posture tracking is a core supporting technology for achieving natural interactions and building immersive experiences. With the widespread adoption of XR devices, scenarios such as virtual fitness, remote collaboration, and immersive gaming have placed higher demands on the accuracy, real-time performance, and robustness of posture tracking.

[0003] Patent application publication number CN117291951A discloses a multi-body posture tracking method based on human key points. The method detects human bodies in a video frame, performs target association matching on the detected human bodies and the tracked targets, screens and filters initial new targets that fail to match, removes false targets, and then uses human key point tracking for real and successfully matched human bodies. The method then expands the tracking area, obtains the coordinates and confidence levels of human key points, determines if there is cross-occlusion between the human bodies and eliminates duplicate tracking, removes the tracked target, and outputs the position and key point information of the tracked target. This process repeats until tracking is complete.

[0004] However, some existing technologies are unable to break through the modal barriers of visual and temporal features, dynamically associate limb contours and joint movement trends, and generate multimodal conditional vectors that accurately guide posture reasoning. At the same time, although the diffusion model has powerful generation capabilities, the original architecture has a slow reasoning speed and is not optimized for the posture degradation mode of XR scenarios. It lacks the integration of the physical laws of human movement. The generated posture may violate physiological common sense and cannot be directly used for interaction. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a real-time human posture tracking method and system for XR scenarios, which solves the problems of being unable to break through the modal barriers of visual and temporal features and the degradation of the diffusion model.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a real-time human posture tracking method for XR scenes, the method specifically comprising the following steps: S1. Obtain the current frame RGB image, the previous N frames of historical posture sequence, and IMU data collected by the XR device, denoise the RGB image, and normalize the historical posture sequence; S2. Extracting visual features of the current frame RGB image through an image encoder, extracting temporal features of the historical posture sequence through a temporal encoder, and dynamically aligning the visual features and temporal features using a cross-attention mechanism to generate a multimodal conditional vector; S3. Inputting the multimodal condition vector into a lightweight diffusion model engine, performing rapid denoising through an inverse diffusion process, and generating a preliminary prediction result of the 3D pose of the current frame. The lightweight diffusion model engine includes a 4-layer U-Net architecture and adopts a 10-step DDIM sampling strategy. S4. Perform Kalman filtering on the preliminary prediction result to smooth the final 3D posture of the current frame.

[0007] As a further solution of the present invention, the value of N in S1 is 5, the historical posture sequence contains 6-degree-of-freedom parameters of 24 joints, and the normalization process is to map the joint coordinates to the range of [-1, 1].

[0008] As a further solution of the present invention, the image encoder adopts the MobileNetV3 network to output 128-dimensional visual features; the temporal encoder adopts the GRU network to output 64-dimensional temporal features; the cross-attention mechanism realizes dynamic alignment of visual features and temporal features through query-key-value mapping, and outputs a 192-dimensional multimodal conditional vector.

[0009] As a further solution of the present invention, the specific method of the reverse diffusion process of the diffusion model engine is as follows: Define the noise scheduling parameters for forward diffusion ∈[1e-4,0.02], T=1000 steps, in the reverse diffusion process, the noise is predicted by 4-layer U-Net , U-Net uses depth-separable convolution and DDIM sampling strategy to reduce the 1000-step denoising process to 10 steps, and each step is calculated by formula Update the hidden state, where P t is the noisy posture at step t, , for The cumulative product of is the noise prediction network, and c is the multimodal vector.

[0010] As a further solution of the present invention, the lightweight diffusion model engine also includes a temporal consistency constraint module, which analyzes the hidden state inheritance and loss function optimization from two aspects, and converts the hidden state h of the t-1 frame back diffusion into t-1 Inject birth noise of frame t, and the hidden state h t-1 Contains the motion memory of the previous frame posture. At the same time, a new joint velocity smoothing term is added to the loss function of model training. Specifically, , where the first Is to ensure the accuracy of noise prediction, the second Is to constrain the current frame joint velocity v t and the joint velocity v of the previous frame t-1 The difference, among which The value of v is 0.3, t is the joint velocity of the current frame, v t-1 is the joint velocity of the previous frame.

[0011] As a further solution of the present invention, the Kalman filter is combined with IMU data for dynamic correction, and the IMU data includes real-time measurement values ​​of the accelerometer and the gyroscope.

[0012] A real-time human posture tracking system for XR scenarios, comprising: The data acquisition module receives the current frame RGB image, the previous N frames of historical posture sequence, and IMU data collected by the XR device, and transmits the collected information to the data preprocessing module; The data preprocessing module is used to perform denoising on the obtained RGB image and normalize the historical posture sequence to obtain the corresponding preprocessed data and transmit it to the multimodal conditional fusion module; The multimodal conditional fusion module includes an image encoder, a temporal encoder, and a cross-attention submodule. The image encoder is used to parse the RGB image to obtain visual features, and the temporal encoder is used to model the motion trend of the historical posture sequence to obtain temporal features. The cross-attention mechanism is introduced to associate the two to generate a multimodal conditional vector, which is then transmitted to the lightweight processing module. A lightweight processing module, which is used to input the obtained multimodal condition vector into the lightweight diffusion model engine and perform fast denoising to generate a preliminary prediction result of the current frame's 3D pose. At the same time, the obtained preliminary prediction result is transmitted to the post-processing module; The post-processing module is used to perform temporal smoothing on the preliminary prediction results using the Kalman filter algorithm to obtain the final 3D pose and transmit it to the information display module; The information display module is used to track according to the final 3D posture obtained.

[0013] As a further solution of the present invention, the image encoder is a MobileNetV3 network, the temporal encoder is a GRU network, the output dimension of the cross-attention submodule is 192 dimensions, the lightweight diffusion model engine adopts INT8 quantization processing, the model size is ≤8MB, and the single-frame processing delay is ≤12ms.

[0014] As a further solution of the present invention, the XR device is AR glasses, VR headset or MR all-in-one machine, and the XR device is equipped with a monocular RGB camera and a 6-axis IMU sensor.

[0015] The present invention provides a method and system for real-time human posture tracking in XR scenarios. Compared with the existing technology, it has the following advantages: The present invention suppresses noise and focuses on the human body area through dynamic Gaussian filtering and ROI cropping for RGB images. Historical posture normalization covers spatial, temporal and physiological constraints to ensure the consistency of physical meaning of input data, realizes multi-dimensional and scene-adaptive preprocessing, and lays a data foundation for subsequent algorithms. The cross-attention mechanism breaks the modal boundary between visual and temporal features, generates multimodal conditional vectors, and simultaneously encodes the current limb appearance and historical movement trends, providing accurate reasoning basis for the diffusion model and realizing dynamic and scene-adaptive feature collaboration.

[0016] The 4-layer U-Net and depthwise separable convolution of the present invention reduce the number of model parameters. The DDIM sampling strategy compresses the thousand-step reasoning to 10 steps and delays single-frame processing. The temporal consistency module enforces posture reasoning to conform to the laws of human motion through hidden state inheritance and joint velocity smoothing loss analysis. The general diffusion model has no physiological constraints and is easy to generate joint teleportation and postures that violate mechanics, making the output posture more natural and reducing inter-frame jitter. The theoretical posture is constructed based on the first 5 frames and dynamically fused with the preliminary prediction of the diffusion model. Details are retained when it matches the motion inertia, and the trajectory is corrected when abnormal jumps occur. The final output 3D posture can resist single-frame noise and ensure movement continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a diagram of the steps and methods of the present invention; Figure 2 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] Example 1

[0020] See also Figure 1 , this application provides a method for real-time human posture tracking in XR scenes, which specifically includes the following steps: Step 1: Use the heterogeneous sensor array built into the XR device to acquire three types of data in real time. These include an RGB image stream (captured at 30fps by a monocular high-definition camera (such as the MetaQuest3's 12-megapixel RGB camera), with a resolution of 1920×1080 and a 90° field of view, covering the main joint areas of the upper body), a historical pose sequence (caching the human pose parameters of the last five frames (each frame contains the 3D coordinates and quaternion rotation parameters of 24 key joints). For example, in a virtual fitness scene, the user's punching poses from the previous five frames are stored), and an IMU data stream (a 6-axis inertial measurement unit (such as the ICM-42688-P) outputs acceleration (±16g) and angular velocity (±2000° / s) data at a 1000Hz sampling rate). The resulting RGB image is denoised to obtain a preprocessed RGB image, and the historical pose sequence is normalized to obtain a preprocessed historical pose sequence. The specific denoising of RGB images includes time domain filtering denoising and region of interest extraction, among which time domain filtering denoising adopts adaptive Gaussian filtering algorithm ( =0.5-1.5 dynamic adjustment), suppressing random noise while maintaining edge features; Region of Interest Extraction predicts the human body position in the current frame based on historical postures, and dynamically crops an 800×800 pixel ROI area to reduce background interference.

[0021] Normalization of historical posture sequences includes spatial normalization, time window alignment, and outlier correction; The spatial normalization specifically projects the joint coordinates into the unit sphere space to eliminate the influence of user height differences; Time window alignment uses a sliding window mechanism to synchronize RGB frames with IMU data, and resamples the IMU's 1000Hz data to 30Hz through linear interpolation to ensure timestamp consistency; Outlier correction means detecting and replacing joint angles that exceed physiological limits (such as elbow flexion exceeding 180°, which is specifically used to effectively avoid posture jumps caused by violent movements in dance training scenarios).

[0022] Step 2: Use a lightweight image encoder (such as the optimized MobileNetV3) to perform feature analysis on the preprocessed RGB image. Specifically, 128-dimensional core features are extracted from the image, which are recorded as visual features. These include edge gradients of limb contours (such as the change in the contour when the arm is swinging), texture information around joints (such as the relationship between elbow wrinkles and joint angles), and illumination-adaptive brightness features (such as shoulder shadows that can still be recognized in strong light). At the same time, a temporal encoder (such as a single-layer GRU network) is used to model the motion trend of the previous 5-frame historical posture sequence to generate 64-dimensional temporal features, which are recorded as temporal features. The specific processing method is as follows: Capture the dynamic laws of limb movement, including joint velocity (such as the change in wrist acceleration from static to swinging), movement continuity (such as the coordinated movement rhythm of the shoulder-elbow-wrist in a punching action), and inertial characteristics of movement direction (such as the natural tendency of the arm to fall after being raised).

[0023] A cross-attention mechanism is introduced to realize the intelligent association of visual features and temporal features, generating a 192-dimensional multimodal conditional vector. Specifically, the mechanism focuses on the associated area through "dynamic weight allocation": when the visual feature shows "hand close to the head", the weight of "shoulder rotation" in the temporal feature will be increased, so that the two can form a synergy in the "arm raising" action; conversely, if the temporal feature shows "legs moving quickly", the weight of the "knee area" in the visual feature will be enhanced.

[0024] Step 3: Input the resulting multimodal conditional vector into the lightweight diffusion model engine. The core of the diffusion model is to simulate the process of "posture degradation and recovery." In reality, occlusion (such as crossed arms blocking the abdomen) and motion blur (such as image smearing caused by rapid arm swings) can blur posture information, like "noise contamination." The inverse diffusion process acts like "intelligent noise reduction"—using the multimodal conditional vector as a "clue," it gradually removes noise, restores the obscured true posture, and performs denoising through the inverse diffusion process. The inverse diffusion process specifically includes the following: Define the noise scheduling parameters for forward diffusion ∈[1e-4,0.02], T=1000 steps, initial step (small ), only adding weak noise, retaining the basic outline of the posture (such as simulating a scene with slight motion blur), the subsequent steps (large ), gradually increasing the noise to make the pose data approach completely random (such as simulating pose loss caused by heavy occlusion); The core of the reverse diffusion process is to predict the noise in the current noisy posture through a 4-layer lightweight U-Net To achieve gradual denoising, we use depthwise separable convolution instead of traditional convolution. The number of channels is dynamically adjusted according to [32, 64, 128, 64]. The shallow layer captures basic features such as joint positions, and the deep layer focuses on limb movement associations (such as shoulder-elbow-wrist collaborative constraints). The input is a noisy posture P. t , current time step t, multimodal condition vector, output prediction noise , using the DDIM sampling strategy, the 1000-step denoising process is reduced to 10 steps, and each step is calculated by formula Update the hidden state, where P tis the noisy posture at step t, , for The cumulative product of is the noise prediction network, c is the multimodal vector; Among them, the lightweight diffusion model engine introduces a temporal consistency constraint module, analyzes from two aspects: hidden state inheritance and loss function optimization, and transforms the hidden state h of the reverse diffusion of the t-1 frame into t-1 Inject birth noise of frame t, and the hidden state h t-1 Contains the motion memory of the previous frame posture (such as arm swing direction, joint rotation trend), and adds a joint velocity smoothing term to the loss function of model training, specifically, , where the first Is to ensure the accuracy of noise prediction, the second Is to constrain the current frame joint velocity v t and the joint velocity v of the previous frame t-1 The difference, among which The value of v is 0.3, t is the joint velocity of the current frame, v t-1 is the joint velocity of the previous frame.

[0025] Step 4: Generate a preliminary prediction of the 3D pose of the current frame based on the diffusion model and perform temporal smoothing through Kalman filtering. The specific processing method is as follows: Based on the posture dynamics model built from the previous five frames, motion trends are accurately analyzed—for example, extracting the continuous changes in velocity and acceleration from arm swing data to deduce the "theoretical posture" of the current frame. When the diffusion model outputs preliminary results (due to interference such as image blur, there may be transient jitter such as jumps in joint position), "dual-state fusion calibration" is initiated: If the preliminary results match the historical motion inertia (such as the arm continuing its upward trajectory), a high confidence weight is assigned to retain posture features such as fingertip details and muscle stretching. Once an abnormal jump is detected (such as a reverse offset of the joint without physical basis), the theoretical posture is used as an anchor point, and dynamic weight distribution is used to weaken the jump noise and correct the posture trajectory, so that the joint movement returns to a smooth curve that conforms to human physiology, thereby suppressing inter-frame jitter from the root.

[0026] Example 2

[0027] See also Figure 2 This application provides a real-time human posture tracking system for XR scenes, which includes: a data acquisition module, a data preprocessing module, a multimodal condition fusion module, a lightweight processing module, a post-processing module and an information display module, and is combined with Figure 2 It can be known that the information between the above functional modules is transmitted in one direction.

[0028] The data acquisition module is used to receive the current frame RGB image, the previous N frames of historical posture sequence and IMU data collected by the XR device, and transmit the collected information to the data preprocessing module.

[0029] The data preprocessing module is used to denoise the obtained RGB image and normalize the historical posture sequence to obtain the corresponding preprocessed data, and transmit it to the multimodal conditional fusion module. The specific processing method is similar to the processing process in step 1; The multimodal conditional fusion module includes an image encoder, a temporal encoder, and a cross-attention submodule. The image encoder is used to perform feature analysis on the RGB image to obtain visual features, and the temporal encoder is used to perform motion trend modeling on the historical posture sequence to obtain temporal features. A cross-attention mechanism is introduced to associate the two to generate a multimodal conditional vector, which is then transmitted to the lightweight processing module. The specific processing method is similar to that of step 2. The lightweight processing module is used to input the obtained multimodal condition vector into the lightweight diffusion model engine and perform fast denoising to generate a preliminary prediction result of the current frame's 3D pose. At the same time, the obtained preliminary prediction result is transmitted to the post-processing module. The specific processing method is similar to the processing process in step 3; The post-processing module is used to perform temporal smoothing on the preliminary prediction results using the Kalman filter algorithm to obtain the final 3D pose and transmit it to the information display module. The specific processing method is similar to that of step 4. The information display module is used to track according to the final 3D posture obtained.

[0030] Some of the data in the above formulas are calculated based on their numerical values ​​and are not substituted into parameter units for calculation. At the same time, the contents not described in detail in this specification belong to the existing technology known to those skilled in the art.

[0031] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A real-time human posture tracking method for XR scenes, characterized by: The method specifically comprises the following steps: S1. Obtain the current frame RGB image, the previous N frames of historical posture sequence, and IMU data collected by the XR device, denoise the RGB image, and normalize the historical posture sequence; S2. Extracting visual features of the current frame RGB image through an image encoder, extracting temporal features of the historical posture sequence through a temporal encoder, and dynamically aligning the visual features and temporal features using a cross-attention mechanism to generate a multimodal conditional vector; S3. Input the multimodal condition vector into the lightweight diffusion model engine, perform rapid denoising through the inverse diffusion process, and generate a preliminary prediction result of the current frame's 3D pose. The lightweight diffusion model engine includes a 4-layer U-Net architecture and adopts a 10-step DDIM sampling strategy. S4. Perform Kalman filtering on the preliminary prediction results to smooth the final 3D pose of the current frame.

2. The method for real-time human posture tracking in XR scenes according to claim 1, characterized in that: The value of N in S1 is 5, the historical posture sequence contains 6-degree-of-freedom parameters of 24 joints, and the normalization process is to map the joint coordinates to the range of [-1, 1].

3. The method for real-time human posture tracking in XR scenes according to claim 1, characterized in that: In step S2, the image encoder adopts the MobileNetV3 network to output 128-dimensional visual features; the temporal encoder adopts the GRU network to output 64-dimensional temporal features; the cross-attention mechanism realizes dynamic alignment of visual features and temporal features through query-key-value mapping, and outputs a 192-dimensional multimodal conditional vector.

4. The method for real-time human posture tracking in XR scenes according to claim 1, characterized in that: In step S3, the reverse diffusion process of the diffusion model engine is specifically as follows: Define the noise scheduling parameters for forward diffusion ∈[1e-4,0.02], T=1000 steps, in the reverse diffusion process, the noise is predicted by 4-layer U-Net , U-Net uses depth-separable convolution and DDIM sampling strategy to reduce the 1000-step denoising process to 10 steps, and each step is calculated by formula Update the hidden state, where P t is the noisy posture at step t, , for The cumulative product of is the noise prediction network, and c is the multimodal vector.

5. The method for real-time human posture tracking in XR scenes according to claim 1, characterized in that: In step S3, the lightweight diffusion model engine also includes a temporal consistency constraint module, which analyzes the hidden state inheritance and loss function optimization from two aspects, and converts the hidden state h of the reverse diffusion of the t-1 frame into t-1 Inject birth noise of frame t, and the hidden state h t-1 Contains the motion memory of the previous frame posture. At the same time, a new joint velocity smoothing term is added to the loss function of model training. Specifically, , where the first Is to ensure the accuracy of noise prediction, the second Is to constrain the current frame joint velocity v t and the joint velocity v of the previous frame t-1 The difference, among which The value of v is 0.3, t is the joint velocity of the current frame, v t-1 is the joint velocity of the previous frame.

6. The method for real-time human posture tracking in XR scenes according to claim 1, characterized in that: In step S4, the Kalman filter is dynamically corrected in combination with IMU data, where the IMU data includes real-time measurements of an accelerometer and a gyroscope.

7. A real-time human posture tracking system for XR scenes, characterized by: include: The data acquisition module receives the current frame RGB image, the previous N frames of historical posture sequence, and IMU data collected by the XR device, and transmits the collected information to the data preprocessing module; The data preprocessing module is used to perform denoising on the obtained RGB image and normalize the historical posture sequence to obtain the corresponding preprocessed data and transmit it to the multimodal conditional fusion module; The multimodal conditional fusion module includes an image encoder, a temporal encoder, and a cross-attention submodule. The image encoder is used to parse the RGB image to obtain visual features, and the temporal encoder is used to model the motion trend of the historical posture sequence to obtain temporal features. The cross-attention mechanism is introduced to associate the two to generate a multimodal conditional vector, which is then transmitted to the lightweight processing module. A lightweight processing module, which is used to input the obtained multimodal condition vector into the lightweight diffusion model engine and perform fast denoising to generate a preliminary prediction result of the current frame's 3D pose. At the same time, the obtained preliminary prediction result is transmitted to the post-processing module; The post-processing module is used to perform temporal smoothing on the preliminary prediction results using the Kalman filter algorithm to obtain the final 3D pose and transmit it to the information display module; The information display module is used to track according to the final 3D posture obtained.

8. The XR scene real-time human posture tracking system according to claim 7, characterized in that: The image encoder is a MobileNetV3 network, the temporal encoder is a GRU network, the output dimension of the cross-attention submodule is 192 dimensions, the lightweight diffusion model engine adopts INT8 quantization processing, the model size is ≤8MB, and the single-frame processing delay is ≤12ms.

9. The XR scene real-time human posture tracking system according to claim 7, characterized in that: The XR device is AR glasses, VR headset or MR all-in-one device, and the XR device is equipped with a monocular RGB camera and a 6-axis IMU sensor.

Images