Extended reality display control method, system, and device based on biomechanical simulation

Abstract

The application relates to an extended reality display control method, system and device based on biomechanical simulation, which introduces a biomechanical model based on user physiological characteristics, constructs motion expectations conforming to individual differences, quantifies sensory conflicts by using the difference between expected postures and visual flow, and actively intervenes in the dizziness mechanism from the root. In addition, the model parameters are adjusted combined with eye movement data feedback, which solves the problems of poor adaptability and passive compensation lag of traditional general models while ensuring the computing efficiency of the mobile terminal, effectively reduces the dizziness of the user, and prolongs the use time of the device.

Description

Extended Reality Display Control Method, System, and Device Based on Biomechanical Simulation Technical Field

[0001] This application relates to the field of extended reality human-computer interaction and biomechanical simulation technology, and in particular to an extended reality display control method, system and device based on biomechanical simulation. Background Technology

[0002] Extended Reality (XR) technology, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), is gradually expanding from simple entertainment applications to fields such as industrial simulation, medical rehabilitation, and education and training. However, in the process of popularizing XR technology, motion sickness (or cybersickness) experienced by users when using head-mounted display devices remains one of the main bottlenecks hindering its long-term use and widespread adoption.

[0003] The physiological causes of XR dizziness are complex. The core cause currently recognized by the academic community is the Sensory Conflict Theory, which states that the visual flow signals received by users in the virtual environment are inconsistent with the actual body movement signals received by the vestibular system and proprioception. This mismatch of sensory information causes confusion in the brain, which in turn leads to discomfort symptoms such as nausea, dizziness, and sweating.

[0004] To alleviate this problem, existing technologies mainly employ the following solutions:

[0005] First, the latency of motion-to-photon can be reduced through hardware upgrades and algorithm optimization, such as using high refresh rate screens and asynchronous spacewarp (ASW) technology. This approach mainly focuses on eliminating dizziness caused by system lag, but it requires high hardware computing power, making it difficult to implement perfectly on mobile XR devices. Furthermore, it cannot resolve sensory conflicts caused by non-latency factors, such as virtual movement while the body remains still.

[0006] Second, a static frame of reference is introduced into the user's field of vision, such as a virtual nose, a cockpit, or a dynamically narrowed field of view (FOV, i.e., tunneling effect) during movement. While this method can alleviate dizziness to some extent, it significantly sacrifices the user's immersion and visual experience, greatly reducing the realism of the virtual environment.

[0007] Third, inertial measurement units (IMUs) are used to predict head movements. However, existing motion prediction algorithms are usually based on a general rigid body motion model, treating all users as the same standard model and ignoring the significant differences in physiological structure among different users.

[0008] In summary, existing XR vertigo suppression solutions generally suffer from the following technical shortcomings:

[0009] Lack of personalized adaptation: The influence of the user's physiological parameters on motion perception is not considered, resulting in a deviation between the motion prediction model and the user's actual proprioception. This deviation will accumulate and exacerbate dizziness during long-term use.

[0010] Passive response: Most solutions passively compensate after detecting violent movement or a delay has occurred, lacking active posture prediction based on biomechanical principles.

[0011] Open-loop control: Existing systems typically cannot sense the user's dizziness in real time, nor can they dynamically adjust the intensity of anti-dizziness strategies based on the user's real-time reactions. This results in adjustments that are either too aggressive and disrupt the experience, or insufficient to suppress dizziness. Summary of the Invention

[0012] To address the aforementioned issues, this application provides an extended reality display control method, system, and device based on biomechanical simulation to suppress XR dizziness with low computational consumption.

[0013] To achieve the above objectives, in a first aspect, the extended reality display control method based on biomechanical simulation provided in this application includes the following steps:

[0014] S100: Acquires the user's physiological characteristic data and collects the head display pose data and eye movement data when the user wears the extended reality device;

[0015] S200: Configure the physiological characteristic data as parameters of the biomechanical model, and input the head display pose data and eye movement data into the biomechanical model to calculate the user's expected body posture data at the current moment;

[0016] S300: Calculate the difference between the expected body posture data and the visual stream data of the currently rendered screen of the extended reality device, and generate a sensory conflict signal;

[0017] S400: In response to the sensory conflict signal, adjust the image rendering parameters of the extended reality device according to a preset strategy, and drive the display unit to display.

[0018] Preferably, the step of calculating the user's expected body posture data at the current moment includes:

[0019] Based on the height and arm length data in the physiological characteristic data, the joint node coordinates and bone length of the virtual skeleton in the biomechanical model are set;

[0020] The head-mounted display pose data and eye-tracking data are subjected to temporal filtering and then used as constraint terms to input the inverse kinematics solver based on the virtual skeleton.

[0021] The head rotation increment is propagated along the kinematic chain from the cervical spine to the trunk, and the gaze direction in the eye movement data is used as the preferred weight term for upper body rotation. Combined with the preset physiological range constraint of joint angles, the angles of each joint of the skeleton are solved.

[0022] Based on the solved joint angles, the expected body posture data, including torso orientation and center of gravity displacement trend, is output.

[0023] Preferably, in step S300, the step of calculating the difference value and generating a sensory conflict signal includes:

[0024] Extract the first motion vector of the expected body posture data and the second motion vector of the virtual camera corresponding to the current rendered screen, respectively.

[0025] The first motion vector and the second motion vector are normalized, and the directional angle between them is calculated.

[0026] Calculate the acceleration difference between the instantaneous acceleration corresponding to the first motion vector and the instantaneous acceleration corresponding to the second motion vector;

[0027] The initial conflict value is obtained by performing a weighted summation operation based on the angle of direction and the difference in acceleration.

[0028] The initial conflict value is subjected to time-domain smoothing filtering, and the processed value is mapped to a preset quantization range as the sensory conflict signal.

[0029] Preferably, in step S400, the step of adjusting the image rendering parameters of the extended reality device includes:

[0030] When the value corresponding to the sensory conflict signal exceeds a preset first threshold, at least one of the following adjustment strategies is executed:

[0031] Based on the intensity of the sensory conflict signal, the current rendering field of view is reduced proportionally, wherein the reduction ratio is positively correlated with the sensory conflict signal;

[0032] Increase the motion damping coefficient of the virtual camera used to generate the rendered image so that the motion speed of the rendered image lags behind the change in the head-mounted display pose.

[0033] Overlay a progressive vignetting effect on the edges of the rendered image, or reduce the visual weight of dynamic elements in the rendered scene.

[0034] Preferably, the method further includes a feedback adjustment step:

[0035] Real-time monitoring of pupil diameter data in the eye movement data, and calculation of the rate of change of pupil diameter;

[0036] When the rate of change of the pupil diameter continues to a preset positive threshold within a first preset time period, it is determined that the user is in a state of high risk of dizziness, and the calculation weight of generating the sensory conflict signal is increased, or the adjustment range of the image rendering parameters is increased.

[0037] When the rate of change of the pupil diameter remains below a preset negative threshold for a second preset duration, it is determined that the user is in a recovery state, and the calculation weight for generating the sensory conflict signal is reduced, or the adjustment range of the image rendering parameters is reduced.

[0038] Preferably, before step S200, the method further includes a model initialization calibration step:

[0039] Guide the user to complete a preset standard movement sequence, which includes at least neck extreme rotation movement and torso swinging movement;

[0040] Collect the user's pose data when performing the standard action sequence to determine the actual range of motion boundaries and motion preference parameters of each joint of the user;

[0041] The joint constraints of the biomechanical model are modified using the actual range of motion boundaries and motion preference parameters.

[0042] Preferably, the biomechanical model is a neural network model or a kinematic chain model based on physical equations, and step S200 is executed in the local processor of the extended reality device, and the duration of a single calculation is less than a preset time threshold.

[0043] Secondly, embodiments of this application also provide an extended reality display system based on biomechanical simulation, comprising:

[0044] The data acquisition module is used to acquire the user's physiological characteristic data, and to collect the head-mounted display posture data and eye movement data when the user wears the extended reality device;

[0045] The posture prediction module is used to configure the physiological feature data as parameters of the biomechanical model, and input the head display pose data and eye movement data into the biomechanical model to calculate the user's expected body posture data at the current moment.

[0046] The conflict detection module is used to calculate the difference between the expected body posture data and the visual stream data of the currently rendered screen of the extended reality device, and generate a sensory conflict signal.

[0047] The rendering control module is used to respond to the sensory conflict signal, adjust the image rendering parameters of the extended reality device according to a preset strategy, and drive the display unit to display the image.

[0048] Preferably, it further includes:

[0049] The feedback adjustment module is used to monitor the pupil diameter data in the eye movement data in real time and calculate the rate of change of the pupil diameter. When the rate of change of the pupil diameter continuously exceeds a preset positive threshold within a first preset time period, it is determined that the user is in a high-risk state of dizziness, and the calculation weight of generating the sensory conflict signal is increased, or the adjustment range of the image rendering parameters is increased. When the rate of change of the pupil diameter continuously falls below a preset negative threshold within a second preset time period, it is determined that the user is in a recovery state, and the calculation weight of generating the sensory conflict signal is reduced, or the adjustment range of the image rendering parameters is decreased.

[0050] Thirdly, embodiments of this application provide an extended reality device, including:

[0051] The device includes a display, an eye-tracking sensor, a memory, and a processor; the memory stores a computer program, and the processor executes the computer program to implement the extended reality display control method based on biomechanical simulation as described in any embodiment of the first aspect.

[0052] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the extended reality display control method based on biomechanical simulation as described in any embodiment of the first aspect.

[0053] The extended reality display control method, system, and device designed in this application, based on biomechanical simulation, constructs motion expectations that conform to individual differences by introducing a biomechanical model based on the user's physiological characteristics. It then quantifies sensory conflict by utilizing the difference between the expected posture and visual flow, achieving proactive intervention at the root of the dizziness mechanism. Furthermore, by combining eye-tracking data feedback to adjust model parameters, this solution addresses the problems of poor adaptability and lag in passive compensation of traditional general-purpose models while ensuring mobile computing efficiency, effectively reducing user dizziness and extending device usage time. Attached Figure Description

[0054] Figure 1 is a flowchart illustrating the extended reality display control method based on biomechanical simulation provided in an embodiment of this application. Detailed Implementation

[0055] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application.

[0056] Firstly, this embodiment provides an extended reality (XR) display control method based on biomechanical simulation. This method is mainly applied to terminal devices such as head-mounted virtual reality (VR) devices, augmented reality (AR) glasses, or mixed reality (MR) headsets. This method actively predicts the user's body posture by constructing a personalized biomechanical model locally, and dynamically compensates for this posture in the rendered image, thereby suppressing dizziness.

[0057] Referring to Figure 1, the method specifically includes the following steps:

[0058] S100: Acquires the user's physiological characteristic data and collects the head-mounted display pose data and eye movement data when the user wears the extended reality device.

[0059] In this embodiment, physiological feature data is mainly used to construct a personalized model. Specifically, the acquired height and arm length data can come from user input settings, imported from scans of external devices, or be inferred from the viewpoint height.

[0060] Meanwhile, by using the inertial measurement unit (IMU) and visual positioning system (SLAM) built into the extended reality device, the head display pose data is collected in real time, such as the head's position coordinates and Euler angles in three-dimensional space: pitch, yaw, and roll. In addition, eye-tracking sensors integrated into the device are used to collect the user's eye movement data in real time, which includes at least the gaze direction and pupil diameter.

[0061] In some embodiments, the method further includes a model initialization calibration step:

[0062] First, the user is guided to complete a preset standard action sequence, which includes at least a neck rotation limit movement and a torso swing movement. For example, when a new user wears the device for the first time or when the user actively triggers calibration, a calibration guidance interface is loaded in the virtual field of view. The extended reality device guides the user to complete the preset standard action sequence through holographic demonstrations or arrow indicators. In this embodiment, the standard action sequence includes at least:

[0063] Neck limit test: For example, prompting users "Please turn your head to the left as far as possible, and then turn your head to the right as far as possible", and "Please try to raise and lower your head as much as possible".

[0064] Trunk swing test: For example, prompt the user to "keep your head facing forward and naturally step in place or turn your upper body left and right".

[0065] Subsequently, during the user's execution of the above actions, pose data is continuously collected as the user performs the standard action sequence to determine the actual range of motion boundaries and motion preference parameters of each joint. For example, the maximum yaw angle of the user's neck to the left is measured to be 55 degrees, and to the right to be 60 degrees, and the lag time or swing amplitude ratio of the torso relative to the head is measured when the user is naturally stepping.

[0066] Finally, the joint constraints of the biomechanical model are modified using the actual range of motion boundaries and motion preference parameters. For example, the rotational constraint range of the cervical joint in the model is updated from the default [-70°, +70°] to the measured [-55°, +60°], thereby providing more accurate posture predictions in subsequent steps.

[0067] S200: Configure the physiological characteristic data as parameters of the biomechanical model, and input the head display pose data and eye movement data into the biomechanical model to calculate the user's expected body posture data at the current moment.

[0068] In this embodiment, the biomechanical model is preferably a lightweight model running on the local processor of the extended reality device, such as the NPU or DSP of an XR chip, with a single inference or computation time of less than a preset time threshold, such as 3ms, to meet the high frame rate requirements of real-time rendering. The biomechanical model can be a pre-trained small neural network model or a kinematic chain model based on rigid body dynamics equations.

[0069] In practice, if a small neural network model is used, the training data sources mainly include: publicly available biomechanical datasets, which use real human motion capture data to extract the correspondence between head-mounted display pose and body posture as training samples; or data generated through physical simulation, which uses a physics engine to simulate the natural movement of human skeletons under gravity and physiological constraints, and generates synthetic data in batches.

[0070] The network is trained using the aforementioned data to learn the nonlinear mapping relationship from head and eye movement inputs to trunk posture outputs, and then deployed after model quantization pruning.

[0071] If a kinematic chain model based on physical equations is adopted, the public dataset or simulation data is used for parameter calibration. That is, by statistically analyzing the motion patterns in the dataset, key parameters in the physical equations, such as joint stiffness coefficients and motion coupling ratios, are optimized, so that the calculation results of the mathematical equations are more consistent with the statistical laws of real human motion.

[0072] Specifically, the steps for calculating the user's expected body posture data at the current moment include:

[0073] S201: Based on the height and arm length data in the physiological characteristic data, set the joint node coordinates and bone length of the virtual skeleton in the biomechanical model.

[0074] In practice, the height and arm length data obtained in step S100 are used as prior knowledge to parametrically construct the virtual skeleton in the biomechanical model. Specifically, the length of the spine (including cervical, thoracic, and lumbar vertebrae) is calculated based on the height data according to ergonomic proportions, such as the head-to-body ratio, and the lever arm lengths from the shoulder joint to the elbow and from the elbow to the wrist are determined based on the arm length data. This allows for the determination of the basic coordinates of the joint nodes and the length constraints of the connecting rods in the three-dimensional space of the virtual skeleton.

[0075] S202: Perform temporal filtering on the head-mounted display pose data and eye-tracking data, and input them as constraint terms into the inverse kinematics solver based on the virtual skeleton.

[0076] In practice, the raw data collected by the sensors, such as attitude and eye-tracking data (gaze direction vector) represented by three-dimensional position coordinates and quaternions or Euler angles, undergoes short-time filtering. In this embodiment, Kalman filtering or complementary filtering algorithms are preferably used to smooth abrupt changes in sensor data, remove measurement noise, and thus obtain a more stable input state quantity.

[0077] S203: Propagate the head rotation increment along the kinematic chain from the cervical spine to the trunk, and use the gaze direction in the eye movement data as the preferred weight term for upper body rotation. Combined with the preset physiological range constraint of joint angles, solve the angle of each joint of the skeleton.

[0078] In practice, the filtered head-mounted display pose is used as a constraint term for the end effector and input into the skeleton-based inverse kinematics (IK) solver. The solution process follows the following biomechanical logic:

[0079] Kinematic chain propagation: The incremental rotation of the head propagates downwards along the biokinematic chains of the cervical and thoracic vertebrae and trunk. This means that when the head rotates, the model simulates how this rotation decays stepwise and propagates to the trunk.

[0080] Fixation point preference constraint: The gaze direction in eye-tracking data is used as a preference or weighting factor for upper body rotation. For example, when the model detects that the user's head is turning to the left and their gaze is also strongly directed to the left, the model infers that the torso is more likely to follow suit and rotate to the left; conversely, if the gaze remains centered, the torso rotation may be smaller.

[0081] Physiological range constraints: The solver presets physiological range constraints for each joint angle. For example, the horizontal rotation angle of the neck relative to the torso is limited to ±70 degrees to avoid generating postures that violate human anatomy. In this embodiment, the physiological range of each joint angle can be obtained based on authoritative medical or ergonomic standard databases. In another preferred embodiment, the physiological range constraints can also be obtained when the user first wears the device or when the user actively triggers calibration. For example, during initial use, the user is guided to perform maximum head and body turning movements, and the joint limit angles of that specific user are recorded in real time, thereby replacing the aforementioned default standard database parameters and achieving more precise personalized constraints.

[0082] S204: Based on the solved joint angles, output the expected body posture data, including torso orientation and center of gravity displacement trend. After considering the above constraints, construct the objective function, and preferably use the least squares optimization algorithm to solve for the optimal angle of each joint, so that the head end pose of the model is as consistent as possible with the sensor observations, while minimizing the energy generated by the joint angle changes, and must satisfy the above physiological range constraints.

[0083] Through the above calculations, the final output includes expected body posture data, including torso orientation, estimated shoulder and hip angles, and the displacement trend of the body's center of gravity. This data can accurately reflect the user's true body state based on biological instincts under the current head-mounted display movement and gaze state.

[0084] S300: Calculate the difference between the expected body posture data and the visual stream data of the currently rendered screen of the extended reality device, and generate a sensory conflict signal.

[0085] This step aims to quantify the degree of inconsistency between the biomechanically anticipated bodily sensations and the visual input presented by the extended reality device. This process is typically performed in each frame rendering loop, and the specific workflow is as follows:

[0086] S301: Extract the first motion vector of the expected body posture data and the second motion vector of the virtual camera corresponding to the current rendered screen.

[0087] First, a first motion vector v1 is extracted from the expected body posture data output in step S200. In a preferred embodiment, this first motion vector can be the expected acceleration vector of the body's center of gravity calculated based on a biomechanical model, or the velocity vector of the torso orientation.

[0088] Simultaneously, the second motion vector v2 of the virtual camera corresponding to the current frame or the next frame is obtained from the graphics rendering pipeline. This vector reflects the user's viewpoint motion state in the virtual scene, such as the real-time motion velocity vector or acceleration vector of the Camera component in game engines like Unity or Unreal Engine.

[0089] S302: Normalize the first motion vector and the second motion vector to eliminate the influence of different dimensions and velocity magnitudes, and calculate the directional angle θ between them.

[0090] The specific calculation formula is as follows:

[0091] θ=arccos(clamp(v1·v2,—1,1))

[0092] Here, v1·v2 represents the dot product of two vectors, and the clamp function is used to restrict the value to the interval [-1,1] to prevent domain overflow caused by numerical calculation errors. In this way, the included angle θ intuitively reflects whether the expected direction of body movement is consistent with the direction of movement of the seen image; for example, if the body is expected to move forward, but the image moves backward, the included angle is close to π, and the conflict is the greatest.

[0093] S303: Calculate the acceleration difference Δa between the instantaneous acceleration corresponding to the first motion vector v1 and the instantaneous acceleration corresponding to the second motion vector v2.

[0094] Δa=|a1—a2|

[0095] Where a1 is the instantaneous acceleration corresponding to the first motion vector v1, and a2 is the instantaneous acceleration corresponding to the second motion vector v2.

[0096] S304: A weighted summation operation is performed based on the included direction angle θ and the acceleration difference Δa to obtain the initial conflict value C. The specific calculation formula is as follows:

[0097]

[0098] Among them, the adjustable weighting coefficients α and β, preferably α=60 and β=40, are used to balance the contribution of directional conflict and acceleration conflict to the total dizziness. Preferably, the reference acceleration constant used for normalization is... Set to 1m / s 2 Or the component of gravitational acceleration g, used to eliminate the influence of dimensions.

[0099] S305: Perform time-domain smoothing filtering on the initial conflict value and map the processed value to a preset quantization range as the sensory conflict signal.

[0100] Preferably, the Exponential Moving Average (EMA) algorithm is used:

[0101] C smoothed =λ·C smoothed_prev +(1—λ)·C

[0102] Among them, C smoothed C is the smoothed value of the current frame. smoothed_prev λ is the smoothing value of the previous frame, and λ is the smoothing factor, for example, 0.8, used to suppress transient noise.

[0103] Finally, the smoothed values ​​are mapped to a preset quantization range, such as a 0-100 percentage scale, to serve as the final output sensory conflict signal. The higher the signal value, the higher the user's current risk of dizziness, and the stronger the necessity for intervention.

[0104] S400: In response to the sensory conflict signal, adjust the image rendering parameters of the extended reality device according to a preset strategy, and drive the display unit to display.

[0105] Based on the sensory conflict signal C generated in step S300, this step takes graded or continuous intervention measures.

[0106] Specifically, a first threshold is preset. When the detected sensory conflict signal C exceeds this threshold, it indicates that the user is in a dizziness risk zone. Rendering compensation will then be performed, and depending on the actual application scenario, such as VR games or AR navigation, a combination of one or more of the following adjustment strategies will be selectively executed:

[0107] Strategy 1: Based on the intensity of the sensory conflict signal C, proportionally reduce the current rendering field of view (FOV), where the reduction ratio is positively correlated with the sensory conflict signal. The calculation formula is as follows:

[0108] FOV new =FOV base ·(1—k1·C normalized )

[0109] Among them, FOV base C is the basic field of view of the equipment. normalized The normalized conflict value is represented by k1, which is a scaling factor ranging from [0, 0.3]. This means that the more intense the conflict, the greater the area of ​​the visual field edge that is clipped or occluded, thus forcing the user to focus on the relatively stable central area.

[0110] Strategy Two: Increase the motion damping coefficient of the virtual camera used to generate the rendered image, so that the motion speed of the rendered image lags behind the changes in the head-mounted display's pose. An example of the calculation formula is as follows:

[0111] Vel new =Vel curr ·(1—k2·C normalized )

[0112] Among them, Vel curr The camera speed should have been in the current frame; k2 is the damping strength coefficient, with a value between [0, 0.7]. This strategy filters out or reduces high-frequency head shaking or rapid turning, thus reducing visual stimulation.

[0113] Strategy 3: In addition to adjusting the physical parameters mentioned above, the system can also perform visual intervention in the post-processing stage:

[0114] Gradual Vignette: A black mask with varying transparency is overlaid on the edge area of ​​the rendered image. The intensity and coverage radius of the mask change dynamically with the collision signal C.

[0115] Dynamic element deweighting: Reduce the texture clarity or contrast of high-speed moving objects in the scene, such as speeding vehicles and particle effects, thereby reducing the interference weight of these dynamic elements on the user's sense of balance.

[0116] Preferably, in order to ensure the consistency of user experience, the adjustment of all the above parameters is not abrupt, but can be discretized according to light / medium / heavy levels, or a smooth transition can be achieved through linear interpolation.

[0117] Through the above adjustments, the visual information received by the user's brain is made to be consistent with the proprioception again, or the interference weight of visual information on balance perception is reduced, thereby driving the display unit to output images that are less likely to cause dizziness.

[0118] In some embodiments, to achieve adaptive adjustment to different user physiological states, this method further includes a continuously running feedback regulation closed loop. This step aims to utilize the non-photoreflective changes in pupil diameter as a physiological indicator of autonomic nervous system stress response to detect precursors of motion sickness. The specific implementation process is as follows:

[0119] First, the pupil diameter data in the eye-tracking data is monitored in real time, and the rate of change of the pupil diameter is calculated. Specifically, the user's pupil diameter data can be sampled at a frequency of 60Hz or 120Hz using an eye-tracking sensor. Then, the first derivative of the pupil diameter with respect to time, i.e., the rate of change of the pupil diameter dD / dt, is calculated, while a positive threshold T is set. expand and a negative threshold T recoverPreferably, T expand =0.2mm / s, T recover =0.05mm / s.

[0120] When the rate of change of the pupil diameter (dD / dt) continuously exceeds the positive threshold T within a first preset time period expand If this state persists for a first preset duration τ, such as 2 seconds, the user is judged to be in a high-risk state of dizziness, i.e., the sympathetic nervous system is excited, and the body exhibits a stress response. In response to this judgment, a higher level of inhibition will be implemented. Specific adjustment actions include:

[0121] Increase the calculation weight of sensory conflict signals: that is, increase the weight coefficient α or β in formula C in step S300, so that the same physical conflict will produce a higher conflict score C, thereby triggering compensation earlier.

[0122] Increase the adjustment range of image rendering parameters: for example, increase the coefficient k1 in the field of view reduction formula in step S400, or increase the damping coefficient k2 of the virtual camera, thereby applying a stronger visual intervention.

[0123] Furthermore, when the rate of change of the pupil diameter remains below the negative threshold T for a second preset duration... recover And maintain the second preset duration τ recover For example, after 10 seconds, it is determined that the user is in a recovery state and has adapted to the current virtual environment. In response to this determination, the calculation weight of generating the sensory conflict signal is gradually reduced, or the adjustment range of the image rendering parameters is reduced, until the default rendering state is restored.

[0124] In this embodiment, in order to prevent the system from frequently switching between different suppression levels, once the enhanced adjustment is triggered, the high suppression state will be forced to last for at least 5 seconds. Even if the pupil data drops in a short period of time, it will not exit immediately, thereby ensuring the continuity and stability of the user's visual experience.

[0125] Secondly, based on the same concept, embodiments of this application also provide an extended reality display system based on biomechanical simulation, which mainly includes:

[0126] The data acquisition module is used to acquire the user's physiological characteristic data, as well as the head-mounted display pose and eye-tracking data when the user wears the extended reality device. This module serves as the system's data entry point, providing the basic data flow for the subsequent computing engine.

[0127] The posture prediction module is used to configure the physiological feature data as parameters of the biomechanical model, and input the head-mounted display pose data and eye-tracking data into the biomechanical model to calculate the user's expected body posture data at the current moment. This module internally deploys an inverse kinematics (IK) solver, which calculates and outputs the user's expected body posture data at the current moment based on preset physiological constraints and kinematic chain transmission logic.

[0128] The conflict detection module calculates the difference between the expected body posture data and the visual stream data of the currently rendered image from the extended reality device, generating a sensory conflict signal. This module quantifies the inconsistency between the expected motion vector and the camera motion vector by calculating the directional angle and acceleration difference, thereby generating the sensory conflict signal.

[0129] The rendering control module is used to respond to the sensory conflict signal, adjust the image rendering parameters of the extended reality device according to a preset strategy, and drive the display unit to display the image.

[0130] Preferably, the system further includes a feedback adjustment module, which is used to monitor the pupil diameter data in the eye movement data in real time and calculate the rate of change of the pupil diameter.

[0131] When the rate of change of the pupil diameter continuously exceeds a preset positive threshold within a first preset time period, it is determined that the user is in a state of high risk of dizziness, and the calculation weight of generating the sensory conflict signal is increased, or the adjustment range of the image rendering parameters is increased.

[0132] When the rate of change of the pupil diameter remains below a preset negative threshold for a second preset duration, it is determined that the user is in a recovery state, and the calculation weight for generating the sensory conflict signal is reduced, or the adjustment range of the image rendering parameters is reduced.

[0133] Thirdly, embodiments of this application provide an extended reality device. This application also provides an extended reality device, which can be a VR all-in-one machine, a VR headset connected to a PC, or AR glasses. The device mainly includes:

[0134] Memory: Used to store computer programs and user physiological characteristic data.

[0135] The processor may include one or more central processing units (CPUs), graphics processing units (GPUs), or neural network processing units (NPUs). The processor executes computer programs stored in memory to implement the steps of the method described in the first aspect.

[0136] Monitor: Used to display the final rendered virtual scene.

[0137] Eye-tracking sensor: Integrated near the display to capture eye movements.

[0138] The extended reality display control method, system, and device based on biomechanical simulation provided in this application introduce a biomechanical model based on user physiological characteristics to construct motion expectations that conform to individual differences. It then quantifies sensory conflict by utilizing the difference between the expected posture and visual flow, achieving proactive intervention at the root of the dizziness mechanism. Furthermore, by combining eye-tracking data feedback to adjust model parameters, this solution ensures mobile terminal computing efficiency while solving the problems of poor adaptability and lag in passive compensation of traditional general-purpose models, effectively reducing user dizziness and extending device usage time.

[0139] In the description of this application, it should be noted that the terms "vertical", "up", "down", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0140] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0141] Finally, it should be noted that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for controlling extended reality displays based on biomechanical simulation, characterized in that, The process includes the following steps: S100: Acquire the user's physiological characteristic data, and collect the head-mounted display pose data and eye-tracking data of the user while wearing the extended reality device; S200: Configure the physiological characteristic data as parameters of a biomechanical model, and input the head-mounted display pose data and eye-tracking data into the biomechanical model to calculate the user's expected body posture data at the current moment; S300: Calculate the difference between the expected body posture data and the visual flow data of the currently rendered image of the extended reality device, and generate a sensory conflict signal; S400: In response to the sensory conflict signal, adjust the image rendering parameters of the extended reality device according to a preset strategy, and drive the display unit to display; Step S300 The steps for calculating the difference value and generating a sensory conflict signal include: extracting a first motion vector from the expected body posture data and a second motion vector from the virtual camera corresponding to the currently rendered screen; normalizing the first motion vector and the second motion vector and calculating the directional angle between them; calculating the acceleration difference between the instantaneous acceleration corresponding to the first motion vector and the instantaneous acceleration corresponding to the second motion vector; performing a weighted summation operation based on the directional angle and the acceleration difference to obtain an initial conflict value; performing temporal smoothing filtering on the initial conflict value and mapping the processed value to a preset quantization range as the sensory conflict signal.

2. The extended reality display control method based on biomechanical simulation according to claim 1, characterized in that, The steps for calculating the user's expected body posture data at the current moment include: setting the joint node coordinates and bone length of the virtual skeleton in the biomechanical model based on the height and arm length data in the physiological characteristic data; performing temporal filtering on the head-mounted display pose data and eye-tracking data, and inputting them as constraints into the inverse kinematics solver based on the virtual skeleton; propagating the head rotation increment along the kinematic chain from the cervical spine to the trunk, and using the gaze direction in the eye-tracking data as a preference weight for upper body rotation, combined with preset physiological range constraints for joint angles, to solve for the angles of each joint of the skeleton; and outputting the expected body posture data, including trunk orientation and center of gravity displacement trend, based on the solved joint angles.

3. The extended reality display control method based on biomechanical simulation according to claim 1, characterized in that, In step S400, the step of adjusting the image rendering parameters of the extended reality device includes: when the value corresponding to the sensory conflict signal exceeds a preset first threshold, executing at least one of the following adjustment strategies: proportionally reducing the current rendering field of view according to the intensity of the sensory conflict signal, wherein the reduction ratio is positively correlated with the sensory conflict signal; increasing the motion damping coefficient of the virtual camera used to generate the rendered image so that the motion speed of the rendered image lags behind the change in the head-mounted display pose; superimposing a progressive vignetting effect on the edge area of ​​the rendered image, or reducing the visual weight of dynamic elements in the rendered scene.

4. The extended reality display control method based on biomechanical simulation according to claim 1, characterized in that, The method also includes a feedback adjustment step: real-time monitoring of pupil diameter data in the eye movement data and calculation of the rate of change of pupil diameter; When the rate of change of the pupil diameter remains above a preset positive threshold for a first preset duration, the user is determined to be in a high-risk state of dizziness, and the calculation weight for generating the sensory conflict signal is increased, or the adjustment range of the image rendering parameters is increased; when the rate of change of the pupil diameter remains below a preset negative threshold for a second preset duration, the user is determined to be in a recovery state, and the calculation weight for generating the sensory conflict signal is reduced, or the adjustment range of the image rendering parameters is decreased.

5. The extended reality display control method based on biomechanical simulation according to claim 1, characterized in that, Before step S200, the method also includes a model initialization and calibration step: guiding the user to complete a preset standard action sequence, the standard action sequence including at least neck limit rotation and trunk swinging; collecting the user's pose data when performing the standard action sequence, and determining the actual range of motion boundaries and motion preference parameters of each joint of the user; The joint constraints of the biomechanical model are modified using the actual range of motion boundaries and motion preference parameters.

6. The extended reality display control method based on biomechanical simulation according to claim 1, characterized in that, The biomechanical model is a neural network model or a kinematic chain model based on physical equations. Step S200 is executed in the local processor of the extended reality device, and the duration of a single calculation is less than a preset time threshold.

7. An extended reality display system based on biomechanical simulation, characterized in that, include: The data acquisition module is used to acquire the user's physiological characteristic data, and to collect the head-mounted display posture data and eye movement data when the user wears the extended reality device; The posture prediction module is used to configure the physiological feature data as parameters of the biomechanical model, and input the head-mounted display pose data and eye-tracking data into the biomechanical model to calculate the user's expected body posture data at the current moment; the conflict detection module is used to calculate the difference between the expected body posture data and the visual flow data of the currently rendered screen of the extended reality device, and generate a sensory conflict signal. The steps for calculating the difference value and generating a sensory conflict signal include: extracting a first motion vector from the expected body posture data and a second motion vector from the virtual camera corresponding to the current rendered image; normalizing the first motion vector and the second motion vector and calculating the directional angle between them; calculating the acceleration difference between the instantaneous acceleration corresponding to the first motion vector and the instantaneous acceleration corresponding to the second motion vector; performing a weighted summation operation based on the directional angle and the acceleration difference to obtain an initial conflict value; performing temporal smoothing filtering on the initial conflict value and mapping the processed value to a preset quantization range as the sensory conflict signal; and a rendering control module, used to respond to the sensory conflict signal, adjust the image rendering parameters of the extended reality device according to a preset strategy, and drive the display unit to display.

8. The extended reality display system based on biomechanical simulation according to claim 7, characterized in that, Also includes: The feedback adjustment module is used to monitor the pupil diameter data in the eye movement data in real time and calculate the rate of change of pupil diameter; When the rate of change of the pupil diameter continuously exceeds a preset positive threshold within a first preset time period, the user is determined to be in a high-risk state of dizziness, and the calculation weight for generating the sensory conflict signal is increased, or the adjustment range of the image rendering parameters is increased; when the rate of change of the pupil diameter continuously falls below a preset negative threshold within a second preset time period, the user is determined to be in a recovery state, and the calculation weight for generating the sensory conflict signal is reduced, or the adjustment range of the image rendering parameters is decreased.

9. An extended reality device, characterized in that, include: Display, eye-tracking sensor, memory, and processor; The memory stores a computer program, and when the processor executes the computer program, it implements the extended reality display control method based on biomechanical simulation as described in any one of claims 1 to 6.

Images