Virtual reality presentation method and apparatus, and XR device and storage medium

Abstract

The present invention relates to the technical field of virtual reality applications. Provided are a virtual reality presentation method and apparatus, and an XR device and a storage medium. The method comprises: acquiring real-time motion data of an XR device; performing calculation on the basis of the real-time motion data, so as to obtain a real-time motion state of the XR device; and on the basis of the real-time motion state, constructing a virtual scene corresponding to the real-time motion state, and displaying the virtual scene on the XR device.

Description

Virtual reality presentation method, device, XR device and storage medium

[0001] Cross-references

[0002] This application refers to Chinese Patent Application No. 2024100844225, filed on January 19, 2024, entitled “Virtual Reality Presentation Method, Apparatus, XR Device and Storage Medium”, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to the field of virtual reality application technology, and in particular to a virtual reality presentation method, apparatus, XR device, and storage medium. Background Art

[0004] The human brain primarily relies on the vestibular, visual, and perceptual systems to sense and control balance. The inner ear contains vestibular receptors, primarily composed of three semicircular canals, the saccule, and the utricle. These organs, collectively known as the otolith organs, are responsible for sensing various aspects of the body's position and speed. A significant number of passengers experience motion sickness when riding in cars or public transportation platforms, as the vestibular system perceives motion inconsistently with the eye.

[0005] In related technologies, the car sensor information is integrated with the sensors of the HMD head-mounted device to calculate the car's motion state and the HMD's motion state relative to the car. The dizziness is alleviated by reducing the delay, and an immersive VR virtual reality experience is provided in the car. However, strict data synchronization between the VR device and the car's sensors is required, which consumes a lot of computing power and makes it difficult to achieve real-time data synchronization. As a result, the effect of using virtual reality to eliminate motion sickness symptoms is poor and the cost is too high.

[0006] Summary of the Invention

[0007] In view of this, the present invention provides a virtual reality presentation method, apparatus, XR device and storage medium to solve the problem that using virtual reality to eliminate motion sickness symptoms is poorly effective and costly.

[0008] In a first aspect, the present invention provides a virtual reality presentation method, the method comprising:

[0009] Get real-time motion data from XR devices;

[0010] Based on real-time motion data, calculate the real-time motion status of the XR device;

[0011] Based on the real-time motion state, a virtual scene corresponding to the real-time motion state is constructed and displayed on the XR device.

[0012] In the present invention, the real-time motion state of the XR device is estimated through the real-time motion data of the XR device, and a corresponding virtual scene is constructed and displayed, so that the user can visually perceive that the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, thereby eliminating or alleviating the user's motion sickness symptoms. There is no need for strict synchronization of data between the XR device and the sensors installed on the motion platform, which greatly reduces the computing power consumption and improves the effect of the XR device in eliminating motion sickness symptoms.

[0013] In an optional implementation, obtaining real-time motion data of an XR device includes:

[0014] Collect real-time motion data using sensors installed on XR devices;

[0015] Based on real-time motion data, determine whether the XR device is affected by external forces other than gravity;

[0016] When the XR device is not affected by any external forces other than gravity, the real-time motion data is initialized.

[0017] In this method, by initializing the real-time motion data when it is stable, the reference world coordinate system and origin of the XR device are determined, preparing for the next step of motion state estimation.

[0018] In an optional embodiment, calculating the real-time motion state of the XR device based on the real-time motion data includes:

[0019] Based on real-time motion data, the sensor bias at the current moment is calculated;

[0020] The real-time motion state of the previous moment is obtained, combined with the bias of the sensor at the current moment, and the real-time motion state of the current moment is obtained by integral recursion.

[0021] In this method, by utilizing the bias at the current moment, the motion state of the XR device at the current moment is estimated, providing posture information for subsequent virtual scene rendering, thereby facilitating the construction of the virtual scene.

[0022] In an optional embodiment, after obtaining the current real-time motion state by integral recursion, the method further includes:

[0023] Using a damped integral method, the damping coefficient of the real-time motion state is adjusted to obtain a first real-time motion state, so as to prevent the first real-time motion state from diverging;

[0024] The first real-time motion state is smoothed to obtain a second real-time motion state.

[0025] In this manner, by utilizing a damped integration method, the divergence of the motion state is avoided and the motion state is smoothed, thereby further improving the accuracy of the processed motion state and making it closer to the actual motion state of the virtual display device.

[0026] In an optional embodiment, calculating the real-time motion state of the XR device based on the real-time motion data further includes:

[0027] Based on real-time motion data, the sensor's zero bias is calculated using a neural network.

[0028] Based on the zero bias, the real-time motion state of the XR device is estimated.

[0029] In this method, by utilizing neural networks, the motion state can be estimated more accurately in the event of large-scale drift, avoiding bias and causing more severe dizziness to the user, thereby further improving the user experience.

[0030] In an optional embodiment, the zero bias of the sensor is obtained by calculating using a neural network, including:

[0031] Obtain the real-time motion data of the sensor and the real-time posture data corresponding to the real-time motion data at the same time to construct a training set;

[0032] The neural network is trained based on the training set to obtain a pre-trained neural network, and the pre-trained neural network is used to calculate the zero bias of the sensor.

[0033] In this way, the neural network is trained with a large amount of data, which further improves the accuracy of motion state estimation in the zero-bias case, thereby avoiding causing more severe dizziness to the user and improving the user experience.

[0034] In an optional embodiment, constructing a virtual scene corresponding to the real-time motion state based on the real-time motion state, and displaying the virtual scene on the XR device includes:

[0035] According to the current timestamp, refresh delay and other delays, based on the real-time motion state, the timestamp of the virtual scene and the motion state corresponding to the virtual scene are calculated;

[0036] Combined with the timestamp of the virtual scene and the motion state corresponding to the virtual scene, the posture of the XR device is adjusted, and the virtual scene corresponding to the real-time motion state is rendered and displayed on the XR device.

[0037] In this method, the XR device adjusts its posture in real time according to the real-time motion state and displays a virtual scene consistent with the real motion state, so that the user can visually perceive that the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, thereby eliminating or alleviating the user's motion sickness symptoms.

[0038] In a second aspect, the present invention provides a virtual reality presentation device, comprising:

[0039] Data acquisition module, used to obtain real-time motion data of the sensor;

[0040] The motion state estimation module is used to calculate the real-time motion state of the XR device based on real-time motion data;

[0041] The scene construction module is used to construct a virtual scene corresponding to the real-time motion state based on the real-time motion state and display the virtual scene on the XR device.

[0042] In a third aspect, the present invention provides an XR device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the virtual reality presentation method of the first aspect or any corresponding embodiment thereof.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the virtual reality presentation method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] FIG1 is a schematic flow chart of a virtual reality presentation method according to an embodiment of the present invention.

[0046] FIG2 is a schematic flow chart of a method for eliminating or alleviating motion sickness based on an inertial sensor according to an embodiment of the present invention.

[0047] FIG3 is a flow chart of another virtual reality presentation method according to an embodiment of the present invention.

[0048] FIG4 is a flow chart of another virtual reality presentation method according to an embodiment of the present invention.

[0049] FIG5 is a flow chart of another method for eliminating or alleviating motion sickness based on an inertial sensor according to an embodiment of the present invention.

[0050] FIG6 is a structural block diagram of a virtual reality presentation device according to an embodiment of the present invention.

[0051] FIG7 is a schematic diagram of the hardware structure of an XR device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0053] Embodiments of the present invention provide a virtual reality presentation method for use in an XR (Extended Reality) device. It should be noted that the execution subject may be a virtual reality presentation device, which can be implemented as part or all of an XR device through software, hardware, or a combination of software and hardware. The XR device may be a VR device, an AR device, an MR device, an HMD, and / or other XR device, including but not limited to a VR headset, an AR headset, and an MR headset. The following method embodiments are all described using an XR device as the execution subject.

[0054] The XR device in this embodiment is suitable for use in scenarios where the user is on a motion platform and is provided with a real motion state experience to eliminate the user's motion sickness symptoms. The motion platforms that can be used include but are not limited to: the back seat or co-pilot seat of a car, a bus, an airplane, a ship, etc. The virtual reality presentation method provided by the present invention does not require strict synchronization of data between the XR device and the sensor installed on the motion platform. The real-time motion state of the XR device is estimated through the real-time motion data of the XR device, and the corresponding virtual scene is constructed and displayed, so that the user can visually perceive that the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, thereby eliminating or alleviating the user's motion sickness symptoms, greatly reducing computing power consumption, and improving the effect of the XR device in eliminating motion sickness symptoms.

[0055] According to an embodiment of the present invention, an embodiment of a virtual reality presentation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in an XR device such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] In some embodiments, a virtual reality presentation method is provided that can be used with the aforementioned XR devices, such as VR devices, AR devices, MR devices, HMDs, and / or other XR devices, including but not limited to VR head-mounted devices, AR head-mounted devices, and MR head-mounted devices. FIG1 is a flow chart of the virtual reality presentation method according to an embodiment of the present invention. As shown in FIG1 , the process includes the following steps:

[0057] Step S101: Acquire real-time motion data of the XR device.

[0058] In the example, real-time measurement data of the inertial sensor in the XR device is obtained. In the example, real-time measurement data of devices such as a magnetometer is also obtained. The real-time measurement data of the inertial sensor includes acceleration and angular velocity. The real-time motion state to be estimated includes position, angle, and velocity.

[0059] Step S102: Calculate the real-time motion state of the XR device based on the real-time motion data.

[0060] In the example, since the real-time measurement data obtained by the inertial sensor has bias and noise, the collected real-time measurement data is filtered, integrated, and processed to calculate the motion state at the current moment.

[0061] Step S103: Based on the real-time motion state, a virtual scene corresponding to the real-time motion state is constructed, and the virtual scene is displayed on the XR device.

[0062] In the example, the human eye can perceive the position, speed, angle, and angular velocity in the virtual space, while the vestibule is not sensitive to absolute position and absolute angle, but is more sensitive to acceleration and angular velocity. Therefore, the XR device display system displays a three-dimensional virtual scene consistent with the real-time motion state based on the real-time motion state. The motion state perceived by the user's vision is consistent with the real motion state perceived by the vestibule, which can eliminate or alleviate motion sickness.

[0063] In one implementation scenario, Figure 2 is a flow chart of a method for eliminating or alleviating motion sickness based on inertial sensors according to an embodiment of the present invention. As shown in Figure 2, the method for eliminating or alleviating motion sickness based on inertial sensors applied to XR devices includes: S1: acquiring real-time measurement data of the inertial sensor and magnetometer (optional) in the VR device, and initializing the motion state after the data stabilizes. S2: filtering, integrating, and other processing the real-time measurement data, estimating the motion state at the current moment, and smoothing the motion state. S3: The VR device display system displays a 3D virtual scene consistent with the relative motion state, and acceleration and angular velocity consistent with the user's vision and vestibular perception.

[0064] The virtual reality presentation method provided in this embodiment calculates the real-time motion state of the XR device through the real-time motion data of the XR device, constructs and displays the corresponding virtual scene, so that the user's visual perception of the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, eliminating or alleviating the user's motion sickness symptoms. There is no need for strict synchronization of data between the XR device and the sensors installed on the motion platform, which greatly reduces computing power consumption and improves the effect of the XR device in eliminating motion sickness symptoms.

[0065] In some embodiments, a virtual reality presentation method is provided that can be used with the aforementioned XR devices, such as VR devices, AR devices, MR devices, HMDs, and / or other XR devices, including but not limited to VR head-mounted devices, AR head-mounted devices, and MR head-mounted devices. FIG3 is a flow chart of another virtual reality presentation method according to an embodiment of the present invention. As shown in FIG3 , the process includes the following steps:

[0066] Step S301: Acquire real-time motion data of the XR device.

[0067] Specifically, the above step S301 includes:

[0068] Step S3011: Use sensors installed on the XR device to collect real-time motion data.

[0069] Step S3012: Based on the real-time motion data, determine whether the XR device is affected by any external force other than gravity.

[0070] Step S3013: When the XR device is not affected by any external force other than gravity, the real-time motion data is initialized.

[0071] In the example, by obtaining real-time motion data from inertial sensors, magnetometers and other devices in the VR device, the motion state is initialized after the real-time motion data stabilizes. The purpose of initializing the motion state (position, speed, acceleration, angle, angular velocity) is to determine the reference world coordinate system and origin of the device, in preparation for the next step of motion state calculation. When the VR device is not affected by external forces other than gravity, initialization can be performed. There are two specific scenarios: when the VR device is stationary or when the VR device is moving in a uniform straight line, real-time motion data is initialized. The initialization method is as follows:

[0072] (1) Select stable IMU inertial sensor data (including acceleration and angular velocity data) over a period of time, calculate the mean and variance of acceleration and angular velocity, and determine whether it is affected by external forces other than gravity. If the VR device is not affected by external forces other than gravity, the axial relationship between the current gravity and the IMU inertial sensor coordinate system can be calculated based on the magnitude of gravity, and then the axial direction of the reference world coordinate system can be initialized.

[0073] (2) Since it is impossible to determine whether the VR device is absolutely stationary or in uniform linear motion based on the data from the IMU inertial sensor itself, the user cannot perceive the absolute position, absolute speed, and absolute angle in a complete VR environment. Therefore, during initialization, the initial position, initial speed, and initial angle of the VR device relative to the reference world system are set to zero or other empirical values ​​to complete the initialization of the real-time motion data.

[0074] In this method, by initializing the real-time motion data when it is stable, the reference world coordinate system and origin of the XR device are determined, preparing for the next step of motion state estimation.

[0075] Step S302 : Calculate the real-time motion state of the XR device based on the real-time motion data.

[0076] Specifically, the above step S302 includes:

[0077] Step S3021: Calculate the sensor bias at the current moment based on the real-time motion data.

[0078] Step S3022: Obtain the real-time motion state at the previous moment, combine it with the sensor bias at the current moment, and obtain the real-time motion state at the current moment through integral recursion.

[0079] In this example, the real-time measurement data is filtered and integrated to calculate the current motion state and smooth the motion state. Specifically, the following steps are performed:

[0080] (1) Since the real-time motion data (acceleration, angular velocity) collected by the original IMU inertial sensor contains bias and noise, the bias and noise need to be processed. The noise processing method is to perform consistency check and low-pass filtering on the original collected acceleration and angular velocity data to eliminate jitter caused by high-frequency interference.

[0081] (2) IMU bias calculation: As in initialization, based on the stable observation state (uniform motion or absolute stillness) over a period of time as a priori, the mean value of acceleration and angular velocity over a period of time is statistically calculated as the IMU bias.

[0082] (3) Motion state estimation: The motion state at the previous moment, the real-time motion data (acceleration, angular velocity) collected by the inertial sensor IMU at the current moment, the gravity component, and the IMU bias are used to obtain the motion state at the current moment through integral recursion.

[0083] In this method, the motion state of the XR device at the current moment is calculated by utilizing the bias at the current moment, providing posture information for subsequent virtual scene rendering, thereby facilitating the construction of the virtual scene.

[0084] Step S3023 : Using a damped integration method, the damping coefficient of the real-time motion state is adjusted to obtain a first real-time motion state, so as to prevent the first real-time motion state from diverging.

[0085] In this example, the motion state obtained by recursive integration can cause low-frequency drift due to IMU noise and inaccurate bias. Therefore, the solution is to use a damped integration method and adjust the damping coefficient to avoid divergence of the motion state.

[0086] Step S3024: Smoothing the first real-time motion state to obtain a second real-time motion state.

[0087] In the example, the jitter of the output motion state can be smoothed using SG or Euro filtering.

[0088] In this manner, by utilizing a damped integration method, the divergence of the motion state is avoided and the motion state is smoothed, thereby further improving the accuracy of the processed motion state and making it closer to the actual motion state of the virtual display device.

[0089] Step S303: Based on the real-time motion state, a virtual scene corresponding to the real-time motion state is constructed, and the virtual scene is displayed on the XR device.

[0090] Specifically, the above step S303 includes:

[0091] Step S3031 , calculating and generating the timestamp of the virtual scene and the motion state corresponding to the virtual scene based on the real-time motion state according to the current timestamp, refresh delay and other delays.

[0092] Step S3032: Adjust the posture of the XR device based on the timestamp of the virtual scene and the motion state corresponding to the virtual scene, and render and display the virtual scene corresponding to the real-time motion state on the XR device.

[0093] The damped integration method described in the present invention may be an integration method based on a first-order complementary filter or a Mahony filtering algorithm.

[0094] In this example, the VR device display system displays a 3D virtual scene consistent with the relative motion state, so that the user's vision and vestibular perception are consistent with the acceleration and angular velocity. Specifically, it includes:

[0095] The VR device display system calculates the timestamp required to generate virtual content based on the current timestamp, refresh delay, other delays, etc., and predicts the required motion state. The method of motion state prediction includes: using the Hot-Winter method based on the current latest motion state, and performing prediction and filtering at the same time. When performing 3D virtual field rendering, the position and posture of the left and right virtual cameras are calculated by the predicted motion state. The calculation method is: the predicted IMU motion state is converted into the operating state of the center of the eyes through the external parameters of the device structure, and the device is translated to the left and right by 1 / 2 of the pupil distance, that is, IPD / 2 (Interpupillary Distance), respectively, to obtain the position and posture of the two virtual cameras. Among them, the 3D virtual scene is produced by software, which can be similar to a highway scene, a city street scene, etc., and the user switches according to personal preferences.

[0096] In this method, the XR device adjusts its posture in real time according to the real-time motion state and displays a virtual scene consistent with the real motion state, so that the user can visually perceive that the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, thereby eliminating or alleviating the user's motion sickness symptoms.

[0097] The virtual reality presentation method provided in this embodiment initializes the real-time motion data when the collected real-time motion data is stable, determines the reference world coordinate system and origin of the XR device, and prepares for the next step of motion state estimation. By using the bias at the current moment, the motion state of the XR device at the current moment is calculated, and the posture information is provided for the subsequent rendering of the virtual scene, which facilitates the construction of the virtual scene. By using the damped integration method to avoid the divergence of the motion state and smooth the motion state, the accuracy of the processed motion state is further improved, which is closer to the real motion state of the virtual display device. According to the real-time motion state, the XR device adjusts the posture in real time and displays a virtual scene consistent with the real motion state, so that the user's visual perception of the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, eliminating or alleviating the user's motion sickness symptoms.

[0098] In some embodiments, a virtual reality presentation method is provided that can be used with the aforementioned XR devices, such as VR devices, AR devices, MR devices, HMDs, and / or other XR devices, including but not limited to VR head-mounted devices, AR head-mounted devices, and MR head-mounted devices. FIG4 is a flow chart of another virtual reality presentation method according to an embodiment of the present invention. As shown in FIG4 , the process includes the following steps:

[0099] Step S401: Acquire real-time motion data of the sensor. Please refer to step S301 of the embodiment shown in FIG3 for details, which will not be repeated here.

[0100] Step S402: Calculate the real-time motion state of the XR device based on the real-time motion data.

[0101] Specifically, the above step S402 includes:

[0102] Step S4021: Based on the real-time motion data, the zero bias of the sensor is calculated using a neural network.

[0103] In some optional implementations, the above step S4021 includes:

[0104] Step a1: Acquire the real-time motion data of the sensor and the real-time pose data corresponding to the same moment to construct a training set.

[0105] Step a2: training the neural network based on the training set to obtain a pre-trained neural network, which is used to calculate the zero bias of the sensor.

[0106] In this example, IMU bias is primarily manifested in varying degrees of acceleration and angular velocity over time when stationary, severely impacting the integral calculation of motion states and causing large-scale drift, which can lead to even more severe dizziness for the user. Calculating IMU bias using a neural network requires training the network. This training involves collecting a large amount of real-time IMU measurement data and the actual position at the corresponding moment. Once training is complete, the trained neural network can be used in real-time on the mobile device.

[0107] In this way, the neural network is trained with a large amount of data, which further improves the accuracy of motion state estimation in the zero-bias case, thereby avoiding causing more severe dizziness to the user and improving the user experience.

[0108] Step S4022: Based on the zero bias, the real-time motion state of the XR device is estimated.

[0109] In the example, a neural network method is used to calculate the zero bias of the inertial sensor (IMU), thereby integrating and solving the motion state; the motion state and motion increment are also smoothed by SG or one-Euro filtering.

[0110] In this method, by utilizing neural networks, the motion state can be estimated more accurately in the event of large-scale drift, avoiding bias and causing more severe dizziness to the user, thereby further improving the user experience.

[0111] In step S403, a virtual scene corresponding to the real-time motion state is constructed based on the real-time motion state, and the virtual scene is displayed on the XR device. For details, please refer to step S303 of the embodiment shown in FIG3 , which will not be repeated here.

[0112] In one implementation scenario, FIG5 is a flow chart of another method for eliminating or alleviating motion sickness based on inertial sensors according to an embodiment of the present invention. As shown in FIG5 , the method for eliminating or alleviating motion sickness based on inertial sensors applied to VRXR devices includes: S1: obtaining real-time measurement data of the inertial sensor and magnetometer (optional) in the VR device, and initializing the motion state after the data stabilizes. S2: Passing the real-time measurement data through a neural network to calculate the zero bias, thereby estimating the motion state at the current moment, and smoothing the motion state. S3: The VR device display system displays a 3D virtual scene consistent with the relative motion state, and acceleration and angular velocity consistent with the user's vision and vestibular perception.

[0113] The virtual reality presentation method provided in this embodiment utilizes a neural network to more accurately calculate motion state in the presence of large-scale drift, avoiding bias that could cause more severe dizziness in the user and further improving the user experience. Training the neural network with a large amount of data further improves the accuracy of motion state estimation in the zero-bias scenario, thereby avoiding more severe dizziness in the user and improving the user experience.

[0114] In some embodiments, a virtual reality presentation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementations. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0115] This embodiment provides a virtual reality presentation device, as shown in FIG6 , including:

[0116] The data acquisition module 601 is used to acquire real-time motion data from the sensor. Please refer to step S101 of the embodiment shown in FIG1 for details, which will not be repeated here.

[0117] The motion state estimation module 602 is configured to calculate the real-time motion state of the XR device based on the real-time motion data. For details, please refer to step S102 of the embodiment shown in FIG1 , which will not be described in detail here.

[0118] The scene construction module 603 is used to construct a virtual scene corresponding to the real-time motion state based on the real-time motion state, and display the virtual scene on the XR device. For details, please refer to step S103 of the embodiment shown in Figure 1, and will not be repeated here.

[0119] In some optional implementations, the data acquisition module 601 includes:

[0120] The data acquisition unit is used to collect real-time motion data using sensors installed on the XR device.

[0121] The external force judgment unit is used to judge whether the XR device is subjected to external forces other than gravity based on real-time motion data.

[0122] The data initialization unit is used to initialize the real-time motion data when the XR device is not affected by any external force other than gravity.

[0123] In some optional implementations, the motion state estimation module 602 includes:

[0124] The bias calculation unit is used to calculate the bias of the sensor at the current moment based on the real-time motion data.

[0125] The first motion state estimation unit is used to obtain the real-time motion state at the previous moment, combine it with the bias of the sensor at the current moment, and obtain the real-time motion state at the current moment through integral recursion.

[0126] In some optional implementations, the motion state estimation module 602 further includes:

[0127] The divergence processing unit is used to adjust the damping coefficient of the real-time motion state by using a damped integration method to obtain the first real-time motion state, so as to avoid divergence of the first motion state.

[0128] The smoothing processing unit is used to perform smoothing processing on the first motion state to obtain a second motion state.

[0129] In some optional implementations, the motion state estimation module 602 further includes:

[0130] The zero bias calculation unit is used to calculate the zero bias of the sensor based on real-time motion data using a neural network.

[0131] The second motion state estimation unit is used to estimate the real-time motion state of the XR device based on the zero bias.

[0132] In some optional implementations, the zero bias calculation unit includes:

[0133] The training set construction subunit is used to obtain the real-time motion data of the sensor and the real posture data corresponding to the real-time motion data at the same time, and construct a training set.

[0134] The network training subunit is used to train the neural network based on the training set to obtain a pre-trained neural network, and the pre-trained neural network is used to calculate the zero bias of the sensor.

[0135] In some optional implementations, the scene construction module 603 includes:

[0136] The motion state prediction unit is used to calculate and generate the timestamp of the virtual scene and the motion state corresponding to the virtual scene based on the real-time motion state according to the current timestamp, refresh delay and other delays.

[0137] The posture adjustment unit is used to adjust the posture of the XR device based on the timestamp of the virtual scene and the motion state corresponding to the virtual scene, and render and display the virtual scene corresponding to the real-time motion state on the XR device.

[0138] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0139] The virtual reality presentation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0140] An embodiment of the present invention further provides an XR device having the virtual reality presentation apparatus shown in FIG6 above.

[0141] Please refer to Figure 7, which is a schematic diagram of the structure of an XR device provided by an optional embodiment of the present invention. As shown in Figure 7, the XR device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses and can be installed on a common motherboard or in other ways as needed. The processor can process instructions executed within the XR device, including instructions stored in or on the memory for displaying graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices. Similarly, multiple XR devices can be connected, with each device providing some of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 uses a single processor 10 as an example.

[0142] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0143] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0144] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the XR device, etc. In addition, the memory 20 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the XR device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0146] The XR device further includes a communication interface 30 for the XR device to communicate with other devices or a communication network.

[0147] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

Claims

1. A virtual reality rendering method, comprising: Obtaining real-time motion data of the XR device; Calculating the real-time motion state of the XR device based on the real-time motion data; And Based on the real-time motion state, constructing a virtual scene corresponding to the real-time motion state and displaying the virtual scene on the XR device.

2. The method according to claim 1, wherein, The obtaining of the real-time motion data of the XR device includes: Collecting the real-time motion data by using sensors installed on the XR device; Based on the real-time motion data, determining whether the XR device is subjected to external forces other than gravity; and When the XR device is not subjected to external forces other than gravity, initializing the real-time motion data.

3. The method according to claim 2, wherein The calculating of the real-time motion state of the XR device based on the real-time motion data includes: Calculating the bias of the sensor at the current moment based on the real-time motion data; and Obtaining the real-time motion state at the previous moment, and integrating and recursively deriving the real-time motion state at the current moment by combining the bias of the sensor at the current moment.

4. The method according to claim 3, wherein After the real-time motion state at the current moment is obtained by integral recursion, the method further includes: Using an integral method with damping to adjust the damping coefficient of the real-time motion state to obtain a first real-time motion state to avoid divergence of the first real-time motion state; and Smoothing the first real-time motion state to obtain a second real-time motion state.

5. The method according to claim 3, wherein The calculating of the real-time motion state of the XR device based on the real-time motion data further includes: Calculating the zero bias of the sensor based on the real-time motion data by using a neural network; and Estimating the real-time motion state of the XR device based on the zero bias.

6. The method according to claim 5, wherein The calculating of the zero bias of the sensor by using a neural network includes: Obtaining the real-time motion data of the sensor and the real pose data at the same moment corresponding to the real-time motion data, and constructing a training set; and Training the neural network based on the training set to obtain a pre-trained neural network, and the pre-trained neural network is used to calculate the zero bias of the sensor.

7. The method according to claim 1, wherein The constructing of the virtual scene corresponding to the real-time motion state based on the real-time motion state and displaying the virtual scene on the XR device includes: Based on the real-time motion state, calculating and generating the time stamp of the virtual scene and the motion state corresponding to the virtual scene according to the current time stamp, the refresh delay and other delays; and Combining the time stamp of the virtual scene and the motion state corresponding to the virtual scene, adjusting the pose of the XR device, and rendering and displaying the virtual scene corresponding to the real-time motion state on the XR device.

8. The method according to any one of claims 1-7, wherein, The XR device is applied to a motion platform.

9. A virtual reality rendering device, comprising: A data acquisition module for acquiring real-time motion data of a sensor; A motion state estimation module for calculating the real-time motion state of the XR device based on the real-time motion data; And A scene construction module, configured to construct a virtual scene corresponding to the real-time motion state based on the real-time motion state, and display the virtual scene on the XR device.

10. An XR device, comprising: A memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the virtual reality rendering method according to any one of claims 1 to 8.

11. A computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the virtual reality rendering method according to any one of claims 1 to 8.

Images