Virtual reality presentation method and device, XR equipment and storage medium

Abstract

The invention relates to the technical field of virtual reality application, and discloses a virtual reality presentation method and device, XR equipment and a storage medium, and the method comprises the steps: obtaining real-time motion data of the XR equipment; calculating to obtain a real-time motion state of the XR equipment based on the real-time motion data; and based on the real-time motion state, constructing a virtual scene corresponding to the real-time motion state, and displaying the virtual scene on the XR equipment. According to the method, the real-time motion state of the XR equipment is calculated, and the corresponding virtual scene is constructed and displayed, so that the real-time motion state in the virtual scene displayed by the XR equipment visually perceived by a user is consistent with the real motion state perceived by the ear vestibule of the user, carsickness symptoms of the user are eliminated or relieved, and the user experience is improved. Strict data synchronization between the XR equipment and the sensor installed on the motion platform is not needed, and the computing power consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality application technology, and in particular to a virtual reality presentation method, apparatus, XR device and storage medium. Background Art

[0002] The human brain mainly relies on the vestibular, visual and self-sensing systems to sense and control balance. The human inner ear has vestibular terminal receptors, which are mainly composed of three semicircular canals, the saccule and the utricle. The saccule and the utricle are collectively called otoliths, which are mainly used to sense various positions and speeds of the human body. When riding in a car or a public sports platform, a large number of passengers will experience motion sickness symptoms because the motion state perceived by the vestibular system is inconsistent with the motion state perceived by the eyes.

[0003] In the related technology, the car sensor information is integrated with the sensors of the HMD head-mounted device to calculate the car's motion state and the HMD's motion state relative to the car. The dizziness is relieved by reducing the delay, and an immersive VR virtual reality experience in the car is provided. This requires strict data synchronization between the VR device and the car's sensors, consumes a lot of computing power, and is difficult to achieve real-time data synchronization, resulting in poor results in eliminating motion sickness symptoms using virtual reality and high costs. Summary of the invention

[0004] In view of this, the present invention provides a virtual reality presentation method, apparatus, XR device and storage medium to solve the problem that the effect of eliminating motion sickness symptoms by using virtual reality is poor and the cost is too high.

[0005] In a first aspect, the present invention provides a virtual reality presentation method, the method comprising:

[0006] Get real-time motion data of XR devices;

[0007] Based on the real-time motion data, the real-time motion state of the XR device is calculated;

[0008] Based on the real-time motion state, a virtual scene corresponding to the real-time motion state is constructed, and the virtual scene is displayed on the XR device.

[0009] In the present invention, the real-time motion state of the XR device is estimated through the real-time motion data of the XR device, and the corresponding virtual scene is constructed and displayed, so that the real-time motion state in the virtual scene displayed by the XR device perceived by the user through vision is consistent with the actual motion state perceived by the user's vestibule, thereby eliminating or alleviating the motion sickness symptoms of the user. There is no need for strict synchronization of data between the XR device and the sensors installed on the motion platform, which greatly reduces the computing power consumption and improves the effect of the XR device in eliminating motion sickness symptoms.

[0010] In an optional implementation, obtaining real-time motion data of an XR device includes:

[0011] Collect real-time motion data using sensors installed on XR devices;

[0012] Based on real-time motion data, determine whether the XR device is affected by external forces other than gravity;

[0013] When the XR device is not affected by any external force other than gravity, the real-time motion data is initialized.

[0014] In this method, the real-time motion data is initialized when the collected real-time motion data is stable, and the reference world coordinate system and origin of the XR device are determined to prepare for the next step of motion state estimation.

[0015] In an optional implementation, calculating the real-time motion state of the XR device based on the real-time motion data includes:

[0016] Based on the real-time motion data, the sensor bias at the current moment is calculated;

[0017] The real-time motion state of the previous moment is obtained, combined with the bias of the sensor at the current moment, and the real-time motion state of the current moment is obtained by integral recursion.

[0018] In this method, the motion state of the XR device at the current moment is estimated by utilizing the bias at the current moment, providing posture information for subsequent virtual scene rendering, thereby facilitating the construction of the virtual scene.

[0019] In an optional implementation, after obtaining the real-time motion state at the current moment by integral recursion, the method further includes:

[0020] Using a damped integral method, the damping coefficient of the real-time motion state is adjusted to obtain a first real-time motion state, so as to avoid divergence of the first real-time motion state;

[0021] The first real-time motion state is smoothed to obtain a second real-time motion state.

[0022] In this manner, by utilizing a damped integration method, the divergence of the motion state is avoided and the motion state is smoothed, thereby further improving the accuracy of the processed motion state and making it closer to the actual motion state of the virtual display device.

[0023] In an optional implementation, calculating the real-time motion state of the XR device based on the real-time motion data further includes:

[0024] Based on real-time motion data, the sensor's zero bias is calculated using a neural network;

[0025] Based on the zero bias, the real-time motion state of the XR device is estimated.

[0026] In this way, by utilizing neural networks, the motion state can be estimated more accurately in the event of large-scale drift, avoiding bias and causing more severe dizziness to the user, thereby further improving the user experience.

[0027] In an optional implementation, the zero bias of the sensor is calculated using a neural network, including:

[0028] Acquire the real-time motion data of the sensor and the real posture data corresponding to the real-time motion data at the same time, and construct a training set;

[0029] The neural network is trained based on the training set to obtain a pre-trained neural network, and the pre-trained neural network is used to calculate the zero bias of the sensor.

[0030] In this way, the neural network is trained with a large amount of data, which further improves the accuracy of motion state prediction under zero bias conditions, thereby avoiding causing more severe dizziness to the user and improving the user experience.

[0031] In an optional implementation, based on the real-time motion state, constructing a virtual scene corresponding to the real-time motion state, and displaying the virtual scene on the XR device includes:

[0032] According to the current timestamp, refresh delay and other delays, based on the real-time motion state, the timestamp of the virtual scene and the motion state corresponding to the virtual scene are calculated and generated;

[0033] Combined with the timestamp of the virtual scene and the motion state corresponding to the virtual scene, the posture of the XR device is adjusted, and the virtual scene corresponding to the real-time motion state is rendered and displayed on the XR device.

[0034] In this method, the XR device adjusts its posture in real time according to the real-time motion state and displays a virtual scene that is consistent with the real motion state, so that the user can visually perceive that the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, thereby eliminating or alleviating the user's motion sickness symptoms.

[0035] In a second aspect, the present invention provides a virtual reality presentation device, the device comprising:

[0036] A data acquisition module, used to acquire real-time motion data of the sensor;

[0037] The motion state estimation module is used to calculate the real-time motion state of the XR device based on the real-time motion data;

[0038] The scene construction module is used to construct a virtual scene corresponding to the real-time motion state based on the real-time motion state, and display the virtual scene on the XR device.

[0039] In a third aspect, the present invention provides an XR device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the virtual reality presentation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the virtual reality presentation method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 4 is a flowchart of a virtual reality presentation method according to an embodiment of the present invention.

[0043] Figure 2 The figure is a flow chart of a method for eliminating or alleviating motion sickness based on an inertial sensor according to an embodiment of the present invention.

[0044] Figure 3 4 is a flow chart of another virtual reality presentation method according to an embodiment of the present invention.

[0045] Figure 4 4 is a flow chart of another virtual reality presentation method according to an embodiment of the present invention.

[0046] Figure 5 4 is a flow chart of another method for eliminating or alleviating motion sickness based on an inertial sensor according to an embodiment of the present invention.

[0047] Figure 6 4 is a structural block diagram of a virtual reality presentation device according to an embodiment of the present invention.

[0048] Figure 7 Schematic diagram of the hardware structure of the XR device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] In the embodiment of the present invention, a virtual reality presentation method is provided, which is used in an XR (Extended Reality) device. It should be noted that the execution subject may be a virtual reality presentation device, which may be implemented as part or all of an XR device through software, hardware, or a combination of software and hardware. The XR device may be a VR device, an AR device, an MR device, an HMD and / or other XR devices, including but not limited to a VR head-mounted device, an AR head-mounted device, an MR head-mounted device, etc. In the following method embodiments, the execution subject is an XR device as an example for explanation.

[0051] The XR device in this embodiment is suitable for use in scenarios where the XR device is on a motion platform, providing the user with a real motion state experience to eliminate the user's motion sickness symptoms. The motion platforms that can be used include, but are not limited to: the back seat or co-pilot of a car, a bus, an airplane, a ship, etc. The virtual reality presentation method provided by the present invention does not require strict synchronization of data between the XR device and the sensor installed on the motion platform. The real-time motion state of the XR device is estimated through the real-time motion data of the XR device, and the corresponding virtual scene is constructed and displayed, so that the user can visually perceive that the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, thereby eliminating or alleviating the user's motion sickness symptoms, greatly reducing computing power consumption, and improving the effect of the XR device in eliminating motion sickness symptoms.

[0052] According to an embodiment of the present invention, an embodiment of a virtual reality presentation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in an XR device such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0053] In this embodiment, a virtual reality presentation method is provided, which can be used for the above-mentioned XR devices, such as VR devices, AR devices, MR devices, HMDs and / or other XR devices, including but not limited to VR head-mounted devices, AR head-mounted devices, and MR head-mounted devices. Figure 1 is a flow chart of a virtual reality presentation method according to an embodiment of the present invention. Figure 1As shown, the process includes the following steps:

[0054] Step S101, obtaining real-time motion data of the XR device.

[0055] In one example, real-time measurement data of an inertial sensor in an XR device is obtained, and optionally, real-time measurement data of devices such as a magnetometer is also obtained, wherein the real-time measurement data of the inertial sensor includes: acceleration, angular velocity; the real-time motion state to be estimated includes: position, angle, and speed.

[0056] Step S102: Calculate the real-time motion state of the XR device based on the real-time motion data.

[0057] In one example, since the real-time measurement data obtained by the inertial sensor has bias and noise, the collected real-time measurement data is filtered, integrated, and processed to calculate the motion state at the current moment.

[0058] Step S103: Based on the real-time motion state, a virtual scene corresponding to the real-time motion state is constructed, and the virtual scene is displayed on the XR device.

[0059] In one example, the human eye can perceive the position, speed, angle, and angular velocity in the virtual space, while the vestibule is not sensitive to absolute position and absolute angle, but is more sensitive to acceleration and angular velocity. Therefore, the XR device display system displays a three-dimensional virtual scene consistent with the real-time motion state based on the real-time motion state. The motion state perceived by the user's vision is consistent with the real motion state perceived by the vestibule, which can eliminate or alleviate motion sickness.

[0060] In one implementation scenario, Figure 2 FIG. 1 is a flow chart of a method for eliminating or alleviating motion sickness based on an inertial sensor according to an embodiment of the present invention. Figure 2 As shown, the method for eliminating or alleviating motion sickness based on inertial sensors applied to XR devices includes: S1: obtaining real-time measurement data of inertial sensors and magnetometers (optional) in VR devices, and initializing the motion state after the data is stable. S2: filtering, integrating, and other processing are performed on the real-time measurement data to estimate the motion state at the current moment and smooth the motion state. S3: The VR device display system displays a 3D virtual scene consistent with the relative motion state, and acceleration and angular velocity consistent with the user's vision and vestibular perception.

[0061] The virtual reality presentation method provided in this embodiment calculates the real-time motion state of the XR device through the real-time motion data of the XR device, constructs and displays the corresponding virtual scene, so that the real-time motion state in the virtual scene displayed by the XR device perceived by the user through vision is consistent with the actual motion state perceived by the user's vestibule, thereby eliminating or alleviating the motion sickness symptoms of the user. There is no need for strict synchronization of data between the XR device and the sensors installed on the motion platform, which greatly reduces the computing power consumption and improves the effect of the XR device in eliminating motion sickness symptoms.

[0062] In this embodiment, a virtual reality presentation method is provided, which can be used for the above-mentioned XR devices, such as VR devices, AR devices, MR devices, HMDs and / or other XR devices, including but not limited to VR head-mounted devices, AR head-mounted devices, and MR head-mounted devices. Figure 3 is a flow chart of another virtual reality presentation method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0063] Step S301, obtaining real-time motion data of the XR device.

[0064] Specifically, the above step S301 includes:

[0065] Step S3011, collect real-time motion data using sensors installed on the XR device.

[0066] Step S3012: Based on the real-time motion data, determine whether the XR device is affected by external forces other than gravity.

[0067] Step S3013: When the XR device is not affected by any external force except gravity, the real-time motion data is initialized.

[0068] In one example, by acquiring real-time motion data from inertial sensors, magnetometers and other devices in the VR device, the motion state is initialized after the real-time motion data stabilizes. The purpose of initializing the motion state (position, speed, acceleration, angle, angular velocity) is to determine the reference world coordinate system and origin of the device, in preparation for the next step of motion state calculation. When the VR device is not affected by external forces other than gravity, it can be initialized. There are two specific scenarios: when the VR device is stationary or when the VR device is moving in a uniform straight line, the real-time motion data is initialized. The initialization method is as follows:

[0069] (1) Select stable IMU inertial sensor data (including acceleration and angular velocity data) over a period of time, calculate the mean and variance of acceleration and angular velocity, and determine whether the device is affected by external forces other than gravity. If the VR device is not affected by external forces other than gravity, the axial relationship between the current gravity and the IMU inertial sensor coordinate system can be calculated based on the magnitude of gravity, and then the axial direction of the reference world coordinate system can be initialized.

[0070] (2) Since it is impossible to determine whether the VR device is absolutely stationary or moving in a uniform linear motion based on the data from the IMU inertial sensor itself, in a complete VR environment, the user cannot perceive the absolute position, absolute speed, and absolute angle. Therefore, during initialization, the initial position, initial speed, and initial angle of the VR device relative to the reference world system are set to zero or other empirical values ​​to complete the initialization of the real-time motion data.

[0071] In this method, the real-time motion data is initialized when the collected real-time motion data is stable, and the reference world coordinate system and origin of the XR device are determined to prepare for the next step of motion state estimation.

[0072] Step S302: Calculate the real-time motion state of the XR device based on the real-time motion data.

[0073] Specifically, the above step S302 includes:

[0074] Step S3021, based on the real-time motion data, calculate and obtain the bias of the sensor at the current moment.

[0075] Step S3022, obtaining the real-time motion state at the previous moment, combining it with the sensor bias at the current moment, and integrating and recursively obtaining the real-time motion state at the current moment.

[0076] In one example, the real-time measurement data is filtered, integrated, etc., the motion state at the current moment is calculated, and the motion state is smoothed, specifically including:

[0077] (1) Since the real-time motion data (acceleration, angular velocity) collected by the original IMU inertial sensor contains bias and noise, the bias and noise need to be processed. The noise processing method is to perform consistency check and low-pass filtering on the original collected acceleration and angular velocity data to eliminate jitter caused by high-frequency interference.

[0078] (2) IMU bias calculation: As in the initialization, based on the stable observation state (uniform motion or absolute stillness) over a period of time as a priori, the mean value of acceleration and angular velocity over a period of time is statistically calculated as the IMU bias.

[0079] (3) Motion state estimation: The motion state at the previous moment, the real-time motion data (acceleration, angular velocity) collected by the inertial sensor IMU at the current moment, the gravity component, and the IMU bias are integrated and recursively calculated to obtain the motion state at the current moment.

[0080] In this method, the motion state of the XR device at the current moment is calculated by utilizing the bias at the current moment, providing posture information for subsequent virtual scene rendering, thereby facilitating the construction of the virtual scene.

[0081] Step S3023: using a damped integral method, adjusting the damping coefficient of the real-time motion state to obtain the first real-time motion state, so as to avoid divergence of the first real-time motion state.

[0082] In one example, the motion state obtained by integral recursion may produce low-frequency drift due to IMU noise and inaccurate bias. Therefore, the solution is to use a damped integral method to adjust the damping coefficient to avoid the divergence of the motion state.

[0083] Step S3024: smoothing the first real-time motion state to obtain a second real-time motion state.

[0084] In one example, the jitter of the output motion state may be smoothed using SG or Euro filtering.

[0085] In this manner, by utilizing a damped integration method, the divergence of the motion state is avoided and the motion state is smoothed, thereby further improving the accuracy of the processed motion state and making it closer to the actual motion state of the virtual display device.

[0086] Step S303: Based on the real-time motion state, a virtual scene corresponding to the real-time motion state is constructed, and the virtual scene is displayed on the XR device.

[0087] Specifically, the above step S303 includes:

[0088] Step S3031, according to the current timestamp, refresh delay and other delays, based on the real-time motion state, calculate and generate the timestamp of the virtual scene and the motion state corresponding to the virtual scene.

[0089] Step S3032: In combination with the timestamp of the virtual scene and the motion state corresponding to the virtual scene, the posture of the XR device is adjusted, and the virtual scene corresponding to the real-time motion state is rendered and displayed on the XR device.

[0090] The damped integration method described in the present invention may be an integration method based on a first-order complementary filter or a Mahony filtering algorithm.

[0091] In one example, the VR device display system displays a 3D virtual scene consistent with the relative motion state according to the relative motion state, so that the user's vision and vestibular perception are consistent with the acceleration and angular velocity. Specifically, it includes:

[0092] The VR device display system calculates the timestamp required to generate virtual content based on the current timestamp, refresh delay, other delays, etc., and predicts the required motion state. The method of motion state prediction includes: using the Hot-Winter method based on the current latest motion state, and performing prediction and filtering at the same time. When rendering a 3D virtual field, the posture of the left and right virtual cameras is calculated by the predicted motion state. The calculation method is: the predicted IMU motion state is converted into the running state of the center of the eyes through the device structure external parameter, and the device is translated to the left and right by 1 / 2 of the pupil distance, that is, IPD / 2 (Interpupillary Distance), respectively, to obtain the posture of the two virtual cameras. Among them, the 3D virtual scene is made by software, which can be similar to a highway scene, a city street scene, etc., and the user switches according to personal preferences.

[0093] In this method, the XR device adjusts its posture in real time according to the real-time motion state and displays a virtual scene that is consistent with the real motion state, so that the user can visually perceive that the real-time motion state in the virtual scene displayed by the XR device is consistent with the real motion state perceived by the user's vestibule, thereby eliminating or alleviating the user's motion sickness symptoms.

[0094] The virtual reality presentation method provided in this embodiment initializes the real-time motion data when the collected real-time motion data is stable, determines the reference world coordinate system and origin of the XR device, and prepares for the next step of motion state estimation. By using the bias at the current moment, the motion state of the XR device at the current moment is calculated, and the posture information is provided for the subsequent rendering of the virtual scene, which facilitates the construction of the virtual scene. By using the damped integration method to avoid the divergence of the motion state and smooth the motion state, the accuracy of the processed motion state is further improved, which is closer to the real motion state of the virtual display device. According to the real-time motion state, the XR device adjusts the posture in real time and displays a virtual scene consistent with the real motion state, so that the user can visually perceive the real-time motion state in the virtual scene displayed by the XR device consistent with the real motion state perceived by the user's vestibule, eliminating or alleviating the motion sickness symptoms of the user.

[0095] In this embodiment, a virtual reality presentation method is provided, which can be used for the above-mentioned XR devices, such as VR devices, AR devices, MR devices, HMDs and / or other XR devices, including but not limited to VR head-mounted devices, AR head-mounted devices, and MR head-mounted devices. Figure 4is a flowchart of another virtual reality presentation method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0096] Step S401, obtain real-time motion data of the sensor. For details, see Figure 3 Step S301 of the illustrated embodiment will not be described in detail here.

[0097] Step S402: Calculate the real-time motion state of the XR device based on the real-time motion data.

[0098] Specifically, the above step S402 includes:

[0099] Step S4021, based on the real-time motion data, the zero bias of the sensor is calculated using a neural network.

[0100] In some optional implementations, the above step S4021 includes:

[0101] Step a1, obtaining the real-time motion data of the sensor and the real posture data corresponding to the real-time motion data at the same time, and constructing a training set.

[0102] Step a2: training the neural network based on the training set to obtain a pre-trained neural network, and the pre-trained neural network is used to calculate the zero bias of the sensor.

[0103] In one example, the zero bias of IMU is mainly manifested in: acceleration and angular velocity at rest will produce different degrees of bias over time, which will seriously affect the integral calculation of the motion state, causing large-scale drift and causing more severe dizziness to the user. To calculate the zero bias of the inertial sensor IMU using a neural network method, the network needs to be trained first. During training, a large amount of real-time measurement data of the IMU inertial sensor and the actual posture at the corresponding time need to be collected. After the training is completed, the trained neural network can be used on the mobile terminal for real-time operation.

[0104] In this way, the neural network is trained with a large amount of data, which further improves the accuracy of motion state prediction under zero bias conditions, thereby avoiding causing more severe dizziness to the user and improving the user experience.

[0105] Step S4022: based on the zero bias, estimate the real-time motion state of the XR device.

[0106] In one example, a neural network method is used to calculate the zero bias of an inertial sensor (IMU), thereby integrating and solving the motion state; the motion state and motion increment are also smoothed by SG or one-Euro filtering.

[0107] In this way, by utilizing neural networks, the motion state can be estimated more accurately in the event of large-scale drift, avoiding bias and causing more severe dizziness to the user, thereby further improving the user experience.

[0108] Step S403: Based on the real-time motion state, a virtual scene corresponding to the real-time motion state is constructed, and the virtual scene is displayed on the XR device. Figure 3 Step S303 of the illustrated embodiment will not be described in detail here.

[0109] In one implementation scenario, Figure 5 is a flow chart of another method for eliminating or alleviating motion sickness based on an inertial sensor according to an embodiment of the present invention, such as Figure 5 As shown, the method for eliminating or alleviating motion sickness based on inertial sensors applied to VRXR devices includes: S1: obtaining real-time measurement data of inertial sensors and magnetometers (optional) in VR devices, and initializing the motion state after the data is stable. S2: passing the real-time measurement data through a neural network to calculate the zero bias, thereby estimating the motion state at the current moment and smoothing the motion state. S3: The VR device display system displays a consistent 3D virtual scene, acceleration and angular velocity consistent with the user's vision and vestibular perception according to the relative motion state.

[0110] The virtual reality presentation method provided in this embodiment uses a neural network to more accurately calculate the motion state in the case of large-scale drift, thereby avoiding bias causing more severe dizziness to the user and further improving the user experience. By training the neural network with a large amount of data, the accuracy of motion state estimation in the case of zero bias is further improved, thereby avoiding causing more severe dizziness to the user and improving the user experience.

[0111] In this embodiment, a virtual reality presentation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0112] This embodiment provides a virtual reality presentation device, such as Figure 6 As shown, including:

[0113] The data acquisition module 601 is used to acquire the real-time motion data of the sensor. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0114] The motion state estimation module 602 is used to calculate the real-time motion state of the XR device based on the real-time motion data. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0115] The scene construction module 603 is used to construct a virtual scene corresponding to the real-time motion state based on the real-time motion state, and display the virtual scene on the XR device. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0116] In some optional implementations, the data acquisition module 601 includes:

[0117] The data acquisition unit is used to collect real-time motion data using sensors installed on the XR device.

[0118] The external force judgment unit is used to judge whether the XR device is subjected to external forces other than gravity based on real-time motion data.

[0119] The data initialization unit is used to initialize the real-time motion data when the XR device is not affected by any external force other than gravity.

[0120] In some optional implementations, the motion state estimation module 602 includes:

[0121] The bias calculation unit is used to calculate the bias of the sensor at the current moment based on the real-time motion data.

[0122] The first motion state estimation unit is used to obtain the real-time motion state at the previous moment, and integrate and recursively obtain the real-time motion state at the current moment in combination with the bias of the sensor at the current moment.

[0123] In some optional implementations, the motion state estimation module 602 further includes:

[0124] The divergence processing unit is used to adjust the damping coefficient of the real-time motion state by using a damped integration method to obtain the first real-time motion state so as to avoid the divergence of the first motion state.

[0125] The smoothing processing unit is used to perform smoothing on the first motion state to obtain a second motion state.

[0126] In some optional implementations, the motion state estimation module 602 further includes:

[0127] The zero bias calculation unit is used to calculate the zero bias of the sensor based on real-time motion data using a neural network.

[0128] The second motion state estimating unit is used to estimate the real-time motion state of the XR device based on the zero bias.

[0129] In some optional implementations, the zero bias calculation unit includes:

[0130] The training set construction subunit is used to obtain the real-time motion data of the sensor and the real posture data corresponding to the real-time motion data at the same time, so as to construct a training set.

[0131] The network training subunit is used to train the neural network based on the training set to obtain a pre-trained neural network, and the pre-trained neural network is used to calculate the zero bias of the sensor.

[0132] In some optional implementations, the scene construction module 603 includes:

[0133] The motion state prediction unit is used to calculate and generate the timestamp of the virtual scene and the motion state corresponding to the virtual scene based on the real-time motion state according to the current timestamp, refresh delay and other delays.

[0134] The posture adjustment unit is used to adjust the posture of the XR device in combination with the timestamp of the virtual scene and the motion state corresponding to the virtual scene, and render and display the virtual scene corresponding to the real-time motion state on the XR device.

[0135] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0136] The virtual reality presentation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0137] The embodiment of the present invention also provides an XR device having the above Figure 6 The virtual reality presentation device shown.

[0138] See also Figure 7 , Figure 7 is a schematic diagram of the structure of an XR device provided by an optional embodiment of the present invention, such as Figure 7As shown, the XR device includes: one or more processors 10, a memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the XR device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple XR devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0139] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0140] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0141] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the XR device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely disposed relative to the processor 10, and these remote memories may be connected to the XR device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0143] The XR device further includes a communication interface 30 for the XR device to communicate with other devices or a communication network.

[0144] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0145] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A virtual reality presentation method, characterized in that: The method comprises: Acquire real-time motion data of the XR device; Based on the real-time motion data, calculate the real-time motion state of the XR device; Based on the real-time motion state, a virtual scene corresponding to the real-time motion state is constructed, and the virtual scene is displayed on the XR device.

2. The method according to claim 1, characterized in that The acquiring real-time motion data of the XR device includes: Collecting the real-time motion data using a sensor installed on the XR device; Based on the real-time motion data, determining whether the XR device is subjected to external forces other than gravity; When the XR device is not affected by any external force other than gravity, the real-time motion data is initialized.

3. The method according to claim 2, characterized in that The calculating the real-time motion state of the XR device based on the real-time motion data includes: Based on the real-time motion data, calculate the bias of the sensor at the current moment; The real-time motion state at the previous moment is obtained, combined with the bias of the sensor at the current moment, and the real-time motion state at the current moment is obtained by integral recursion.

4. The method according to claim 3, characterized in that After the real-time motion state at the current moment is obtained by the integral recursion, the method further includes: Using a damped integral method, the damping coefficient of the real-time motion state is adjusted to obtain a first real-time motion state, so as to avoid divergence of the first real-time motion state; The first real-time motion state is smoothed to obtain a second real-time motion state.

5. The method according to claim 3, characterized in that: The calculating the real-time motion state of the XR device based on the real-time motion data further includes: Based on the real-time motion data, a zero bias of the sensor is calculated using a neural network; Based on the zero bias, the real-time motion state of the XR device is estimated.

6. The method according to claim 5, characterized in that The method of obtaining the zero bias of the sensor by using a neural network calculation includes: Acquire the real-time motion data of the sensor and the real posture data corresponding to the real-time motion data at the same time, and construct a training set; The neural network is trained based on the training set to obtain a pre-trained neural network, and the pre-trained neural network is used to calculate the zero bias of the sensor.

7. The method according to claim 1, characterized in that The constructing a virtual scene corresponding to the real-time motion state based on the real-time motion state, and displaying the virtual scene on the XR device comprises: According to the current timestamp, refresh delay and other delays, based on the real-time motion state, calculate and generate the timestamp of the virtual scene and the motion state corresponding to the virtual scene; In combination with the timestamp of the virtual scene and the motion state corresponding to the virtual scene, the posture of the XR device is adjusted, and the virtual scene corresponding to the real-time motion state is rendered and displayed on the XR device.

8. The method according to any one of claims 1 to 7, characterized in that: The XR device is applied on a motion platform.

9. A virtual reality presentation device, characterized in that: The device comprises: A data acquisition module, used to acquire real-time motion data of the sensor; A motion state estimation module, used to calculate the real-time motion state of the XR device based on the real-time motion data; A scene construction module is used to construct a virtual scene corresponding to the real-time motion state based on the real-time motion state, and display the virtual scene on the XR device.

10. An XR device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the virtual reality presentation method according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the virtual reality presentation method according to any one of claims 1 to 8.

Images