A virtual reality system and method

Abstract

The application relates to a virtual reality system and method, and relates to the technical field of virtual reality.The system comprises a VR rendering host module, a space positioning module, a wireless transmission module, a VR terminal module and a control module; the space positioning module collects position data and attitude data of the VR terminal module and determines six-degree-of-freedom positioning data according to the position data and the attitude data; the six-degree-of-freedom positioning data carrying a timestamp is sent to the control module; the control module predicts the attitude at the next moment based on the received six-degree-of-freedom positioning data and sends the attitude prediction result to the VR rendering host module; the VR rendering host module renders and encodes to generate virtual reality video data packets based on the attitude prediction result; the video data packets carrying the timestamp are sent to the VR terminal module via the wireless transmission module; the VR terminal module decodes the received video data packets based on the timestamp to obtain a decoded picture, and outputs a display picture after performing space compensation processing on the decoded picture.

Description

Technical Field

[0001] This application relates to the technical field of VR devices, and in particular to a virtual reality system and method. Background Technology

[0002] Virtual reality systems present immersive 3D images to users through head-mounted displays. In large-space applications, users need to move freely within an area of ​​hundreds of square meters and interact naturally with the virtual scene.

[0003] In large-space application scenarios, users move freely in the scene at walking speed. In addition to head rotation, the viewpoint also shifts due to body displacement. When the end-to-end latency exceeds a certain threshold, the screen content becomes misaligned with the user's actual perspective. That is, the user feels that the screen is lagging behind their own movement, which can cause dizziness and loss of direction after prolonged use. Summary of the Invention

[0004] In order to at least partially solve the above-mentioned technical problems, this application provides a virtual reality system.

[0005] In a first aspect, the virtual reality system provided in this application includes: a VR rendering host module, a spatial positioning module, a wireless transmission module, a VR terminal module, and a control module;

[0006] The spatial positioning module collects the position and attitude data of the VR terminal module and determines six-degree-of-freedom positioning data based on the position and attitude data; the six-degree-of-freedom positioning data with timestamps is then sent to the control module.

[0007] The control module predicts the attitude at the next moment based on the received six-degree-of-freedom positioning data and sends the attitude prediction result to the VR rendering host module;

[0008] The VR rendering host module renders and encodes virtual reality video data packets based on the pose prediction results; the video data packets carrying the timestamp are then sent to the VR terminal module via the wireless transmission module.

[0009] The VR terminal module decodes the received video data packets based on the timestamp to obtain the decoded image, performs spatial compensation processing on the decoded image, and then outputs the display image.

[0010] By adopting the above technical solutions, the consistency of posture during screen display is improved, reducing user dizziness caused by asynchrony between posture and screen. Attached Figure Description

[0011] Figure 1 This is a system diagram of a virtual reality system according to an embodiment of this application;

[0012] Figure 2 This is a flowchart of a virtual reality method according to an embodiment of this application. Detailed Implementation

[0013] The following is in conjunction with the appendix Figure 1-2 The present application will be further described with reference to specific embodiments:

[0014] In one embodiment, a virtual reality system includes: a VR rendering host module 10, a spatial positioning module 20, a wireless transmission module 30, a VR terminal module 40, and a control module 50; wherein, the spatial positioning module 20 collects position data and attitude data of the VR terminal module 40 and determines six-degree-of-freedom positioning data based on the position data and attitude data; sends the six-degree-of-freedom positioning data carrying a timestamp to the control module 50; the control module 50 predicts the attitude at the next moment based on the received six-degree-of-freedom positioning data and sends the attitude prediction result to the VR rendering host module 10; the VR rendering host module 10 renders and encodes virtual reality video data packets based on the attitude prediction result; sends the video data packets carrying the timestamp to the VR terminal module 40 via the wireless transmission module 30; the VR terminal module 40 decodes the received video data packets based on the timestamp to obtain a decoded image, performs spatial compensation processing on the decoded image, and outputs a display image.

[0015] Specifically, the spatial positioning module collects position and attitude data from the VR terminal module, determines six-degree-of-freedom (6DOF) positioning data based on the position and attitude data, adds a timestamp to each positioning data, and sends it to the control module. The control module receives the timestamped 6DOF positioning data, predicts the attitude for the next moment based on historical positioning data, and outputs the attitude prediction result. This attitude prediction result includes the predicted 3D head position and the predicted 3D head orientation angle, reflecting the expected pose of the VR terminal module at the next display moment. The control module sends this attitude prediction result to the VR rendering host module. The VR rendering host module receives the attitude prediction result and uses it as the predicted pose for the rendering camera. At the predicted pose, it performs 3D scene rendering to generate a frame of virtual reality image. After rendering, the VR rendering host module performs video encoding on the rendered image, encapsulating the encoded bitstream into video data packets. Each video data packet carries a timestamp corresponding to the predicted pose. The wireless transmission module sends the video data packets generated by the VR rendering host module to the VR terminal module via wireless communication. The VR terminal module receives the aforementioned video data packets and performs timing alignment based on the timestamps carried in the video data packets. Subsequently, the VR terminal module decodes the received video data packets to obtain the decoded image. The VR terminal module obtains its actual head posture at the current moment, compares it with the posture at the rendering moment corresponding to the decoded image, and performs spatial compensation transformation on the decoded image based on the posture difference to align the output display image with the actual head posture, eliminating the posture deviation between the rendering moment and the display moment.

[0016] Through the above technical solutions, posture prediction and spatial compensation are added during the VR device positioning process. Posture prediction provides the expected viewing posture for the rendering, making the rendered image as close as possible to the actual viewpoint at the time of display. Spatial compensation performs final alignment of the image with the latest real posture on the terminal side, which improves the posture consistency at the time of image display and reduces the user's dizziness caused by the asynchronous posture and image.

[0017] In some embodiments, the spatial positioning module 20 includes an inertial measurement unit 21, an infrared positioning unit 22, and an ultra-wideband positioning unit 23; determining six-degree-of-freedom positioning data includes:

[0018] The acceleration and angular velocity data collected by the inertial measurement unit 21 are accumulated and integrated to obtain the relative attitude and displacement changes between two adjacent moments;

[0019] The global reference system coordinates of the VR terminal module 40 are obtained based on the infrared positioning unit 22 and the ultra-wideband positioning unit 23;

[0020] The Mahalanobis distance of the global reference frame coordinates is calculated, and outlier detection is performed based on the Mahalanobis distance. When the global reference frame coordinates are outliers, the observation update of the extended Kalman filter model is cut off, and only the relative attitude and displacement changes are used as predicted values ​​for short-term extrapolation to estimate the six-degree-of-freedom positioning data. When the global reference frame coordinates are normal, the relative attitude and displacement changes are used as predicted values, and the global reference frame coordinates are used as observed values ​​to update the extended Kalman filter model to estimate the six-degree-of-freedom positioning data. When the VR terminal module 40 moves between the coverage areas of multiple positioning base stations, the positioning results of different base stations are weighted and averaged to obtain the six-degree-of-freedom positioning data.

[0021] Specifically, the spatial positioning module performs cumulative integration on the acceleration and angular velocity data collected by the inertial measurement unit (IMU), with the integration time interval matched to the output frequency of the IMU. The first integration of acceleration yields the velocity change, the second integration yields the position change, and the first integration of angular velocity yields the attitude angle change. The integration results between two adjacent sampling times constitute the relative attitude and displacement changes.

[0022] The infrared positioning unit captures the spatial position of infrared markers set on the VR terminal module by deploying multiple infrared cameras in space, and calculates the three-dimensional coordinates of the VR terminal module in the global coordinate system based on the spatial position of the infrared markers. The ultra-wideband positioning unit determines the global coordinates of the VR terminal module by measuring the signal flight time between the ultra-wideband tag on the VR terminal module and multiple fixed ultra-wideband base stations and using trilateration.

[0023] The spatial positioning module calculates the Mahalanobis distance of the currently acquired global reference frame coordinates relative to the historical observation distribution. The calculated Mahalanobis distance is compared with a preset anomaly threshold: when the Mahalanobis distance is less than or equal to the threshold, the global reference frame coordinates are considered normal, and the module enters the extended Kalman filter normal fusion mode. At this time, the spatial positioning module uses relative attitude and displacement changes as predicted values ​​from the extended Kalman filter model for state prediction, and uses the global reference frame coordinates as observed values ​​for observation updates. After fusion, six-DOF positioning data is output.

[0024] When the Mahalanobis distance exceeds the threshold, the global reference frame coordinates are determined to be outliers. The spatial positioning module cuts off the observation update channel to the extended Kalman filter model and uses only the relative attitude and displacement changes as prediction values ​​for short-term recursion to estimate the six-degree-of-freedom positioning data. The Mahalanobis distance threshold mentioned above can be set based on the statistical characteristics of historical normal observation data.

[0025] When the VR terminal module moves within a large space, from the coverage area of ​​the first positioning base station to the coverage area of ​​the second positioning base station, the spatial positioning module receives the positioning results from the two base stations; it then performs a weighted average of the positioning results from the two base stations according to their signal quality indicators, and fuses and outputs a smoothly transitioning six-degree-of-freedom positioning data.

[0026] If only inertial measurement unit (IMU) integration is used for calculation, the measurement noise of angular velocity and acceleration accumulates with the number of integrations, resulting in positioning drift within minutes. If only infrared or ultra-wideband (UWB) positioning is used to obtain absolute coordinates, the positioning will completely fail if the infrared marker is blocked or the UWB signal is reflected by multipath. In this application, the IMU provides short-period incremental information, while infrared and UWB positioning provide absolute references that do not accumulate over time. During normal operation, the incremental information maintains the continuity of inter-frame positioning, and the absolute reference periodically corrects the accumulated deviation of the incremental information. When infrared or UWB fails, if only the fusion weight of abnormal observations is reduced, even if the weight is reduced to a very small value, the abnormal values ​​will still inject residual errors into the filter state. This application uses the method of cutting off the observation channel rather than reducing the weight, so that the abnormal values ​​do not participate in the state update, and the filter only maintains the output with inertial incremental information for a short time.

[0027] In some embodiments, the extended Kalman filter model is updated using the global reference frame coordinates as observations to estimate the six-degree-of-freedom positioning data, including:

[0028] The state vector of the extended Kalman filter model is constructed, including three-dimensional position, three-dimensional velocity, three-dimensional attitude, and the zero bias error of the inertial measurement unit 21. Based on the relative attitude and displacement changes, the state equation is predicted and deduced to obtain the prior state estimate and the prior covariance matrix. The Kalman gain is calculated and the observation residual is calculated in combination with the global reference frame coordinates. The prior state estimate is updated based on the Kalman gain and the observation residual, and the posterior state estimate is output. The three-dimensional position and the three-dimensional attitude in the posterior state estimate are obtained as the six-degree-of-freedom positioning data. The zero bias error in the posterior state estimate is obtained and fed back to the cumulative integration processing stage of the acceleration and angular velocity data.

[0029] Specifically, the state vector of the extended Kalman filter model includes three-dimensional position, three-dimensional velocity, three-dimensional attitude, and the zero bias error of the inertial measurement unit; the total dimension of the extended Kalman filter model is the sum of the above components; by incorporating the zero bias error of the inertial measurement unit into the state vector for online estimation, the extended Kalman filter model can not only track the motion state of the VR terminal module but also track the error state of the sensor itself.

[0030] Based on the relative attitude and displacement changes output in step S201, a predictive derivation of the state equations is performed. The state prediction can employ, for example, the mechanical arrangement equations of a strapdown inertial navigation system, mapping acceleration and angular velocity to changes in position, velocity, and attitude, and propagating an estimate of the zero bias error. The prediction step outputs prior state estimates and a prior covariance matrix, which reflects the uncertainty of the predicted values.

[0031] The Kalman gain matrix is ​​calculated using the prior covariance matrix, the observation noise covariance matrix, and the observation matrix. This can be understood as: the Kalman gain matrix measures the confidence in the observation data during observation updates; the greater the observation noise, the smaller the Kalman gain, and the lower the filter's confidence in the observation. The observation residuals are obtained by subtracting the predicted values ​​from the global reference frame coordinates and the state equation. The prior state estimate is then corrected by weighting the observation residuals with the Kalman gain matrix, and the posterior state estimate is output. The three-dimensional position and three-dimensional attitude components are extracted from the posterior state estimate as the six-degree-of-freedom positioning data for this output.

[0032] The zero-bias error component is extracted from the posterior state estimate and fed back to the inertial data preprocessing stage. The online estimate of the zero-bias error described above removes the sensor drift component from the original measurement data, making the input data for subsequent integration calculations closer to the actual motion and suppressing the accumulation rate of integration error.

[0033] Through the above technical solutions, the extended Kalman filter differs from simple weighted averaging or complementary filtering. It models the system state as a stochastic process, propagating the state and its uncertainty in each prediction step, and calculating the optimal gain based on the observation noise level in the observation update. When the noise level of the inertial measurement unit is high, the covariance of the state prediction increases, the Kalman gain tilts towards the observation direction, and the filter trusts the global reference coordinates more. When the observation noise is large, the gain tilts towards the prediction direction, and the inertial recursion results are trusted more. Through the extended Kalman filter, six-degree-of-freedom positioning data in the best statistical sense can be obtained instead of a mixture of fixed weights.

[0034] In some embodiments, cutting off the observation updates to the extended Kalman filter model and using only the relative attitude and displacement changes as predicted values ​​for short-term extrapolation to estimate the six-degree-of-freedom positioning data includes: obtaining the posterior state estimate output by the extended Kalman filter model at the most recent normal moment before the anomaly occurred, as the starting reference point for short-term extrapolation; freezing the error covariance update of the extended Kalman filter model and stopping the calculation of Kalman gain; using the starting reference point as a basis, using the acceleration and angular velocity data output in real time by the inertial measurement unit 21 for recursion, and outputting the extrapolated attitude and position as the six-degree-of-freedom positioning data; continuously calculating the Mahalanobis distance of the newly acquired global reference frame coordinates until the Mahalanobis distance recovers to below the anomaly threshold, then unfreezing and resuming the observation updates to the extended Kalman filter model.

[0035] Specifically, when an anomaly in the global reference frame coordinates is detected, the spatial positioning module retrieves the posterior state estimate output by the extended Kalman filter model at the most recent normal moment before the anomaly occurred, and uses it as the starting reference point for short-term extrapolation; this starting reference point contains the optimal state estimate result under normal fusion mode.

[0036] The spatial positioning module freezes the error covariance matrix of the extended Kalman filter model, stopping the prediction update of the covariance and the calculation of the Kalman gain; the purpose of the above freezing operation is to cut off the channel for abnormal observations to transmit erroneous information to the state estimation.

[0037] Starting from the initial reference point, the space positioning module uses only the acceleration and angular velocity data output in real time by the inertial measurement unit to perform recursive calculations through strapdown inertial navigation mechanics. No external observation corrections are introduced during this recursive process, and the calculated three-dimensional attitude and three-dimensional position are output as the current six-degree-of-freedom positioning data. Although the accuracy of the recursion decreases as the calculation time increases, the accuracy can be maintained at a high level in the first few hundred milliseconds of the calculation, which is sufficient to cover the duration of most occlusion and interference events.

[0038] The spatial positioning module continuously calculates the Mahalanobis distance of the newly acquired global reference frame coordinates. When the Mahalanobis distance of several consecutive frames recovers to below the anomaly threshold, it is determined that the global reference has returned to normal. At this point, the frozen state of the covariance matrix is ​​released, and the current result of pure inertial recursion is used as the prediction value of the extended Kalman filter model, restoring normal input to the observation update channel.

[0039] Through the above technical solution, when infrared or ultra-wideband positioning outputs abnormal values ​​due to external interference, if the filter continues to perform observation updates, the erroneous position information will cause jumps in the state vector, leading to the user's perception of sudden image shifts or jitter. The solution in this application cuts off the observation input and freezes covariance propagation: since the covariance matrix carries the filter's perception of its own state uncertainty, if the covariance continues to propagate and the Kalman gain is calculated during the abnormal period, even without new observation input, the covariance matrix will expand due to the superposition of process noise, resulting in an abnormally large gain for the first normal observation value when returning to normal, causing significant overshoot in the state; freezing covariance propagation allows the filter to maintain the confidence level at the last moment before the abnormality, and upon recovery, it smoothly converges back to the normal state with a moderate gain, improving the positioning accuracy of six-degree-of-freedom positioning data.

[0040] In some embodiments, predicting the pose at the next moment includes:

[0041] A Kalman filter algorithm is used to perform linear prediction on historical posture data to obtain a linear prediction result; the historical posture data is then input into a long short-term memory neural network model for nonlinear prediction to obtain a nonlinear prediction result; the motion state of the VR terminal module 40 is evaluated; when the motion state is stable, the fusion weight of the linear prediction result is increased; when the motion state is violent, the fusion weight of the nonlinear prediction result is increased; the linear prediction result and the nonlinear prediction result are then weighted and fused to obtain the final posture prediction result.

[0042] Specifically, the Kalman filter algorithm is used to perform linear prediction on historical attitude data. The Kalman filter algorithm models the motion of the VR terminal module as a constant velocity or constant acceleration linear model, and extrapolates the linear prediction value of the next moment based on several recent historical attitude data points. The advantage of this linear prediction is that it has a small computational load and low prediction residual for steady motion.

[0043] When nonlinear prediction is required, the control module inputs a series of posture data from historical moments into a pre-trained long short-term memory neural network model. This model learns the nonlinear characteristics of posture changes from the historical posture data through its internal forget gate, input gate, and output gate structure. These nonlinear characteristics include turning points in motion trends, acceleration, and deceleration patterns. The long short-term memory neural network model outputs the posture prediction value for the next moment as the nonlinear prediction result. This nonlinear prediction can capture complex motion patterns that the Kalman filter algorithm cannot model.

[0044] The control module analyzes the magnitude and frequency of changes in posture data in real time over a recent period to assess the current motion state of the VR terminal module. When it is determined that the current motion state is stable, the weight coefficient of the linear prediction result in the fusion is increased. When it is determined that the current motion state is violent, the weight coefficient of the nonlinear prediction result is increased. After fusion, the final posture prediction result is output.

[0045] The above technical solutions address the issue that when a user performs actions such as rapid head turning or sudden squatting, the head movement involves acceleration changes and nonlinear transitions. In these cases, Kalman filtering predictions lag behind the actual movement by one to two sampling periods. While Long Short-Term Memory (LSTM) neural network models learn posture change patterns from historical data through gating structures, they have better predictive capabilities for nonlinear movement transitions. However, the output of LSM models lacks the noise smoothing properties of Kalman filtering, potentially leading to high-frequency jitter during stable movements. The solution proposed in this application addresses this: increasing the fusion ratio of the Kalman output during stable movements to obtain smooth and stable predictions; and increasing the fusion ratio of the network model during vigorous movements to improve responsiveness to rapid changes, ensuring high accuracy in posture prediction across various movement states.

[0046] In some embodiments, video encoding of the virtual reality scene includes: disabling bidirectional prediction B-frames, using only I-frames and P-frames for encoding, or adopting a full I-frame encoding mode and setting the image group length to 1; dividing each frame of the virtual reality scene into multiple independent image segments, and encoding and transmitting each image segment independently; allocating a first bitrate to the central region of the field of view and a second bitrate to the edge region of the field of view, wherein the first bitrate is greater than the second bitrate; and encoding the corresponding regions according to the allocated first bitrate and second bitrate.

[0047] Specifically, the encoder of the VR rendering host module disables bidirectional prediction B-frames and uses only I-frames as keyframes and P-frames as forward prediction frames for encoding; or it adopts a full I-frame encoding mode and sets the image group length to 1, that is, each frame is encoded as an independent keyframe; by disabling B-frames, the decoding waiting delay introduced by the frame reordering buffer is eliminated, so that each frame can be transmitted and decoded immediately after encoding.

[0048] Each rendered virtual reality frame is divided into multiple independent image segments of equal size along the horizontal and vertical directions. The encoder performs encoding operations on each segment independently, and there is no encoding dependency between the segments. After each segment is encoded, it is immediately encapsulated into an independent data packet and sent. The VR terminal module can start decoding as soon as it receives a segment, without having to wait for all segments of the entire frame to arrive.

[0049] Based on the characteristics of human vision, the clarity of the central area of ​​the field of view has a much greater impact on the user experience than the edge area; the VR rendering host module allocates a higher first bitrate to the central area of ​​the field of view to ensure the image quality of the central field of view; while allocating a lower second bitrate to the edge area of ​​the field of view to optimize bandwidth utilization under the condition of limited overall bitrate budget.

[0050] The encoder of the VR rendering host module encodes the central and edge regions separately according to the allocated first and second bitrates. The parameters for the two regions can be set independently during encoding: the central region uses a smaller quantization step size to retain more detail, while the edge region uses a larger quantization step size to allow for more information compression. The encoded bitstreams are then transmitted in fragment order.

[0051] The above technical solutions can improve the compression rate by using B-frames for bidirectional prediction in standard video coding. However, the encoding and decoding of B-frames require waiting for the preceding and following reference frames to be ready, introducing at least one additional frame of delay. In VR low-latency applications, the waiting time of this frame reordering buffer will be added to the end-to-end latency.

[0052] This application disables B-frames or uses full I-frame encoding, sacrificing compression efficiency for zero inter-frame latency. Independent encoding of image segments allows the VR terminal module to begin decoding immediately upon receiving the first segment's data packet, eliminating the need to wait for all segments of the entire frame to arrive, thus shortening the overall decoding time. Furthermore, segments in the central region are marked with high priority due to their position index, receiving a larger bit budget in bitrate allocation, while segments in the edge regions receive fewer bits. This concentrates limited encoding bandwidth in the user's visually sensitive areas, improving the user's perception of image clarity without increasing the overall bitrate, thereby enhancing the user experience.

[0053] In some embodiments, sending the video data packet to the VR terminal module 40 includes: real-time monitoring of network status parameters; the status parameters include round-trip time, packet loss rate, jitter parameters, and available bandwidth; adjusting the redundancy of forward error correction according to the packet loss rate: disabling forward error correction when the packet loss rate is below a first threshold; setting a first level of redundancy when the packet loss rate is between the first and second thresholds; setting a second level of redundancy when the packet loss rate is above the second threshold, wherein the second level of redundancy is greater than the first level of redundancy; after adjusting the forward error correction redundancy, adjusting the video encoding bitrate according to the available bandwidth; when the available bandwidth is insufficient, triggering feedback to the VR rendering host module 10: progressively reducing the field of view of the rendered image and encoding and transmitting the central region of the reduced field of view; prioritizing video data, audio data, positioning data, and interactive data, and transmitting them in priority order according to a service quality scheduling strategy.

[0054] Specifically, the wireless transmission module collects the round-trip time, packet loss rate, jitter parameters, and available bandwidth of the current wireless channel in real time. The wireless transmission module compares the real-time measured packet loss rate with two threshold levels: when the packet loss rate is lower than the first threshold, forward error correction is turned off to save bandwidth for increasing the code rate; when the packet loss rate is between the first and second thresholds, the first level of redundancy is enabled to add an appropriate amount of redundant packets; when the packet loss rate is higher than the second threshold, the second level of redundancy is enabled to use stronger error correction coding to meet higher packet loss recovery capabilities.

[0055] Since adjusting the forward error correction redundancy consumes some available bandwidth, after determining the forward error correction parameters, the wireless transmission module feeds back the current upper limit of the encoding bitrate to the VR rendering host module based on the remaining available bandwidth. The VR rendering host module then adjusts the target bitrate for video encoding accordingly. When the available bandwidth drops to an insufficient level to support the minimum acceptable bitrate at the current resolution, the wireless transmission module triggers a bandwidth shortage notification, causing the VR rendering host module to progressively reduce the field of view of the rendered image, encoding and transmitting only the central area of ​​the reduced field of view. The reduction in the field of view is proportional to the bandwidth shortfall, and the field of view is gradually expanded back to the normal range after bandwidth is restored.

[0056] The wireless transmission module prioritizes video, audio, location, and interactive data. Video data is given the highest priority to ensure the continuity of the video stream; location and interactive data are given the next highest priority; and audio data is given a lower priority because its bitrate is much lower than video and it has a higher tolerance for short interruptions. The wireless transmission module uses a priority queue scheduling strategy to send data in priority order, sending lower priority data only after the higher priority video / location data has been sent.

[0057] The above technical solution disables forward error correction when the packet loss rate is extremely low, allocating all bandwidth to the video bitrate to ensure image clarity; when the packet loss rate increases, redundancy is increased in two stages, trading bandwidth for error correction capability; when the available bandwidth is further insufficient, the field of view is narrowed to encode only the central area of ​​transmission, concentrating the limited bitrate on the core area that the user is looking at rather than distributing it evenly across the entire image, thus avoiding uniform blurring of the entire image under insufficient bandwidth conditions.

[0058] In some embodiments, the output display screen after performing spatial compensation processing includes: before displaying each frame, acquiring the latest head posture data and performing a two-dimensional deformation transformation on the decoded screen based on the latest head posture data, so that the decoded screen matches the current head posture data; comparing the rendering frame rate of the video data packet with the display refresh rate of the VR terminal module 40; when the rendering frame rate is lower than the display refresh rate, acquiring the previously received frame and the current frame; analyzing the previously received frame and the current frame to obtain motion vectors; performing spatial calculation based on the motion vectors to synthesize an intermediate frame between the previously received frame and the current frame; inserting the intermediate frame into the corresponding time node of the display data line, so that the refresh rate of the final display screen after inserting the intermediate frame matches the native refresh rate of the VR terminal module 40 and does not increase the rendering load of the VR rendering host module 10.

[0059] Specifically, before each display refresh, the VR terminal module acquires the latest head pose data at the current moment through the inertial measurement unit. This latest pose data is compared with the pose data at the rendering moment corresponding to the currently displayed decoded image, and the pose difference between the two is calculated. The pose difference includes the rotation angle difference around the three coordinate axes. Based on this pose difference, the VR terminal module performs a two-dimensional deformation transformation on the entire decoded image. This can be understood as mapping the decoded image as a texture map onto the imaging plane with the current pose as the viewpoint, resampling and interpolating the image pixels to align the output image with the current real-world viewpoint.

[0060] The VR terminal module continuously compares the rendering frame rate of the video data packets sent by the VR rendering host module with the native display refresh rate of the VR terminal module's display panel. This rendering frame rate is determined by the upper limit of the bitrate fed back by the wireless transmission module and the rendering performance of the VR rendering host module. The display refresh rate is a hardware parameter of the VR terminal module's display panel. When the rendering frame rate is lower than the display refresh rate, the VR terminal module retrieves two consecutive decoded frames from the buffer: the previous frame and the current frame. It then performs pixel-by-pixel motion estimation analysis on the two frames, calculating the displacement vector of each pixel block between the two frames to obtain the motion vector field of the entire image. The motion estimation algorithm can employ block matching or optical flow methods, with computational overhead controlled within the capabilities of the VR terminal module's embedded computing unit.

[0061] Based on the obtained motion vector field, the VR terminal module performs bidirectional motion compensation interpolation on the previous and current frames. For the time nodes that need to be generated, the motion vector of each pixel block is scaled according to the time ratio. Pixel values ​​are sampled from the corresponding positions in the previous and current frames and weighted and mixed to synthesize intermediate frames. The synthesized intermediate frame image is inserted into the correct time position between the previous and current frames in the display data line, so that the refresh rate of the final displayed image is improved to match the native refresh rate of the VR terminal module.

[0062] With the above technical solution, after the VR rendering host module completes rendering with the predicted posture and sends it to the VR terminal module, the user's head may have rotated slightly during the transmission process. If the decoded image is displayed directly, the user will see an image rendered based on the predicted posture several frames ago, which will have a posture deviation from the actual posture. In this application, a two-dimensional deformation transformation is performed on the decoded image: the image is regarded as a texture attached to the virtual sphere corresponding to the posture at the rendering time, and the texture coordinates are remapped and interpolated based on the rotation angle difference to make the output image match the current real viewpoint. The computational amount of this transformation is much less than re-rendering a frame, and it can be completed on the processor of the VR terminal module during the idle time of the display refresh cycle. When the rendering frame rate is lower than the display refresh rate, the VR terminal module synthesizes an intermediate frame based on the motion vector between two adjacent frames and inserts it into the display data line. Since the synthesis of the intermediate frame does not use any resources of the VR rendering host module, it improves the perceived smoothness on the terminal side rather than the actual rendering frame rate.

[0063] In some embodiments, predicting the posture at the next moment further includes prediction optimization based on user intent, specifically including: collecting the user's historical posture data and motion trajectory and combining them with the layout of interactive elements in the virtual reality screen to construct a user behavior analysis model; monitoring the user's gaze focus, interaction intent, and environmental change information in the current virtual reality screen in real time and inputting them into the user behavior analysis model to obtain an intent prediction result, the intent prediction result including the predicted high-probability movement direction and target area at the next moment; using the intent prediction result as prior information and performing a weighted fusion with the linear prediction result of the Kalman filter algorithm and the nonlinear prediction result of the long short-term memory neural network model, wherein: when the confidence of the intent prediction result is higher than a preset threshold, the fusion weight of the intent prediction result is increased, so that the final posture prediction result obtained is biased towards the high-probability movement direction; when a sudden change in user behavior or an unclear intent is detected, the fusion result of the linear prediction result and the nonlinear prediction result is restored to be the main result.

[0064] Specifically, the user's historical posture data and movement trajectory are collected and combined with the layout information of interactive elements in the virtual reality screen to construct a user behavior analysis model. The aforementioned layout of interactive elements refers to the spatial distribution of interactive objects, task objectives, path guidance markers, and other elements in the virtual scene. The user behavior analysis model learns the user's movement decision-making patterns in various interactive scenarios and establishes a mapping relationship from scene context to movement intention.

[0065] The control module acquires eye-tracking data or head orientation data from the VR terminal module in real time, determines the projection position of the user's current gaze focus in the virtual scene, and detects currently visible interactive elements, task prompts, and environmental change events in the virtual scene. This real-time monitoring information, along with recent historical posture data, is input into the user behavior analysis model. The user behavior analysis model outputs an intent prediction result, which includes the predicted high-probability movement direction and target area for the next moment.

[0066] The control module performs a weighted fusion of the intent prediction result with the linear prediction result of the Kalman filter algorithm and the nonlinear prediction result of the long short-term memory neural network model. The fusion weights are adjusted according to the confidence score of the intent prediction result. When the confidence score is higher than the preset threshold, it indicates that the user behavior pattern is highly consistent with the model's expectations. At this time, the weight coefficient of the intent prediction result in the fusion is increased, so that the posture prediction result is biased towards the high-probability movement direction. When a sudden change in user behavior or unclear intent is detected, the control of the intent prior is withdrawn, and the fusion result of linear and nonlinear predictions is restored to be the main one.

[0067] With the above technical solution, when a user faces multiple interactive options, such as multiple entrances or objects to be operated in a virtual scene, motion inertia alone cannot determine which direction the user will choose. This solution uses the constructed user behavior analysis model as additional inputs, taking the spatial layout of scene interactive elements and the user's gaze focus as additional inputs. By learning the interaction decision patterns of a large number of users, the user behavior analysis model establishes a relationship between which interactive element the current gaze point stays near and the probability of moving towards that element. When a user hesitates at a fork in the road, inertial prediction tends to maintain the current direction, while intention prediction may give a left turn tendency judgment because the gaze point is biased towards the left target.

[0068] In some embodiments, the training process of the user behavior analysis model includes: collecting posture data, gaze data, and interaction behavior labels of multiple users in typical virtual reality scenes to construct a spatiotemporal behavior dataset; using a bidirectional long short-term memory network to encode the bidirectional temporal dependency between the posture data and gaze data, and combining an attention network layer to identify key interaction elements; training the user behavior analysis model through reinforcement learning, and using the mean square error between the predicted trajectory and the actual trajectory and the rendering resource utilization rate as a joint reward function.

[0069] Specifically, operational data from multiple different users in various typical virtual reality scenarios are collected as training samples. The data for each user includes temporal data of head posture, temporal data of eye gaze points, temporal data of controller or glove interaction devices, and action labels actually performed by the user at each time point. The temporal data of all users are aligned with the interactive elements in the virtual scene to construct a spatiotemporal behavior dataset with behavioral labels.

[0070] The pose and gaze data in the constructed dataset are input into a bidirectional long short-term memory network for encoding. The bidirectional long short-term memory network contains a forward propagation layer and a backward propagation layer. The forward layer processes the data in forward chronological order to capture feature dependencies from the past to the present moment. The backward layer processes the data in reverse chronological order to capture contextual relationships from the future to the present moment. The bidirectional encoded features are then fed into an attention network layer, which identifies the most critical interaction time points for behavior prediction by calculating the weight coefficients of each time step.

[0071] The model's input is the user's current state data, including pose, gaze point, and scene information. The model output is the predicted motion direction and probability distribution of the target region for the next moment. The model samples the predicted action based on the output probability distribution and compares it with the user's actual action at that moment. Training can use a joint reward function to evaluate the quality of each prediction step: the reward function consists of the positional and directional errors between the predicted trajectory and the actual trajectory, as well as the rendering wait time saved by pre-allocating rendering resources based on the prediction results; the joint reward function simultaneously constrains prediction accuracy and rendering resource utilization efficiency.

[0072] In some embodiments, adjusting the rendering resource allocation of the VR rendering host module 10 according to the target area further includes: generating virtual pre-rendered anchor points on the predicted high-probability motion path, wherein the position of the virtual pre-rendered anchor points is adjusted in real time according to the confidence level and motion speed; the VR rendering host module 10 performs low-precision rendering of the area covered by the virtual pre-rendered anchor points in advance and caches the generated low-precision rendering frames to edge computing nodes; when the user's actual movement approaches the virtual pre-rendered anchor points, the rendering is loaded and upgraded to high-precision rendering in real time; the wireless transmission module 30 monitors the current available network bandwidth, and when the available network bandwidth is lower than the bandwidth threshold, video data of the area covered by the virtual pre-rendered anchor points is transmitted first, and the transmission priority of the corresponding video data is increased to the highest level through the service quality scheduling strategy.

[0073] Specifically, the control module generates several virtual pre-rendered anchor points along the predicted motion path in the virtual scene based on the high-probability motion direction and target area in the intent prediction results. The spatial layout density of the anchor points is adjusted according to the confidence level of the intent prediction and the user's current movement speed. It can be understood that the higher the confidence level, the denser the anchor points are distributed along the prediction direction and the more concentrated they are towards the target area. The faster the movement speed, the larger the layout spacing of the anchor points to cover a wider range of motion.

[0074] The VR rendering host module reads the rendering parameters of each generated anchor point and performs low-precision rendering on the coverage area of ​​each anchor point in turn. This low-precision rendering adopts strategies such as reducing resolution, simplifying the shading model and reducing geometric details to significantly reduce the rendering time of a single frame, so that the VR rendering host module can complete the pre-rendering of multiple anchor points in the interval between normal rendering of the current frame. The generated low-precision rendering frames are cached to the edge computing nodes.

[0075] When the control module detects that the user's actual movement is close to the area covered by the virtual pre-rendered anchor point, the edge computing node loads the cached low-precision rendering frame to the VR rendering host module or performs high-precision upgrade rendering locally; the upgrade strategy performs super-resolution reconstruction or adds high-precision detail rendering based on the cached low-precision frame, so that the picture can quickly reach high quality the moment the user arrives.

[0076] The wireless transmission module monitors the available network bandwidth in real time. When the available bandwidth drops below the bandwidth threshold, the wireless transmission module identifies video segment data located in the virtual pre-rendered anchor point coverage area in the current frame and temporarily raises the transmission priority of this part of the video segment data to the highest level through the quality of service scheduling strategy, giving it priority over video and audio data in other areas.

[0077] Through the above technical solution, low-precision rendering is completed in advance and cached near the user before the user reaches the target area; when the user actually enters the area, the rendering changes from high-precision rendering from scratch to low-precision frame upgrade based on the cache, shortening the rendering delay perceived by the user; the transmission priority improvement when bandwidth is limited ensures that the picture of the area that the user is about to view can be delivered first even under bad network conditions, maintaining the smoothness and picture quality of key pictures.

[0078] In some embodiments, the wireless transmission module 30 receives the intent prediction result and parses out the high-probability motion direction and target area for the next moment;

[0079] Based on the current network topology, identify the wireless transmission paths in the high-probability motion direction and predict the network load on the wireless transmission path in future time periods. If the predicted network load in the high-probability motion direction exceeds a preset load threshold, perform at least one of the following optimization operations: adjust the base station transmit power or frequency resource allocation on the corresponding path; divert video data to backup paths with loads lower than the preset load threshold using a multipath transmission protocol, and adjust the diversion ratio according to the latency and packet loss rate of different paths; trigger edge computing nodes to preload video data in the target area; monitor the actual network load in real time, and gradually release the corresponding optimization resources when the load of the predicted path recovers to below the preset load threshold.

[0080] Specifically, the wireless transmission module receives the intent prediction results from the control module, and parses them to obtain the user's high-probability movement direction and target scene area at the next moment; based on the wireless network topology of the deployment space, the wireless transmission module determines the arrangement of wireless access points or base stations that the user will pass through in sequence when moving from the current position along the high-probability movement direction, as well as the transmission path between each access point; the wireless transmission module predicts the network load of each path in the future time period based on the historical load data of each access point and the number of currently connected terminals.

[0081] The wireless transmission module compares the predicted load of each path with a preset load threshold. If the load of a path in a high-probability movement direction exceeds the threshold, the network is at risk of increased congestion or even outage due to user arrival, triggering at least one of the following optimization operations: adjusting the transmission power or frequency resource allocation of the base station on the corresponding path to increase the network's carrying capacity in advance; enabling multi-path transmission protocols for traffic offloading; and notifying edge computing nodes to preload video data of the target area.

[0082] When the predicted load on the main path is too high and there is a backup path with a lower load, the wireless transmission module enables the multi-path transmission protocol. Specifically, the multi-path transmission protocol splits the video data stream into two or more sub-streams proportionally, which are then sent to the VR terminal module in parallel via the main path and the backup path, respectively. The splitting ratio of each path is adjusted in real time according to its current latency and packet loss rate.

[0083] The wireless transmission module continuously monitors the actual network load. When the load of the predicted path recovers to below the preset load threshold, it gradually releases the frequency resources and transmission power margin reserved for that path and restores the normal network configuration.

[0084] With the above technical solution, before a user moves to the coverage area of ​​a base station, the system already knows that the user will move towards that base station and estimates the connection load that the base station will increase. If the estimated load exceeds the threshold, the system will complete the pre-boost of transmission power or pre-allocation of frequency resources before the user actually arrives. These time-consuming resource configuration operations are completed in advance in the background.

[0085] In some embodiments, the execution process of the multipath transmission protocol includes: calculating the data splitting ratio between the wireless transmission path as the primary path and the backup path based on the confidence level of the intent prediction result and the user's movement speed, wherein: when the confidence level is higher than a confidence threshold and the movement speed is lower than a speed threshold, the transmission ratio of the primary path is set to be no less than a preset first ratio; when the confidence level is lower than the confidence threshold or the movement speed is higher than the speed threshold, the transmission ratio of the backup path is increased; and differential redundancy coding is added to the data packets split to different paths: lightweight forward error correction is used for data packets allocated to the primary path, and weighted concatenated error correction codes are used for data packets allocated to the backup path.

[0086] Specifically, the wireless transmission module obtains the confidence level of the intent prediction result and the user's current moving speed from the control module; based on the latency and packet loss rate of the main path and the backup path, it calculates the traffic splitting ratio in combination with the confidence level and speed parameters; when the confidence level is higher than the confidence level threshold and the moving speed is lower than the speed threshold, it indicates that the predicted movement direction is highly reliable and the user is moving slowly. At this time, the transmission ratio of the main path is not lower than the preset first ratio, and most of the traffic is retained on the main path; when the confidence level is lower than the confidence level threshold or the moving speed is higher than the speed threshold, the uncertainty of the predicted direction is high or the user may move across areas in a short period of time. At this time, the transmission ratio of the backup path is increased, and a traffic channel is pre-established on the backup path to prevent the main path from suddenly deteriorating.

[0087] The wireless transmission module employs different levels of forward error correction coding protection for data packets allocated to the primary and backup paths. Lightweight forward error correction coding is used for data packets on the primary path: a small number of redundant check packets are added to the original data packets, with the redundancy controlled at a low level, because the channel quality of the primary path is usually better and the VR terminal module can quickly recover from a single packet loss on the primary path. Weighted concatenated error correction codes are used for data packets on the backup path: two layers of redundant coding, concatenating inner and outer codes, provide stronger error protection capabilities, because the channel quality of the backup path may fluctuate more.

[0088] The above technical solution trusts the main path when the direction is certain and the movement is slow, maximizing the bandwidth utilization of the main path; when the direction is uncertain or the movement is fast, the diversion ratio of the backup path is increased, achieving a balance between overall bandwidth overhead and transmission reliability.

[0089] In some embodiments, the edge computing node preloads video data of the target area, including: the edge computing node parses the scene ID and viewpoint parameters corresponding to the target area according to the received intent prediction result; based on the scene ID and viewpoint parameters, it retrieves and preloads the corresponding pre-rendered video frames from the cloud resource library into the local solid-state drive cache, and starts the differential encoding module; when the user actually enters the target area, if the real-time rendering delay exceeds a preset delay threshold, it directly decodes the locally cached pre-rendered video frames, and transmits only the difference data between the current frame and the pre-rendered video frames through the differential encoding module.

[0090] Specifically, the edge computing node receives the intent prediction result from the control module and parses out the target area information from it; the target area is assigned a unique scene ID in the virtual scene, identifying the set of 3D scene assets corresponding to the area; the viewpoint parameters associated with the target area are parsed out, including the predicted virtual camera position coordinates, orientation angle, and field of view; the scene ID and viewpoint parameters together constitute the query key for retrieving resources from the cloud library.

[0091] The edge computing node initiates a retrieval request to the cloud resource library using the parsed scene ID as an index; the cloud resource library stores pre-rendered video frames generated for each scene ID and viewpoint; the edge computing node downloads the retrieved pre-rendered video frames to its local solid-state drive cache; the edge computing node starts the differential encoding module and completes the differential encoding preparation between the currently cached frame and the newly retrieved frame in the background.

[0092] When the control module detects that the user's actual position has entered the trigger range of the target area, it compares the time required for the VR rendering host module to complete real-time high-precision rendering with a preset delay threshold: if the real-time rendering delay is lower than the threshold, the result of real-time rendering by the VR rendering host module is used directly without enabling pre-loaded data; if the real-time rendering delay exceeds the threshold, the locally cached pre-rendered video frames are directly decoded; then, the difference data between the current frame and the pre-rendered frame is calculated by the differential encoding module, and only the difference data is transmitted to the VR terminal module; the VR terminal module merges the locally decoded pre-rendered frame with the received difference data to reconstruct a high-precision display image.

[0093] By leveraging the proximity of edge computing nodes to the user, the pre-rendered frames are downloaded from the cloud to the local solid-state drive cache after predicting the target area the user is about to enter. When the user actually enters the target area and real-time rendering cannot meet the latency requirements, the local pre-rendered frames are used and differential encoding is used to transmit only the changed parts. This greatly reduces the latency from when the user enters the area to when they see the high-precision image, and reduces the amount of data transmitted to a scale that only contains difference information.

[0094] In some embodiments, predicting the attitude at the next moment includes adjusting the prediction time advance, and the specific steps are as follows: obtaining the current network transmission delay measured in real time by the wireless transmission module 30; calculating and updating the prediction time advance based on the current network transmission delay, the video encoding time of the host, and the decoding time of the terminal, wherein the prediction time advance represents the interval from the current sampling moment to the target time node of the actual display output of the screen; and inputting the prediction time advance as a time variable into the Kalman filter algorithm and the long short-term memory neural network model to generate the attitude prediction result aligned with the target time node.

[0095] Specifically, the control module obtains the current network transmission latency measurement value from the wireless transmission module in real time; obtains the video encoding time of the current frame from the VR rendering host module; and obtains the decoding time from the VR terminal module. The control module calculates the predicted timing advance based on the sum of the current network transmission latency, video encoding time, and decoding time, representing the interval from the current sampling time to the target time node of the actual display output of the image. The update of the predicted timing advance is not an instantaneous jump, but is smoothed by an exponentially weighted moving average to avoid frequent jumps in the predicted target time caused by instantaneous fluctuations of the latency component.

[0096] The control module inputs the calculated predicted time advance as an explicit time parameter into the Kalman filter algorithm and the Long Short-Term Memory (LSTM) neural network model. In the Kalman filter algorithm, the predicted time advance is used as a parameter value for the time step in the state transition matrix: the time interval in the original state transition equation is replaced with the current predicted time advance value, and the number of steps in the state prediction recursion and the time step size of each step are adjusted accordingly. In the LSM neural network model, the predicted time advance is concatenated into the features of each input time step as additional time axis information to enable the model to perceive the target time range of the prediction. After the time parameters are injected, both models generate attitude prediction results aligned with the target time node.

[0097] Through the above technical solution, the accuracy of attitude prediction depends not only on the prediction algorithm itself, but also on whether the prediction time range matches the actual link delay. If the advance time of attitude prediction is fixed at an empirical value, when the network transmission delay or encoding time deviates from this empirical value due to changes in scene complexity, a time deviation will occur between the time corresponding to the prediction result and the actual display time of the screen. This will increase the attitude difference that needs to be corrected in the spatial compensation stage, and may even exceed the compensation range of the two-dimensional deformation transformation. This solution can adapt to delay changes under different network conditions and rendering loads.

[0098] It should be noted that the above embodiments are only used to illustrate this application and are not intended to limit the technical solutions described in this application. Although this specification has described this application in detail with reference to the above embodiments, those skilled in the art should understand that they can still make modifications or equivalent substitutions to this application. All technical solutions and improvements that do not depart from the spirit and scope of this application should be covered within the scope of the claims of this application.

Claims

1. A virtual reality system, characterized in that, include: VR rendering host module (10), spatial positioning module (20), wireless transmission module (30), VR terminal module (40) and control module (50); The spatial positioning module (20) collects the position data and attitude data of the VR terminal module (40) and determines the six degrees of freedom positioning data based on the position data and attitude data; and sends the six degrees of freedom positioning data with timestamps to the control module (50). The control module (50) predicts the attitude at the next moment based on the received six-degree-of-freedom positioning data and sends the attitude prediction result to the VR rendering host module (10). The VR rendering host module (10) renders and encodes virtual reality video data packets based on the posture prediction results; The video data packet carrying the timestamp is sent to the VR terminal module (40) via the wireless transmission module (30). The VR terminal module (40) decodes the received video data packet based on the timestamp to obtain a decoded screen, performs spatial compensation processing on the decoded screen, and outputs a display screen.

2. The virtual reality system according to claim 1, characterized in that, The spatial positioning module (20) includes an inertial measurement unit (21), an infrared positioning unit (22), and an ultra-wideband positioning unit (23). Determine the six-degree-of-freedom positioning data, including: The acceleration and angular velocity data collected by the inertial measurement unit (21) are accumulated and integrated to obtain the relative attitude and displacement changes between two adjacent update times; The global reference system coordinates of the VR terminal module (40) are obtained based on the infrared positioning unit (22) and the ultra-wideband positioning unit (23); Calculate the Mahalanobis distance of the global reference system coordinates and perform outlier detection based on the Mahalanobis distance; When the global reference frame coordinates are outliers, the observation update of the extended Kalman filter model is cut off, and only the relative attitude and displacement changes are used as prediction values ​​for short-term extrapolation to estimate the six-degree-of-freedom positioning data. When the global reference frame coordinates are normal, the relative attitude and displacement changes are used as predicted values, and the global reference frame coordinates are used as observed values ​​to update the extended Kalman filter model, thereby estimating the six-degree-of-freedom positioning data. When the VR terminal module (40) moves between the coverage areas of multiple positioning base stations, the positioning results of different base stations are weighted and averaged to obtain the six degrees of freedom positioning data.

3. A virtual reality system according to claim 2, characterized in that, The extended Kalman filter model is updated using the global reference frame coordinates as observations to estimate the six-degree-of-freedom positioning data, including: Construct the state vector of the extended Kalman filter model, the state vector including three-dimensional position, three-dimensional velocity, three-dimensional attitude and the zero bias error of the inertial measurement unit (21); Based on the relative attitude and displacement changes, the state equation is performed to predict and deduce the prior state estimate and the prior covariance matrix. Calculate the Kalman gain and combine it with the global reference frame coordinates to calculate the observation residuals; Based on the Kalman gain and the observation residual, the prior state estimate is updated by observation, the posterior state estimate is output, and the three-dimensional position and the three-dimensional attitude in the posterior state estimate are extracted as the six-degree-of-freedom positioning data. Specifically, the zero bias error in the posterior state estimate is extracted and fed back to the cumulative integration processing stage of the acceleration and angular velocity data.

4. A virtual reality system according to claim 3, characterized in that, By cutting off observation updates to the extended Kalman filter model and using only the relative attitude and displacement changes as prediction values ​​for short-term extrapolation, the six-degree-of-freedom positioning data is estimated, including: Extract the posterior state estimate output by the extended Kalman filter model at the most recent normal moment before the anomaly occurred, and use it as the starting reference point for short-time extrapolation; Freeze the error covariance update of the extended Kalman filter model and stop calculating the Kalman gain; based on the starting reference point, use the acceleration and angular velocity data output in real time by the inertial measurement unit (21) to perform recursion and output the calculated attitude and position as the six-degree-of-freedom positioning data; The Mahalanobis distance of the newly acquired global reference frame coordinates is continuously calculated until the Mahalanobis distance recovers to below the anomaly threshold. Then, the freeze is lifted and the observation update of the extended Kalman filter model is resumed.

5. A virtual reality system according to claim 4, characterized in that, Predicting the attitude at the next moment, including: The Kalman filter algorithm is used to perform linear prediction on historical attitude data to obtain linear prediction results. The historical posture data is input into a long short-term memory neural network model for nonlinear prediction to obtain the nonlinear prediction result. The motion state of the VR terminal module (40) is evaluated based on the rate of change of current acceleration and angular velocity collected by the inertial measurement unit (21). When the motion state is steady motion, increase the fusion weight of the linear prediction result; when the motion state is violent motion, increase the fusion weight of the nonlinear prediction result. The final attitude prediction result is obtained by weighted fusion of the linear prediction result and the nonlinear prediction result.

6. A virtual reality system according to claim 5, characterized in that, Video encoding of the virtual reality footage includes: Disable bidirectional prediction B-frames, use only I-frames and P-frames for encoding, or use full I-frame encoding mode and set the group length to 1; Each frame of the virtual reality scene is divided into multiple independent screen segments, and each screen segment is independently encoded and transmitted; A first bit rate is allocated to the central region of the field of view, and a second bit rate is allocated to the edge region of the field of view, wherein the first bit rate is greater than the second bit rate; The corresponding regions are encoded according to the allocated first code rate and second code rate.

7. A virtual reality system according to claim 6, characterized in that, Sending the video data packet to the VR terminal module (40) includes: Real-time monitoring of network status parameters; these status parameters include round-trip time, packet loss rate, jitter parameters, and available bandwidth; The redundancy of forward error correction is adjusted according to the packet loss rate: forward error correction is turned off when the packet loss rate is lower than a first threshold; a first level of redundancy is set when the packet loss rate is between the first threshold and a second threshold; a second level of redundancy is set when the packet loss rate is higher than the second threshold, and the second level of redundancy is greater than the first level of redundancy. After the forward error correction redundancy adjustment, the video coding bitrate is adjusted according to the available bandwidth; When the available bandwidth is insufficient, a feedback is triggered to the VR rendering host module (10): the field of view of the rendered image is gradually reduced and the center area of ​​the reduced field of view is encoded and transmitted. Video data, audio data, location data, and interactive data are prioritized and transmitted in order of priority using a quality of service scheduling strategy.

8. A virtual reality system according to claim 7, characterized in that, After performing spatial compensation processing, the output display screen includes: Before each frame is displayed, the latest head pose data is acquired and the decoded image is subjected to two-dimensional deformation transformation based on the latest head pose data, so that the decoded image matches the current head pose data. Compare the rendering frame rate of the video data packet with the display refresh rate of the VR terminal module (40); When the rendering frame rate is lower than the display refresh rate, extract the previously received frame and the current frame. Motion vectors are obtained by analyzing the previous frame and the current frame. Spatial calculation is performed based on the motion vectors to synthesize an intermediate frame between the previous frame and the current frame. The intermediate frame is inserted into the corresponding time node of the display sequence so that the refresh rate of the final display screen after the insertion of the intermediate frame matches the native refresh rate of the VR terminal module (40) and does not increase the rendering load of the VR rendering host module (10).

9. A virtual reality system according to claim 8, characterized in that, Predicting the attitude at the next moment also includes: Collect users' historical posture data and movement trajectories, and combine them with the layout of interactive elements in virtual reality visuals to build a user behavior analysis model; The system monitors the user's gaze focus, interaction intent, and environmental changes in the current virtual reality scene in real time and inputs them into the user behavior analysis model to obtain intent prediction results. The intent prediction results include the predicted high-probability movement direction and target area for the next moment. The intention prediction result is used as prior information and weighted and fused with the linear prediction result of the Kalman filter algorithm and the nonlinear prediction result of the long short-term memory neural network model. When the confidence of the intention prediction result is higher than a preset threshold, the fusion weight of the intention prediction result is increased so that the final posture prediction result is biased towards the high-probability motion direction. When a sudden change in user behavior or unclear intent is detected, the system will restore the system based on the fusion result of the linear prediction and the nonlinear prediction.

10. A virtual reality method, based on the system according to any one of claims 1-9, characterized in that, The method includes: Collect position and attitude data of the VR terminal module (40) and determine six-degree-of-freedom positioning data based on the position and attitude data; The attitude at the next moment is predicted based on the received six-degree-of-freedom positioning data; The virtual reality video data packet is rendered and encoded based on the posture prediction results; the video data packet carrying the timestamp is sent to the VR terminal module (40). The VR terminal module (40) decodes the received video data packet based on the timestamp to obtain a decoded screen, performs spatial compensation processing on the decoded screen, and outputs a display screen.

Images