VR Virtual-Real Fusion Interaction System and Method Based on Large Spatial Positioning and Dynamic Rendering

By using tightly coupled extended Kalman filtering and long short-term memory networks to compensate for pose, combined with directed distance fields and distributed haptic feedback, the positioning and rendering latency issues of VR virtual-real interaction systems in large spaces are solved, achieving a highly robust, safe, and immersive multi-person VR experience.

CN122336094BActive Publication Date: 2026-11-17FUTURE VISION CULTURE TECHNOLOGY (WUHAN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202610446716.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-04-07
Publication Date
2026-11-17
Estimated Expiration
2046-04-07

AI Technical Summary

Technical Problem

Existing VR virtual-real interaction systems struggle to balance positioning accuracy and rendering latency in large spaces, and resource competition is severe during multi-person interaction, resulting in insufficient robustness and security of the experience.

Method used

By employing tightly coupled extended Kalman filtering to fuse SLAM visual features with UWB/Bluetooth AoA data, integrating a long short-term memory network model for pose compensation, and constructing a risk field function through a directed distance field to dynamically generate boundary warning images, combined with a distributed haptic feedback array and a predictive resource pre-allocation mechanism, highly robust localization and multimodal interaction are achieved.

Benefits of technology

Maintaining stable positioning in scenes with limited visual textures and fast movement, dynamic boundary warnings enhance safety, predictive resource scheduling ensures a low-latency, high-quality experience, and multimodal interaction improves immersion and security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122336094B_ABST
    Figure CN122336094B_ABST
Patent Text Reader

Abstract

This invention discloses a VR virtual-real fusion interaction system and method based on large-space positioning and dynamic rendering. The system includes: a physical space modeling module to generate a spatial risk field function; a hybrid positioning engine that uses tightly coupled extended Kalman filtering to fuse SLAM visual features and UWB / Bluetooth AoA data to calculate pose, and integrates a long short-term memory network for predictive compensation in case of visual failure; a spatial synchronization and dynamic scheduling module to allocate rendering and bandwidth resources; a virtual-real fusion rendering unit that dynamically generates a fused video stream containing virtual scenes and boundary warnings; a low-latency transmission network that distributes the video stream based on WiFi 7 with a latency of no more than 15ms; and a terminal presentation and closed-loop correction module that displays and transmits IMU data to correct pose. This invention achieves highly robust positioning, dynamic boundary warning, predictive resource scheduling, and multimodal interaction closed loop, significantly improving the immersion and safety of large-space multi-person VR experiences.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of VR virtual-real fusion technology, and in particular to a VR virtual-real fusion interactive system and method based on large spatial positioning and dynamic rendering. Background Technology

[0002] With the development of virtual reality technology, location-based large-space (LBE) VR experiences are gradually becoming a new hotspot in the cultural tourism and entertainment industry. Such systems require multiple users to move freely within a shared physical space and interact with virtual scenes and other users in real time, placing extremely high demands on positioning accuracy, rendering latency, and spatial security.

[0003] Existing VR positioning technologies are mainly divided into two categories: one is optical positioning relying on external base stations (such as SteamVR Tracking), which, while highly accurate, is complex to deploy, costly, and suffers from occlusion issues, making it difficult to cover large spaces of hundreds of square meters; the other is visual positioning technology based on SLAM, such as Chinese patent CN121661335A, which uses a differentiated feature weighting method to suppress interference from dynamic and low-confidence features, and point-line joint modeling to enhance the stability and observability of low-texture scenes. However, visual SLAM is prone to failure in scenarios with rapid movement, drastic changes in lighting, and scarce textures, leading to positioning drift or interruption, which seriously affects the continuity of multi-user simultaneous experiences. In terms of multi-user interaction, existing systems mostly adopt an architecture of centralized server rendering followed by video stream distribution. However, as the number of concurrent users increases, the competition for GPU rendering resources and network bandwidth intensifies, easily leading to frame drops or transmission latency fluctuations, making it difficult to guarantee a low-latency, high-quality multi-user synchronous experience.

[0004] To address the aforementioned issues, this invention proposes a VR virtual-real fusion interaction system and method based on large-space positioning and dynamic rendering. It achieves highly robust continuous positioning by fusing SLAM visual features with UWB / Bluetooth AoA data through tightly coupled extended Kalman filtering and integrating a long short-term memory network model for predictive compensation in case of visual failure. It dynamically generates boundary warning images based on a directed distance field-constructed risk field function, enabling refined visual expression of risk levels. A predictive resource pre-allocation mechanism solves resource contention issues under concurrent user scenarios. Furthermore, it integrates a distributed haptic feedback array to construct a multimodal interactive closed loop, significantly improving the immersion and safety of large-space multi-user VR experiences. Summary of the Invention

[0005] In view of the shortcomings in the background technology, the purpose of this invention is to propose a VR virtual-real fusion interaction system and method based on large spatial positioning and dynamic rendering, so as to solve the problems of poor robustness, security and poor user experience in existing VR virtual-real interaction.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] On the one hand, the present invention provides a VR virtual-real fusion interactive system based on large spatial positioning and dynamic rendering, including a physical space modeling module, a hybrid positioning engine, a spatial synchronization and dynamic scheduling module, a virtual-real fusion rendering unit, a low-latency transmission network, and a terminal presentation and closed-loop correction module.

[0008] The physical space modeling module, configured on a local server, is used to acquire the three-dimensional geometric structure of the offline physical space and generate a risk field function that characterizes the continuous distribution of spatial boundaries and obstacles.

[0009] The hybrid positioning engine is deployed on a local server and multiple VR headsets to collect SLAM visual feature point data and UWB / Bluetooth AoA angle of arrival data of each VR headset in real time. It uses tightly coupled extended Kalman filtering to fuse the data and calculate the six-degree-of-freedom pose of each VR headset. It also integrates a long short-term memory network model to output the predicted pose for compensation when visual features fail.

[0010] The spatial synchronization and dynamic scheduling module is used to generate a global spatial synchronization signal based on the six degrees of freedom pose, and to dynamically allocate independent GPU rendering pipeline resources and network bandwidth resources to each VR headset.

[0011] The virtual-real fusion rendering unit is deployed on a local server and is equipped with a multi-view real-time rendering engine. It is used to render a fused video stream containing virtual scene images and dynamic boundary warning images for each VR headset device according to the six degrees of freedom pose and the global space synchronization signal. The dynamic boundary warning images are dynamically generated based on the risk field function.

[0012] The low-latency transmission network is built on the WiFi 7 protocol and is configured with forward error correction coding and dynamic bitrate adjustment mechanism to synchronously distribute the fused video stream to the corresponding VR headset with an end-to-end latency of no more than 15ms.

[0013] The terminal presentation and closed-loop correction module is integrated into each VR headset device, including at least one display screen, for receiving and decoding the fused video stream for display, and simultaneously acquiring six-axis motion data of the VR headset device's built-in IMU in real time and transmitting it back to the hybrid positioning engine for closed-loop correction of the six degrees of freedom pose.

[0014] Preferably, the hybrid positioning engine further includes an adaptive weight adjuster, which is used to dynamically adjust the measurement noise covariance matrix of SLAM visual feature point data and UWB angle of arrival data in tightly coupled extended Kalman filtering based on the visual texture richness and UWB signal strength of the current environment.

[0015] Preferably, the risk field function is a continuously differentiable function constructed based on a directed distance field. The function value of the risk field function is jointly determined by the directed distance between the current position of the VR headset and the nearest physical obstacle and the surface normal vector of the nearest physical obstacle. The transparency, color, and flashing frequency of the dynamic boundary warning screen are dynamically determined by the absolute value and sign of the directed distance. When the directed distance is negative, the warning screen flashes bright red.

[0016] Preferably, the spatial synchronization and dynamic scheduling module further includes a predictive resource pre-allocation unit, which is used to pre-allocate the computing resources and network bandwidth required for rendering the next frame to the corresponding VR headset device 100ms in advance, based on the predicted pose output by the long short-term memory network model.

[0017] Preferably, the virtual-real fusion rendering unit is also used to generate multi-person synchronous interaction events and trigger unified visual effects and haptic driving signals when it detects that the virtual viewpoints of different VR headsets are within a preset interaction distance and the angle between their line of sight is less than 30°.

[0018] Preferably, the system further includes a distributed haptic feedback array, which consists of piezoelectric ceramic vibration units deployed under the floor of the physical space and linear resonant motors deployed on the surface of the walls of the physical space. The array is used to receive the haptic drive signal and generate vibration frequency and amplitude at a specified physical location that match the type of virtual interactive event.

[0019] Preferably, the low-latency transmission network is further configured with an intelligent packet loss retransmission mechanism. The intelligent packet loss retransmission mechanism is based on real-time channel quality assessment, selectively retransmits only key frame data packets, and adopts a frame copying and filling strategy when non-key frames are lost.

[0020] Preferably, the system further includes an offline calibration module, which is used to guide the user to wear a VR headset to traverse the entire physical space during the system initialization phase, collect SLAM feature points and UWB signal strength fingerprints of the covered space, construct a spatial fingerprint map, and simultaneously optimize the accuracy of the risk field function.

[0021] Preferably, the VR headset has a 4K resolution and a 90Hz refresh rate display screen, and the VR headset also integrates an eye-tracking sensor for real-time capture of the user's gaze point. The virtual-real fusion rendering unit uses the highest quality rendering for the gaze area and a reduced resolution gaze point rendering technique for the edge field of view based on the user's gaze point data captured in real time by the eye-tracking sensor, so as to reduce the rendering load.

[0022] On the other hand, the present invention provides a VR virtual-real fusion interaction method based on large spatial positioning and dynamic rendering, which is implemented based on the above-mentioned VR virtual-real fusion interaction system based on large spatial positioning and dynamic rendering, and includes the following steps:

[0023] S1. Obtain the three-dimensional geometric structure of the entity space through the physical space modeling module, and generate a risk field function based on the directed distance field;

[0024] S2. Real-time acquisition of SLAM visual feature point data and UWB / Bluetooth AoA angle of arrival data of each VR headset through a hybrid positioning engine, front-end fusion using tightly coupled extended Kalman filter, calculation of the six-degree-of-freedom pose of each VR headset, and output of compensating pose when visual features fail based on a long short-term memory network prediction model.

[0025] S3. Generate a global spatial synchronization signal based on the six degrees of freedom pose, and dynamically allocate rendering computing resources and network bandwidth resources to each VR headset;

[0026] S4. Render a fused video stream containing virtual scene images and dynamic boundary warning images for each VR headset device, wherein the transparency, color and flashing frequency of the dynamic boundary warning images are dynamically determined by the directional distance gradient value of the risk field function at the current position of the VR headset device.

[0027] S5. Through a low-latency transmission network based on WiFi 7, and by employing forward error correction and dynamic bitrate adjustment mechanisms, the fused video stream is synchronously distributed to the corresponding VR headset with an end-to-end latency of no more than 15ms.

[0028] S6. Present the fused video stream through the display screen of the VR headset, while simultaneously collecting IMU data and eye-tracking data from the VR headset in real time, sending them back to the hybrid positioning engine for closed-loop correction, and triggering foveated rendering optimization;

[0029] S7. When a multi-user virtual interaction event is detected, a synchronous interaction signal is generated to drive the distributed haptic feedback array to produce haptic feedback that matches the event type.

[0030] Further, in step S2, the fusion process of the tightly coupled extended Kalman filter includes:

[0031] S201. Establish the state vector ,in, For location, For speed, For attitude quaternions, and These are the biases for the accelerometer and gyroscope, respectively. For the first The three-dimensional coordinates of a visual feature point;

[0032] S202. Use IMU data for prediction; the prediction model is as follows:

[0033]

[0034]

[0035]

[0036] in, Let be a rotation matrix. and For IMU measurements, The gravitational vector, symbol Represents quaternion multiplication;

[0037] S203. The first update is performed using visual feature point observation. The three-dimensional feature points are projected onto the camera plane and compared with the position of the feature points extracted from the image to obtain the residual.

[0038] S204. Perform a second update using UWB / Bluetooth AoA observation, based on the current position of the VR headset. and base station location Calculate the predicted distance The predicted angle of arrival is compared with the actual measured value to obtain the residual;

[0039] S205. By extending the update step of the Kalman filter, the visual residual and the UWB residual are fused to estimate the state vector and IMU bias online.

[0040] Further, in step S4, the process of generating the dynamic boundary warning screen includes:

[0041] S401. Based on the directed distance field, extract multi-layer isosurfaces with distance thresholds of 0.1m, 0.3m, and 0.5m;

[0042] S402. The multi-layer isosurfaces extracted in S401 with distance thresholds of 0.1m, 0.3m, and 0.5m are rendered as red, yellow, and green semi-transparent boundary meshes, respectively. The transparency of the boundary mesh is inversely proportional to the absolute value of the directed distance. When the directed distance is less than 0.1m, the boundary mesh is highlighted and flashing red. The directed distance refers to the directed distance between the current position of the VR headset and the nearest physical obstacle.

[0043] Compared with existing technologies, the VR virtual-real fusion interaction system and method based on large-space positioning and dynamic rendering provided by this invention has the following beneficial effects:

[0044] (1) Highly robust hybrid positioning and continuous compensation: Tightly coupled extended Kalman filter is used to fuse SLAM visual features and UWB / Bluetooth AoA data, and a long short-term memory network model is integrated to output predicted pose for compensation when visual failure occurs. Even in scenarios with scarce visual texture and fast movement, it can still maintain continuous and stable six-degree-of-freedom pose calculation, significantly reducing image jitter and drift.

[0045] (2) Dynamic boundary warning: Based on the directed distance field, a risk field function is constructed to dynamically generate boundary warning images. The transparency, color and flashing frequency of the images change in real time with the directed distance between the user and the obstacle, so that the user can intuitively perceive the spatial boundary, effectively prevent collision accidents, and improve the safety of free movement in large spaces.

[0046] (3) Prediction-driven resource scheduling: Based on the predicted pose output by the long short-term memory network, GPU rendering pipeline and network bandwidth resources are pre-allocated to each VR headset to solve the resource competition problem in multi-person concurrent scenarios, ensure stable output of multi-channel fused video streams, and guarantee low latency and high image quality multi-person synchronous experience.

[0047] (4) Multimodal interaction closed loop: Integrating a distributed haptic feedback array, visual effects and haptic driving signals are triggered simultaneously when multi-person interaction events are detected. Combined with eye tracking and gaze point rendering to optimize visual load, an immersive interaction closed loop of "visual-auditory-touch" multimodal fusion is constructed, which significantly improves the user's sense of presence and interaction realism.

[0048] (5) System-level engineering optimization: SLAM feature points and UWB fingerprints are collected through the offline calibration module to construct a spatial fingerprint map to optimize the accuracy of the risk field. The low-latency transmission network is configured with forward error correction and intelligent packet loss retransmission to achieve an end-to-end latency of no more than 15ms, ensuring the system stability and feasibility in high-concurrency and complex environments, and possessing extremely high commercial practical value. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the VR virtual-real fusion interaction system based on large spatial positioning and dynamic rendering according to the present invention;

[0051] Figure 2 This is a flowchart illustrating the workflow of the VR virtual-real fusion interaction method based on large-space positioning and dynamic rendering of this invention. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0053] Example 1: As Figure 1 As shown, this embodiment of the invention provides a VR virtual-real fusion interactive system based on large-space positioning and dynamic rendering, including a physical space modeling module, a hybrid positioning engine, a spatial synchronization and dynamic scheduling module, a virtual-real fusion rendering unit, a low-latency transmission network, and a terminal presentation and closed-loop correction module, wherein:

[0054] The physical space modeling module, configured on a local server, is used to acquire the three-dimensional geometric structure of offline physical spaces and generate risk field functions that characterize the continuous distribution of spatial boundaries and obstacles.

[0055] The hybrid positioning engine is deployed on a local server and multiple VR headsets to collect SLAM visual feature point data and UWB / Bluetooth AoA angle of arrival data of each VR headset in real time. It uses tightly coupled extended Kalman filter to fuse the data and calculate the six-degree-of-freedom pose of each VR headset. It also integrates a long short-term memory network model to output the predicted pose for compensation when visual features fail.

[0056] The spatial synchronization and dynamic scheduling module is used to generate a global spatial synchronization signal based on the six degrees of freedom pose, and to dynamically allocate independent GPU rendering pipeline resources and network bandwidth resources to each VR headset.

[0057] The virtual-real fusion rendering unit is deployed on a local server and is equipped with a multi-view real-time rendering engine. It is used to render a fused video stream containing virtual scene images and dynamic boundary warning images for each VR headset device according to the six degrees of freedom pose and the global space synchronization signal. The dynamic boundary warning images are dynamically generated based on the risk field function.

[0058] A low-latency transmission network, built on the WiFi 7 protocol and configured with forward error correction coding and dynamic bitrate adjustment mechanism, is used to synchronously distribute the fused video stream to the corresponding VR headset device with an end-to-end latency of no more than 15ms.

[0059] The terminal presentation and closed-loop correction module is integrated into each VR headset device, including at least one display screen, for receiving and decoding the fused video stream for display, and simultaneously acquiring six-axis motion data of the VR headset device’s built-in IMU in real time and transmitting it back to the hybrid positioning engine for closed-loop correction of the six degrees of freedom pose.

[0060] Specifically, in this embodiment 1, the hybrid positioning engine further includes an adaptive weight adjuster, which is used to dynamically adjust the measurement noise covariance matrix of SLAM visual feature point data and UWB angle of arrival data in tightly coupled extended Kalman filtering according to the visual texture richness of the current environment and the UWB signal strength.

[0061] Specifically, in this embodiment 1, the risk field function is a continuously differentiable function constructed based on the directed distance field. Its function value is jointly determined by the directed distance between the current position of the VR headset and the nearest physical obstacle and the surface normal vector of the nearest physical obstacle. The transparency, color and flashing frequency of the dynamic boundary warning screen are dynamically determined by the absolute value and sign of the directed distance. When the directed distance is negative, the dynamic boundary warning screen flashes red.

[0062] Specifically, in this embodiment 1, the spatial synchronization and dynamic scheduling module further includes a predictive resource pre-allocation unit, which is used to pre-allocate the computing resources and network bandwidth required for rendering the next frame to the corresponding VR headset device 100ms in advance based on the predicted pose output by the long short-term memory network model.

[0063] Specifically, in this embodiment 1, the virtual-real fusion rendering unit is also used to generate multi-person synchronous interaction events and trigger unified visual effects and haptic driving signals when it detects that the virtual viewpoints of different VR headsets are within a preset interaction distance and the angle between the lines of sight is less than 30°.

[0064] Specifically, in this embodiment 1, the VR virtual-real fusion interaction system based on large space positioning and dynamic rendering also includes a distributed haptic feedback array, which consists of a piezoelectric ceramic vibration unit deployed under the floor of the physical space and a linear resonant motor deployed on the surface of the wall of the physical space. It is used to receive haptic drive signals and generate vibration frequency and amplitude that match the type of virtual interactive event at a specified physical location.

[0065] Specifically, in this embodiment 1, the low-latency transmission network is also configured with an intelligent packet loss retransmission mechanism. This intelligent packet loss retransmission mechanism is based on real-time channel quality assessment, selectively retransmits only key frame data packets, and adopts a frame copying filling strategy when non-key frames are lost.

[0066] Specifically, in this embodiment 1, the VR virtual-real fusion interaction system based on large spatial positioning and dynamic rendering also includes an offline calibration module. The offline calibration module is used to guide the user to wear a VR headset to traverse the entire physical space during the system initialization phase, collect SLAM feature points and UWB signal strength fingerprints of the covered space, construct a spatial fingerprint map, and simultaneously optimize the accuracy of the risk field function.

[0067] Specifically, in this embodiment 1, the VR headset's display screen has a 4K resolution and a 90Hz refresh rate, and the VR headset also integrates an eye-tracking sensor to capture the user's gaze point in real time. The virtual-real fusion rendering unit uses the highest image quality to render the gaze area based on the user's gaze point data captured in real time by the eye-tracking sensor, and uses a gaze point rendering technology with reduced resolution for the edge field of view to reduce the rendering load.

[0068] The VR virtual-real fusion interaction system based on large-space positioning and dynamic rendering provided in this invention adopts a physical space modeling module, a hybrid positioning engine, a spatial synchronization and dynamic scheduling module, a virtual-real fusion rendering unit, a low-latency transmission network, and a terminal presentation and closed-loop correction module. This system achieves VR virtual-real fusion interaction through large-space positioning and dynamic rendering. It fuses SLAM visual features with UWB / Bluetooth AoA data using tightly coupled extended Kalman filtering to calculate pose, and integrates a long short-term memory network for predictive compensation in case of visual failure. It dynamically allocates rendering and bandwidth resources, generates a fused video stream containing virtual scenes and boundary warnings, displays it, and transmits back IMU data to correct pose. This achieves highly robust VR virtual-real fusion positioning, dynamic boundary warning, predictive resource scheduling, and multimodal interaction closed-loop, significantly improving the immersion and safety of large-space multi-person VR experiences. Through design, simulation, and verification, a modular product is formed, enabling rapid portability between different platforms and accelerating the product development process.

[0069] Example 2: Figure 2 As shown, this embodiment of the invention provides a VR virtual-real fusion interaction method based on large spatial positioning and dynamic rendering, which is specifically implemented using the VR virtual-real fusion interaction system based on large spatial positioning and dynamic rendering as described in Embodiment 1, and specifically includes the following steps:

[0070] S1. Obtain the three-dimensional geometric structure of the entity space through the physical space modeling module, and generate a risk field function based on the directed distance field;

[0071] S2. Real-time acquisition of SLAM visual feature point data and UWB / Bluetooth AoA angle of arrival data through a hybrid positioning engine, front-end fusion using tightly coupled extended Kalman filter, calculation of the six-degree-of-freedom pose of each VR headset, and output of compensating pose when visual features fail based on a long short-term memory network prediction model.

[0072] S3. Generate a global spatial synchronization signal based on the six degrees of freedom pose, and dynamically allocate rendering computing resources and network bandwidth resources to each VR headset;

[0073] S4. Render a fused video stream containing virtual scene images and dynamic boundary warning images for each VR headset device, wherein the transparency, color and flashing frequency of the dynamic boundary warning images are dynamically determined by the directional distance gradient value of the risk field function at the current position of the VR headset device.

[0074] S5. Through a low-latency transmission network based on WiFi 7, and by employing forward error correction and dynamic bitrate adjustment mechanisms, the fused video stream is synchronously distributed to the corresponding VR headset with an end-to-end latency of no more than 15ms.

[0075] S6. Present the fused video stream through the display screen of the VR headset, while simultaneously collecting IMU data and eye-tracking data from the VR headset in real time, sending them back to the hybrid positioning engine for closed-loop correction, and triggering foveated rendering optimization;

[0076] S7. When a multi-user virtual interaction event is detected, a synchronous interaction signal is generated to drive the distributed haptic feedback array to produce haptic feedback that matches the event type.

[0077] Furthermore, in step S2 of this embodiment 2, the fusion process of the tightly coupled extended Kalman filter includes:

[0078] S201. Establish the state vector ,in, For location, For speed, For attitude quaternions, and These are the biases for the accelerometer and gyroscope, respectively. For the first The three-dimensional coordinates of a visual feature point;

[0079] S202. Use IMU data for prediction; the prediction model is as follows:

[0080]

[0081]

[0082]

[0083] in, Let be a rotation matrix. and For IMU measurements, The gravitational vector, symbol Represents quaternion multiplication;

[0084] S203. The first update is performed using visual feature point observation. The three-dimensional feature points are projected onto the camera plane and compared with the position of the feature points extracted from the image to obtain the residual.

[0085] S204. Perform a second update using UWB / Bluetooth AoA observation, based on the current position of the VR headset. and base station location Calculate the predicted distance The predicted angle of arrival is compared with the actual measured value to obtain the residual;

[0086] S205. By extending the update step of the Kalman filter, the visual residual and the UWB residual are fused to estimate the state vector and IMU bias online.

[0087] Furthermore, in step S4 of this embodiment 2, the process of generating the dynamic boundary warning screen includes:

[0088] S401. Based on the directed distance field, extract multi-layer isosurfaces with distance thresholds of 0.1m, 0.3m, and 0.5m;

[0089] S402. Render the multi-layer isosurfaces extracted in S401 with distance thresholds of 0.1m, 0.3m, and 0.5m as red, yellow, and green semi-transparent boundary meshes, respectively. The transparency of the boundary mesh is inversely proportional to the absolute value of the directed distance. When the directed distance is less than 0.1m, the boundary mesh is highlighted and flashing red. The directed distance refers to the directed distance between the current position of the VR headset and the nearest physical obstacle.

[0090] like Figure 2 As shown, through spatial modeling, pose calculation, resource scheduling, virtual and real rendering, low-latency transmission, display and correction, and interactive feedback, highly robust positioning, dynamic boundary warning, predictive resource scheduling, and multimodal interaction closed loop of VR virtual and real fusion are achieved, which significantly improves the immersion and safety of large-space multi-person VR experience.

[0091] To further illustrate the VR virtual-real fusion interaction system and method based on large spatial positioning and dynamic rendering provided by the present invention, a specific application embodiment is used below as an example to explain how the VR virtual-real fusion interaction system based on large spatial positioning and dynamic rendering provided by the present invention works.

[0092] 1. System Hardware Architecture

[0093] This embodiment provides a VR virtual-real fusion interactive system based on large-space positioning and dynamic rendering, deployed in an offline LBE experience venue (such as a VR experience store in Wuhan MixC), with a venue area of ​​180 square meters, containing physical obstacles such as pillars and walls. The system hardware configuration is as follows:

[0094] Local server: It adopts a high-performance workstation, configured with an Intel Core i9-13900K CPU, 64GB DDR5 memory, NVIDIA RTX 4090 graphics card (24GB VRAM), 2TB NVMe SSD, and runs the Ubuntu 22.04 operating system on the server, running the core software modules.

[0095] VR headset: It uses 20 PICO 4 Ultra Enterprise all-in-one devices. Each headset is equipped with a Qualcomm Snapdragon XR2 Gen 2 chip, 12GB of RAM, 256GB of flash storage, dual 2.56-inch displays (2160×2160 resolution per eye, 4320×2160 overall, 90Hz refresh rate), and integrates four-eye SLAM cameras, IMU (six-axis), and UWB / Bluetooth AoA positioning tags.

[0096] UWB / Bluetooth positioning base stations: Six UWB / Bluetooth AoA positioning base stations (using Decawave DWM1000 modules) are deployed at the four corners and center of the venue ceiling. The base stations are connected to the server via PoE switches to send ranging signals and measure the angle of arrival.

[0097] Network equipment: Deploy two Xiaomi AX9000 routers (supporting WiFi 7, dual 10G Ethernet ports) and connect them to the server via wired connection to form a Mesh network, ensuring low-latency communication between the VR headset and the server.

[0098] Haptic feedback array: 32 piezoelectric ceramic vibration units (each unit is 10cm×10cm and spaced 1.5m apart) are embedded under the floor, and 16 linear resonant motors are installed on the wall surface. All units are connected to the haptic drive controller via CAN bus, and the controller communicates with the server via USB.

[0099] Broadcast control terminal: A Xiaomi Redmi Pad SE tablet (11-inch, 8G+128G) with the PICO large-space broadcast control system installed, used by operators to monitor and control the experience process.

[0100] 2. Implementation of the Physical Space Modeling Module

[0101] Spatial scanning: During the system initialization phase, staff wearing a PICO headset (or using a handheld LiDAR scanner) traverse the entire venue to collect point cloud data. The point cloud data includes the three-dimensional coordinates (x, y, z) and color information of each point, with an accuracy of ±1cm.

[0102] 3D Reconstruction: The point cloud is meshed using the Poisson surface reconstruction algorithm to generate a triangular mesh model, and the boundaries of physical obstacles such as walls, columns, and the ground are identified. The reconstructed model is stored in OBJ format.

[0103] Directed distance field calculation: Based on the reconstructed mesh model, the directed distance field of the entire space is calculated. The space is discretized into a 5mm×5mm×5mm voxel mesh. For each voxel center point p, the directed distance d(p) to the nearest obstacle surface is calculated: if p is in free space, d(p) is positive, and its value is the distance to the nearest surface; if p is inside an obstacle, d(p) is negative, and its value is the inverse of the distance to the nearest surface. The three-dimensional directed distance field function is obtained. .

[0104] Risk field function generation: Defining the risk field function ,in, Functions of type Sigmoid:

[0105]

[0106] We set k=2.0, which causes the risk value to increase sharply when the distance to the surface is close. The value of R(p) ranges from [-1,1], where a negative value indicates that the obstacle has been penetrated.

[0107] 3. Detailed Implementation of the Hybrid Positioning Engine

[0108] The hybrid positioning engine uses a tightly coupled extended Kalman filter (EKF) to fuse SLAM visual feature point data with UWB / Bluetooth AoA angle of arrival data, and integrates an LSTM prediction model for pose compensation.

[0109] 3.1 Data Acquisition and Preprocessing

[0110] Visual feature points: The SLAM camera of each head-mounted display acquires images at 30fps, extracts ORB feature points, and calculates the three-dimensional coordinates (local coordinate system) of the feature points in space through multi-view geometry. Each feature point includes its position and descriptor in the image.

[0111] UWB / Bluetooth AoA data: The UWB base station sends ranging pulses, and the head-mounted display tag receives them and calculates the distance to the base station based on the time difference of arrival. Simultaneously, the base station measures the angle of arrival (AoA) using its antenna array. Each base station provides a distance r. i and azimuth (θ)i , φ i Data update rate: 20Hz.

[0112] IMU data: The headset has a built-in six-axis IMU (accelerometer + gyroscope) that collects data at 200Hz, providing angular velocity ω and acceleration a.

[0113] 3.2 Tightly Coupled EKF Design

[0114] The state vector is defined as: Where P is position (3D), v is velocity (3D), q is attitude quaternion (4D), and b is position (3D). a and b g These are the biases for the accelerometer and gyroscope (each in 3 dimensions), f i Let be the three-dimensional coordinates (3m-dimensional) of the i-th visual feature point. The dimension of the state vector varies with the number of feature points.

[0115] Prediction steps: Predict the state using IMU data and a kinematic model:

[0116]

[0117]

[0118]

[0119] in, Let be a rotation matrix. and For IMU measurements, The gravitational vector, symbol This represents quaternion multiplication.

[0120] Update steps:

[0121] Visual observation: When a feature point is detected, it is projected onto the current camera plane and compared with the position of the feature point extracted from the image to obtain the residual.

[0122] UWB / AoA Observation: Based on the current position p of the head-mounted display and the base station position s i Calculate the predicted distance And the predicted angle of arrival is compared with the actual measured value.

[0123] EKF updates the state vector by minimizing the residuals while simultaneously estimating the IMU bias online. UWB data plays a dominant role when visual features are sparse, while UWB assists in eliminating drift when visual features are abundant.

[0124] 3.3 LSTM Prediction Model

[0125] To address scenarios where visual features are completely ineffective (such as rapid rotation or low-light environments), we trained an LSTM network for pose prediction. The network structure is as follows:

[0126] Input: IMU data (6-dimensional) over the past 200ms (i.e., 40 time steps, each step 5ms) and UWB measurement residuals (input the residuals if UWB is valid, otherwise set them to zero).

[0127] Hidden layers: Two LSTM layers, each with 128 units.

[0128] Output: The pose change increment in the next 50ms (position increment Δp and attitude increment Δq, a total of 7 dimensions).

[0129] Training data is generated by collecting a large number of motion trajectories (including various failure scenarios) in actual venues, and cross-validation is used to ensure generalization ability.

[0130] In EKF, if the number of visual feature points is less than a threshold (e.g., 10) for three consecutive frames, LSTM compensation is activated: the current IMU sequence is input into LSTM to obtain the predicted pose increment, which replaces the prediction step of EKF until visual recovery.

[0131] 3.4 Adaptive Weight Adjustment

[0132] The observation noise covariance matrix R of EKF is dynamically adjusted according to the environment. The visual quality factor α is defined. v (Based on the number and uniformity of feature points) and UWB signal quality factor α u (Based on signal strength and multipath interference level), the noise covariance R of visual observation is... v Multiply by weight 1 / α v The noise covariance R of UWB observations u Multiply by weight 1 / α u When α v As R approaches 0, v When the size is extremely large, visual observation is suppressed; conversely, when the size is extremely small, visual observation is suppressed.

[0133] 4. Spatial Synchronization and Dynamic Scheduling Module

[0134] This module is responsible for generating global spatial synchronization signals and dynamically allocating GPU rendering resources and network bandwidth to each head-mounted display device.

[0135] Spatial synchronization: The server broadcasts a timestamp at 1ms intervals, and all head-mounted displays synchronize their local clocks to ensure that all pose data has a unified time stamp. Based on the pose data, a synchronization signal is generated to mark the start of each frame's rendering.

[0136] Resource pre-allocation: Based on the pose predicted by LSTM, the rendering resources required for the next frame are allocated to each headset 100ms in advance.

[0137] GPU rendering pipeline: Allocate an independent rendering thread for each headset and reserve corresponding video memory bandwidth.

[0138] Network bandwidth: Based on the predicted video stream bitrate requirements (depending on scene complexity), a corresponding bandwidth (e.g., 80MHz bandwidth, specific queue priority) is reserved for each headset on the WiFi 7 router through a QoS mechanism. If the total bandwidth is insufficient, it is dynamically adjusted according to priority (e.g., according to user VIP level).

[0139] 5. Implementation of the Virtual-Real Fusion Rendering Unit

[0140] The virtual-real fusion rendering unit runs on the server and is custom-developed based on the Unity 3D engine to achieve real-time rendering from multiple perspectives.

[0141] Virtual Scenes: Multiple large-scale VR films (such as Mars, pyramids, Jurassic Park, etc.) are pre-loaded, each film containing 3D models, animations, special effects, and other resources. The corresponding scene is loaded based on the current experience content.

[0142] Dynamic Boundary Warning: While rendering the virtual scene, the risk field value R(p) is calculated based on the current position p of the headset, and a dynamic boundary mesh is rendered. The boundary mesh is extracted based on isosurfaces of the directed distance field. For example, multiple isosurfaces with d(p) = 0.5m, 0.3m, 0.1m are used, and displayed as green, yellow, and red semi-transparent meshes respectively. The transparency alpha is adjusted according to R(p).

[0143]

[0144] When R(p) is negative (walls have been penetrated), a red flashing grid is displayed at a frequency of 5Hz.

[0145] Gaze-based rendering: If the headset supports eye tracking (such as the optional accessory for PICO 4 Ultra Enterprise), the highest quality (4K) is used for the gaze area (within 15° of the center of the field of view) based on the user's gaze point, while the edge field of view is rendered at a lower resolution (such as 1080p) to reduce the GPU load.

[0146] Multi-user interaction: The server continuously monitors the pose of all headsets. If the distance between the virtual viewpoints of any two headsets is less than 1.5m and the angle between their line of sight is less than 30°, it is determined to be a social interaction event, triggering corresponding special effects (such as a halo appearing when two people are close) and generating haptic drive signals.

[0147] Each rendered frame is a 1920×1920 (single eye) RGB image, encoded as an H.265 video stream, with the bitrate dynamically adjusted according to the scene (20-50Mbps).

[0148] 6. Implementation of Low-Latency Transmission Networks

[0149] The transmission network is built using WiFi 7 (802.11be) technology, with the following main features:

[0150] Multi-link operation (MLO): Simultaneously using the 2.4GHz and 5GHz / 6GHz frequency bands to transmit data, improving throughput and reliability.

[0151] Forward Error Correction (FEC): Redundant coding is added to each frame of video data, and the receiving end can correct no more than 10% of packet loss.

[0152] Dynamic bitrate adjustment: The video encoding bitrate is dynamically adjusted based on real-time channel quality (such as RSSI and packet loss rate). If the packet loss rate exceeds 5%, the bitrate is reduced; if the channel recovers, the bitrate is increased.

[0153] Intelligent packet loss retransmission: Selective retransmission is performed on key frames (I-frames), and if non-key frames (P-frames) are lost, the previous frame is copied to fill the gap, thus avoiding lag.

[0154] Actual performance: With 20 headsets running concurrently, the end-to-end latency is stable at 12-15ms, and the packet loss rate is less than 0.5%.

[0155] 7. Terminal Presentation and Closed-Loop Correction Module

[0156] The headset receives and decodes the video stream for display, while simultaneously continuously acquiring IMU data (200Hz) and transmitting it back to the server. The server updates the EKF based on the latest IMU data, achieving closed-loop correction. Furthermore, the headset's built-in SLAM camera can also transmit keyframe images, allowing the server to optimize the visual SLAM map.

[0157] 8. Offline calibration module

[0158] Run the offline calibration procedure during initial deployment or spatial relocation:

[0159] (1) Staff members wearing head-mounted displays walk along a predetermined path, covering the entire space.

[0160] (2) The system records SLAM feature points and UWB signal strength fingerprints (each location corresponds to the RSSI and AoA of each base station) to construct a spatial fingerprint map.

[0161] (3) At the same time, use tools such as laser rangefinders to measure the actual obstacle positions and optimize the accuracy of the risk field function.

[0162] (4) Store the calibration data in the database for reference during real-time positioning.

[0163] 9. Multi-user interaction and haptic feedback

[0164] When two users approach each other and shake hands in a virtual environment:

[0165] (1) The system detects an interactive event and generates a tactile driving signal: triggers the piezoelectric ceramic unit at the corresponding position under the floor to vibrate at a frequency of 50Hz to simulate footsteps; at the same time, it triggers the linear resonant motor in the wall to generate a gentle vibration to simulate environmental feedback.

[0166] (2) The server will broadcast the interactive events synchronously to all headsets, triggering visual effects (such as a halo appearing when shaking hands).

[0167] 10. Methods and Steps

[0168] This embodiment describes a virtual-real fusion interaction method based on the above system, including the following steps:

[0169] S1. Spatial Modeling: Obtain the three-dimensional structure of the entity space through the physical space modeling module, and generate a risk field function based on the directed distance field.

[0170] S2. Pose Calculation: Real-time acquisition of SLAM visual feature point data and UWB / Bluetooth AoA angle of arrival data is achieved through a hybrid positioning engine. Front-end fusion is performed using a tightly coupled extended Kalman filter to calculate the six-DOF pose of each VR headset. Compensation pose is output based on an LSTM prediction model when vision fails.

[0171] S3. Resource Scheduling: Generate a global spatial synchronization signal based on the pose and dynamically allocate rendering computing resources and network bandwidth resources for each headset.

[0172] S4. Virtual-Real Rendering: Render a fused video stream containing virtual scene images and dynamic boundary warning images for each headset. The transparency, color, and flashing frequency of the dynamic boundary warning images are dynamically determined by the directional distance gradient value of the risk field function at the current position of the headset.

[0173] S5. Low-latency transmission: Through a low-latency transmission network based on WiFi 7, and employing forward error correction and dynamic bitrate adjustment mechanisms, the fused video stream is synchronously distributed to the corresponding head-mounted display with an end-to-end latency of no more than 15ms.

[0174] S6. Display and Correction: The fused video stream is presented through the head-mounted display screen, while IMU data and eye-tracking data are collected in real time and sent back to the hybrid positioning engine for closed-loop correction and to trigger gaze point rendering optimization.

[0175] S7. Interactive Feedback: When a multi-user virtual interaction event is detected, a synchronous interaction signal is generated to drive the distributed haptic feedback array to produce haptic feedback that matches the event type.

[0176] Through the above specific application examples, it is illustrated that this system calculates pose by fusing SLAM visual features and UWB / Bluetooth AoA data through tightly coupled extended Kalman filtering, and integrates a long short-term memory network for predictive compensation when visual failure occurs. It dynamically allocates rendering and bandwidth resources, generates a fused video stream containing virtual scenes and boundary warnings, displays and transmits IMU data back to correct the pose, and achieves highly robust positioning, dynamic boundary warning, predictive resource scheduling, and multimodal interaction closed loop in VR virtual-real fusion. This significantly improves the immersion and safety of large-space multi-person VR experiences.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A VR virtual-real fusion interactive system based on large spatial positioning and dynamic rendering, characterized in that: It includes a physical space modeling module, a hybrid positioning engine, a spatial synchronization and dynamic scheduling module, a virtual-real fusion rendering unit, a low-latency transmission network, and a terminal presentation and closed-loop correction module; The physical space modeling module, configured on a local server, is used to acquire the three-dimensional geometric structure of the offline physical space and generate a risk field function that characterizes the continuous distribution of spatial boundaries and obstacles. The hybrid positioning engine is deployed on a local server and multiple VR headsets to collect SLAM visual feature point data and UWB / Bluetooth AoA angle of arrival data of each VR headset in real time. It uses tightly coupled extended Kalman filtering to fuse the data and calculate the six-degree-of-freedom pose of each VR headset. It also integrates a long short-term memory network model to output the predicted pose for compensation when visual features fail. Each VR headset integrates a SLAM camera, an IMU, and a UWB / Bluetooth AoA positioning tag. The UWB / Bluetooth AoA angle of arrival data includes the distance provided by each UWB / Bluetooth AoA positioning base station deployed in the physical space. and azimuth ; The state vector of the tightly coupled extended Kalman filter is: ,in, For location, For speed, For attitude quaternions, and These are the biases for the accelerometer and gyroscope, respectively. For the first The three-dimensional coordinates of visual feature points are given, and the dimension of the state vector varies with the number of visual feature points. The tightly coupled extended Kalman filter uses IMU data for prediction. A first update is performed using visual feature point observations, projecting the three-dimensional feature points onto the camera plane and comparing them with the feature point positions extracted from the image to obtain the visual residual. A second update is performed using UWB / Bluetooth AoA observations, based on the current position of the VR headset. and base station location Calculate the predicted distance and the predicted angle of arrival, compared with the actual measured distance. and the azimuth angle The UWB measurement residuals are obtained by comparison; through the update step of extended Kalman filtering, the visual residuals and the UWB measurement residuals are fused to estimate the state vector and IMU bias online; The hybrid positioning engine also includes an adaptive weight adjuster, which is used to determine a visual quality factor based on the number and uniformity of visual feature points. The UWB signal quality factor is determined based on the UWB signal strength and multipath interference level. The measurement noise covariance of visual observation Multiply by weight The measurement noise covariance of UWB observations Multiply by weight To dynamically adjust the measurement noise covariance matrix of SLAM visual feature point data and UWB / Bluetooth AoA angle of arrival data in the tightly coupled extended Kalman filter; The input to the Long Short-Term Memory (LSTM) network model is the IMU data from the past 200ms and the UWB measurement residual. The UWB measurement residual is input when the UWB observation is valid; otherwise, it is set to zero. The output of the LTM network model is the pose change increment for the next 50ms, which includes the position increment. and attitude increment The visual feature failure refers to the number of visual feature points being lower than a threshold for three consecutive frames. When the visual feature fails, the hybrid localization engine inputs the current IMU sequence into the long short-term memory network model to obtain the predicted pose increment, and replaces the prediction step of the tightly coupled extended Kalman filter with the predicted pose increment until vision is recovered. The spatial synchronization and dynamic scheduling module is used to generate a global spatial synchronization signal based on the six degrees of freedom pose, and dynamically allocate independent GPU rendering pipeline resources and network bandwidth resources to each VR headset. The virtual-real fusion rendering unit is deployed on a local server and is equipped with a multi-view real-time rendering engine. It is used to render a fused video stream containing virtual scene images and dynamic boundary warning images for each VR headset device based on the six degrees of freedom pose and global space synchronization signal. The dynamic boundary warning images are dynamically generated based on the risk field function. The low-latency transmission network is built on the WiFi 7 protocol and is configured with forward error correction coding and dynamic bitrate adjustment mechanism to synchronously distribute the fused video stream to the corresponding VR headset with an end-to-end latency of no more than 15ms. The terminal presentation and closed-loop correction module is integrated into each VR headset device, including at least one display screen, for receiving and decoding the fused video stream for display, and for real-time acquisition of six-axis motion data from the VR headset device’s built-in IMU and transmitting it back to the hybrid positioning engine for closed-loop correction of the six degrees of freedom pose.

2. The VR virtual-real fusion interactive system based on large spatial positioning and dynamic rendering as described in claim 1, characterized in that: The risk field function is a continuously differentiable function constructed based on a directed distance field. The function value of the risk field function is jointly determined by the directed distance between the current position of the VR headset and the nearest physical obstacle and the surface normal vector of the nearest physical obstacle. The transparency, color, and flashing frequency of the dynamic boundary warning screen are dynamically determined by the absolute value and sign of the directed distance. When the directed distance is negative, the dynamic boundary warning screen flashes bright red.

3. The VR virtual-real fusion interactive system based on large spatial positioning and dynamic rendering as described in claim 1, characterized in that: The spatial synchronization and dynamic scheduling module also includes a predictive resource pre-allocation unit, which is used to pre-allocate the computing resources and network bandwidth required for rendering the next frame to the corresponding VR headset device 100ms in advance, based on the predicted pose output by the long short-term memory network model.

4. The VR virtual-real fusion interactive system based on large spatial positioning and dynamic rendering as described in claim 1, characterized in that: The virtual-real fusion rendering unit is also used to generate multi-person synchronous interaction events and trigger unified visual effects and haptic driving signals when it detects that the virtual viewpoints of different VR headsets are within a preset interaction distance and the angle between their line of sight is less than 30°.

5. The VR virtual-real fusion interactive system based on large spatial positioning and dynamic rendering as described in claim 4, characterized in that: The system also includes a distributed haptic feedback array, which consists of a piezoelectric ceramic vibration unit deployed under the floor of the physical space and a linear resonant motor deployed on the surface of the wall of the physical space. The distributed haptic feedback array is used to receive the haptic drive signal and generate a vibration frequency and amplitude that match the type of virtual interactive event at a specified physical location.

6. The VR virtual-real fusion interactive system based on large spatial positioning and dynamic rendering as described in claim 1, characterized in that: The VR headset has a 4K resolution and a 90Hz refresh rate display. It also integrates an eye-tracking sensor to capture the user's gaze point in real time. The virtual-real fusion rendering unit uses the highest quality rendering for the gaze area and a reduced resolution gaze point rendering technique for the edge field of view based on the user's gaze point data captured in real time by the eye-tracking sensor, so as to reduce the rendering load.

7. A VR virtual-real fusion interaction method based on large spatial positioning and dynamic rendering, implemented based on the VR virtual-real fusion interaction system based on large spatial positioning and dynamic rendering as described in any one of claims 1-6, characterized in that, The method includes the following steps: S1. Obtain the three-dimensional geometric structure of the entity space through the physical space modeling module, and generate a risk field function based on the directed distance field; S2. SLAM visual feature point data and UWB / Bluetooth AoA angle-of-arrival data for each VR headset are collected in real time using a hybrid positioning engine. Tightly coupled extended Kalman filtering is used for front-end fusion to calculate the six-DOF pose of each VR headset. A long short-term memory network prediction model is used to output a compensating pose when visual features fail. The UWB / Bluetooth AoA angle-of-arrival data includes the distance provided by each UWB / Bluetooth AoA positioning base station deployed in the physical space. and azimuth ; The fusion process of the tightly coupled extended Kalman filter includes: S201. Establish the state vector ,in, For location, For speed, For attitude quaternions, and These are the biases for the accelerometer and gyroscope, respectively. For the first The three-dimensional coordinates of a visual feature point, wherein the dimension of the state vector varies with the number of visual feature points; S202. Using IMU data for prediction, the prediction model is as follows: ; ; ; in, Let be a rotation matrix. and For IMU measurements, The gravitational vector, symbol Represents quaternion multiplication; S203. The first update is performed using visual feature point observation. The three-dimensional feature points are projected onto the camera plane and compared with the positions of feature points extracted from the image to obtain the visual residual. S204. Perform a second update using UWB / Bluetooth AoA observation, based on the current position of the VR headset. and base station location Calculate the predicted distance and the predicted angle of arrival, compared with the actual measured distance. and the azimuth angle The UWB measurement residuals were obtained by comparison; S205. Determining the visual quality factor based on the number and uniformity of visual feature points. The UWB signal quality factor is determined based on the UWB signal strength and multipath interference level. The measurement noise covariance of visual observation Multiply by weight The measurement noise covariance of UWB observations Multiply by weight By extending the Kalman filter update step, the visual residual and the UWB measurement residual are fused to estimate the state vector and IMU bias online. S206. Using the IMU data from the past 200ms and the UWB measurement residual as input to the Long Short-Term Memory (LSTM) network prediction model, when the UWB observation is invalid, the UWB measurement residual is set to zero. The LTM network prediction model outputs the pose change increment for the next 50ms, where the pose change increment includes the position increment. and attitude increment When the number of visual feature points in three consecutive frames is lower than the threshold, the visual feature is determined to be invalid. The current IMU sequence is input into the Long Short-Term Memory Network prediction model to obtain the predicted pose increment, and the prediction step of the tightly coupled extended Kalman filter is replaced by the predicted pose increment until vision is recovered. S3. Generate a global spatial synchronization signal based on the six degrees of freedom pose, and dynamically allocate rendering computing resources and network bandwidth resources to each VR headset; S4. Render a fused video stream containing virtual scene images and dynamic boundary warning images for each VR headset device, wherein the transparency, color and flashing frequency of the dynamic boundary warning images are dynamically determined by the directed distance gradient value of the risk field function at the current position of the VR headset device. S5. Using a low-latency transmission network based on WiFi 7 and employing forward error correction and dynamic bitrate adjustment mechanisms, the fused video stream is synchronously distributed to the corresponding VR headset with an end-to-end latency of no more than 15ms. S6. Present the fused video stream through the display screen of the VR headset, while simultaneously collecting the IMU data and eye-tracking data of the VR headset in real time, sending them back to the hybrid positioning engine for closed-loop correction, and triggering foveated rendering optimization. S7. When a multi-user virtual interaction event is detected, a synchronous interaction signal is generated to drive the distributed haptic feedback array to produce haptic feedback that matches the event type.

8. The VR virtual-real fusion interaction method based on large spatial positioning and dynamic rendering according to claim 7, characterized in that, In step S4, the process of generating the dynamic boundary warning screen includes: S401. Based on the directed distance field, extract multi-layer isosurfaces with distance thresholds of 0.1m, 0.3m, and 0.5m; S402. Render the multi-layer isosurfaces extracted in S401 with distance thresholds of 0.1m, 0.3m, and 0.5m as red, yellow, and green semi-transparent boundary meshes, respectively. The transparency of the boundary mesh is inversely proportional to the absolute value of the directed distance. When the directed distance is less than 0.1m, the boundary mesh is highlighted and flashing red. The directed distance refers to the directed distance between the current position of the VR headset and the nearest physical obstacle.

Citation Information

Patent Citations

  • Dynamic semantic vision SLAM (Simultaneous Localization and Mapping) method based on point and line feature adaptive weighting

    CN121661335A

  • Security area setting method and device and virtual reality equipment

    CN113284258A

  • Tight coupling odometer method and system based on UWB and visual fusion

    CN117629187A

  • VR large-space positioning interaction system based on multi-modal perception

    CN120469587A