A VR large space positioning interaction system based on multimodal perception

Abstract

The present invention relates to the field of virtual reality positioning and discloses a VR large-space positioning interaction system based on multimodal perception. The system comprises the following steps: deploying multimodal sensors to acquire sensor data, extracting and preprocessing visual features, matching and establishing a sparse point cloud map, performing scale calibration, and constructing an environment model; pre-judging a UWB signal path based on the environment model, optimizing and compensating for NLOS states, observing and updating through degradation detection, and fusing the predicted state changes; constructing an interactive perception network to track hands and the whole body; using a layered tactile system to achieve tactile feedback, and generating tactile effects through vibration frequency mapping; utilizing beamforming technology to improve transmission efficiency, using an edge cloud server cluster to render a virtual scene, and pre-rendering the scene in advance to offset network and rendering delays; and designing an online calibration mechanism to correct system errors through visual loop detection, dynamic correction of UWB beacons, and IMU drift compensation.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality positioning technology, and in particular to a VR large-space positioning interaction system based on multimodal perception. Background Art

[0002] The VR large-space positioning and interaction system based on multimodal perception is one of the core breakthrough directions in the field of virtual reality technology. Its goal is to achieve high-precision positioning and natural interaction of users in a large physical space by integrating data from multiple sensors such as vision, inertia, ultrasound, and depth perception.

[0003] Existing technologies rely on passive reflective markers or active LED markers, and use multiple cameras to capture the position of marker points to achieve millimeter-level accuracy. IMU is used to compensate for the short-term motion blur of visual SLAM, and visual loop detection is used to suppress the cumulative error of IMU. Visual SLAM is prone to failure in low light, strong light or dynamic occlusion; UWB is affected by non-line-of-sight errors, and its accuracy decreases after penetrating walls; multimodal data fusion requires high-performance edge computing equipment, and multi-target tracking in dynamic environments requires complex algorithms for joint probabilistic data association, which can easily lead to excessive latency and motion sickness; existing technologies have low recognition rates for hand occlusion and rapid movements; complex gestures rely on predefined templates and lack generalization capabilities. Vibration handles can only simulate simple touch and cannot restore the material of the object or the direction of the force. When multiple people interact at the same time, network transmission delays and uneven rendering loads can easily lead to asynchronous movements, destroying the sense of immersion; existing systems mostly rely on preset collision volumes, which make it difficult to accurately simulate the interaction between complex human postures and virtual objects;

[0004] In view of this, it is necessary to provide a VR large-space positioning interaction system based on multimodal perception. Summary of the Invention

[0005] The purpose of the present invention is to provide a VR large space positioning interaction system based on multimodal perception. To solve the above-mentioned problems in the prior art, the present invention is implemented through the following technical solutions:

[0006] The embodiment of the present invention provides a VR large-space positioning interaction system based on multimodal perception, which specifically includes the following modules:

[0007] Perception deployment module: deploys multimodal sensors, acquires images and performs visual feature extraction, obtains lidar data for preprocessing, matches visual feature points to build a sparse point cloud map, uses UWB beacons to calibrate the map, and semantically annotates objects to build an environment model;

[0008] Fusion positioning module: Acquires UWB, VIO, and LiDAR data based on the environmental model, pre-judges the UWB signal path based on the environmental model, optimizes and compensates for NLOS errors, observes and updates UWB, VIO, and LiDAR data through degradation detection, and fuses and predicts state changes;

[0009] Interactive feedback module: uses sensor fusion technology to build an interactive perception network to track hands and the entire body; uses a layered tactile system to achieve tactile feedback, uses vibration frequency mapping to generate tactile effects, and uses voice recognition combined with ultrasound and light-based technologies for interaction and attention perception;

[0010] Interaction Optimization Module: Deploys millimeter-wave base stations and uses beamforming technology to improve transmission efficiency. Edge cloud server clusters render virtual scenes, obtain user historical trajectories to predict the coordinates of the next frame's 3D point cloud, and pre-render scenes in advance to offset network and rendering delays.

[0011] Calibration and compensation module: Design an online calibration mechanism to monitor and correct system errors in real time through visual loop detection, UWB beacon dynamic correction, and IMU drift compensation.

[0012] Beneficial effects of the present invention:

[0013] 1. The sparse point cloud map constructed by ORB-SLAM3 is scale-calibrated by providing a global absolute coordinate reference using UWB beacons, solving the scale drift problem in traditional SLAM mapping. The pre-built semantic environment model is used to predict the UWB signal propagation path, and the ranging error in the NLOS state is dynamically compensated through weighted least squares optimization. The degradation state of LiDAR or VIO data is determined in real time through covariance matrix analysis, triggering UWB data for position correction, and dynamically fusing multi-source data with selective Kalman filtering. The visual feature extraction of ORB-SLAM3 and the pre-processing of LiDAR point clouds are combined to optimize the camera pose through reprojection error, and the map is semantically annotated. The DBoW3 bag-of-words model is used to realize visual loop detection, and the dynamic correction of UWB beacons and IMU drift compensation are combined to optimize the global pose by minimizing the reprojection error and ranging residual error.

[0014] 2. Design a layered tactile architecture of base layer + enhancement layer + force feedback layer, and restore the tactile effect through vibration frequency mapping and contact force-current conversion formula; combine ultrasonic ranging and light-based measurement to achieve submillimeter line of sight positioning, support gaze interaction and attention perception; use the Whisper model to implement local offline voice recognition, with command priority higher than manual operation, and real-time response without the need for a network; utilize the high bandwidth and low latency characteristics of 5G-A millimeter waves, combine edge cloud rendering with the OpenXR standard, and achieve multi-brand headset compatibility and dynamic compression of the visual cone area; predict the next frame position based on the user's historical motion trajectory, and pre-render the scene in advance to offset network and rendering delays; synchronize all sensor clocks through the PTP protocol and hardware delay compensation; design a multi-layer calibration mechanism for visual loop detection, UWB anchor point periodic calibration, and IMU zero-bias compensation during static periods, and achieve global pose correction by minimizing reprojection error and ranging residual. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 This is a flowchart of a VR large-space positioning interaction system based on multimodal perception provided by Example 1 of the present invention;

[0017] Figure 2 This is a structural diagram of a VR large-space positioning interaction system based on multimodal perception provided by Example 2 of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0019] Example 1

[0020] like Figure 1 As shown, the embodiment of the present invention provides a VR large space positioning interaction system based on multimodal perception, which specifically includes the following modules:

[0021] Perception deployment module: deploys multimodal sensors, acquires images and performs visual feature extraction, obtains lidar data for preprocessing, matches visual feature points to build a sparse point cloud map, uses UWB beacons to calibrate the map, and semantically annotates objects to build an environment model;

[0022] In a specific embodiment, ultra-wideband (UWB) anchor points are installed at the four corners and ceiling of the target space, and the distance between the anchor points is strictly controlled to be ≤5 meters.

[0023] Select the UWB anchor point of the built-in DW1000 module. The DW1000 module provides a global absolute coordinate reference, providing a precise coordinate reference for subsequent positioning and mapping.

[0024] It should be noted that the DW1000 module is an ultra-wideband (UWB) wireless communication module launched by Decawave. Based on the DW1000 chip design, it is designed for high-precision positioning and low-power communication scenarios. Anchor points are deployed in the four corners and ceiling of the space to form a relatively even distribution, ensuring that each area in the space can receive signals from at least three anchor points, thereby achieving triangulation positioning.

[0025] Deploy a solid-state laser radar with a scanning frequency of 10 Hz to scan the spatial environment in real time and generate point cloud data;

[0026] Install the solid-state laser radar at a fixed height in the space. For example, install the solid-state laser radar below the ceiling to obtain a scanning field of view, reduce obstructions, and ensure that the outlines and structures of objects in the space are fully captured.

[0027] Arrange a visual sensor network, select Intel-Real-SenseD455 as the visual sensor, and set the frame rate to 60fps;

[0028] Reasonably distribute visual sensors in the space to ensure full coverage of the space for dynamic obstacle detection and loop detection;

[0029] It should be noted that the distribution of visual sensors needs to take into account the overlap of perspectives so that repeated scenes can be accurately identified during subsequent loop closure detection;

[0030] Incremental SLAM mapping:

[0031] The ORB-SLAM3 algorithm is used to extract visual features from images acquired by the visual sensor. ORB feature point detection and descriptor calculation are used to obtain visual feature points in the image. At the same time, the point cloud data generated by the lidar is preprocessed to remove noise points before matching with the visual feature points.

[0032] Get the pixel coordinates of the visual feature points in the image using the formula:

[0033]

[0034] Get the visual reprojection error ,in, is the pixel coordinate of the feature point in the image, which is obtained by collecting the image and performing feature detection by the visual sensor; is the camera intrinsic parameter matrix, which includes the focal length and principal point coordinates of the camera, and is determined through the camera calibration process; is the camera’s pose transformation matrix, is the rotation matrix, describing the rotation posture of the camera, is the translation vector, describing the position of the camera; is the three-dimensional point cloud coordinate, which is the spatial point coordinate obtained by SLAM mapping;

[0035] The visual reprojection error is used to calculate the reprojection error of the 3D point cloud on the image plane. By optimizing this error, the camera pose is adjusted to improve the mapping accuracy.

[0036] Construct sparse point cloud maps by fusing vision and LiDAR data;

[0037] During the construction process, the map is scaled using the global absolute coordinates provided by the UWB beacon to ensure the scale consistency of the map;

[0038] UWB ranging model:

[0039]

[0040] in, The coordinates of the UWB anchor point are fixed values ​​obtained through precise deployment and measurement in space in the early stage. The line-of-sight error is a small error in the line-of-sight situation because the signal propagation path is not blocked. In the non-line-of-sight situation, the signal propagation path is blocked and the error is large. The Kalman filter is used to estimate and compensate for the error in real time to improve the ranging accuracy.

[0041] Specifically, the known coordinate points obtained by the UWB beacon are compared with the coordinate points estimated during the SLAM process, the scale factor is calculated, and the map is scaled and adjusted;

[0042] Semantic annotation:

[0043] Use deep learning models such as Point-Net or MaskR-CNN to semantically classify walls, doors, and obstacles in the constructed point cloud map;

[0044] Label the training data, which includes the categories and locations of various objects. After labeling, the point cloud data is fed into the trained model, which outputs the semantic labels for each point and generates a structured environment model.

[0045] The structured environment model includes the category, location, and shape information of objects in the space, providing rich environmental information for subsequent path planning and interaction.

[0046] Fusion positioning module: Based on the environmental model, it obtains UWB, VIO, and LiDAR data through sensors, pre-judges the UWB signal path based on the environmental model, optimizes and compensates for the NLOS state, observes and updates the UWB, VIO, and LiDAR data through degradation detection, and fuses and predicts state changes;

[0047] The UWB tag is worn on the user's ankle or handle. The tag's built-in DW1000 module communicates with the UWB anchor point in space to obtain centimeter-level absolute position information in real time.

[0048] Based on the constructed environmental model, pre-determine whether the propagation path of the UWB signal is blocked;

[0049] Specifically, if there are obstacles on the path, that is, in the NLOS state, the weighted least squares optimization method is used to calculate the distance noise standard deviation of each anchor point. Assign weights to the ranging data. The smaller the noise standard deviation, the higher the weight. The user's position is estimated by minimizing the weighted sum of squared errors. , compensate for NLOS error

[0050] Wear the tag on the ankle or handle, which reflects the user's overall motion trajectory while reducing the impact of occlusion from other parts of the body on the signal;

[0051] The VIO module is integrated into a VR headset, which includes an IMU and a monocular camera.

[0052] The gyroscope measures the angular velocity of the headset, the accelerometer measures the acceleration of the headset, and the monocular camera is used to capture images of the surrounding environment;

[0053] Through the VIO algorithm, combined with IMU and camera data, the 6-DOF pose is output, that is, the position and attitude information of the headset;

[0054] LiDAR scans the environment in real time, acquires point cloud data, and detects dynamic obstacles such as other users and moving objects through point cloud processing algorithms;

[0055] The detection process includes point cloud segmentation and cluster analysis steps to separate dynamic objects from the static environment, providing a basis for path planning and obstacle avoidance;

[0056] Selective fusion based on the status of different sensors;

[0057] Detect whether LiDAR or VIO data has degradation, including insufficient feature points and IMU drift. Use the detection mechanism to trigger UWB data to correct the position.

[0058] Degradation detection is done by the formula:

[0059]

[0060] Get degradation detection value ,in, is the current covariance matrix, reflecting the uncertainty of state estimation; is the initial covariance matrix, which is set to a smaller value, indicating that the uncertainty of the initial state is lower; is the sum of the diagonal elements of the current covariance matrix, is the sum of the diagonal elements of the initial covariance matrix;

[0061] when When it is greater than 2, it indicates that the uncertainty of the current state estimation has increased significantly, triggering vision / LiDAR fusion to utilize more sensor data to reduce the uncertainty;

[0062] Visual observation uses the PnP algorithm to calculate the reprojection error based on the correspondence between the 3D point cloud and the 2D image feature points, and then update the posture;

[0063] LiDAR observation uses a point cloud matching algorithm to match the currently scanned point cloud with the point cloud in the map to optimize the global pose;

[0064] Selective Kalman Filter: Defining the state vector ,in, Represents the three-dimensional point cloud coordinates, consisting of three-dimensional coordinates (x, y, z); Indicates the speed of state change, also a three-dimensional vector; Represents quaternion attitude, which is used to describe the rotational attitude of an object. Compared with Euler angles, it avoids the gimbal lock problem. Represents angular velocity, which is the angular velocity vector of the object's rotation;

[0065] Through the prediction equation:

[0066]

[0067] Get predicted state changes ,in, is the state transfer matrix, which is used to describe the state transfer relationship from time k−1 to time k, considering the kinematic characteristics of the object; As the input matrix, the angular velocity Translated into the effect on the state vector; is process noise, reflecting the uncertainty of the system model and external interference;

[0068] Interactive feedback module: uses sensor fusion technology to build an interactive perception network to track hands and the entire body; uses a layered tactile system to achieve tactile feedback, uses vibration frequency mapping to generate tactile effects, and uses voice recognition combined with ultrasound and light-based technologies for interaction and attention perception;

[0069] Hand tracking: Use Leap Motion or inertial sensors to obtain hand motion data; Leap Motion uses visual sensors to capture hand posture and finger movements in real time, while inertial sensors use IMUs to measure hand acceleration and angular velocity;

[0070] The extended Kalman filter is used to fuse skeleton data with IMU information. First, a skeleton model of the hand is established, and the sensor data is mapped to the position and posture of the skeleton joints. The extended Kalman filter is used to filter out noise and errors, and obtain hand tracking data with a delay of less than 50ms.

[0071] Full body posture tracking: Full body posture monitoring using OptiTrackLite or low-cost IMU arrays;

[0072] OptiTrackLite uses optical marker positioning technology to obtain the position of each joint in the body, while the IMU array uses sensors worn on various parts of the body to measure acceleration, angular velocity, and magnetic field strength.

[0073] Combined with the human kinematics model, the motion relationship between the body joints is established, and the joint angles are predicted through the model to achieve real-time tracking of the whole body posture;

[0074] Implementing tactile feedback using a layered tactile system consisting of a base layer, an enhancement layer, and a force feedback layer.

[0075] The base layer uses the CyberEngine ultra-wideband motor built into the handle. The ultra-wideband motor has a preset operating frequency range of 50-500Hz and a power consumption of 0.12W, simulating vibration and recoil effects.

[0076] For example, when a user touches an object in a virtual environment, the ultra-wideband motor generates vibrations of different frequencies and intensities according to the hardness of the object and the force of the collision;

[0077] The enhancement layer uses a flexible electronic skin that fits the palm and analyzes contact pressure and texture through visual images;

[0078] Sensors on the surface of the flexible electronic skin sense pressure distribution and texture changes during contact, converting these pressure distribution and texture changes into electrical signals to restore the tactile feel of the material. For example, the cold and hard feel of metal and the soft and soft feel of cloth can be restored.

[0079] The force feedback layer uses pneumatic muscles or shape memory alloy-driven exoskeleton gloves to provide directional feedback of force;

[0080] When the user pushes or pulls an object in a virtual environment, the exoskeleton glove applies a corresponding force based on the object's weight and the direction of resistance, allowing the user to feel the real force difference;

[0081] Based on tactile feedback, tactile effects are generated using vibration frequency mapping;

[0082] Vibration frequency mapping: Map low-frequency vibrations to collision effects. When the user collides with a virtual object, the controller motor generates low-frequency vibrations to simulate the impact of the collision.

[0083] High-frequency vibration is mapped to the friction effect. When the user slides on the surface of the virtual object, the high-frequency vibration simulates the delicate feeling of friction.

[0084] The contact force is obtained based on tactile feedback. The contact force includes the weight and resistance of the object. The obtained contact force is converted into the driving current of the ultra-wideband motor through the formula:

[0085]

[0086] Get the driving current ,in, is the contact force, which is measured by sensors or calculated by physical models in the virtual environment; is the sensitivity coefficient, which is calibrated according to the characteristics of the tactile feedback device and reflects the conversion ratio between contact force and current; Bias current is used to adjust the initial current value to ensure that the device is in the correct working state when there is no contact force;

[0087] Through this formula, the contact force is converted into a corresponding current signal, driving the ultra-wideband motor of the tactile feedback device to produce the corresponding tactile effect;

[0088] The Whisper model is used to implement offline speech recognition. The Whisper model runs on the local device without relying on the network, improving the real-time performance and reliability of speech recognition.

[0089] Common interactive commands are pre-trained into the model. When users issue voice commands, the model recognizes and interprets them in real time. These commands take precedence over manual operations, ensuring that users can quickly trigger corresponding functions through voice.

[0090] Integrated hybrid gaze tracking technology, combining ultrasonic and light-based measurement methods, for sub-millimeter gaze positioning;

[0091] Ultrasonic sensors are used to measure the distance between the eyes and the device, and light-based measurement is used to detect the direction of eye gaze. By combining the two, the user's gaze position can be accurately determined;

[0092] Supports gaze interaction, where users select or activate a virtual object by gazing at it, while also enabling attention perception, adjusting the display and interaction of the virtual environment based on the user's gaze direction;

[0093] Interaction Optimization Module: Deploys millimeter-wave base stations and uses beamforming technology to improve transmission efficiency. Edge cloud server clusters render virtual scenes, obtain user historical trajectories to predict the coordinates of the next frame's 3D point cloud, and pre-render scenes in advance to offset network and rendering delays.

[0094] Deploy 5G-A millimeter wave base stations within the system deployment site to ensure that users always maintain high-speed, low-latency network connections while on the move;

[0095] The high frequency characteristics of the millimeter wave band make the signal wavelength shorter, achieving a narrower beam width. The beamforming technology improves the signal transmission efficiency and anti-interference ability.

[0096] Edge cloud is deployed on a server cluster at the edge of the site to run rendering tasks, using the powerful computing power of edge servers to render virtual scenes;

[0097] Compatible with multiple brands of headsets through the OpenXR standard, achieving seamless connection and data interaction between different models of VR headsets, improving system compatibility and scalability;

[0098] The video stream compression adopts MPEG-OMAF format, which supports efficient encoding of 360-degree panoramic videos;

[0099] High-resolution encoding of the user's visual cone area ensures a clear and smooth display of the area the user is currently focusing on;

[0100] Degrade the quality of transmission in non-viewing areas to reduce data volume and save bandwidth;

[0101] During the encoding process, the viewing cone area is dynamically adjusted according to the user's line of sight to improve compression efficiency;

[0102] When synchronizing interactive data, all sensor data and rendering instructions carry precise timestamps, and timestamp alignment is used to ensure the temporal consistency of the data.

[0103] A prediction compensation algorithm uses a kinematic model to predict the next frame position based on the user's historical motion trajectory, and pre-renders the scene on the edge server in advance to offset network and rendering delays, ensuring that the scene the user sees is synchronized with the actual action.

[0104] Get the user's 3D point cloud coordinates, state change speed, and state change acceleration of the previous frame using the formula:

[0105]

[0106] Get the predicted 3D point cloud coordinates ,in, The user's previous frame position; is the speed of the previous frame; is the acceleration of the previous frame; is the initial prediction time, which is the network round-trip delay plus the rendering delay;

[0107] The predicted 3D point cloud coordinates are used to predict the next frame's position based on the user's historical motion state, allowing the scene to be rendered in advance to reduce the impact of latency on user experience.

[0108] Calibration and compensation module: Design an online calibration mechanism to monitor and correct system errors in real time through visual loop detection, UWB beacon dynamic correction, and IMU drift compensation;

[0109] The DBoW3 bag-of-words model is used to process images collected by the visual sensor. First, the ORB feature points of the image are extracted, and the bag-of-words model is constructed to convert the image into a bag-of-words vector.

[0110] By comparing the bag-of-words vector of the current image with the bag-of-words vector of historical images, repeated scenes are identified. When repeated scenes are detected, global pose optimization is triggered to adjust the system's pose estimation and eliminate accumulated errors.

[0111] A UWB anchor point with a known position is set in space as a calibration point. The user regularly approaches the UWB anchor point. The system obtains the user's UWB tag position at that time and compares it with the known position of the anchor point to calculate the tag's offset parameters, which include position deviation and attitude deviation. The offset parameters are automatically updated to correct the positioning error of the UWB tag.

[0112] Use the user's stationary period to perform zero-bias calibration on the accelerometer and gyroscope;

[0113] In a static state, the output of the accelerometer should theoretically be the acceleration due to gravity. The zero bias of the accelerometer is obtained by calculating the deviation between the average output value and the acceleration due to gravity.

[0114] The output of the gyroscope should be zero when it is stationary. The gyroscope's zero bias is obtained by calculating the average output value, and the IMU data is compensated for the zero bias to reduce drift error.

[0115] Specifically, through the formula:

[0116]

[0117] Minimize the correction error Perform global pose optimization, where is the reprojection error of the i-th visual feature point, reflecting the error of visual positioning; is the ranging residual of the jth UWB anchor point, reflecting the error of UWB positioning;

[0118] By minimizing the sum of the squares of these two errors and optimizing the pose estimation of visual and UWB data simultaneously, the compensated and corrected global pose is obtained.

[0119] Synchronize the clocks of all sensors through the PTP protocol to ensure that the timestamps of the sensor data are consistent;

[0120] Obtain the camera's timestamp, measure the hardware delay compensation from sensor data acquisition to transmission to the system, sum the camera's timestamp with the hardware delay compensation to obtain a corrected timestamp, and correct the timestamp to ensure the time consistency of the data and provide an accurate time reference for multimodal data fusion.

[0121] Example 2

[0122] like Figure 2 As shown, an embodiment of the present invention provides a VR large space positioning interaction method based on multimodal perception, which specifically includes the following steps:

[0123] Step 1: Deploy multimodal sensors, acquire images and perform visual feature extraction, obtain lidar data for preprocessing, match visual feature points to create a sparse point cloud map, use UWB beacons to calibrate the map, and semantically annotate objects to build an environment model.

[0124] Step 2: Acquire UWB, VIO, and LiDAR data based on the environmental model, pre-judge the UWB signal path based on the environmental model, optimize and compensate for the NLOS state error, observe and update the UWB, VIO, and LiDAR data through degradation detection, and fuse the predicted state changes;

[0125] Step 3: Use sensor fusion technology to build an interactive perception network to track the hands and the whole body; use a layered tactile system to achieve tactile feedback, use vibration frequency mapping to generate tactile effects, and use speech recognition combined with ultrasound and light-based technologies for interaction and attention perception;

[0126] Step 4: Deploy millimeter wave base stations and use beamforming technology to improve transmission efficiency. Edge cloud server clusters render virtual scenes, obtain user historical trajectories, predict the 3D point cloud coordinates for the next frame, and pre-render the scene in advance to offset network and rendering delays.

[0127] Step 5: Design an online calibration mechanism to monitor and correct system errors in real time through visual loop detection, UWB beacon dynamic correction, and IMU drift compensation.

[0128] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention; the above formulas are all dimensionless and numerical calculations, and the formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field based on actual conditions and historical experience, and can be adjusted according to actual conditions; the above is only a preferred embodiment of the present invention and is not used to limit the present invention. All changes and improvements made according to the scope of application of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A VR large space positioning interaction system based on multimodal perception, characterized by: Includes the following modules: Perception deployment module: deploys multimodal sensors, acquires images and performs visual feature extraction, obtains lidar data for preprocessing, matches visual feature points to build a sparse point cloud map, uses UWB beacons to calibrate the map, and semantically annotates objects to build an environment model; Fusion positioning module: Acquires UWB, VIO, and LiDAR data based on the environmental model, pre-judges the UWB signal path based on the environmental model, optimizes and compensates for NLOS errors, observes and updates UWB, VIO, and LiDAR data through degradation detection, and fuses and predicts state changes; Interactive feedback module: uses sensor fusion technology to build an interactive perception network to track hands and the entire body; uses a layered tactile system to achieve tactile feedback, uses vibration frequency mapping to generate tactile effects, and uses voice recognition combined with ultrasound and light-based technologies for interaction and attention perception; Interaction Optimization Module: Deploys millimeter-wave base stations and uses beamforming technology to improve transmission efficiency. Edge cloud server clusters render virtual scenes, obtain user historical trajectories to predict the coordinates of the next frame's 3D point cloud, and pre-render scenes in advance to offset network and rendering delays. Calibration and compensation module: Design an online calibration mechanism to monitor and correct system errors in real time through visual loop detection, UWB beacon dynamic correction, and IMU drift compensation.

2. A VR large space positioning interaction system based on multimodal perception according to claim 1, characterized in that: The method for matching visual feature points is: The ORB-SLAM3 algorithm is used to extract visual features from images acquired by the visual sensor. ORB feature point detection and descriptor calculation are used to obtain visual feature points in the image. At the same time, the point cloud data generated by the lidar is preprocessed to remove noise points before matching with the visual feature points. Get the pixel coordinates of the visual feature points in the image using the formula: Get the visual reprojection error ,in, is the pixel coordinate of the feature point in the image; is the camera intrinsic parameter matrix; is the camera’s pose transformation matrix, is the rotation matrix, is the translation vector; is the 3D point cloud coordinate.

3. The VR large space positioning interaction system based on multimodal perception according to claim 1 is characterized in that: The method for performing scale calibration is: Use the global absolute coordinates provided by the UWB beacon to calibrate the map scale to ensure the scale consistency of the map; UWB ranging model: in, is the coordinate of the UWB anchor point, The line-of-sight error is estimated and compensated in real time through Kalman filtering; The known coordinate points obtained by the UWB beacon are compared with the coordinate points estimated during the SLAM process to calculate the scale factor and adjust the map scale.

4. The VR large space positioning interaction system based on multimodal perception according to claim 1 is characterized in that: The method for optimizing the compensation error is: Based on the constructed environmental model, pre-determine whether the propagation path of the UWB signal is blocked; If there are obstacles on the path, that is, in the NLOS state, the weighted least squares optimization method is used to calculate the distance noise standard deviation of each anchor point. Assign weights to the ranging data. The smaller the noise standard deviation, the higher the weight. The user's position is estimated by minimizing the weighted sum of squared errors. , compensate for NLOS error.

5. The VR large space positioning interaction system based on multimodal perception according to claim 1 is characterized in that: The degradation detection method is: Selective fusion based on the status of different sensors; Detect whether LiDAR or VIO data has degradation, including insufficient feature points and IMU drift. Use the detection mechanism to trigger UWB data to correct the position. Degradation detection is done by the formula: Get degradation detection value ,in, is the current covariance matrix, is the initial covariance matrix, is the sum of the diagonal elements of the current covariance matrix, is the sum of the diagonal elements of the initial covariance matrix; when When it is greater than 2, it indicates that the uncertainty of the current state estimation increases significantly, triggering the fusion of vision and LiDAR; Visual observation uses the PnP algorithm to calculate the reprojection error and update the pose based on the correspondence between the 3D point cloud and the 2D image feature points; LiDAR observations use a point cloud matching algorithm to match the currently scanned point cloud with the point cloud in the map.

6. The VR large space positioning interaction system based on multimodal perception according to claim 1, characterized in that: The method for predicting state changes is: Define the state vector ,in, Represents the three-dimensional point cloud coordinates, consisting of three-dimensional coordinates (x, y, z); Indicates the speed of state change, also a three-dimensional vector; Represents quaternion attitude, which is used to describe the rotational attitude of an object. Compared with Euler angles, it avoids the gimbal lock problem. Represents angular velocity, which is the angular velocity vector of the object's rotation; Through the prediction equation: Get predicted state changes ,in, is the state transition matrix, As the input matrix, the angular velocity Translated into the effect on the state vector; is the process noise.

7. The VR large space positioning interaction system based on multimodal perception according to claim 1 is characterized in that: The method for realizing tactile feedback of the layered tactile system is as follows: Implementing tactile feedback using a layered tactile system consisting of a base layer, an enhancement layer, and a force feedback layer. The base layer uses the CyberEngine ultra-wideband motor built into the handle. The ultra-wideband motor has a preset operating frequency range of 50-500Hz and a power consumption of 0.12W, simulating vibration and recoil effects. The enhancement layer uses a flexible electronic skin that fits the palm and analyzes contact pressure and texture through visual images; Sensors on the surface of the flexible electronic skin sense pressure distribution and texture changes during contact, converting these pressure distribution and texture changes into electrical signals to restore the tactile feel of the material. The force feedback layer uses pneumatic muscles or shape memory alloy-driven exoskeleton gloves to provide directional feedback of force.

8. The VR large space positioning interaction system based on multimodal perception according to claim 1 is characterized in that: The method for generating the tactile effect is: The contact force is obtained based on tactile feedback. The contact force includes the weight and resistance of the object. The obtained contact force is converted into the driving current of the ultra-wideband motor through the formula: Get the driving current ,in, is the contact force, is the sensitivity coefficient, is the bias current; The contact force is converted into a corresponding current signal, driving the ultra-wideband motor of the tactile feedback device to produce the corresponding tactile effect.

9. The VR large space positioning interaction system based on multimodal perception according to claim 1, characterized in that: The method for predicting the coordinates of the next frame of three-dimensional point cloud is: Based on the user's historical motion trajectory, a kinematic model is used to predict the next frame position and pre-render the scene on the edge server in advance; Get the user's 3D point cloud coordinates, state change speed, and state change acceleration of the previous frame using the formula: Get the predicted 3D point cloud coordinates ,in, The user's previous frame position; is the speed of the previous frame; is the acceleration of the previous frame; is the initial prediction time; Through the predicted 3D point cloud coordinates, the position of the next frame is predicted according to the user's historical motion status, and the scene is rendered in advance.

10. The VR large space positioning interaction system based on multimodal perception according to claim 1, characterized in that: The method for correcting the system error is: Use the user's stationary period to perform zero-bias calibration on the accelerometer and gyroscope; In a static state, the output of the accelerometer should theoretically be the acceleration due to gravity. The zero bias of the accelerometer is obtained by calculating the deviation between the average output value and the acceleration due to gravity. The output of the gyroscope should be zero when it is stationary. The gyroscope's zero bias is obtained by calculating the average output value, and the IMU data is compensated for the zero bias to reduce drift error. Specifically, through the formula: Get the minimized correction error Perform global pose optimization, where is the reprojection error of the i-th visual feature point, is the ranging residual of the j-th UWB anchor point; By minimizing the sum of the squares of these two errors and optimizing the pose estimation of visual and UWB data simultaneously, the compensated and corrected global pose is obtained.

Images