Scenic spot immersive viewing and tour guide system based on unmanned aerial vehicle and XR head-mounted display

Abstract

The invention relates to the technical field of intelligent viewing, and discloses a scenic spot immersive viewing and tour guide system based on an unmanned aerial vehicle and an XR head-mounted display, which comprises an unmanned aerial vehicle unit, a signal transmission link, the XR head-mounted display, a sound and picture synchronization module, a safety redundancy unit and a multi-axis self-rotating body sensing seat unit, long-distance and complex-terrain 4K real-time image transmission is achieved through a mixed signal transmission link, the time delay is smaller than or equal to 120 ms, the frame loss rate is smaller than or equal to 0.2%, and stable and smooth first-view-angle far and close shot images are presented in cooperation with a multi-point holder slow motion algorithm; xR head-mounted display is combined with an IMU attitude stabilization algorithm to inhibit dizziness, and stereo explanation audios are synchronously output to realize sound and picture consistency; the multi-axis self-rotating body sensing seat simulates postures such as climbing, descending, yawing and rolling in real time according to flight data, dynamic feedback such as back pushing feeling and weightlessness feeling is provided, and tourists can obtain the three-in-one viewing effect of'real-time high-definition pictures + multi-sense immersion experience + precise science popularization explanation 'without moving.

Description

Technical Field

[0001] The present invention relates to the field of intelligent viewing technology, and in particular to an immersive viewing and tour guide system for a scenic area based on a drone and an XR head display. Background Art

[0002] Existing scenic spots offer three main methods for viewing high-altitude or remote scenic spots: ground / cableway sightseeing, aerial promotional videos / recorded VR videos, and drone live broadcasts combined with large screens. However, all have significant limitations. Ground / cableway sightseeing, restricted by terrain, makes it difficult to reach optimal viewing heights or angles, lacking the detail and immersive experience of landscapes with significant vertical or elevation differences. Aerial promotional videos / recorded VR videos are pre-recorded, lacking real-time and immersive experiences, and offer no tactile feedback, resulting in a lack of immersion. Drone live broadcasts combined with large screens can suffer from unstable signal links, high latency, a lack of interactivity, lack of educational depth, and lack of dynamic feedback in complex environments.

[0003] Therefore, there is an urgent need for an immersive viewing and tour guide system for scenic spots based on drones and XR headsets to address the shortcomings of existing technologies. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention proposes an immersive viewing and tour guide system for scenic spots based on drones and XR headsets. Through the real-time image transmission of drones, three-dimensional presentation of XR headsets and linkage with multi-axis somatosensory seats, the problems of insufficient immersion and poor interactivity in the existing technology are solved.

[0005] In order to solve the above technical problems, the basic technical solutions proposed by the present invention are:

[0006] An immersive scenic area viewing and tour guide system based on drones and XR headsets includes a drone unit, a signal transmission link, an XR headset, an audio-visual synchronization module, a safety redundancy unit, and a multi-axis self-rotating somatosensory seat unit.

[0007] Preferably, in the drone unit, the aircraft flies fully automatically according to a preset route, the gimbal adopts a "multi-point gimbal slow motion algorithm" based on calculus, and adjusts the pitch angle, yaw angle and roll angle in real time according to the waypoint, and the onboard image acquisition module outputs a 4K / 60fps real-time video stream;

[0008] The core logic of the multi-point pan-tilt easing algorithm is:

[0009] Assume the waypoint sequence is {P i (x i ,y i ,z i )}, calculate the current waypoint P in real time n With the next waypoint P n+1 Space vector

[0010] The calculus interpolation method is used to dynamically adjust the gimbal pitch angle θ and yaw angle φ so that the lens optical axis is always aligned with the target landscape. The formula is: φ(t)=arctan 2(y n+1 -y n ,x n+1 -x n )·k φ where d xy is the horizontal distance, k θ ,k φ is the smoothing coefficient, ranging from 0.1 to 0.5.

[0011] Preferably, the signal transmission link includes a wireless image transmission link and a wired image transmission link. The wireless link completes the video stream transmission from the drone to the ground control station, and the wired link completes the video transmission from the control station to the XR headset. An active image conversion module is provided in the link to achieve high-definition, low-latency, stable real-time image transmission of ≥50m.

[0012] Wireless link: Using 2.4GHz / 5.8GHz image transmission module to achieve video transmission from the drone to the ground control station, with an effective distance of 8km-15km and a delay of ≤80ms;

[0013] Wired link: Connect the control station and XR headset via HDMI 2.0 cable / CAT6 8-core twisted pair cable / optical fiber, with a transmission distance of ≥ 50m, and equipped with an active image conversion and amplification module.

[0014] Preferably, the XR headset receives real-time image streams via HDMI-IN, has a built-in IMU attitude stabilization algorithm, a FOV≈50° to suppress dizziness, and has a stereo audio playback function.

[0015] Preferably, the audio and video synchronization module matches the explanation audio with the drone route time point one by one. When the drone flies to scenic spot A, at instant t1, the module is triggered at t1 and plays the explanation of scenic spot A, with an end-to-end delay of ≤120ms.

[0016] Preferably, the safety redundancy unit sets an electronic fence: flight radius <15km, altitude <1500m; triggers automatic return after the drone loses contact for 5s; and adopts GNSS+RTK high-precision positioning.

[0017] Preferably, the multi-axis self-rotating somatosensory seat unit adopts a multi-degree-of-freedom electric platform and is equipped with a self-rotating ring, which can rotate continuously 360°; the seat controller communicates with the audio and video synchronization module through the bus, and outputs the corresponding posture in real time according to the height change, pitch angle, yaw angle, and roll angle of the waypoints in the route; the seat has a load capacity of 120kg, a maximum pitch / roll rate ≥20° / s, a rotation rate ≥30° / s, a motion response time ≤100ms, and is equipped with a seat belt and an emergency stop button.

[0018] The beneficial effects of the present invention are:

[0019] The present invention realizes 4K real-time image transmission under complex terrain through a hybrid signal transmission link, with a delay of ≤120ms and a frame loss rate of ≤0.2%. It is combined with a "multi-point pan-tilt de-motion algorithm" to present stable and smooth first-person perspective distant and near-view images; the XR headset is combined with an IMU attitude stabilization algorithm to suppress dizziness, and synchronously outputs stereo audio commentary to achieve consistent sound and picture; the multi-axis self-rotating somatosensory seat simulates climbing, descending, yaw, rolling and other postures in real time according to flight data, and provides dynamic feedback such as push-back and weightlessness, so that tourists can obtain the three-in-one viewing effect of "real-time high-definition images + multi-sensory immersive experience + accurate science popularization explanation" without moving. The system's modular design is adaptable to multiple types of scenic spots, with both rapid deployment capabilities and functional scalability. Through physiological feedback adjustment and semantic enhancement technology, it improves the comfort of individual user experience and the efficiency of information acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a system block diagram of the present invention's immersive scenic area viewing and tour guide system based on drones and XR head displays. DETAILED DESCRIPTION

[0021] The following will be combined with the Figure 1 The technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.

[0022] Please refer to Figure 1 , an immersive scenic spot viewing and tour guide system based on drones and XR headsets, including:

[0023] Drone unit: The aircraft flies fully automatically along a pre-set route. The gimbal uses a calculus-based multi-point gimbal easing algorithm to adjust pitch, yaw, and roll angles in real time according to waypoints, simulating smooth manual camera movement. The onboard image acquisition module outputs a 4K / 60fps real-time video stream.

[0024] A binocular vision obstacle avoidance module is added to the front end of the drone, using the YOLOv5 target detection algorithm to identify obstacles (such as trees and power lines), and combined with the depth camera to generate a point cloud map. The obstacle avoidance logic is: when the obstacle distance d is detected <5m, the path replanning algorithm is triggered, and a detour path is generated based on the A* algorithm; the obstacle avoidance priority is: vertical avoidance > horizontal avoidance, ensuring that landscape photography is not affected.

[0025] The core logic of the multi-point pan-tilt easing algorithm is:

[0026] Assume the waypoint sequence is {P i (x i ,y i ,z i )}, calculate the current waypoint P in real time n With the next waypoint P n+1 Space vector

[0027] The calculus interpolation method is used to dynamically adjust the gimbal pitch angle θ and yaw angle φ so that the lens optical axis is always aligned with the target landscape. The formula is: φ(t)=arctan 2(y n+1 -y n ,x n+1 -x n )·k φ where d xy is the horizontal distance, k θ ,k φ is the smoothing coefficient (range 0.1-0.5).

[0028] Signal transmission link: This includes both wireless and wired image transmission links. The wireless link transmits the video stream from the drone to the ground control station. Wireless link: This uses a 2.4GHz / 5.8GHz image transmission module (such as the DJIO4) to transmit video from the drone to the ground control station, with an effective range of ≥1km and a latency of ≤80ms.

[0029] A wired link transmits video from the control station to the XR headset, and an active image conversion module is incorporated into the link to achieve high-definition, low-latency, stable real-time image transmission over distances of 50 meters or more. Wired Link: The control station and XR headset are connected via HDMI 2.0 cable, CAT6 8-core twisted pair cable, or optical fiber, with a transmission distance of 50 meters or more. This link is equipped with an active image conversion and amplification module (supporting 4K 60fps signal amplification and noise reduction).

[0030] The video bit rate is dynamically adjusted according to the real-time packet loss rate of the wireless link. The formula is: When the packet loss rate is greater than 10%, it automatically switches to 720p / 30fps low-definition mode to ensure video smoothness.

[0031] XR headset: Receives real-time image streams via HDMI-IN (or equivalent). Features built-in IMU stabilization and a 50° FOV to mitigate motion sickness. Also includes stereo audio playback for synchronized viewing of scenic spot guides.

[0032] The audio and video synchronization module aligns the audio commentary with the time points of the drone's flight path. When the drone reaches Scenic Spot A (time t1), the module triggers and plays the commentary at t1, achieving audio and video synchronization with an end-to-end latency of ≤120ms. When the drone reaches the trigger point for the scenic spot, a timestamp t1 is generated and transmitted to the ground control station via a wireless link. The control station then triggers the audio playback synchronously, with an end-to-end latency of ≤120ms and an error of ≤±50ms.

[0033] Real-time speech recognition (ASR) is performed on the explanation audio to extract keywords (such as "waterfall" and "height"); corresponding elements in the picture are identified through computer vision algorithms, and when the matching degree is greater than 80%, augmented reality annotation is triggered (such as superimposing height data on the waterfall picture).

[0034] Safety Redundancy Unit: Set an electronic fence with a flight radius of less than 15km and an altitude of less than 1500m. Automatic return to home is triggered after the drone loses contact for 5 seconds. GNSS+RTK high-precision positioning ensures accuracy in high-altitude environments.

[0035] If the drone loses contact for more than 5 seconds, the return route planning will be automatically triggered, giving priority to the shortest route back to the starting point. Emergency Braking: The ground control station is equipped with an emergency stop button that can remotely lock the drone's attitude or force a landing.

[0036] Multi-axis self-rotating somatosensory seat unit: The seat utilizes a multi-degree-of-freedom electric platform (up, down, pitch, roll, and yaw) and features a self-rotating ring for continuous 360° rotation. The seat controller communicates with the audio and video synchronization module via a bus, outputting real-time attitude information based on the altitude, pitch, yaw, and roll angles of pre-programmed waypoints along the route: Climb: Lean back 5–15° and ascend, simulating a push-back sensation; Descent: Lean forward 5–10° and descend, simulating a weightless sensation; Yaw: The seat rotates ±30°; Roll: Tilts left and right ±20°. The seat has a load capacity of 120kg, a maximum pitch / roll rate ≥20° / s, a rotation rate ≥30° / s, and a motion response time ≤100ms. The seat is equipped with a seatbelt and an emergency stop button.

[0037] The following is a clear and complete description of the technical solutions in conjunction with the embodiments of the present invention.

[0038] Example 1

[0039] The immersive scenic area viewing and tour guide system based on drones and XR headsets provided in this embodiment includes a drone unit, a signal transmission link, an XR headset, an audio and video synchronization module, a safety redundancy unit, and a multi-axis self-rotating somatosensory seat unit.

[0040] The drone unit's aircraft flies fully autonomously along a pre-planned route. The gimbal uses a multi-point gimbal jog algorithm to adjust its posture in real time, achieving smooth camera movement. The onboard image acquisition module captures a 4K real-time video stream. A wireless link transmits the video stream from the drone to the ground control station, while a wired link transmits the video stream from the control station to the XR headset. An active image conversion module ensures high-definition, low-latency transmission over long distances.

[0041] The XR headset receives a real-time image stream, uses a built-in algorithm to suppress motion sickness, and simultaneously plays a stereo audio commentary synchronized with the visuals. The audio and video synchronization module precisely aligns the audio commentary with the timing of the drone's flight path, achieving end-to-end audio and video synchronization with a latency of ≤120ms.

[0042] The safety redundancy unit ensures the safety of drone flight through electronic fences, automatic return and dual positioning technology. The multi-axis self-rotating somatosensory seat unit adjusts the seat posture in real time according to the flight posture of the drone, providing tourists with realistic somatosensory feedback.

[0043] Example 2

[0044] Based on Example 1, the system can be further optimized. For example, eye tracking can be added to the XR headset to adjust the drone's camera angle based on the visitor's line of sight, providing a more personalized viewing experience. Multilingual commentary can be added to the audio and video synchronization module to meet the needs of different visitors. The control algorithm of the multi-axis self-rotating somatosensory seat unit can be optimized to improve the accuracy and response speed of seat posture adjustment.

[0045] Based on the disclosure and teachings of the above description, those skilled in the art may also make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and modifications and variations of the present invention should also fall within the scope of protection of the claims of the present invention. In addition, although certain specific terms are used in this description, these terms are only for convenience of description and do not constitute any limitation to the present invention.

Claims

1. An immersive scenic viewing and tour guide system based on drones and XR head-mounted displays, characterized by: It includes a drone unit, a signal transmission link, an XR headset, an audio and video synchronization module, a safety redundancy unit and a multi-axis self-rotating somatosensory seat unit.

2. The immersive scenic area viewing and tour guide system based on drones and XR head displays according to claim 1 is characterized in that: In the drone unit, the aircraft flies fully automatically along a preset route. The gimbal uses a calculus-based "multi-point gimbal slow motion algorithm" to adjust the pitch, yaw, and roll angles in real time according to the waypoints. The onboard image acquisition module outputs a 4K / 60fps real-time video stream. The core logic of the multi-point pan-tilt easing algorithm is: Assume the waypoint sequence is {P i (x i ,y i ,z i )}, calculate the current waypoint P in real time n With the next waypoint P n+1 Space vector The calculus interpolation method is used to dynamically adjust the gimbal pitch angle θ and yaw angle φ so that the lens optical axis is always aligned with the target landscape. The formula is: φ(t)=arctan 2(y n+1 -y n ,x n+1 -x n )·k φ where d xy is the horizontal distance, k θ ,k φ is the smoothing coefficient, ranging from 0.1 to 0.

5.

3. The immersive scenic area viewing and tour guide system based on drones and XR head displays according to claim 1 is characterized in that: The signal transmission link includes a wireless image transmission link and a wired image transmission link. The wireless link completes the video stream transmission from the drone to the ground control station, and the wired link completes the video transmission from the control station to the XR headset. An active image conversion module is set in the link to achieve high-definition, low-latency, stable real-time image transmission of ≥50m; Wireless link: Using 2.4GHz / 5.8GHz image transmission module to achieve video transmission from the drone to the ground control station, with an effective distance of 8km-15km and a delay of ≤80ms; Wired link: Connect the control station and XR headset via HDMI 2.0 cable / CAT6 8-core twisted pair cable / optical fiber, with a transmission distance of ≥ 50m, and equipped with an active image conversion and amplification module.

4. The immersive scenic area viewing and tour guide system based on drones and XR head displays according to claim 1 is characterized in that: The XR headset receives real-time image streams via HDMI-IN, has a built-in IMU attitude stabilization algorithm, a FOV≈50° to suppress dizziness, and has stereo audio playback capabilities.

5. The immersive scenic area viewing and tour guide system based on drones and XR head displays according to claim 1 is characterized in that: The audio and video synchronization module matches the explanation audio with the drone route time point one by one. When the drone flies to scenic spot A, at instant t1, the module is triggered at t1 and plays the explanation of scenic spot A, with an end-to-end delay of ≤120ms.

6. The scenic area immersive viewing and tour guide system based on drones and XR head displays according to claim 1 is characterized in that: The safety redundancy unit sets an electronic fence: flight radius <15km, altitude <1500m; triggers automatic return after the drone loses contact for 5 seconds; and uses GNSS+RTK high-precision positioning.

7. The scenic area immersive viewing and tour guide system based on drones and XR head displays according to claim 1 is characterized in that: The multi-axis self-rotating somatosensory seat unit uses a multi-degree-of-freedom electric platform and a self-rotating ring, which can rotate continuously 360 degrees. The seat controller communicates with the audio and video synchronization module through a bus, and outputs the corresponding attitude in real time according to the altitude change, pitch angle, yaw angle, and roll angle of the waypoints in the route. The seat has a load capacity of 120kg, a maximum pitch / roll rate ≥20° / s, a rotation rate ≥30° / s, a motion response time ≤100ms, and is equipped with a seat belt and an emergency stop button.

Images