Immersive eVTOL interactive experience control system based on time-space synchronization

Abstract

The invention discloses an immersive eVTOL interactive experience control system and method based on time-space synchronization. The system and method are deployed in a ground terminal workstation. The system comprises a data receiving module, a space-time synchronization module, an immersion consistency control module, a multi-channel feedback output module and an immersion quality tuning module. According to the system, eVTOL flight data is received in real time, a time-space synchronous interaction environment is constructed, and immersive interaction between a user and eVTOL is achieved. The immersion consistency control module carries out fusion processing on the interaction feedback information, and improves the continuity and reality of interaction feedback; the multi-channel feedback output module integrates visual, auditory and tactile information output to enhance the immersion experience of the user; and the quality optimization module automatically optimizes interaction parameters based on user feedback, and improves the immersion matching degree. According to the method, the immersion and practicability of eVTOL man-machine interaction are improved, and the method has a good application prospect.

Description

Technical Field

[0001] The present invention relates to virtual reality, flight control and human-computer interaction technologies, and in particular to an immersive eVTOL interactive experience control system based on time-space synchronization, which is applied to scenarios such as remote control, flight monitoring and simulation training of eVTOL aircraft. Background Art

[0002] With the development of electric vertical take-off and landing (eVTOL) vehicles, demand for applications such as remote control, flight monitoring, and simulation training continues to increase. However, existing eVTOL flight control systems face numerous challenges during remote control, primarily manifesting in flight data transmission delays, image synchronization issues, delayed control feedback, and a lack of immersion. These issues seriously impact the control experience and flight safety.

[0003] While some virtual reality (VR)-based flight simulation systems exist, these systems generally lack high-precision spatiotemporal synchronization technology, resulting in delays or misalignments between flight data and feedback information, affecting the operator's true perception of flight status. Furthermore, most existing systems rely on a single feedback channel (such as visual or force feedback), failing to provide a fully immersive experience and limiting their application in eVTOL remote control.

[0004] To address these issues, existing technologies offer some solutions for multimodal feedback and sensor data synchronization. However, these solutions suffer from insufficient synchronization accuracy, delayed feedback response, and low immersion and interactive realism. Therefore, improving the immersion and realism of eVTOL remote control systems has become a pressing technical challenge.

[0005] To address the above issues, the present invention proposes an immersive eVTOL interactive experience control system based on spatiotemporal synchronization. By introducing spatiotemporal synchronization technology, an immersive consistency control module, and a multi-channel feedback mechanism, it effectively solves problems such as data synchronization, feedback lag, and insufficient immersion. It significantly improves the operator's immersive perception and interactive realism, and provides a more efficient, stable, and realistic eVTOL remote control and simulation training solution. By enhancing the authenticity and safety of the immersive eVTOL driving experience and optimizing cost control capabilities, this system lays the foundation for large-scale implementation, thereby accelerating the transition of the low-altitude economy from the experimental stage to commercial application. Summary of the Invention

[0006] This paper addresses the technical bottlenecks of existing eVTOL remote control systems, such as flight data transmission delays, asynchronous multimodal feedback, and insufficient immersive experience authenticity. It proposes an immersive eVTOL interactive experience control system based on spatiotemporal synchronization. By building a multi-level collaborative control architecture, this system optimizes the entire process from data acquisition and state synchronization to immersive feedback, specifically addressing the following technical challenges:

[0007] 1) The problem of misalignment between images and sensor data caused by the lack of unified spatiotemporal benchmarks of multi-source heterogeneous data;

[0008] 2) The problem of fragmented immersive experience caused by the single-channel feedback mechanism;

[0009] 3) Fixed feedback parameters are difficult to adapt to the personalized needs of different users’ perception preferences;

[0010] 4) Mismatch between control instructions and state feedback caused by accumulated delays in remote control.

[0011] The system's core innovation lies in the construction of a four-dimensional collaborative control mechanism: "data acquisition - decision generation - multimodal feedback - experience optimization." This creates a ground control system with high-precision synchronization capabilities and personalized immersive interaction. Its technical architecture comprises multiple key functional modules that are efficiently coupled and operate collaboratively, comprehensively enhancing the immersive experience and control responsiveness during remote control of eVTOL aircraft.

[0012] The intelligent data reception and fusion module uses a multi-threaded parallel processing mechanism to receive and efficiently process multi-dimensional sensor data streams from the eVTOL aircraft in real time. The received data includes flight status information, such as inertial navigation parameters such as three-axis attitude angle, linear acceleration, angular velocity, altitude, and airspeed; environmental perception information, such as lidar point cloud data, millimeter-wave radar ranging signals, and depth images generated by binocular vision; and high-resolution airborne video data, including 4K / 60fps high-definition image sequences captured by forward-looking, downward-looking, and surround-looking cameras. The module uses an adaptive Kalman filter method to process multi-source heterogeneous data, improving data alignment efficiency and feature fusion effects, thereby enhancing the system's real-time responsiveness and accuracy during flight status perception.

[0013] The high-precision spatiotemporal synchronization module innovatively employs a layered spatiotemporal synchronization strategy in its technical implementation. For time synchronization, the system introduces a precision clock protocol to establish a globally unified time base. Combined with a forward prediction compensation algorithm, this effectively reduces temporal jitter generated during network transmission, enabling high-precision time synchronization between video image frames and sensor data. For spatial synchronization, the system constructs a dynamic spatial coordinate system based on the pilot's first-person perspective and employs a quaternion interpolation and fusion algorithm to achieve unified representation of multi-source attitude data in spatial coordinates, improving the consistency and accuracy of the system during spatial synchronization.

[0014] The immersive consistency control engine comprises two levels of intelligent processing modules: a flight event recognition unit and a feedback strategy matching unit. The flight event recognition unit employs a pattern recognition algorithm based on a deep convolutional neural network, enabling real-time analysis and classification of flight state characteristics. It effectively identifies a variety of standard flight maneuvers and abnormal flight states, with high recognition accuracy and response efficiency. The feedback strategy matching unit incorporates a built-in feedback action library containing a large number of preset multimodal interaction templates, each integrating visual, force, and auditory feedback parameters. Visual parameters include field of view angle change curves, lens shake patterns, and heads-up display prompt styles; force parameters include six-degree-of-freedom seat motion trajectories and joystick damping coefficients; and auditory parameters encompass three-dimensional sound positioning methods and voice prompt priority setting strategies. This engine ensures that control commands are precisely matched to feedback content, creating a consistent, immersive interactive experience.

[0015] The multi-channel collaborative feedback system overcomes the limitations of traditional systems that rely on a single feedback channel. It establishes a collaborative coupling mechanism for three feedback channels: visual, force, and auditory, and achieves a high response speed. The visual feedback channel uses a virtual engine to construct a flight cockpit rendering system, supporting high-resolution and high-frame-rate dynamic image rendering and providing an adjustable field of view to adapt to different scenario requirements. The force feedback channel outputs multi-band cockpit vibration feedback through a six-degree-of-freedom electric platform to simulate attitude changes and body response during flight. The auditory feedback channel uses a spatial sound field reconstruction algorithm to support multi-channel sound source localization and enhance the spatial perception of sound and image. The three feedback channels can be driven collaboratively to provide users with a consistent multimodal, low-latency interactive experience.

[0016] The immersive experience quality optimization system adopts a closed-loop control strategy and consists of a QoE evaluation engine and an adaptive regulator. The QoE evaluation engine can monitor multiple immersive experience-related indicators in real time during the interaction process, including image rendering delay, force feedback response lag, and user physiological comfort. Based on these evaluation results, the adaptive regulator uses a fuzzy logic control algorithm to dynamically adjust the output intensity, response rhythm, and priority parameters of each feedback channel to optimize the consistency of multi-channel feedback, improve the system's adaptability, and coordinate with the user's interactive experience.

[0017] The user physiological state monitoring system integrates multimodal biosensors to collect and model users' physiological responses during interaction in real time. The collected data includes cardiovascular indicators, eye movement characteristics, and operational behavior information. Based on this collected physiological and behavioral data, the system constructs a personalized adaptation model and dynamically adjusts feedback intensity and response strategies accordingly, achieving effective adaptation for different user groups and optimizing the immersive interactive experience.

[0018] Compared with existing solutions, this invention achieves technical improvements in synchronization accuracy, immersive feedback, system adaptability, and training efficiency. By introducing a spatiotemporally coupled hierarchical synchronization algorithm, the system significantly reduces the synchronization error range found in traditional immersive feedback control schemes, effectively improving the response speed of the "perception-decision-execution" chain and keeping control latency close to or below the human perception threshold. This, in turn, helps reduce the operator's cognitive burden and improve operational efficiency.

[0019] In terms of immersive experience, this system enhances the user's spatial orientation, posture judgment, and anomaly recognition capabilities through the coordinated stimulation of vision, force perception, and hearing. This significantly improves spatial orientation accuracy, shortens emergency response time, and reduces manipulation error rates. Furthermore, the system utilizes machine learning technology to build user profiles and supports a variety of personalized configurations, including enhanced visual cues and interactive guidance for novices, enhanced force feedback and tactical perception for experts, and a special adaptation mode for users with vestibular sensitivity, thereby enhancing the system's adaptability to humans and its scope of application.

[0020] In both simulation training and practical applications, this system effectively shortens the user's operational learning cycle and flight adaptation time. It improves the efficiency of mastering basic flight skills, enhances the effectiveness of training for handling special conditions, and optimizes the control transition and adaptation process between different eVTOL aircraft models. As a result, this system provides a more efficient, safe, and immersive solution for flight training and remote control in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a block diagram of the overall structure of the eVTOL interactive experience control system provided by an embodiment of the present invention;

[0022] Figure 2 An immersive interactive feedback flow chart provided by an embodiment of the present invention;

[0023] Figure 3 This is a hardware structure diagram of a terminal workstation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] To better illustrate the technical solutions of the present invention, embodiments of the present invention are described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are intended only to illustrate specific applications of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent replacements and technical improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0025] The immersive eVTOL interactive experience control system, based on spatiotemporal synchronization, provided by this invention is deployed at a ground workstation terminal and is widely used in scenarios such as remote control of eVTOL aircraft, flight monitoring, and flight simulation training. By integrating multiple advanced technologies, the system optimizes flight data processing and feedback control, significantly enhancing pilot immersion and control precision.

[0026] In this system, the data receiving module is responsible for receiving multimodal flight data from the eVTOL aircraft in real time, including image and video streams, the aircraft's attitude angle, velocity, acceleration, wind speed, air pressure, temperature and humidity, and data collected by other sensors. This data is integrated into a multi-source data fusion unit, which can perform time alignment and feature-level fusion of various heterogeneous data sources such as inertial navigation systems, radar equipment, and altimeters. Through this efficient data fusion processing, the system can significantly improve the accuracy and robustness of flight status perception, ensuring that pilots receive accurate flight information under all environmental conditions.

[0027] The spatiotemporal synchronization module plays an important role in this system. This module adopts an innovative hierarchical synchronization strategy to ensure that data from different sensors and devices are highly consistent in time and space. In the time dimension, the system builds a global unified time base based on a precise clock protocol, and combines it with a forward prediction compensation algorithm to effectively reduce the time synchronization error caused by network transmission jitter. This module ensures that the time synchronization deviation between the video image frame and the sensor data is within a low-latency range, which helps to reduce the cognitive burden on the pilot during the operation process. In terms of spatial synchronization, the system constructs a dynamic reference coordinate system centered on the pilot's viewpoint and uses quaternion interpolation to achieve spatial consistency fusion of multi-source attitude data. This technology effectively controls the spatial synchronization error of the aircraft's flight state and ensures the accurate spatial alignment of the sensor data.

[0028] The immersive consistency control module is the core component of this system, responsible for generating immersive feedback that matches the flight action in real time based on the flight status data. This module mainly consists of two parts: the flight event recognition unit and the feedback matching unit. The flight event recognition unit uses a deep convolutional neural network to perform real-time analysis of flight status characteristics, and can identify a variety of flight actions including takeoff, climb, turn, landing and sudden braking. Based on the recognition results, the system promptly triggers the feedback matching unit and selects a composite feedback template that adapts to the current flight action from the preset feedback action library. The selected template covers a variety of feedback forms, including visual effects, force feedback and auditory feedback. This multimodal feedback mechanism ensures that pilots have a continuous and consistent immersive experience, enhancing the temporal consistency of flight events and the sense of reality of perception.

[0029] The multi-channel feedback output module is a key component of this system. It transcends traditional single-feedback methods and employs three feedback channels: vision, force, and auditory. The visual feedback channel uses a virtual reality engine to create dynamic first-person flight footage of the aircraft, supporting high-resolution and high-frame-rate rendering and dynamic adjustment of the field of view. The force feedback channel utilizes a six-degree-of-freedom motorized platform to deliver integrated low-frequency vibration and high-frequency tremor feedback, simulating the dynamic effects of acceleration, air resistance, and fuselage vibration perceived by the pilot during flight. The auditory feedback channel utilizes three-dimensional spatial sound reconstruction technology based on head-related transfer functions to accurately simulate the spatial position of multiple sound sources and achieve precise sound and image localization. These feedback channels work together to ensure that pilots are able to perceive changes in flight status through multi-sensory stimulation, creating a realistic and highly immersive interactive experience.

[0030] To further optimize immersion and user experience, an immersive quality tuning module was developed. This module uses a closed-loop adjustment mechanism to monitor and optimize the quality of the immersive experience in real time. Its core components include a QoE evaluation engine and an adaptive regulator. The QoE evaluation engine tracks and analyzes multiple key indicators during the interaction process, such as image rendering latency, force feedback lag, and user physiological comfort. Based on these real-time evaluation results, the adaptive regulator dynamically adjusts the feedback parameters of each channel, including channel weight, timing response, and feedback strength, to ensure that the system can be adaptively tuned to different operating environments and user states. The introduction of this module greatly enhances the system's intelligence, enabling pilots to enjoy the most appropriate immersive feedback experience under different flight missions and operating conditions.

[0031] The system also integrates a user physiological monitoring module to collect information about the pilot's physiological state, such as heart rate, galvanic skin response, eye movement patterns, and facial expressions. This physiological data, when integrated with the QoE assessment engine, can provide pilots with personalized feedback adjustment strategies. Especially during prolonged operations or complex tasks, the system can automatically adjust feedback intensity or reduce certain strong stimuli based on the user's physiological responses, thereby minimizing motion sickness and fatigue and enhancing pilot comfort and safety.

[0032] like Figure 3 As shown, an embodiment of the present invention also provides a complete hardware structure for an immersive ground workstation. This workstation integrates multiple hardware devices, including a control seat that supports attitude linkage, a head-mounted VR terminal, a multi-channel surround sound system, a high-resolution interactive control panel, and multimodal physiological sensor equipment. These components work together through a unified system platform to create a highly immersive and responsive virtual interactive space, providing pilots with an immersive operating environment and a highly realistic flight simulation experience.

[0033] In practical applications, this system can be widely used in remote control, real-time monitoring, and high-fidelity simulation training for eVTOL aircraft. By introducing a unified spatiotemporal reference model, an immersive feedback mapping mechanism, and a dynamic quality of experience tuning system, this system effectively addresses technical challenges such as image latency, action misalignment, and feedback fragmentation that exist in traditional remote control systems, significantly improving control precision, response speed, and user immersion. Whether used for flight control, flight training, or emergency operations, this system provides a highly reliable and immersive interactive experience.

[0034] During the implementation of the present invention, by integrating a variety of advanced technologies, the system effectively solves many problems in the remote control and simulation training of traditional eVTOL aircraft. The high-precision spatiotemporal synchronization, immersive interactive feedback, and intelligent adjustment mechanism provided by the system enable pilots to achieve higher accuracy and a stronger sense of immersion in flight operations. These innovative technologies not only improve the control accuracy in flight missions, but also significantly enhance the user's operating experience and comfort. With the rapid development of the air traffic field, the immersive eVTOL interactive experience control system based on spatiotemporal synchronization will play a key role in future aircraft control, training, and human-computer interaction, and has broad application prospects and far-reaching industrialization potential.

Claims

1. An immersive eVTOL interactive experience control system based on time-space synchronization, deployed at a ground workstation, characterized by: include: The data receiving module is used to obtain multi-modal flight data uploaded by the eVTOL aircraft in real time, and has multi-source data fusion and pre-processing functions; The spatiotemporal synchronization module is used to synchronize the multimodal flight data in a unified time and space manner to generate standardized flight status data. An immersive consistency control module, which is used to identify flight events based on standardized flight status data and generate corresponding immersive interactive feedback instructions; Multi-channel feedback output module, used to drive visual, force, and auditory devices to provide multimodal immersive feedback synchronized with flight events; The immersive quality tuning module is used to dynamically adjust the multi-channel feedback strategy based on the experience quality indicator evaluation results to achieve intelligent optimization of immersion.

2. The system according to claim 1, wherein: The data receiving module integrates a multi-source data fusion unit and a data preprocessing unit, and can perform time alignment and feature fusion on heterogeneous data from inertial measurement units, radars, lidars, altimeters, anemometers, and optical cameras, and filter, denoise, and format standardize the raw flight data.

3. The system according to claim 1, wherein: The space-time synchronization module includes a timestamp calibration unit and an attitude solution unit. The timestamp calibration unit eliminates the transmission delay between different data sources through an interpolation algorithm, and the attitude solution unit constructs a unified spatial reference system with the ground workstation as the origin based on the quaternion algorithm.

4. The system according to claim 1, wherein: The immersive consistency control module includes a flight event recognition unit and a feedback matching unit. The flight event recognition unit identifies flight actions through a preset threshold or a machine learning model, and the feedback matching unit calls a composite feedback solution containing visual effects, force vibration curves and auditory prompts from a pre-built feedback action library.

5. The system according to claim 1, wherein: The multi-channel feedback output module includes a VR visual rendering channel, an action execution channel, and an auditory prompt channel. It can synchronously output first-person perspective three-dimensional scenes, electric seat posture linkage feedback, and spatial sound effects with direction prompt functions, and dynamically adjust feedback parameters according to the urgency of the flight event.

6. The system according to claim 1, wherein: The immersive quality tuning module includes a QoE (Quality of Experience) evaluation engine and a policy regulator, which can evaluate multiple key indicators of immersive experience including picture synchronization, motion feedback delay, and immersive comfort in real time, and dynamically adjust the priority weight, visual rendering parameters and force feedback intensity of each feedback channel.

7. The system according to claim 1, characterized in that Also includes: The user physiological monitoring module is used to collect the operator's heart rate variability data through wrist-worn or seat-integrated sensors, capture pupil movement trajectory through the built-in camera of the head-mounted device, record the force and frequency of joystick operation through pressure sensors, and input physiological signals into the QoE evaluation engine for personalized adjustment of feedback strategies.

8. The system according to claim 1, characterized in that Also includes: The data feedback interface module is used to transmit user behavior data such as operation command response time, feedback discomfort event type and frequency, and physiological signal abnormality records to the eVTOL aircraft main control system to optimize the autonomous control algorithm and human-machine collaboration strategy.

9. The system according to claim 1, characterized in that Also includes: The system is integrated and deployed on an immersive ground workstation that includes a six-degree-of-freedom electric control seat, a 4K resolution VR headset, a 5.1-channel spatial audio system, and a highly sensitive tactile feedback joystick. The various hardware components achieve low-latency data interconnection via Gigabit Ethernet or Fiber Channel.

10. The system according to claim 1, characterized in that Also includes: The composite feedback templates pre-stored in the feedback action library are determined by collecting multimodal data in a real flight environment and through expert scoring or user testing. They include optimized combinations of image jitter amplitude, seat vibration frequency, and warning tone parameters, and are used to enhance the immersive realism of various flight actions.

Images