Mixed reality flight training motion simulation system for virtually constructing scene

Abstract

The invention relates to the technical field of flight simulation training facilities, in particular to a mixed reality flight training motion simulation system for virtually constructing a scene, which comprises a cabin module, a mixed reality imaging module, a six-degree-of-freedom motion platform module and an environment perception and calculation center module, according to the system, through the design of the full-size flight control assembly and the multi-mode sensing unit, a highly real control environment is provided for a pilot. In the real-time training process, the system simulates a real flight fault environment through the fault injection module, randomly sets fault events such as engine failure, modifies kinetic model parameters in real time, and trains the emergency processing capability of a pilot. Meanwhile, the self-adaptive difficulty regulation and control system dynamically adjusts PID gains based on an accumulated value of operation errors of the trainees, and a training environment with higher adaptability is provided.

Description

Technical Field

[0001] The invention relates to the technical field of flight simulation training facilities, and in particular to a mixed reality flight training motion simulation system of a virtual construction scene. Background Art

[0002] As an important tool for pilot training, traditional flight simulators play an indispensable role in the aviation industry. However, with the rapid development of aviation technology and the increasing complexity of flight missions, traditional flight simulators have exposed a series of limitations and technical bottlenecks in meeting the needs of modern flight training. These bottlenecks not only limit the improvement of pilot training effects, but also restrict the development of the aviation industry to a certain extent. In order to solve these problems, it is urgent to develop more advanced flight simulation technology to achieve a higher degree of immersion, more realistic training scenarios, and richer flight operation experience. The following are the specific limitations of traditional technology and the key issues to be solved by this solution.

[0003] First, the lack of visual immersion in traditional flight simulators is an important issue that needs to be addressed urgently. Traditional simulators usually use fixed projection screens or circular screen projections to present virtual scenes. However, this method has significant technical limitations. The field of view of fixed projection screens is limited and lacks full circumferential coverage, which results in pilots being unable to obtain a 360° full-view perception similar to that in a real flight environment during training. Even if some high-end simulators introduce circular screen projections, their longitudinal field of view generally does not exceed ±30°, which cannot truly simulate the pilot's visual experience of observing the horizon or clouds in the air. In addition, the resolution limitations of traditional projection technology and its weak interactivity with the physical cockpit environment also make it difficult for pilots to establish a fully immersive sense of situational substitution visually. This lack of visual immersion directly affects the pilot's ability to recognize flight attitude, instrument position, and external environment, thereby reducing the training effect.

[0004] Secondly, the motion simulation capabilities of traditional motion simulators are limited, and mechanical platforms with limited degrees of freedom cannot accurately reproduce the dynamic behavior under complex flight conditions. Current mainstream flight simulators usually use a motion platform with three degrees of freedom (pitch, roll, and vertical displacement). Its original design intention was to provide basic motion feedback under limited hardware cost and engineering complexity. However, this three-degree-of-freedom design shows obvious inaccuracy when faced with complex flight dynamics scenarios. When the aircraft experiences severe roll-pitch coupling effects, turbulent airflow, or high angle of attack flight, it is difficult for the motion platform to reproduce accurate physical motion feedback, especially under conditions of high acceleration and multi-axis coupled loads. This limitation can cause the pilot's vestibular perception (the balance receptors in the inner ear) to be out of touch with the visual scene generated by the simulator, resulting in motion illusions or motion sickness, further affecting the authenticity and experience of training.

[0005] Third, the training scenarios and emergency simulation capabilities of traditional flight simulators are relatively limited. Usually, the virtual flight scenarios of traditional simulators are generated based on pre-recording or static programming, and their content and logic are highly solidified. Environmental parameters such as terrain, weather, and traffic targets are often fixed and lack the ability to be dynamically adjusted. This not only causes pilots to face repetitive and monotonous training scenarios in multiple training sessions, but also makes it difficult for simulators to adapt to the needs of new flight missions or complex flight environments in a timely manner. In addition, in terms of emergency simulation, traditional systems can often only simulate single fault scenarios through simple parameter triggers (such as engine thrust loss or hydraulic system failure), and cannot present more complex and random multi-fault linkage effects. In real flights, meteorological conditions such as low-altitude wind shear, lightning or thunderstorms may overlap with mechanical failures. This highly dynamic and complex situation is difficult to reproduce in traditional simulators, which directly affects the training effect and psychological preparation of pilots to deal with complex special situations.

[0006] In summary, the above limitations of traditional flight simulators are mainly attributed to the lag of hardware architecture and software design. At the hardware level, traditional simulators mainly rely on mechanical motion platforms and limited visual projection systems, making it difficult to achieve accurate motion feedback and full-scene visual presentation that match the dynamic characteristics of modern aircraft. At the software level, the virtual environment modeling and dynamic control logic of traditional simulators are relatively simple, lacking the ability to simulate real physical environments and complex scenarios in high precision and real time. In addition, the feedback systems of traditional simulators are mostly single (mainly visual feedback), and do not fully integrate multimodal perception technologies such as touch and force to enhance the pilot's immersive experience.

[0007] In response to the above problems, the present invention proposes a flight training simulation device based on mixed reality (MR) technology, a multi-degree-of-freedom motion platform and a high-precision environmental perception system. This solution can not only provide a highly immersive visual experience and dynamic interaction through mixed reality technology, but also accurately reproduce complex flight dynamics behaviors through a high-performance motion platform with six degrees of freedom. At the same time, the present invention has significant innovations in dynamic environment physical modeling and multimodal perception fusion, and can update the consistency of virtual scenes and physical motion in real time, thereby maximizing the restoration of operating experience and environmental perception under real flight conditions. In addition, by introducing improved fault injection and adaptive difficulty control logic, the device can simulate more complex sudden special situations and multi-fault collaborative scenarios, and enhance the pilot's actual combat training capabilities. These technological breakthroughs make the present invention not only far superior to traditional simulators in terms of vision, motion and scene diversity, but also significantly improve training efficiency and reliability, and have important industrial application value. Summary of the invention

[0008] The purpose of the present invention is to provide a mixed reality flight training motion simulation system with a virtual construction scene, which provides a real, immersive, flexible and efficient flight training platform for pilots through high-precision force feedback, mixed reality imaging, efficient computing and closed-loop control, dynamic scene generation and personalized training environment. It provides comprehensive support for the emergency response ability and control skills of pilots, greatly improving the efficiency and effect of flight training.

[0009] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0010] A mixed reality flight training motion simulation system for a virtual construction scene, comprising:

[0011] Cockpit module, mixed reality imaging module, six-degree-of-freedom motion platform module and environmental perception and computing center module;

[0012] The cockpit module includes: full-size flight control components, including force feedback side sticks, rudder pedals and throttle consoles. The force feedback side sticks capture the displacement of the stick at a sampling rate of 1kHz through embedded strain gauges, and provide real-time feedback of the control resistance through a PID control algorithm;

[0013] The multimodal perception unit includes an eye-tracking camera, an electromyography sensor array and a tactile feedback seat. The eye-tracking camera uses infrared capture technology to accurately capture the trainee's gaze position, the electromyography sensor array detects the trainee's muscle activity signals, and the tactile feedback seat uses a vibration unit matrix to simulate the physical sensation caused by real overload during flight.

[0014] The mixed reality imaging module includes:

[0015] The optical system consists of a binocular penetrating MR headset, equipped with a 4K high-definition display, with a field of view of ≥120°. The IMU of the headset is precisely calibrated with the infrared positioning base station, and the positioning accuracy is ≤0.1mm@5m;

[0016] The projection subsystem consists of a ring-screen laser projection array, covering a panoramic field of view of 360° horizontally and ±60° vertically. The projection image is dynamically matched with the head display image through the SLAM algorithm;

[0017] The rendering engine uses ray tracing technology to achieve highly realistic visual effects and integrates multimodal perception data (such as eye tracking, electromyography signals, and tactile feedback) to optimize rendering effects.

[0018] The six-degree-of-freedom motion platform module adopts the Stewart platform design and is controlled by a hydraulic driver to achieve angular motion of pitch ±30° and roll ±30°, as well as vertical displacement of ±0.5m on the Z axis. The maximum acceleration of the platform is ≥2g, and the step response time is less than 80ms.

[0019] The environment perception and computing center module adopts FPGA and GPU heterogeneous computing design. FPGA handles the preprocessing of high-frequency sensor data, and GPU performs complex simulation and real-time rendering operations of MR scenes. The computing delay is less than 5ms.

[0020] Furthermore, the central controller performs closed-loop control through a 10ms sampling period to ensure strict time synchronization between motion signals and rendered images. Relying on the IEEE 1588 precise time synchronization protocol, multi-level calibration of the cockpit, motion platform and MR display coordinate system is performed with an error of less than 0.05°. The visual depth information and tactile seat signals are comprehensively processed through the Kalman filter algorithm.

[0021] Beneficial effects of the present invention:

[0022] The system of the present invention provides a highly realistic control environment for pilots through the design of full-size flight control components and multimodal perception units. Specifically, the force feedback side stick accurately captures the rod displacement at a sampling rate of 1kHz through embedded strain gauges, and adjusts the resistance of the side stick in real time through the PID control algorithm, so that the pilot can feel the control force consistent with the actual flight. This high-precision force feedback mechanism enhances the realism of training; in addition, the multimodal perception unit achieves a comprehensive perception of the pilot's attention, muscle activity and body state through the combination of eye tracking, electromyography sensors and tactile feedback seats. The eye tracking camera uses infrared capture technology to accurately obtain the pilot's gaze point position, combined with time series analysis to evaluate its attention distribution; the electromyography sensor array extracts the characteristic parameters of the electromyography signal through Fourier transform to analyze the pilot's operating intention; the tactile feedback seat simulates the overload feeling in flight through the vibration unit matrix, which enhances the pilot's immersion. This fusion of multimodal feedback makes the training not limited to visual and auditory simulation, but covers the pilot's whole body perception, further improving the comprehensiveness and immersion of the training.

[0023] The system of the present invention realizes the dynamic scene simulation of virtual and real combination through the collaborative work of mixed reality imaging module and six-degree-of-freedom motion platform. The mixed reality imaging module adopts binocular penetrating MR head display, equipped with 4K high-definition display and wide field of view, providing pilots with comprehensive visual immersion. In the optical system, the IMU of the head display and the laser positioning base station are precisely calibrated, combined with the SLAM algorithm to realize the dynamic registration of the virtual scene and the real environment, ensuring the consistency of the virtual and real space. The ring screen laser projection array covers 360° horizontal field of view and ±60° vertical field of view, and uses high-resolution DLP chip and ray tracing technology to generate realistic light and shadow effects, further enhancing the visual reality. The six-degree-of-freedom motion platform is designed through the Stewart platform, combined with hydraulic drive and high-rigidity aluminum alloy structure, to achieve a vertical displacement of ±30° in pitch, ±30° in roll and ±0.5m in Z axis. The maximum acceleration of the platform reaches ≥2g, and the step response time is less than 80ms, which can simulate the real flight dynamic overload experience. This high-precision motion simulation allows pilots to feel real flight movements during the operation process, thereby improving the effectiveness and realism of training.

[0024] The system of the present invention realizes precise and coordinated dynamic control through the efficient computing power of the environmental perception and computing center and the design of a closed-loop control chain. The environmental perception and computing center adopts a combination design of FPGA and GPU heterogeneous computing units. The FPGA is used to process the preprocessing of high-frequency sensor data, and the GPU performs complex simulation real-time rendering, with a computing delay of less than 5ms. This efficient computing power ensures the real-time performance of the system when processing large amounts of data, and provides strong support for the generation and rendering of dynamic scenes. In addition, the system designs a closed-loop control chain between sensor data, central control and motion platform servo valves and MR rendering engines. The central controller performs closed-loop control through a 10ms sampling period to ensure strict time synchronization between motion signals and rendering images. The spatial registration protocol relies on the IEEE 1588 precise time synchronization protocol to perform multi-level calibration of the cockpit, motion platform and MR display coordinate system, with an error of less than 0.05°. The Kalman filter algorithm is used for the comprehensive processing of visual depth information and tactile seat signals, further improving the stability and consistency of the feedback signal. This precise closed-loop control and efficient computing power enable the system to respond to the pilot's operational input in real time and provide feedback that is highly synchronized with the virtual environment, ensuring the continuity and authenticity of the training.

[0025] The system of the present invention provides flexible and efficient training support for pilots through dynamic scene generation and the design of personalized training environment. In the initialization stage, the system uses the dynamic environment description file in XML format to generate a virtual airspace, including complex elements such as weather, terrain and traffic targets, to provide pilots with a variety of training scenarios. The rigid body transformation matrix of the cockpit and MR display coordinate system is accurately established by the checkerboard calibration method to ensure the spatial consistency of the system. In the real-time training process, the system simulates the real flight fault environment through the fault injection module, randomly sets fault events such as engine failure, modifies the dynamic model parameters in real time, and trains the emergency handling ability of pilots. At the same time, the adaptive difficulty control system dynamically adjusts the PID gain based on the cumulative value of the student's operation error to provide a more adaptable training environment. This dynamic scene generation and personalized training design not only improves the flexibility and diversity of training, but also enables the training content to be adjusted in real time according to the specific performance of the students, further improving the efficiency and effect of training.

[0026] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0028] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0029] Figure 2 It is a schematic diagram of the work flow structure of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Example 1

[0032] The mixed reality flight training motion simulation system of a virtual construction scene described in this embodiment includes:

[0033] The cockpit module consists of a full-size flight control assembly and a multimodal sensing unit. The force feedback side rod in the full-size flight control assembly accurately captures the rod displacement and provides real-time feedback on the control resistance through a strain gauge at a sampling rate of 1kHz. The rudder pedal and throttle console monitor and feedback various flight control commands in real time through sensors; the eye tracking camera in the multimodal sensing unit can accurately capture the trainee's gaze position and provide real-time attention monitoring and analysis through a high-precision infrared capture device. The electromyography sensor array is used to detect the trainee's muscle activity signals in order to better understand and evaluate the control intention. The tactile feedback seat can simulate the various physical sensations caused by real overload during flight through the built-in vibration unit matrix, greatly enhancing the sense of immersion.

[0034] The mixed reality imaging module consists of an optical system and a projection subsystem. In the optical system, the binocular penetrating MR headset is equipped with a 4K high-definition display and has a wide field of view of ≥120°, ensuring the full visual immersion of the trainees. The IMU of the headset can be precisely calibrated with the infrared positioning base station, which uses laser texture capture technology to achieve a positioning accuracy of ≤0.1mm@5m, thereby ensuring the precise consistency of virtual and real space. The ring screen laser projection array covers a panoramic field of view of 360° horizontally and ±60° vertically. Using a high-resolution DLP chip, the projected image is perfectly matched with the headset image through the SLAM algorithm to ensure the smoothness and coherence of the image during dynamic scene transitions.

[0035] The six-degree-of-freedom motion platform adopts the Stewart platform design. The platform is controlled by a hydraulic drive to achieve angular motion of ±30° in pitch and ±30° in roll, as well as a vertical displacement of ±0.5m on the Z axis. Its high-rigidity aluminum alloy structure and high-pressure hydraulic cylinder design ensure excellent load capacity and dynamic response. The platform has excellent dynamic performance, can provide a maximum acceleration of ≥2g, and its step response time is less than 80ms, effectively simulating the real dynamic overload experience in flight.

[0036] The environmental perception and computing center provides efficient computing support for the entire system. The combined design of FPGA and GPU heterogeneous computing units is adopted. FPGA is used to process the preprocessing of high-frequency sensor data, while GPU performs real-time rendering operations for complex simulations and MR scenes, providing a computing delay of less than 5ms. In addition, the synchronous processing of various sensor data is achieved through the PCIe 4.0 data bus, ensuring that the system has extremely high real-time performance when performing dynamic closed-loop control.

[0037] In the connection logic between modules, the system has designed a closed-loop control chain that connects sensor data, central control, motion platform servo valve and MR rendering engine. The central controller performs closed-loop control through a 10ms sampling period to ensure strict time synchronization between motion signals and rendering images. The spatial registration protocol is implemented based on the IEEE 1588 precise time synchronization protocol, and the multi-level calibration of the cockpit, motion platform and MR display coordinate system has an accuracy of less than 0.05°. In multimodal feedback fusion, the Kalman filter algorithm is used for the comprehensive processing of visual depth information and tactile seat signals to ensure the temporal and spatial consistency of the feedback signal with the virtual environment.

[0038] During the initialization phase, the system generates a virtual airspace using a dynamic environment description file in XML format, including complex elements such as weather, terrain, and traffic targets. During initialization, the rigid body transformation matrix of the cockpit and MR display coordinate system is accurately established through the checkerboard calibration method. The preloading of dynamic parameters is set according to the aerodynamic derivatives of the target training model, providing a preset for the transfer function of the motion platform.

[0039] During real-time training, the force feedback rod and rudder foot of the cockpit module are responsible for capturing the trainee's control input and converting it into instructions that can be solved by the system. The central controller solves the instantaneous motion state of the aircraft based on the complete six-degree-of-freedom rigid body dynamics formula to achieve accurate dynamic simulation. In the fusion of virtual and real scenes, the MR headset and the ring screen projection dynamically adjust the depth alignment of the virtual and physical scenes according to the platform's real-time posture data to achieve a coherent and natural visual presentation. In multi-channel feedback, the tactile seat generates a specific vibration waveform based on the overload coefficient, and the force feedback rod provides a resistance effect consistent with the aerodynamic load of the virtual rudder surface.

[0040] The system also implements special situation simulation through the fault injection module, randomly sets fault events such as engine failure, and modifies the dynamic model parameters in real time. The design of adaptive difficulty control is based on the cumulative value of the trainee's operation error, and dynamically adjusts the system's PID gain to provide a more adaptable training environment. Overall, the present invention provides an unprecedented virtual reality training platform for pilots through highly integrated design and precise control.

[0041] Example 2

[0042] In the mixed reality flight training motion simulation system of a virtual construction scene described in this embodiment, the cockpit module includes a full-size flight control component and a multi-modal perception unit;

[0043] The full-scale flight control assembly includes a force feedback side stick, rudder pedals, and a throttle platform. The force feedback side stick captures the stick displacement through a strain gauge sensor at a sampling rate of 1kHz. The resistance change of the strain gauge is proportional to the strain. The resistance change ΔR can be expressed as ΔR = k·∈·R0, where k is the sensitivity coefficient of the strain gauge, ∈ is the strain, and R0 is the initial resistance. The tiny resistance change is converted into a voltage signal through a Wheatstone bridge circuit. The real-time feedback mechanism adjusts the resistance of the side stick through a PID control algorithm. The control law is Where e(t)=Ft-Fa is the error, Kp, Ki, Kd are the proportional, integral and differential gains respectively.

[0044] The rudder pedal and throttle console are equipped with angle sensors and position sensors to monitor the control input in real time. The angle sensor uses a rotary encoder, and its output θ is linearly related to the actual angle, expressed as Where N is the number of pulses of the encoder, and Nmax is the maximum number of pulses. The linear mapping of the sensor signal S to the command value C is C = m·S + b, where m and b are mapping coefficients that need to be determined through calibration.

[0045] The multimodal perception unit includes eye tracking, myoelectric sensor array and tactile feedback chair. Eye tracking captures the trainee's gaze point through infrared light source and camera array. Through geometric optics method, the gaze point position (x, y) can be calculated by formula y c Calculate, where (x p ,y p ) is the pupil center, (x c ,y c ) is the corneal reflection point, f is the focal length, and z is the distance from the eye to the screen. Attention analysis calculates the dwell time T and frequency F of the fixation point through time series analysis, which is expressed as Where ti is the duration of each fixation, n is the number of fixations, T total For total time.

[0046] The myoelectric sensor array detects the muscle activity signals of the trainees. The characteristic parameters of the myoelectric signal, such as amplitude and frequency, are extracted through amplification and filtering circuits. The amplitude A and frequency f0 of the myoelectric signal are extracted through Fourier transform, expressed as A=max(|F(s(t))|), f=argmax(|F(s(t))|), where F(s(t)) is the Fourier transform of the signal. Using the support vector machine (SVM) classifier, the feature vector x is input for classification, and the classification function is where α i is the Lagrange multiplier, y i is the category label, K(x i ,x) is the kernel function and b is the bias.

[0047] The tactile feedback seat has a built-in vibration unit matrix that controls the vibration intensity and frequency through PWM signals to simulate the overload feeling during flight. The vibration frequency f and intensity A are calculated through the overload coefficient G, expressed as f=k1·G, A=k2·G, where k1 and k2 are proportional coefficients, calibrated through experiments.

[0048] The cockpit module includes data acquisition, signal processing, command generation, feedback control, attention and intention analysis, and feedback fusion. Sensor data is acquired through high-frequency sampling, and the data is filtered (such as Kalman filtering) and feature extracted. The processed signal is converted into system instructions through a mapping function, and the force feedback and tactile feedback are adjusted using a closed-loop control algorithm. Through eye movement and electromyography data, the trainee's attention and control intention are analyzed, and multi-modal feedback is integrated to ensure the synchronization of the virtual environment and physical control.

[0049] Example 3

[0050] The present embodiment describes a mixed reality flight training motion simulation system for a virtual constructed scene, wherein the mixed reality imaging module includes an optical system, a projection subsystem, and a rendering engine.

[0051] 1. Optical system

[0052] Binocular see-through MR headset

[0053] IMU attitude update

[0054] The IMU (Inertial Measurement Unit) built into the headset updates the user's posture in real time by detecting acceleration and angular velocity. The IMU collects data hundreds of times per second to ensure the real-time and accuracy of posture updates. The posture data is represented in the form of quaternions, and drift is eliminated through optimization algorithms to ensure stability and reliability.

[0055] Laser positioning base station calibration

[0056] The laser positioning base station achieves high-precision positioning by emitting laser textures to capture feature points in the environment. The error of laser positioning is controlled at the millimeter level to ensure that the position data of the headset is accurate. The calibration process uses multi-sensor fusion technology to combine IMU data and laser positioning results to further optimize positioning accuracy.

[0057] 2. Projection subsystem

[0058] Ring screen laser projection array

[0059] SLAM Algorithm

[0060] The SLAM (Simultaneous Localization and Mapping) algorithm achieves dynamic registration of virtual scenes and real environments through real-time positioning and environmental modeling. The core of the SLAM algorithm lies in the extraction and matching of feature points. The SIFT (Scale Invariant Feature Transform) algorithm is used to detect feature points in the projection image, and accurate pose estimation is achieved through optimization algorithms.

[0061] SIFT feature point detection

[0062] The SIFT algorithm extracts feature points of an image through a multi-scale Gaussian pyramid to ensure the stability and repeatability of features at different scales. The description of feature points is implemented through edge response functions to ensure that high matching accuracy can be maintained under changes in lighting and perspective.

[0063] Dynamic scene transitions

[0064] The smoothness of dynamic scenes depends on the fusion of multi-sensor data. The projection system achieves smooth scene transitions by updating the pose data of the virtual scene in real time and combining it with the posture changes of the head display. The update of dynamic scenes uses an interpolation algorithm to ensure the continuity and smoothness of visual presentation.

[0065] 3. Rendering Engine

[0066] Ray Tracing

[0067] The rendering engine achieves highly realistic visual effects through ray tracing technology. The ray tracing algorithm simulates the propagation path of light in the scene, calculates the intersection of light and objects, as well as physical phenomena such as reflection and refraction, and generates realistic light and shadow effects.

[0068] Multimodal perception fusion

[0069] The rendering engine achieves personalized visual presentation by fusing multimodal perception data (such as eye tracking, electromyography, and tactile feedback). The fusion process of multimodal data ensures the synchronization of virtual scenes and user perception through weight distribution and feature vector matching.

[0070] In this embodiment, the posture and position data of the head display are collected through the IMU and the laser positioning base station; the SIFT algorithm is used to extract the feature points of the projection image to achieve feature matching between the real environment and the virtual scene.

[0071] In this embodiment, the posture data of the head display is updated in real time through the SLAM algorithm to achieve dynamic alignment of virtual and real scenes; the positioning accuracy and stability of the SLAM algorithm are optimized by combining the IMU and laser positioning results.

[0072] In this embodiment, the virtual scene is rendered in real time based on the updated posture data, and a highly realistic visual effect is achieved through ray tracing technology; the rendering effect is optimized through multimodal perception data to ensure the personalization and synchronization of the visual presentation.

[0073] In this embodiment, an interpolation algorithm is used to achieve a smooth transition of the projected image, thereby ensuring the continuity of the user's visual presentation during movement; the posture data of the virtual scene is updated in real time, and the dynamic adjustment of the scene is achieved by combining multi-sensor data.

[0074] Example 3

[0075] The mixed reality flight training motion simulation system of a virtual construction scene described in this embodiment includes:

[0076] The full-size flight control assembly includes force feedback side sticks, rudder pedals and throttle. The force feedback side stick uses embedded strain gauges to accurately capture the displacement of the stick at a sampling rate of 1kHz, and provides real-time feedback of the control resistance. The force feedback side stick ensures that the pilot can feel the real aircraft control force, thereby improving the realism of the training. The rudder pedals and throttle console monitor the pilot's control input in real time through high-precision sensors, and convert these commands into instructions that the system can resolve to ensure the accuracy of the flight simulation.

[0077] The multimodal perception unit includes an eye-tracking camera, an electromyographic sensor array, and a tactile feedback seat. The eye-tracking camera uses high-precision infrared capture technology to accurately capture the pilot's gaze point and provide real-time attention monitoring analysis. The electromyographic sensor array is used to detect the pilot's muscle activity signals, which helps to evaluate the pilot's operational intentions and physical state. The tactile feedback seat simulates the physical sensations caused by real overload during flight through a built-in vibration unit matrix, enhancing the pilot's sense of immersion.

[0078] The optical system uses a binocular penetrating MR headset, equipped with a 4K high-definition display, with a wide field of view of ≥120°, ensuring the pilot's full visual immersion. The IMU of the headset is precisely calibrated with the infrared positioning base station, and the laser texture capture technology is used. The positioning accuracy reaches ≤0.1mm@5m, ensuring the precise consistency of the virtual and real space. It ensures that when the pilot uses the MR headset, the virtual environment and the actual environment can be seamlessly combined to provide a real and coherent visual experience.

[0079] The projection subsystem uses a circular screen laser projection array, covering a panoramic field of view of 360° horizontally and ±60° vertically. Using a high-resolution DLP chip, the projected image is perfectly matched with the head display image through the SLAM algorithm, ensuring smooth and coherent images during dynamic scene transitions. It not only provides a wide field of view, but also adjusts the projected image in real time when the pilot moves, ensuring visual consistency and smoothness.

[0080] The six-degree-of-freedom motion platform adopts the Stewart platform, which is controlled by a hydraulic drive to achieve angular motion of ±30° in pitch and ±30° in roll, as well as a vertical displacement of ±0.5m on the Z axis. The platform adopts a high-rigidity aluminum alloy structure and a high-pressure hydraulic cylinder to ensure excellent load capacity and dynamic response. The maximum acceleration of the platform reaches ≥2g, and the step response time is less than 80ms, which can effectively simulate the real dynamic overload experience in flight. It allows pilots to feel the real flight movement during training, improving the effectiveness and realism of training.

[0081] The environmental perception and computing center provides efficient computing support for the entire system, and adopts a combination design of FPGA and GPU heterogeneous computing units. FPGA is used to process the preprocessing of high-frequency sensor data, ensuring the efficiency and real-time performance of data processing. The GPU performs real-time rendering operations for complex simulations and MR scenes, with a computing delay of less than 5ms, ensuring the smoothness of the visual experience. In addition, the PCIe 4.0 data bus is used to achieve synchronous processing of various sensor data, ensuring that the system has extremely high real-time performance when performing dynamic closed-loop control. This allows the entire system to maintain efficient computing capabilities when processing large amounts of data, ensuring the continuity and realism of training.

[0082] The system is equipped with a closed-loop control chain between sensor data, central control, motion platform servo valve and MR rendering engine. The central controller performs closed-loop control through a 10ms sampling period to ensure strict time synchronization between motion signals and rendering images. The spatial registration protocol is implemented based on the IEEE 1588 precise time synchronization protocol, and multi-level calibration is performed on the cockpit, motion platform and MR display coordinate system with an error accuracy of less than 0.05°. In multimodal feedback fusion, the Kalman filter algorithm is used for the comprehensive processing of visual depth information and tactile seat signals to ensure the temporal and spatial consistency of the feedback signal with the virtual environment. The collaborative work between the modules is ensured, and the overall performance of the system and the realism of the training are guaranteed.

[0083] During the initialization phase, the system uses a dynamic environment description file in XML format to generate a virtual airspace, including complex elements such as weather, terrain, and traffic targets. The rigid body transformation matrix of the cockpit and MR display coordinate system is accurately established through the checkerboard calibration method to ensure the spatial consistency of the system. The preload of dynamic parameters is set according to the aerodynamic derivatives of the target training model to provide a preset for the transfer function of the motion platform. This allows the system to quickly enter the training state and provide a personalized training environment.

[0084] During real-time training, the force feedback rod and rudder foot of the cockpit module are responsible for capturing the pilot's operational input and converting it into instructions that can be solved by the system. The central controller solves the instantaneous motion state of the aircraft based on the complete six-degree-of-freedom rigid body dynamics formula to achieve accurate dynamic simulation. In the fusion of virtual and real scenes, the MR headset and the ring screen projection dynamically adjust the depth alignment of the virtual and physical scenes according to the platform's real-time posture data to achieve a coherent and natural visual presentation. In multi-channel feedback, the tactile feedback seat generates a specific vibration waveform based on the overload coefficient, while the force feedback rod provides a resistance effect consistent with the aerodynamic load of the virtual rudder surface, enhancing the realism and immersion of the training.

[0085] The system also uses the fault injection module to simulate special situations, randomly set fault events such as engine failure, and modify the dynamic model parameters in real time to simulate the real flight fault environment. The design of adaptive difficulty control is based on the cumulative value of the student's operation error, and dynamically adjusts the system's PID gain to provide a more adaptable training environment. This makes the training more challenging and targeted, and can effectively improve the pilot's ability to respond.

[0086] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A mixed reality flight training motion simulation system with a virtual construction scene, characterized in that: include: Cockpit module, mixed reality imaging module, six-degree-of-freedom motion platform module and environmental perception and computing center module; The cockpit module includes: full-size flight control components, including force feedback side sticks, rudder pedals and throttle consoles. The force feedback side sticks capture the displacement of the stick at a sampling rate of 1kHz through embedded strain gauges, and provide real-time feedback of the control resistance through a PID control algorithm; Multimodal perception unit, including eye-tracking camera, myoelectric sensor array and tactile feedback seat. The eye-tracking camera captures the trainee's gaze position through infrared, the myoelectric sensor array detects the trainee's muscle activity signals, and the tactile feedback seat simulates the physical feeling caused by real overload during flight through a vibration unit matrix; The mixed reality imaging module includes an optical system, a projection subsystem, and a rendering engine; The six-degree-of-freedom motion platform module adopts the Stewart platform design and is controlled by a hydraulic drive to achieve angular motion of ±30° in pitch and ±30° in roll, as well as a vertical displacement of ±0.5m on the Z axis. The maximum acceleration of the platform is ≥2g, and the step response time is less than 80ms. The environment perception and computing center module adopts FPGA and GPU heterogeneous computing design. FPGA handles the preprocessing of high-frequency sensor data, and GPU performs complex simulation and real-time rendering operations of MR scenes. The computing delay is less than 5ms.

2. The mixed reality flight training motion simulation system of the virtual construction scene as claimed in claim 1, characterized in that: The central controller performs closed-loop control through a 10ms sampling period to ensure strict time synchronization between motion signals and rendered images. Relying on the IEEE 1588 precise time synchronization protocol, the cockpit, motion platform and MR display coordinate system are calibrated at multiple levels with an error of less than 0.05°. The visual depth information and tactile seat signals are comprehensively processed through the Kalman filter algorithm.

3. The mixed reality flight training motion simulation system of the virtual construction scene as claimed in claim 1, characterized in that: The cockpit module includes a full-scale flight control assembly and a multi-modal perception unit; The full-scale flight control assembly includes a force feedback side stick, rudder pedals, and a throttle platform. The force feedback side stick captures the stick displacement through a strain gauge sensor at a sampling rate of 1kHz. The resistance change of the strain gauge is proportional to the strain, and the resistance change ΔR is expressed as ΔR=k·∈·R0, where k is the sensitivity coefficient of the strain gauge, ∈ is the strain, and R0 is the initial resistance. The tiny resistance change is converted into a voltage signal through a Wheatstone bridge circuit. The real-time feedback mechanism adjusts the resistance of the side stick through a PID control algorithm, and the control law is: Where e(t) = Ft-Fa is the error, Kp, Ki, Kd are proportional, integral and differential gains respectively; The rudder pedal and throttle console are equipped with angle sensors and position sensors to monitor the control input in real time; the angle sensor uses a rotary encoder, and its output θ is linearly related to the actual angle, expressed as Where N is the number of pulses of the encoder, and Nmax is the maximum number of pulses; the linear mapping of the sensor signal S to the command value C is C = m·S+b, where m and b are mapping coefficients, which are determined by calibration; The multimodal perception unit includes eye tracking, electromyographic sensor array and tactile feedback chair; eye tracking captures the trainee's gaze point through infrared light source and camera array; through geometric optics method, the gaze point position (x, y) is calculated by formula Calculate, where (x p ,y p ) is the pupil center, (x c ,y c ) is the corneal reflection point, f is the focal length, and z is the distance from the eye to the screen; attention analysis calculates the dwell time T and frequency F of the fixation point through time series analysis, which is expressed as Where ti is the duration of each fixation, n is the number of fixations, T total is the total time; The electromyographic sensor array detects the muscle activity signals of the trainees; the characteristic parameters of the electromyographic signal, including amplitude and frequency, are extracted through amplification and filtering circuits; the amplitude A and frequency f0 of the electromyographic signal are extracted through Fourier transform, expressed as A=max(|F(s(t))|), f=argmax(|F(s(t))|), where F(s(t)) is the Fourier transform of the signal; the feature vector x is input for classification using the support vector machine (SVM) classifier, and the classification function is where α i is the Lagrange multiplier, y i is the category label, K(x i ,x) is the kernel function, b is the bias; The tactile feedback seat has a built-in vibration unit matrix that controls the vibration intensity and frequency through PWM signals to simulate the overload feeling during flight; the vibration frequency f and intensity A are calculated through the overload coefficient G, expressed as f=k1·G, A=k2·G, where k1 and k2 are proportional coefficients calibrated through experiments.

Images