Flight training system

Abstract

The invention relates to the technical field of flight training, in particular to a flight training system. The flight joystick receives operation action data input by a target user based on a preset training instruction; the button control table receives button action data input by a target user based on a preset training instruction; the electronic equipment receives parameter setting data which is input by a target user based on a preset training instruction and is used for setting flight parameters; the multi-axis gravity seat performs six-degree-of-freedom movement based on the operation action data, the button action data and the parameter setting data so as to simulate the flight attitude of the aircraft; and the electronic equipment generates a target virtual driving scene based on the operation action data, the button action data and the parameter setting data, and controls the dynamic scene display equipment to display the target virtual driving scene. The target virtual driving scene is combined with the actual operation process. And the training effect is improved in the flight training process.

Description

Technical Field

[0001] The present invention relates to the technical field of flight training, and particularly to a flight training system. Background Art

[0002] Flight training, as the core professional training for aviation troops, plays a crucial role in improving the technical and tactical levels of flight personnel in flying aircraft and operating on-board equipment and weapons. Its scope covers pilot flight training, as well as the training of professional personnel such as air navigators, communicators, and gunners, and also includes the coordinated training among aircrew members.

[0003] Traditional flight training methods usually conduct flight training in a training simulator based on fixed driving scenarios, thus lacking the combination of driving scenarios and actual operations and being unable to provide feedback on the training actions of users, resulting in poor training effects.

[0004] Therefore, how to improve the training effect during flight training has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a flight training system to solve the problem of how to improve the training effect during flight training.

[0006] In a first aspect, the present invention provides a flight training system, which includes a multi-axis gravity seat, a flight control stick, a button control console, a parameter setting interface, a dynamic scene display device, and an electronic device; wherein, the multi-axis gravity seat, the flight control stick, and the dynamic scene display device are all communicatively connected to the electronic device; wherein: The flight control stick is used to receive the operation action data input by the target user based on a preset training instruction, and the operation action data includes the execution sequence corresponding to each operation action and the operation force corresponding to each operation action; The button control console is used to receive the button action data input by the target user based on a preset training instruction; The electronic device is used to receive the parameter setting data input by the target user for setting flight parameters based on a preset training instruction; The multi-axis gravity seat is used to perform six-degree-of-freedom movement based on the operation action data, the button action data, and the parameter setting data, so as to simulate the flight attitude of an aircraft; The electronic device is further used to generate a target virtual driving scene based on the operation action data, the button action data, and the parameter setting data, and control the dynamic scene display device to display the target virtual driving scene.

[0007] The flight training system provided by the embodiment of the present application, the flight joystick receives the operation action data input by the target user based on the preset training instructions. The button control console receives the button action data input by the target user based on the preset training instructions. So that the flight training system can delicately capture the subtle differences in the operations of the target user on flight attitude adjustment, power control, etc., and provide accurate data support for subsequent simulations. The button action data received by the button control console records the operations of the user on various system switches and function switches of the aircraft. The parameter setting data received by the electronic device includes parameter settings such as flight altitude, speed, and heading. The comprehensive collection of these multi-dimensional data completely restores the operation process of the user in flight training, enables the system to accurately understand the user's operation intention, and lays a solid foundation for simulating flight attitudes and generating virtual scenarios. The multi-axis gravity seat is used to perform six-degree-of-freedom motion based on the operation action data, button action data, and parameter setting data, so as to simulate the flight attitude of the aircraft. Let the target user truly feel the change of flight attitude from the physical perception. This highly realistic physical simulation breaks the limitations of traditional two-dimensional simulation training, makes the target user feel as if they are in a real cockpit, greatly enhances the immersion of the training, helps the target user to quickly get familiar with the physical feelings during flight, and improves the ability to judge and control flight attitudes. The electronic device generates a target virtual driving scene based on the operation action data, button action data, and parameter setting data, and controls the dynamic scene display device to display the target virtual driving scene. It ensures the accuracy of the generated target virtual driving scene and realizes the combination of the target virtual driving scene and the actual operation process. Each component of the above flight training system realizes data interaction and collaborative work by communicating with the electronic device. This collaborative mechanism makes the training process smoother and more realistic, and every operation of the target user can get timely and accurate feedback, promoting the target user to quickly learn and master flight skills, thus improving the training effect during the flight training process.

[0008] In an alternative embodiment, the flight training system further includes an air flow control system, and the air flow control system includes a plurality of air flow valves and air flow pressure sensors; the air flow control system is communicatively connected to the electronic device, wherein: The electronic device is configured to determine the external environment data corresponding to the preset training instruction based on the preset training instruction; and determine the target air flow direction and target air flow speed corresponding to the aircraft based on the external environment data and the flight attitude; The air flow control system is configured to detect the current air flow direction and current air flow speed based on each air flow pressure sensor; and control the opening or closing of each air flow valve according to the differences between the target air flow direction and target air flow speed and the current air flow direction and current air flow speed.

[0009] In an alternative embodiment, the multi-axis gravity seat includes a superconducting magnetic levitation device, a six-degree-of-freedom electric seat, and an adjustable seat belt array; wherein: The six-degree-of-freedom electric seat is installed on the superconducting magnetic levitation device. Through the Meissner effect generated by superconducting materials in a preset temperature environment, the six-degree-of-freedom electric seat floats above the superconducting magnetic levitation device; The six-degree-of-freedom electric seat is configured with multiple linear drive units composed of permanent magnets and electromagnetic coils to complete six-degree-of-freedom motion; The adjustable seat belt array is connected to the six-degree-of-freedom electric seat through multiple independent electric tightening devices, and adjusts the pressure applied to the target user according to the flight attitude.

[0010] In an alternative embodiment, the flight training system further includes a data acquisition device, and the data acquisition device is communicatively connected to an electronic device, wherein: The data acquisition device is used to collect corresponding physiological time-series data, electroencephalogram time-series data, eye movement image data, and facial image data of the target user during training; The electronic device is used to input the physiological time-series data, electroencephalogram time-series data, eye movement image data, and facial image data into a preset workload recognition model, and output the current workload level corresponding to the target user; The electronic device is further used to generate a backup virtual driving scenario based on preset training instructions, operation action data, button action data, and parameter setting data; adjust the complexity of the backup initial virtual driving scenario according to the target cognitive load level to generate a target virtual driving scenario.

[0011] In an alternative embodiment, the preset workload recognition model includes a first feature recognition network and a second feature recognition network. The electronic device is used to: Input the physiological time-series data and electroencephalogram time-series data into the first feature extraction network to generate time-series modal features; Input the facial image data and eye movement part image data into the second feature extraction network to generate visual modal features; Perform fusion processing on the time-series modal features and visual modal features to generate target fusion features; Based on the target fusion features, output the current workload level corresponding to the target user.

[0012] In an alternative embodiment, the electronic device is used to use the visual modal features as the first query matrix, and the time-series modal features as the first key matrix and the first value matrix; Calculate the first dependence weight of the visual modal features on the time-series modal features; Use the time-series modal features as the second query matrix, and the visual modal features as the second key matrix and the second value matrix; Calculate the second dependence weight of the temporal modal features on the visual modal features; Multiply the temporal modal features by the first dependence weight to obtain the target temporal features; Multiply the visual modal features by the second dependence weight to obtain the target visual features; Merge the target temporal features and the target visual features to generate the target fusion features.

[0013] In an alternative embodiment, an electronic device is configured to generate an initial virtual driving scenario based on a preset training instruction; Based on the initial virtual driving scenario, generate a backup virtual driving scenario based on operation action data, button action data, and parameter setting data.

[0014] In an alternative embodiment, an electronic device is configured to perform semantic recognition on a preset training instruction and extract key information from the preset training instruction; the key information includes at least one of a flight mission type, a flight area, and meteorological conditions; Using each key information as an initial node, the dependence relationship between each key information as an initial edge, and setting attribute information for each initial node, construct an initial graph structure; Perform a convolution operation on the initial graph structure to obtain initial node features corresponding to each initial node in the graph structure; Generate an initial virtual driving scenario based on each initial node feature.

[0015] An in-depth training environment helps the driver better understand and master the operation key points of different flight missions under various conditions.

[0016] In an alternative embodiment, an electronic device is configured to abstract operation action data, button action data, and parameter setting data into update nodes and establish update edge connections with each initial node in the initial graph structure; Determine the weights and directions of each update edge according to the association relationship between the operation action data, button action data, parameter setting data and the preset training instruction, and update the attribute information of each updated node to generate a target graph structure; Perform a convolution operation on the target graph structure to obtain target node features corresponding to each target node in the graph structure; the target nodes include each initial node and each update node; The feature matrices of each target node are input into a preset recurrent neural network at each time step to obtain the current target state features; Adjust the initial virtual driving scenario based on the current target state features to generate a backup virtual driving scenario.

[0017] In an alternative embodiment, the electronic device is further configured to input physiological time-series data, electroencephalogram time-series data, eye movement image data, facial image data, and the target cognitive load level into a meta-learner; The meta-learner obtains an initial cognitive load-scenario response model; the initial cognitive load-scenario response model includes an input layer, a hidden layer, and an output layer; the input layer is responsible for receiving data, and the number of its neurons is set according to the data feature dimension; the hidden layer consists of multiple layers of neurons and processes the data through complex connection weights; the output layer then outputs the final result; The meta-learner adjusts the initial cognitive load-scenario response model based on the physiological time-series data, electroencephalogram time-series data, eye movement image data, and facial image data to generate a target cognitive load-scenario response model corresponding to the target user; Input the target cognitive load level into the target cognitive load-scenario response model to output a scenario complexity adjustment parameter; the scenario complexity adjustment parameter includes fault type adjustment, fault occurrence frequency adjustment, and environment complexity adjustment: Adjust the complexity of the standby initial virtual driving scenario based on the scenario complexity adjustment parameter to generate a target virtual driving scenario. Description of the Drawings

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is a schematic structural diagram of a flight training system according to an embodiment of the present invention; Figure 2 It is a schematic structural diagram of another flight training system according to an embodiment of the present invention; Figure 3 It is a schematic structural diagram of yet another flight training system according to an embodiment of the present invention; Figure 4 It is a schematic flowchart of outputting the current workload level corresponding to the target user according to an embodiment of the present invention; Figure 5 It is a schematic flowchart of generating a standby virtual driving scenario according to an embodiment of the present invention; Figure 6 It is a schematic flowchart of generating a target virtual driving scenario according to an embodiment of the present invention.

[0020] Wherein: multi-axis gravity seat 1; superconducting magnetic levitation device 11; six-degree-of-freedom electric seat 12; adjustable seat belt array 13; flight joystick 2; button control console 3; parameter setting interface 4; dynamic scene display device 5; electronic device 6; air flow control system 7; air flow valve 71; air flow pressure sensor 72. Detailed implementation mode

[0021] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0022] The embodiment of the present application provides a flight training system, as Figure 1 shown, the flight training system includes a multi-axis gravity seat 1, a flight joystick 2, a button control console 3, a parameter setting interface 4, a dynamic scene display device 5, and an electronic device 6; wherein, the multi-axis gravity seat 1, the flight joystick 2, and the dynamic scene display device 5 are all communicatively connected to the electronic device 6; wherein: The flight joystick 2 is configured to receive operation action data input by a target user based on a preset training instruction, and the operation action data includes an execution sequence corresponding to each operation action and an operation force corresponding to each operation action.

[0023] The button control console 3 is configured to receive button action data input by a target user based on a preset training instruction.

[0024] The electronic device 6 is configured to receive parameter setting data input by a target user for setting flight parameters based on a preset training instruction.

[0025] The multi-axis gravity seat 1 is configured to perform six-degree-of-freedom motion based on the operation action data, the button action data, and the parameter setting data, so as to simulate the flight attitude of an aircraft.

[0026] The electronic device 6 is further configured to generate a target virtual driving scene based on a preset training instruction, the operation action data, the button action data, and the parameter setting data, and control the dynamic scene display device 5 to display the target virtual driving scene.

[0027] Specifically, the flight training system consists of a multi-axis gravity seat 1, a flight joystick 2, a button control console 3, a parameter setting interface 4, a dynamic scenario display device 5, and an electronic device 6. The multi-axis gravity seat 1, the flight joystick 2, and the dynamic scenario display device 5 are communicatively connected to the electronic device 6 to ensure real-time and accurate data transmission, enabling the electronic device 6 to obtain user operation information and control relevant devices to respond, thus constructing an organically collaborative training system.

[0028] As the core input device for the user to control the aircraft, the flight joystick 2 is equipped with various sensors (such as force sensors and angle sensors). When the target user operates according to the preset training instructions, the force sensor captures the operation force corresponding to the operation action in real time, and the angle sensor records the execution sequence and direction change of the operation action, converts these physical operations into electrical signals, and encodes them into operation action data. For example, when the user pulls the lever to take off, the joystick converts information such as the magnitude of the lever pull force, the change in lever angle, and the operation time sequence into operation action data, and transmits it to the electronic device 6 through the communication connection, providing basic data for subsequent scenario generation and seat simulation.

[0029] The button control console 3 is arranged with various function buttons corresponding to the control of different systems and functions of the aircraft. When the user presses a button according to the preset training instructions, the circuit system of the button control console 3 detects the change in the pressed state of the button and converts it into button action data, including information such as the button identifier and the press time. For example, when the user presses the button to turn on the autopilot mode, the button control console 3 encodes the action information of the button and transmits it to the electronic device 6 through the communication link, and the electronic device 6 thereby knows the user's operation intention for the aircraft system function.

[0030] The electronic device 6 receives the parameter setting data input by the target user through the parameter setting interface 4 for setting flight parameters. The parameter setting interface 4 is usually presented in the form of a touch screen, a knob, a key combination, etc. The user sets parameters such as flight altitude, speed, heading, and engine power on the interface. The input data is converted into digital signals by the signal processing module of the interface and transmitted to the electronic device 6. The electronic device 6 analyzes and stores these data, which serves as an important basis for generating virtual driving scenarios and simulating flight postures.

[0031] After receiving the preset training instructions, operation action data, button action data, and parameter setting data, the electronic device 6 first performs data preprocessing. Then, the electronic device 6 generates a target virtual driving scenario based on the preprocessed preset training instructions, operation action data, button action data, and parameter setting data.

[0032] After the electronic device 6 generates the target virtual driving scenario, it transmits the target scenario data to the dynamic scenario display device 5 (such as a large display screen, virtual reality helmet) through a graphics interface (such as HDMI, DisplayPort). After receiving the data, the dynamic scenario display device 5 uses its own display driver and graphics processing unit (GPU) to decode and render the data, and presents the target virtual driving scenario to the user in the form of an image or video. By watching the display device, the user seems to be in a real flight environment, combined with the physical simulation of the multi-axis gravity seat 1, to obtain a highly immersive flight training experience.

[0033] Through the above principles and work processes, the flight training system realizes a complete closed-loop from user operation input to virtual scenario presentation and physical simulation, providing an efficient and realistic training environment for pilots, and helping pilots improve their flight skills and the ability to handle various flight situations.

[0034] The flight training system provided by the embodiment of the present application, the flight control stick 2 receives the operation action data input by the target user based on the preset training instructions. The button control console 3 receives the button action data input by the target user based on the preset training instructions. So that the flight training system can finely capture the subtle differences in the operations of the target user on flight attitude adjustment, power control, etc., and provide accurate data support for subsequent simulations. The button action data received by the button control console 3 records the operations of the user on various system switches and function switches of the aircraft. The parameter setting data received by the electronic device 6 includes parameter settings such as flight altitude, speed, and heading. The comprehensive collection of these multi-dimensional data completely restores the operation process of the user in flight training, enables the system to accurately understand the user's operation intention, and lays a solid foundation for simulating flight attitudes and generating virtual scenarios. The multi-axis gravity seat 1 is used to perform six-degree-of-freedom motion based on the operation action data, button action data, and parameter setting data, so as to simulate the flight attitude of the aircraft. Let the target user truly feel the change of flight attitude from the physical perception. This highly realistic physical simulation breaks the limitations of traditional two-dimensional simulation training, makes the target user seem to be in a real cockpit, greatly enhances the immersion of the training, helps the target user to quickly get familiar with the physical feelings during flight, and improves the judgment and control ability of flight attitudes. The electronic device 6 generates a target virtual driving scene based on the operation action data, button action data, and parameter setting data, and controls the dynamic scene display device 5 to display the target virtual driving scene. It ensures the accuracy of the generated target virtual driving scene and realizes the combination of the target virtual driving scene and the actual operation process. Each component of the above flight training system is communicatively connected to the electronic device 6 to achieve data interaction and collaborative work. This collaborative mechanism makes the training process smoother and more realistic, and every operation of the target user can get timely and accurate feedback, promoting the target user to quickly learn and master flight skills, thereby improving the training effect during flight training.

[0035] In an alternative embodiment of the present application, as Figure 2 shown, the flight training system further includes an airflow control system 7, and the airflow control system 7 includes a plurality of airflow valves 71 and an airflow pressure sensor 72; the airflow control system 7 is communicatively connected to the electronic device 6, wherein: The electronic device 6 is configured to determine the external environment data corresponding to the preset training instruction based on the preset training instruction; and determine the target airflow direction and target airflow speed corresponding to the aircraft based on the external environment data and the flight attitude.

[0036] The airflow control system 7 is configured to detect the current airflow direction and the current airflow speed based on each airflow pressure sensor 72; and control the opening or closing of each airflow valve 71 according to the difference between the target airflow direction and target airflow speed and the current airflow direction and current airflow speed.

[0037] Specifically, during actual flight, the external environment in which the aircraft is located (such as weather, altitude, etc.) and its own flight attitude (such as pitch, roll, yaw, etc.) will affect the airflow around the aircraft. Therefore, a plurality of air valves 71 are provided on the side of the multi-axis gravity seat 1 to simulate the direct action of the airflow on the human body during flight (such as the wind pressure during climbing and diving). Airflow pressure sensors 72 can be installed around the multi-axis gravity seat 1 to monitor the changes in the airflow around the multi-axis gravity seat 1 in real time and feedback to the electronic device to adjust the simulation parameters. It directly acts on the trainee's body to enhance the realism of the changes in gravity and acceleration (such as simulating airflow impact and crosswind interference). By simulating the external environment data corresponding to the preset training instructions and combining with the flight attitude of the aircraft, the target airflow direction and speed can be determined, enabling the trainer to feel the airflow changes close to real flight and enhancing the realism and immersion of the training. In addition, in cooperation with the six-degree-of-freedom movement of the multi-axis gravity seat 1, a multi-modal feedback of "movement + airflow" is realized to improve the training fidelity.

[0038] The airflow control system 7 uses the airflow pressure sensors 72 to detect the current airflow direction and speed in real time, compares it with the target value, and adjusts the state of the air valve 71 according to the difference, forming a feedback control loop. This feedback mechanism can continuously optimize the airflow state to make it as close as possible to the target value to ensure the accuracy and stability of the simulation.

[0039] Specific working process: The electronic device 6 first receives the preset training instruction and determines the corresponding external environment data according to this instruction. For example, if the training instruction is to simulate a flight scenario on a sunny day at an altitude of 5000 meters, the electronic device 6 will obtain the corresponding environmental parameters such as weather, air pressure, and temperature.

[0040] Then, the electronic device 6 combines the determined external environment data and the flight attitude of the aircraft simulated by the multi-axis gravity seat 1, and calculates the target airflow direction and target airflow speed corresponding to the aircraft in the current state through a specific algorithm or model. For example, when the aircraft is in a climbing attitude and there is a certain crosswind outside, it is necessary to calculate the airflow direction and speed under the combined influence.

[0041] The airflow pressure sensors 72 in the airflow control system 7 continuously detect the direction and speed of the current airflow. These sensors are distributed at relevant positions of the training system, can accurately sense the real-time changes in the airflow, and transmit the detected data to the electronic device 6.

[0042] The electronic device 6 compares the target air flow direction and speed with the current air flow direction and speed, and calculates the difference between the two. The air flow control system 7 controls the opening or closing of each air flow valve 71 based on these differences. If the current air flow speed is lower than the target speed, more air flow valves 71 may be opened to increase the air flow rate; if the air flow direction deviates from the target direction, the air flow direction will be changed by adjusting the air flow valves 71 at different positions to gradually approach the target value. Through continuous detection, comparison and adjustment, the air flow state is continuously optimized to provide the trainer with a realistic air flow feeling that conforms to the current flight state.

[0043] For the flight training system provided by the embodiments of the present application, the electronic device 6 determines the external environment data corresponding to the preset training instruction based on the preset training instruction; and determines the target air flow direction and target air flow speed corresponding to the aircraft based on the external environment data and the flight attitude. Thus, the corresponding air flow conditions can be accurately set according to different training scenarios and requirements. The air flow control system 7 detects the current air flow direction and current air flow speed based on each air flow pressure sensor 72; and controls the opening or closing of each air flow valve 71 according to the difference between the target air flow direction and target air flow speed and the current air flow direction and current air flow speed. It can accurately simulate the air flow conditions faced by the aircraft under different external environments and flight attitudes during flight, such as the air flow changes during takeoff, landing, and encountering air flow turbulence, allowing the pilot to feel the realistic air flow effect, enhancing the immersion and authenticity of the training. In addition, when simulating some dangerous air flow conditions, the pilot can be familiar with the coping methods in a safe training environment, avoiding danger due to lack of experience when encountering similar situations during actual flight. At the same time, the precise control of the air flow control system 7 can also prevent damage to the training equipment or personnel caused by improper air flow simulation. The pilot can truly feel the impact of the air flow on flight during training, better understand and master the control characteristics of the aircraft under different air flow conditions, which helps to improve flight skills and reaction speed, making the training effect closer to actual flight and shortening the transition time from training to actual flight.

[0044] In an alternative embodiment of the present application, as Figure 3 shown, the multi-axis gravity seat 1 includes a superconducting magnetic levitation device 11, a six-degree-of-freedom electric seat 12, and an adjustable seat belt array 13; where: The six-degree-of-freedom electric seat 12 is installed on the superconducting magnetic levitation device 11, and due to the Meissner effect generated by the superconducting material in the preset temperature environment, the six-degree-of-freedom electric seat 12 floats above the superconducting magnetic levitation device 11; The six-degree-of-freedom electric seat 12 is configured with a linear drive unit composed of multiple permanent magnets and electromagnetic coils to complete six-degree-of-freedom movement; The adjustable seat belt array 13 is connected to the six-degree-of-freedom electric seat 12 through multiple independent electric tightening devices, and adjusts the pressure applied to the target user according to the flight attitude.

[0045] Specifically, the superconducting magnetic levitation device 11 utilizes the Meissner effect generated by superconducting materials in a preset temperature environment to achieve magnetic levitation. The Meissner effect means that when a superconducting material is in the superconducting state, it will completely repel the magnetic field, preventing the magnetic field lines from penetrating the superconductor, thereby generating an upward levitation force that enables the six-degree-of-freedom electric seat 12 to stably levitate above the superconducting magnetic levitation device 11. This levitation method can reduce mechanical friction and improve the accuracy and flexibility of movement. The six-degree-of-freedom electric seat 12 achieves six-degree-of-freedom movement by configuring multiple linear drive units composed of permanent magnets and electromagnetic coils. According to Ampere's force principle, when an electric current passes through the electromagnetic coil, it will be subjected to a force in the magnetic field, and the permanent magnet provides a stable magnetic field. By precisely controlling the magnitude and direction of the current in the electromagnetic coil, the linear drive unit can generate forces of different directions and magnitudes, thereby driving the six-degree-of-freedom electric seat 12 to perform translation (movement along the X, Y, and Z axes) and rotation (rotation around the X, Y, and Z axes) in space, simulating various flight attitudes of the aircraft.

[0046] The adjustable seat belt array 13 is connected to the six-degree-of-freedom electric seat 12 through multiple independent electric tightening devices. The principle is that according to the flight attitude simulated by the six-degree-of-freedom electric seat 12, the electronic device 6 calculates the binding force required by the target user in different attitudes, and then controls the electric tightening device to adjust the tightness of the seat belt, thereby applying an appropriate pressure to the target user, enabling the user to feel the gravity and acceleration changes matching the flight attitude, and enhancing the realism and immersion of flight training.

[0047] The specific working process is as follows: When the flight training system is started, the superconducting magnetic levitation device 11 cools the superconducting material to the preset temperature environment, making it enter the superconducting state and generating the Meissner effect. At this time, the six-degree-of-freedom electric seat 12 stably levitates above the superconducting magnetic levitation device 11 under the action of the levitation force generated by the Meissner effect, providing a frictionless support basis for subsequent movements.

[0048] The electronic device 6 calculates the aircraft flight attitude to be simulated according to the preset training instructions of the flight training and relevant data (such as operation action data, button action data, parameter setting data, etc.). Then, the electronic device 6 sends a control signal to the linear drive unit of the six-degree-of-freedom electric seat 12. By controlling the current in the electromagnetic coil, the permanent magnet and the electromagnetic coil interact with each other to generate corresponding forces and torques, driving the six-degree-of-freedom electric seat 12 to perform six-degree-of-freedom movements, accurately simulating various flight attitudes of the aircraft, such as takeoff, landing, turning, climbing, diving, etc.

[0049] In addition, during the process of the six-degree-of-freedom electric seat 12 simulating flight postures, the electronic device 6 will monitor the posture information of the seat in real time and calculate the pressure required by the target user in the current posture according to a preset algorithm. Then, the electronic device 6 sends instructions to the electric tightening devices of the adjustable seat belt array 13 to control the tightening or loosening of the seat belts. Each electric tightening device works independently and can precisely adjust the tension of the seat belt according to the force requirements of different parts of the target user's body, so as to apply appropriate pressure to the target user in different flight postures and let the user experience a realistic flight.

[0050] In the flight training system provided by the embodiment of the present application, the six-degree-of-freedom electric seat 12 is installed on the superconducting magnetic levitation device 11. Through the Meissner effect generated by the superconducting material in a preset temperature environment, the six-degree-of-freedom electric seat 12 is levitated above the superconducting magnetic levitation device 11, realizing contactless levitation support. This levitation method avoids the friction and wear caused by traditional mechanical support, reduces energy loss, improves the stability and reliability of the system, and also reduces the maintenance cost. The six-degree-of-freedom electric seat 12 is configured with multiple linear drive units composed of permanent magnets and electromagnetic coils to complete six-degree-of-freedom movement. Thus, it can precisely control the movement of the seat in six degrees of freedom, including front-back, left-right, up-down translation, and pitch, roll, and yaw rotation. This enables the seat to accurately simulate various flight postures, providing a highly realistic flight experience for users and helping pilots better adapt to posture changes in actual flight during training. The linear drive unit has the characteristic of rapid response and can quickly adjust the position and posture of the seat according to the change of flight posture, following the dynamic process of the simulated flight in real time, reducing delay, and improving the user's immersion and training effect. In addition, it can bear a large load, meet the needs of users of different body types, and ensure the stability and safety of the seat when simulating high-intensity flight actions, ensuring the safety of users during the training process. The adjustable seat belt array 13 is connected to the six-degree-of-freedom electric seat 12 through multiple independent electric tightening devices and adjusts the pressure applied to the target user according to the flight posture. This can provide personalized restraint and support for users, ensuring that the user's body can be properly fixed in different flight postures, improving comfort and safety. Dynamically adjusting the pressure of the seat belt according to the flight posture can enable users to more truly feel the gravity change and body force in flight, further enhancing the immersion of the simulated flight and enabling users to better adapt to the physiological feelings in actual flight during training.

[0051] During the simulated flight process, especially when performing some difficult or dangerous flight actions, the adjustable seat belt array 13 can effectively restrain the user in time, preventing the user from displacing or getting injured on the seat, providing additional safety protection for the user.

[0052] In an alternative embodiment of the present application, the flight training system further includes a data acquisition device, which is communicatively connected to the electronic device 6, where: The data acquisition device is configured to collect corresponding physiological time-series data, electroencephalogram (EEG) time-series data, eye movement image data, and facial image data of the target user during the training process; The electronic device 6 is configured to input the physiological time-series data, EEG time-series data, eye movement image data, and facial image data into a preset workload recognition model, and output the current workload level corresponding to the target user.

[0053] The electronic device 6 is further configured to generate a backup virtual driving scenario based on a preset training instruction, operation action data, button action data, and parameter setting data; adjust the complexity of the backup virtual driving scenario according to the target cognitive load level to generate a target virtual driving scenario.

[0054] Specifically, the data acquisition system may include a wearable human factors physiological recorder, a hydroelectrode EEG system, a wearable eye tracker, and a camera device. Among them, the wearable human factors physiological recorder may be an ErgoLAB intelligent wearable human factors physiological recorder, which is a set of wearable multi-parameter vital sign comprehensive detectors that can be worn on various parts of the human body. It can real-time monitor physiological indicators such as RESP respiratory rate, HR heart rate, ECG electrocardiogram changes, EDA skin conductance changes, PPG pulse changes, and EMG electromyogram of the human body, and can also real-time extract the posture changes of the human body and GPS spatio-temporal behavior trajectories and spatial position data. Thus, the electronic device 6 can collect the corresponding physiological time-series data of the target user based on the ErgoLAB intelligent wearable human factors physiological recorder.

[0055] The hydroelectrode EEG system may be a Semi-Dry EEG hydroelectrode system. The Semi-Dry EEG hydroelectrode system consists of 32 EEG electrodes, a reference electrode, a wireless EEG signal amplifier, an EEG cap, and electrode cotton pads. The sampling rate is as high as 32 kHz, the resolution is 24 bit, and it supports 10 - 1000 times of signal programmable amplification. Thus, the electronic device 6 can collect EEG time-series data based on the Semi-Dry EEG hydroelectrode system.

[0056] The wearable eye tracker may be a Tobii Glasses 2 wearable eye tracker. The Tobii Glasses 2 wearable eye tracker is a wearable eye tracker with wireless real-time observation function, designed specifically for research in real-world environments. The sampling rate of the eye tracker is 50 Hz or 100 Hz. Thus, the electronic device 6 can collect the corresponding eye movement image data of the target user based on the Tobii Glasses 2 wearable eye tracker.

[0057] The imaging device can collect the facial image data corresponding to the target user.

[0058] Then, the electronic device 6 inputs the physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data into a preset workload recognition model, and outputs the current workload level corresponding to the target user.

[0059] In addition, the electronic device 6 generates a standby virtual driving scenario based on a preset training instruction, operation action data, button action data, and parameter setting data; adjusts the complexity of the standby virtual driving scenario according to the target cognitive load level to generate a target virtual driving scenario.

[0060] In the flight training system provided by the embodiments of the present application, the data acquisition device collects the physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data corresponding to the target user during the training process. The electronic device 6 inputs the physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data into a preset workload recognition model, and outputs the current workload level corresponding to the target user. This ensures the accuracy of the output current workload level. Precise workload recognition enables the training system to understand the user's stress and cognitive state during training in real time, avoiding the situation where the training difficulty is too high, which may cause the user to have excessive anxiety and fatigue, affecting the training effect, and also preventing the training difficulty from being too low, making the training lack challenge. Then, it is also used to generate a standby virtual driving scenario based on a preset training instruction, operation action data, button action data, and parameter setting data, ensuring the accuracy of the generated standby virtual driving scenario. In addition, the complexity of the standby virtual driving scenario is adjusted according to the target cognitive load level to generate a target virtual driving scenario. The complexity of the standby virtual driving scenario is adjusted according to the target cognitive load level to generate a target virtual driving scenario. When the user's workload is low, the system increases the scene complexity, such as introducing complex meteorological conditions, multi-system fault combinations, etc., to increase the training difficulty and stimulate the user's learning potential; when the user's workload is too high, the system reduces the scene complexity, reduces interference factors, and simplifies the task process to help the user relieve stress and maintain a good training state. This dynamic adjustment mechanism makes the training scenario always match the user's cognitive load, enhances the user's immersion and participation, effectively improves the training effect, and makes the training more targeted and practical.

[0061] In an alternative embodiment of the present application, the preset workload recognition model includes a first feature recognition network and a second feature recognition network. As Figure 4 shown, the above-mentioned "inputting the physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data into a preset workload recognition model, and outputting the current workload level corresponding to the target user" may include the following steps: Step S101: Input the physiological time-series data and the electroencephalogram time-series data into the first feature extraction network to generate time-series modal features.

[0062] Specifically, the first feature extraction network includes a first sub-feature extraction branch, a second sub-feature extraction branch, and a first sub-feature fusion network. The above step S101 may include the following steps: Step a1: Input the physiological time-series data into the first sub-feature extraction branch and input the electroencephalogram time-series data into the second sub-feature extraction branch.

[0063] Step a2: The first sub-feature extraction branch extracts features from the physiological time-series data to generate a physiological signal sequence.

[0064] Among them, the physiological signal sequence includes at least one of respiratory data features, electrocardiogram data time-domain features, electrocardiogram data frequency-domain features, electrocardiogram data non-linear features, and electromyogram data features.

[0065] Specifically, before extracting features from the physiological time-series data, the physiological time-series data can first be preprocessed.

[0066] Exemplarily, the preprocessing steps for electrocardiogram data and respiratory data include low-pass filtering, power frequency interference elimination, and baseline drift elimination. Among them, low-pass filtering is mainly used to filter out high-frequency data interference such as muscle electrical signals, then a 50Hz notch filter is used to eliminate power frequency interference, and finally baseline drift is eliminated. The preprocessing of skin electrical data only includes low-pass filtering and power frequency interference elimination.

[0067] Then, the first sub-feature extraction branch extracts features from the physiological time-series data. The specific extraction process can be as follows: Electrocardiogram data and respiratory data are periodic data. Among them, the preprocessing of electrocardiogram data also includes the determination of R-wave peaks. The time taken for each heartbeat can be determined by adjacent R-wave peaks, and the heart rate can be determined by taking the reciprocal of the heartbeat time. The R-wave peak detection method is the threshold method. The principle of the threshold method is that first, the first sub-feature extraction branch determines all the peaks of the electrocardiogram data based on the peak function, and then the peaks with a peak value greater than a certain threshold are determined as R-wave peak values. After determining the R-wave peak values, the heartbeat duration can be calculated, and then the instantaneous heart rate is equal to the reciprocal of the heartbeat duration, and the average heart rate is equal to the average of all instantaneous heart rates.

[0068] Respiratory data preprocessing also includes the determination of respiratory peaks and troughs. Among them, the respiratory peak refers to the turning point from the end of inspiration to the start of expiration, and the respiratory trough refers to the turning point from the end of expiration to the start of inspiration. The determination of respiratory peaks and troughs is based on the magnitudes of expiratory amplitude and inspiratory amplitude. The principle of the method for determining respiratory peaks and troughs is as follows: First, the first sub-feature extraction branch uses the peak function to determine all peaks and troughs of the respiratory data. Then, the expiratory amplitude and inspiratory amplitude are obtained, and it is judged whether each expiratory amplitude and inspiratory amplitude is less than a threshold. For those expiratory amplitudes and inspiratory amplitudes less than the threshold, the corresponding peaks and troughs should be deleted. Finally, the respiratory peaks and troughs are obtained.

[0069] In the process of electrocardiogram data feature extraction, time-domain features, frequency-domain features, and non-linear features of electrocardiogram data are calculated based on RR interval data. Among them, the time-domain features include , , SDNN, RMSSD, SDSD, pNN50, pNN20, the frequency-domain features include TP, ULF, VLF, LF, HF, pLF, pHF, LF / HF, and the non-linear features include the SD1 value and SD2 value of the Poincaré scatter plot of RR interval data. The descriptions of each feature are shown in Table 1.

[0070] Table 1 Electrocardiogram data features and their meanings

[0071] In the process of electromyogram data feature extraction, in order to fully extract the features of electrodermal activity data, the diff function in the first sub-feature extraction branch obtains the first-order difference and second-order difference of EDA data. Finally, the mean, median, standard deviation, minimum value, and maximum value of EDA data, the first-order difference of EDA data, and the second-order difference of EDA data are calculated respectively, with a total of 15 features. In the process of respiratory data feature extraction, in order to fully extract the features of electrodermal activity data, the first sub-feature extraction branch first obtains the respiratory peak, respiratory trough, expiratory amplitude, expiratory time, inspiratory amplitude, inspiratory time, respiratory time (determined by the respiratory peak interval), and respiratory time (determined by the respiratory trough interval). Then, the diff function in the first sub-feature extraction branch obtains the first-order difference and second-order difference of the above respiratory data. Finally, the mean, median, standard deviation, minimum value, and maximum value of respiratory data, the first-order difference of respiratory data, and the second-order difference of respiratory data are calculated respectively, with a total of 120 features.

[0072] Finally, the first sub-feature extraction branch combines the obtained features to generate a physiological signal sequence.

[0073] Step a3: Based on the sparse attention scoring method, feature extraction is performed on the physiological signal sequence to generate a physiological signal matrix.

[0074] Specifically, step a3 may include the following steps: Step a31: Perform a linear transformation on the physiological signal sequence to generate an initial query matrix, an initial key matrix, and an initial value matrix.

[0075] Specifically, the first sub-feature extraction branch can convert the physiological signal sequence into a vector representation that can be processed by the model. Assuming that the length of the physiological signal sequence is T and the dimension of the physiological signal feature at each time step is d, the physiological signal sequence can be represented as a matrix X ∈ R T×d . Then, the first sub-feature extraction branch can use a linear transformation to map the input matrix X to the initial query matrix Q, the initial key matrix K, and the initial value matrix V respectively. Specifically, through three learnable weight matrices Wq ∈ R d×dk , Wk ∈ R d×dk and Wv ∈ R d×dv for linear transformation, we get: Q = XW q ∈ R T×dk ; K = XW k ∈ R T×dk ; V = XW v ∈ R T×dv .

[0076] where dk and dv are the dimensions of the initial query matrix, the initial key matrix, and the initial value matrix respectively.

[0077] Step a32: Calculate the similarity scores between each query vector in the initial query matrix and all key vectors in the initial key matrix.

[0078] Specifically, the first sub-feature extraction branch can calculate the similarity score S(q i between each query vector q in the initial query matrix Q and all key vectors k j in the initial key matrix K. Usually, the dot product similarity is used, that is, S(q i , k j ) = q i T k j i . j i T

[0079] Then, perform a softmax operation on the similarity scores to obtain a standard attention probability distribution. Assume the formula is as follows: .

[0080] where, is to scale the dot product similarity to avoid the problem of gradient vanishing or explosion.

[0081] Step a33: Introduce the physiological signal-specific prior and calculate the sparsity metric values corresponding to each query vector in the initial query matrix based on the similarity scores.

[0082] Specifically, for the i-th query vector q in the initial query matrix Q of the first sub-feature extraction branch i , calculate its sparsity metric value M(q i ) according to the formula. Suppose the formula is as follows: ; where is the logarithmic form of the softmax operation on the similarity scores S(q i and all key vectors k j , which measures the global correlation of the query vector q i ,k j ) with the entire sequence. i is the average value of the similarity scores calculated for the query vector q i and all key vectors k j , representing the average correlation of the query vector q i with the sequence. , where HRV_mask(q i ) is the attention enhancement mask for the heart rate variability-related channels. If the query vector q i is related to heart rate variability, the value of this mask is a positive number, otherwise it is 0. λ is the adaptive weight coefficient used to adjust the influence of this part.

[0083] Step a34: Sort each query vector according to the sparsity metric value and select the top-u target query vectors with the highest sparsity metric values.

[0084] Specifically, the first sub-feature extraction branch sorts each query vector according to the calculated sparsity metric values corresponding to each query vector and selects the top-u target query vectors with the highest sparsity metric values. Among them, u = c • log(T) (c is an adjustable constant).

[0085] Step a35: Form a sparse query matrix based on the top-u target query vectors.

[0086] Specifically, the first sub-feature extraction branch forms a sparse query matrix based on the top-u target query vectors.

[0087] Step a36: Generate a physiological signal matrix based on the sparse query matrix, the initial key matrix, and the initial value matrix.

[0088] ​Specifically, the electronic device 6 uses the filtered sparse query matrix to calculate the modified attention. The specific formula is as follows: .

[0089] Among them, is the sparse query matrix. In this way, only the attention between the sparse query matrix and the initial key matrix K is calculated, thereby reducing the computational complexity from O(T 2 ) to O(uT).

[0090] Then, based on the modified attention, a physiological signal matrix is generated.

[0091] Step a4, perform feature extraction on the physiological signal matrix based on multi-period convolutional kernels, and output physiological modality features.

[0092] Specifically, assume that the physiological signal matrix is X, and its dimension is [N, T], where N represents the number of signal channels, and T represents the number of sampling points, that is, the length of the time series.

[0093] According to the characteristics and analysis purposes of physiological signals, a set of convolutional kernels with different periods are designed. The size, shape, and weights of these convolutional kernels are usually obtained through training and learning. Assume that there are a total of K convolutional kernels, and the dimension of each convolutional kernel is [N, L k , where L k represents the length of the k-th convolutional kernel, corresponding to different time periods. For example, shorter convolutional kernels may be used to capture high-frequency details in physiological signals, while longer convolutional kernels are used to extract low-frequency trends and periodic features.

[0094] Convolve each convolutional kernel with the physiological signal matrix X respectively. For the k-th convolutional kernel, the calculation method of its convolution result Yk with X is as follows: Y k =X×W k +b k . Among them, ∗ represents the convolution operation, W k is the weight matrix of the k-th convolutional kernel, and b k is the bias term. The result Y k of the convolution operation is a feature matrix with a dimension of [N, T - L k +1], which represents the feature response of the physiological signal after being filtered by the k-th convolutional kernel.

[0095] Fuse the output feature matrices Y1, Y2,..., Y K of all convolutional kernels. Common fusion methods include concatenation, summation, etc. For example, through the concatenation operation, all feature matrices can be concatenated in the channel dimension to obtain a matrix with a dimension of [N, (T - L1 + 1) + (T - L2 + 1) +... + (T - L KThe fused feature matrix Y of [+1)], that is, the physiological modality feature.

[0096] To better extract the physiological modality feature, some subsequent processes can also be performed on the basis of the fused feature matrix Y, such as pooling operations, application of non-linear activation functions, etc. Pooling operations can reduce the dimension of features while retaining important feature information; non-linear activation functions can increase the non-linear expression ability of the model, making the extracted features more representative.

[0097] After the above steps, the finally obtained feature matrix is the physiological modality feature. It contains the local and global features of physiological signals at different time scales, and can reflect the internal patterns and laws of physiological signals.

[0098] Step a5, the second sub-feature extraction branch calculates the energy spectral density and power spectral density corresponding to the EEG time series data.

[0099] Specifically, the electronic device 6 can choose to analyze the power spectral density (PSD) and energy spectral density (ESD) corresponding to the EEG time series data as EEG analysis indicators. For the analysis of dynamic signals, since there are many frequency bands in the EEG time series data and it is difficult to effectively divide them in the time domain, frequency domain analysis is more commonly used. Usually, the method of analyzing the spectrum is adopted. Five frequency bands are extracted by this method as [δ (1 - 4Hz), θ (4 - 8Hz), α (8 - 14Hz), β (14 - 30Hz), and γ (30 - 75Hz). By arranging the harmonics decomposed from the original signal in descending order of energy, an energy spectrum can be formed, which represents the energy distribution of different frequency components in the signal.

[0100] Specifically, energy spectral density analysis is a method of EEG frequency domain analysis. In the process of analyzing dynamic signals, since the EEG time series data itself often contains multiple frequency bands and it is very difficult to make a detailed division in the time domain, frequency domain analysis has become a commonly used means. If the harmonics decomposed from the original signal are arranged in descending order of energy, an energy spectrum is formed, which represents the energy distribution of different frequency components in the signal. The formula is as follows: ; Among them, is the energy spectral density, are the respective frequency components in the signal, is the Fourier transform of the signal, is the complex conjugate of.

[0101] PSD (Power Spectral Density) is a measure that calculates the proportion of the average power distribution of a random variable per unit frequency, which is a form of the mean square value. Its advantage is that it can transform the amplitude of the EEG time series signal that originally fluctuates with time into the power spectrum of the EEG time series signal that changes with frequency, thus visualizing the distribution and conversion of the EEG time series rhythm. The second sub-feature extraction branch calculates the power spectral density of the EEG signal based on the pwelch function. It is based on the Welch method and estimates the PSD of the signal through the segmented average periodogram method. This method selects the pwelch function and uses a Hamming window of 512 samples, an overlap of 256 samples, and a fast Fourier transform (FFT) of 1024 data points to estimate the spectrum. Finally, the power spectrum is obtained by taking the modulus square of the spectrum, as shown in the formula: ; ; where, is the power spectral density, is the number of samples, is the result of the Fourier transform after windowing. is the energy in the frequency interval, f h is the upper limit of the frequency interval, f i is the lower limit of the frequency interval.

[0102] Step a6, based on the energy spectral density and the power spectral density, extract features from the EEG time series data and output the EEG modality features.

[0103] Specifically, the above step a6 may include the following steps: Step a61, map each frequency point in the EEG time series data to the first initial frequency band, the second initial frequency band, and the third initial frequency band.

[0104] Specifically, the electronic device 6 can map each frequency point in the EEG time series data to the first initial frequency band, the second initial frequency band, and the third initial frequency band. Exemplarily, the first initial frequency band is α (8 - 12 Hz), the second initial frequency band is β (13 - 30 Hz), and the third initial frequency band is γ (30 - 100 Hz).

[0105] Step a62, based on the energy spectral density and the power spectral density, calculate the energy mean value and the power mean value corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band respectively.

[0106] Specifically, based on the energy spectral density and the power spectral density, calculate the energy mean value and the power mean value corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band respectively, to obtain the frequency domain feature X eeg ∈R T×B×Deeg (B = 3 is the number of frequency bands).

[0107] Taking the power spectral density as an example, for the α frequency band, its power mean value , F α is the number of frequency points within the α frequency band. These band characteristics can reflect the EEG activity intensity in different frequency ranges. In cognitive load research, the power changes in different frequency bands are closely related to the cognitive state. For example, an increase in β wave power may be related to high cognitive load.

[0108] Step a63: Calculate the weight information corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band respectively, based on the energy mean values and power mean values corresponding to them.

[0109] Specifically, the weight information α corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band respectively is obtained by processing the energy mean values and power mean values corresponding to them through a fully connected layer f b , S′(q,k b ) = S(q,k b ) • α f b .

[0110] Exemplarily, the electronic device 6 can convert the energy mean value or power mean value of each frequency band into a weight through the softmax function. Assuming that the energy mean value is used to calculate the weight, first calculate S = E1 + E2 + E3, then the weight w1 of the first initial frequency band is w1 = E1 / S, the weight w2 of the second initial frequency band is w2 = E2 / S, the weight w3 of the third initial frequency band is w3 = E3 / S, and w1 + w2 + w3 = 1.

[0111] For example, if the power of the β frequency band continuously increases within a certain period of time, after being processed by the fully connected layer and the Softmax function, the weight α f β of the β frequency band will increase, making the model pay more attention to the information of the β frequency band during self-attention calculation, so as to focus on the frequency band activated during high load. The time-frequency cross-attention uses a two-dimensional attention matrix, combines the distribution of the energy spectral density or power spectral density at different time steps and frequency bands, and captures the dominant frequency band at a specific time point. For example, during the task peak period, the energy or power of the γ frequency band shows an explosive growth, and this change can be more accurately captured through time-frequency cross-attention.

[0112] Step a64: Multiply the first initial frequency band, the second initial frequency band, and the third initial frequency band by the corresponding weight information respectively to generate the first target frequency band, the second target frequency band, and the third target frequency band.

[0113] Specifically, multiply the first initial frequency band, the second initial frequency band, and the third initial frequency band by their corresponding weight information respectively to generate a first target frequency band, a second target frequency band, and a third target frequency band.

[0114] Step a65: Extract the phase information corresponding to the EEG time series data through Hilbert transform to generate a phase matrix.

[0115] Specifically, the electronic device 6 convolves the Hilbert kernel with the EEG time series data x(t) to obtain the signal y(t) after Hilbert transform. From the frequency domain perspective, Hilbert transform shifts the positive frequency components of the signal to the right by 90 degrees and the negative frequency components to the left by 90 degrees. Based on this, the original EEG time series data x(t) and its signal y(t) after Hilbert transform can form an analytic signal z(t) = x(t) + jy(t). Through the analytic signal, the phase information can be calculated. In actual operation, applying Hilbert transform to the EEG time series data after power frequency filtering and short-time Fourier transform can obtain the phase information at each frequency point at different time points, providing data for the phase dimension for constructing three-dimensional features.

[0116] Step a66: Based on the first target frequency band, the second target frequency band, the third target frequency band, and the phase matrix, integrate the information of the time dimension, frequency dimension, and phase dimension corresponding to the EEG time series data to construct a time-frequency-phase three-dimensional feature and generate an EEG modal feature.

[0117] Specifically, based on the first target frequency band, the second target frequency band, the third target frequency band, and the phase matrix, integrate the information of the time dimension, frequency dimension, and phase dimension corresponding to the EEG time series data to construct a time-frequency-phase three-dimensional feature.

[0118] Assume that the time step is T, there are three target frequency bands in the frequency dimension, namely the first target frequency band, the second target frequency band, and the third target frequency band, and the dimension of the phase matrix is T×F (F is the number of frequency points). A three-dimensional tensor Feature∈R can be constructed. T ×3×2+T×F where the first dimension represents time, the first three elements of the second dimension respectively correspond to the energy spectral density and power spectral density information of the three target frequency bands, and the subsequent elements correspond to the elements of the phase matrix. Specifically, it is expressed as follows: ; where, is the corresponding energy spectral density, is the corresponding power spectral density, is the phase matrix.

[0119] Step a7: The first sub-feature fusion network fuses the physiological modal feature and the EEG modal feature to generate a time series modal feature.

[0120] Specifically, the first sub-feature fusion network splices the physiological modality features and the electroencephalogram modality features, or performs weighted fusion processing to generate temporal modality features.

[0121] Step S102: Input the facial image data and the eye movement image data into the second feature extraction network to generate visual modality features.

[0122] Specifically, the second feature extraction network includes a third sub-feature extraction branch, a fourth sub-feature extraction branch, and a second sub-feature fusion network. The above step S102 may include the following steps: Step b1: Input the facial image data into the third sub-feature extraction branch, and input the eye movement image data into the fourth sub-feature extraction branch.

[0123] Step b2: The dynamic feature extraction branch in the third sub-feature extraction branch extracts the optical flow features of the facial image data and outputs the facial optical flow features corresponding to the facial image data.

[0124] Specifically, the facial image data is at least three frames. The above step b2 may include the following steps: Step b21: Perform object detection on each frame of facial image data to determine the region of interest in each frame of facial image data.

[0125] Among them, the region of interest includes the orbicularis oculi muscle, the orbicularis oris muscle, and the frontalis muscle.

[0126] Specifically, the third sub-feature extraction branch may perform object detection on each frame of facial image data based on an object detection algorithm to determine the region of interest in each frame of facial image data.

[0127] Step b22: Based on a preset optical flow algorithm, calculate the motion vectors corresponding to each facial image data according to the position information of each region of interest in the corresponding facial image data.

[0128] Among them, the motion vector includes a horizontal displacement and a vertical displacement.

[0129] Among them, the preset optical flow algorithm may be the Lucas-Kanade (LK) optical flow algorithm, or the Farneback optical flow algorithm, or other optical flow algorithms. The embodiments of the present application do not make specific limitations on the preset optical flow algorithm.

[0130] Specifically, the third sub-feature extraction branch may convert two consecutive frames of facial images It, It+1 ∈ R H×W×3 (RGB three channels) into single-channel images Gt, Gt+1 ∈ R H×W, eliminate color interference. Then, use a 3×3 Gaussian kernel for noise reduction to reduce the impact of high-frequency noise on optical flow calculation.

[0131] Next, the third sub-feature extraction branch can calculate the motion vectors corresponding to each facial image data based on the Farneback optical flow algorithm according to the position information of each region of interest in the corresponding facial image data.

[0132] Then, normalize the motion vectors to [-1, 1] and align them with the original frame timestamps to form an optical flow feature sequence: Flowt = [Ut, Vt] ∈ R H×W×2 .

[0133] Step b23, calculate the motion amplitude, motion direction, and motion acceleration corresponding to the motion vectors.

[0134] Specifically, the third sub-feature extraction branch calculates the motion amplitude and the motion direction as additional feature channels (a total of 4 channels: U, V, M, θ) to enrich the representation dimension of motion features.

[0135] In addition, the third sub-feature extraction branch calculates the second-order difference ΔFlow t = Flow t - Flow t-1 for three consecutive frames of motion vectors to capture the motion acceleration corresponding to the motion vectors and enhance the sensitivity to sudden facial expression changes (such as rapid eyelid opening and closing when surprised).

[0136] Step b24, fuse the motion vectors, motion amplitude, motion direction, and motion acceleration to generate facial optical flow features.

[0137] Specifically, concatenate the motion vectors, motion amplitude, motion direction, and motion acceleration to generate facial optical flow features.

[0138] Step b3, the static feature extraction branch in the third sub-feature extraction branch performs static feature extraction on the facial image data and outputs the facial static features corresponding to the facial image data.

[0139] Specifically, the static feature extraction branch includes multiple parallel branches; the above step b3 can include the following steps: Step b31, the static feature extraction branch performs feature extraction on the facial image data to obtain initial static features.

[0140] Specifically, the static feature extraction branch performs feature extraction on the facial image data based on the convolution kernels in the convolutional layer to obtain initial static features.

[0141] Exemplarily, the size of the convolutional kernel in the convolutional layer is 3×3. Slide this convolutional kernel over the facial image data and perform a convolution operation on each local area, that is, multiply the corresponding elements and sum them to obtain a new feature value. By using multiple different convolutional kernels, different features of the image, such as edges, textures, etc., can be extracted to obtain the initial static features.

[0142] Perform preliminary feature extraction on the input facial image data, and convert the original facial image pixel information into a more representative feature map. This process can be regarded as a simple feature abstraction of the image, preparing for subsequent multi-scale feature extraction.

[0143] Step b32, input the initial static features into each first parallel branch.

[0144] Then, input the initial static features into each first parallel branch.

[0145] Step b33, the first parallel branch weights the initial static features based on both the channel and spatial dimensions to obtain weighted static features.

[0146] Specifically, the first parallel branch can compress the feature map of each channel into a single value through a global average pooling operation to obtain the global statistical information of the channel. Then, input these statistical information into a multi-layer perceptron (MLP), and the MLP will learn how to generate a suitable weight for each channel based on these statistical information. This weight can be regarded as a quantitative representation of the importance of the channel.

[0147] In addition, the first parallel branch can perform a sliding convolution operation on the feature map using a convolutional kernel, and generate spatial weights by learning the parameters of the convolutional kernel. These convolutional kernels can capture the feature correlations within the local spatial region, thereby generating a suitable weight value for each spatial position. Additionally, some attention mechanisms, such as a spatial attention module, can be combined to automatically learn the importance distribution of spatial positions.

[0148] Then, perform channel dimension weighting and spatial dimension weighting respectively to obtain channel weights and spatial weights. Then, fuse these two weights. For example, they can be combined through simple multiplication or addition operations to obtain a comprehensive weight matrix. Finally, multiply this weight matrix by the initial static features to obtain the weighted static features.

[0149] Step b34, perform local feature extraction on the weighted static features based on a preset number of multi-size local feature extraction modules to obtain local static features at various scales.

[0150] Specifically, for each multi-size local feature extraction module, multi-scale parallel convolution is utilized. By cascading two groups of 1x3 and 3x1 asymmetric convolution layers, the receptive field is extended to a 5x5 feature region, and local feature extraction is performed on the weighted static features to obtain local static features at various scales.

[0151] Step b35: Concatenate the local static features to generate multi-scale local features.

[0152] Specifically, concatenate the local static features at various scales to generate multi-scale local features.

[0153] Exemplarily, the extracted multi-scale initial feature M(x) can be expressed by the formula: ; where , , represent convolution operations with different receptive field sizes respectively, represents the average pooling layer, which is used to extract features of different scales of the face. is the operation of connecting these multi-scale features along the channel dimension. In addition, bottleneck structures are constructed using convolution before and after the operation to reduce the computational parameters of multi-scale feature extraction, and finally generate multi-scale local features as follows, that is: .

[0154] Step b36: Based on the global feature shrinking attention module, perform shrinking processing on the multi-scale local features to generate target local features.

[0155] Specifically, the global feature shrinking attention module first takes the absolute value of the multi-scale local feature , and then obtains a new feature map through global pooling to summarize and compress the global information of the facial features. Finally, the feature map is simplified to a one-dimensional vector as shown in the following formula: .

[0156] where represents the absolute value operation, W and H respectively represent the width and height of the feature map, will generate a feature map with C channels, and the value in each channel represents the average value of all pixels in the corresponding channel dimension feature map.

[0157] Then, the of the global information passes through two convolutions and the sigmoid function to obtain a normalized scaling parameter As shown below, the value range of the parameter is (0, 1), so that the obtained threshold will not be too large and will always be positive, that is: .

[0158] Among them, is the scaling parameter, and z is the output of the two-layer convolution. To ensure that the threshold of the shrinkage function is positive and the threshold is not too large so as to zero a large number of features, the threshold t can be expressed as: . In this way, different samples can obtain different thresholds, so that the learned high-level features can become more discriminative.

[0159] After obtaining the threshold, features with absolute values lower than the threshold are removed from the multi-scale local features , and features with absolute values greater than the threshold are shrunk towards 0. The processed features can be expressed as follows: .

[0160] After shrinking, the current irrelevant features can be set to zero, thereby alleviating the redundancy problem brought by multi-scale information and suppressing the noise irrelevant to the facial emotion recognition task, while strengthening the relationship between local features and global features.

[0161] Finally, a cross-layer data path is used to connect the original features before multi-scale feature extraction with the output of the global feature shrinkage attention module, which helps the network retain and reuse richer global features, while preventing the problem of gradient disappearance and network degradation in the deep neural network. The method is shown in the following formula: ; Among them, is the expression feature map that fuses facial local and global features, that is, the target local feature is the original facial feature map output by the previous layer, represents the local features extracted by the multi-scale local feature extraction module, represents the facial features processed by the global feature shrinkage attention module.

[0162] Step b37, input the target local feature into the next parallel branch, and so on until the last parallel branch is processed.

[0163] Specifically, input the target local feature into the next parallel branch, and so on until the last parallel branch is processed. The processing process can be referred to above and will not be elaborated here.

[0164] Step b38, perform global pooling on the results output by each parallel branch, and output the facial static features based on the fully connected layer.

[0165] Specifically, the results output by each parallel branch are globally pooled, and the facial static features are output based on the fully connected layer.

[0166] Step b4: Integrate the facial optical flow features and the facial static features to generate facial fusion features.

[0167] Specifically, the electronic device 6 can perform global average pooling on the motion amplitude M corresponding to the motion vector to obtain the motion saliency vector S move ∈R C , which represents the motion intensity of the features of each channel.

[0168] Generate the channel weight w through the fully connected layer move ∈(0,1) C , and perform weighted fusion on the facial static features and the facial optical flow features to generate facial fusion features. The specific formula is as follows: F final =wmove⊙Fstatic+(1 - w move )⊙Fflow.

[0169] Among them, Fstatic is the facial optical flow feature, and Fflow is the facial static feature.

[0170] Step b5: The gaze point extraction branch in the fourth sub - feature extraction branch extracts features from the eye movement image data and outputs the eye movement trajectory corresponding to the eye movement image data.

[0171] Specifically, the gaze point extraction branch identifies the eye movement image data to determine the eye key points in the eye movement image data. Then, based on a tracking algorithm (such as the KCF algorithm, MedianFlow algorithm, etc.), the eye key points are tracked. The tracking algorithm calculates the motion information of the eyes according to the position changes of the eye key points between adjacent frames. During the tracking process, the position information of the eye key points at each moment is recorded.

[0172] Then, according to the recorded position information of the eye key points, these points are connected to obtain the eye movement trajectory.

[0173] Step b6: Extract features from the eye movement trajectory to determine the gaze point distribution features corresponding to the eye movement image data.

[0174] Specifically, extract features from the eye movement trajectory, including basic statistical features, spatial distribution features, time - series features, and motion features.

[0175] Specifically, the basic statistical features include: Number of fixation points: Count the total number of fixation points in the eye movement image, which reflects the number of visual attention points of the subject during the observation. Average fixation time: Calculate the average value of the durations of all fixation points, which can reflect the degree of attention of the subject to different contents and the information processing speed. Fixation point density: Divide the image into several sub-regions, calculate the ratio of the number of fixation points in each sub-region to the area of the region, so as to understand the distribution density of fixation points in different regions of the image.

[0176] Regarding the spatial distribution features, they include: Central tendency: Calculate the mean value of the fixation point coordinates to obtain the central position of the fixation point distribution, which can reflect the overall visual focus of the subject when observing the image. Dispersion degree: By calculating the standard deviation or variance of the fixation point coordinates, measure the degree of dispersion of the fixation points around the central position. The larger the standard deviation or variance, the more dispersed the fixation points are, and the less concentrated the visual attention of the subject is. Spatial entropy: Used to describe the degree of disorder of the fixation point distribution. The higher the spatial entropy value, the more uniform and random the distribution of fixation points in space; the lower the entropy value, the more concentrated the fixation points are in certain specific regions.

[0177] Regarding the time series features, they include: Time interval between fixation points: Calculate the time interval between adjacent fixation points and analyze its distribution, which can understand the frequency and rhythm of the visual attention transfer of the subject. Serial correlation: By calculating the autocorrelation function or cross-correlation function, analyze the correlation of fixation points in the time series and judge whether there are periodic or trend changes.

[0178] Regarding the motion features, they include: Motion speed: According to the distance and time interval between adjacent fixation points, calculate the motion speed of the fixation points. The speed change can reflect the visual search strategy and attention allocation of the subject during the observation process. Motion direction: Determine the motion direction between each fixation point, and count the number of motions or the time proportion in different directions, which helps to analyze the visual scanning pattern of the subject, such as whether there is a preferred scanning direction.

[0179] Then, perform principal component analysis on the extracted multiple features, convert the high-dimensional feature space into a low-dimensional space, find out several principal components that have the greatest influence on the fixation point distribution features, realize the dimensionality reduction and visualization of the features, and more intuitively display the main patterns of the fixation point distribution. Combine the extracted various features into a feature vector to generate the fixation point distribution features. Comprehensively describe the fixation point distribution features corresponding to the eye movement image data for subsequent classification, recognition or correlation analysis with other data.

[0180] Step b7, the eye movement extraction branch in the fourth sub-feature extraction branch extracts features from the eye movement image data and outputs the eye movement features corresponding to the eye movement image data.

[0181] Specifically, the eye movement extraction branch calculates the movement speeds of the pupil in the horizontal and vertical directions based on the changes in the pupil positions in consecutive eye movement images. The pupil movement speed can reflect the visual search speed of the subject and the speed of attention transfer. For example, when quickly browsing an image, the pupil movement speed is usually relatively fast.

[0182] The eye movement extraction branch calculates the rotation angle of the eyeball based on the relative position changes between the pupil and the iris, as well as the movement information of the periorbital feature points. The eyeball rotation angle can help analyze the fixation direction of the subject and the changes in the visual focus, which is of great significance for understanding the subject's visual cognitive process.

[0183] The eye movement extraction branch detects the state changes of eye closure in the eye movement images and counts the number of blinks within a unit time, that is, the blink frequency. The blink frequency can reflect the fatigue level, attention concentration level, etc. of the subject. Generally speaking, when fatigued or inattentive, the blink frequency will increase.

[0184] Then, the pupil movement speed, the eyeball rotation angle, and the blink frequency are fused to form a comprehensive eye movement feature vector. A simple splicing method can be used to connect different types of feature vectors together; more complex fusion algorithms can also be used, such as feature fusion based on principal component analysis, feature fusion based on neural networks, etc., to reduce the feature dimension, remove the correlation between features, and improve the representativeness and robustness of the features.

[0185] Step b8: Incorporate the fixation point distribution feature and the eye movement feature to generate an eye movement fusion feature.

[0186] Specifically, the fourth sub-feature extraction branch incorporates the fixation point distribution feature and the eye movement feature to generate an eye movement fusion feature. A simple splicing method can be used to connect the fixation point distribution feature and the eye movement feature together; more complex fusion algorithms can also be used, such as feature fusion based on principal component analysis, feature fusion based on neural networks, etc., to reduce the feature dimension, remove the correlation between features, and improve the representativeness and robustness of the features.

[0187] Step b9: Fuse the facial fusion feature and the eye movement fusion feature to generate a visual modality feature.

[0188] Specifically, the facial fusion feature and the eye movement fusion feature are fused to generate a visual modality feature. A simple splicing method can be used to connect the facial fusion feature and the eye movement fusion feature together; more complex fusion algorithms can also be used, such as feature fusion based on principal component analysis, feature fusion based on neural networks, etc., to reduce the feature dimension, remove the correlation between features, and improve the representativeness and robustness of the features.

[0189] Step S103: Perform fusion processing on the temporal modal features and visual modal features to generate target fusion features.

[0190] In an alternative embodiment of the present application, the above step S103 may include the following steps: Step S1031: Use the visual modal features as the first query matrix, and the temporal modal features as the first key matrix and the first value matrix.

[0191] Specifically, the electronic device 6 uses the visual modal features as the first query matrix, and the temporal modal features as the first key matrix and the first value matrix.

[0192] Step S1032: Calculate the first dependence weight of the visual modal features on the temporal modal features.

[0193] Specifically, by calculating the dot product of the first query matrix Q and the first key matrix K, the similarity between the visual modal features and the temporal modal features is measured. The specific calculation formula is: sim = Q • K^T. Where sim i,j represents the similarity score between the i-th visual modal feature and the j-th temporal modal feature.

[0194] To avoid unstable gradients caused by too large dot product results, the similarity scores are usually scaled. The scaling factor is , that is: .

[0195] Then, apply the Softmax function to the scaled similarity scores to convert them into a probability distribution, thereby obtaining the first dependence weight matrix W. The calculation formula of the Softmax function is: .

[0196] Step S1033: Use the temporal modal features as the second query matrix, and the visual modal features as the second key matrix and the second value matrix.

[0197] Specifically, use the temporal modal features as the second query matrix, and the visual modal features as the second key matrix and the second value matrix.

[0198] Step S1034: Calculate the second dependence weight of the temporal modal features on the visual modal features.

[0199] Specifically, referring to the above calculation method for calculating the first dependence weight, calculate the second dependence weight, which will not be elaborated here.

[0200] Step S1035: Multiply the first dependence weight by the temporal modal features to obtain the target temporal features.

[0201] Specifically, the electronic device 6 multiplies the temporal modal features by the first dependence weight to obtain the target temporal features.

[0202] Step S1036: Multiply the visual modal features by the second dependence weight to obtain the target visual features.

[0203] Specifically, the electronic device 6 multiplies the visual modal features by the second dependence weight to obtain the target visual features.

[0204] Step S1037: Combine the target temporal features and the target visual features to generate the target fusion features.

[0205] Specifically, the electronic device 6 combines the target temporal features and the target visual features to generate the target fusion features.

[0206] Step S104: Output the current workload level corresponding to the target user based on the target fusion features.

[0207] Specifically, the preset workload recognition model outputs the current workload level corresponding to the target pilot based on the target fusion features.

[0208] The flight training system provided by the embodiments of the present application inputs physiological time-series data and electroencephalogram time-series data into a first feature extraction network to generate time-series modal features, thereby ensuring the accuracy of the generated time-series modal features. The facial image data and eye movement image data are input into a second feature extraction network to generate visual modal features, ensuring the accuracy of the generated visual modal features. In addition, inputting data into different networks according to modalities avoids the problem of increased complexity caused by the mixing of multiple types of data. The characteristics and processing methods of different modal data vary greatly. Separated processing allows each network to focus on the feature extraction of specific types of data, reducing the burden of network training and calculation and improving data processing efficiency. At the same time, the mutual interference between different modal data is reduced, making feature extraction more accurate and efficient. The visual modal features are used as the first query matrix, and the time-series modal features are used as the first key matrix and the first value matrix; calculate the first dependence weight of the visual modal features on the time-series modal features, ensuring the accuracy of the calculated first dependence weight. The time-series modal features are used as the second query matrix, and the visual modal features are used as the second key matrix and the second value matrix; calculate the second dependence weight of the time-series modal features on the visual modal features, ensuring the accuracy of the calculated second dependence weight. Multiply the time-series modal features by the first dependence weight to obtain target time-series features, and multiply the visual modal features by the second dependence weight to obtain target visual features. This weighting operation can strengthen the part with a higher correlation with the other modal features, making the target features better reflect the association between different modalities. In addition, through the weighting operation, the part with a lower correlation with the other modal features is weakened, reducing the redundant information in the features and improving the quality and efficiency of the features. Merge the target time-series features and the target visual features to generate target fusion features, completing the deep fusion of the visual modality and the time-series modality. The fused features integrate the advantageous information of the two modalities and can more comprehensively and accurately reflect the working state of the pilot. For example, combining visual information such as facial expressions and eye movements and time-series information such as physiological signals and electroencephalogram signals can evaluate the pilot's cognitive load, fatigue level, etc. from multiple perspectives. Based on the target fusion features, output the current workload level corresponding to the target user. The target fusion features contain information from multiple aspects and can more accurately reflect the actual workload of the pilot. Compared with evaluating based only on single-modal features, this method can reduce the probability of misjudgment and missed judgment and provide more reliable workload evaluation results for pilots, flight management personnel, and related systems.

[0209] In an alternative embodiment of the present application, as Figure 5 shown, the above "generating a backup virtual driving scenario based on a preset training instruction, operation action data, button action data, and parameter setting data" may include the following steps: Step S201, generating an initial virtual driving scenario based on a preset training instruction.

[0210] In an alternative embodiment, the above step S201 may include the following steps: Step S2011, perform semantic recognition on the preset training instruction, and extract key information from the preset training instruction.

[0211] Among them, the key information includes at least one of the flight mission type, flight area, and meteorological conditions.

[0212] Specifically, the electronic device 6 performs preliminary cleaning on the preset training instruction text, removing some meaningless characters, punctuation marks, spaces, etc. For example, redundant spaces, line breaks, etc. in the instruction are removed to make the text more concise.

[0213] Then, the continuous text sequence is segmented into meaningful words or phrases. For Chinese instructions, tools such as Jieba segmentation can be used; for English instructions, simple segmentation can be performed according to spaces. For example, "Perform an air patrol mission in North China under thunderstorm weather" will be segmented into "in", "thunderstorm weather", "under", "in", "North China", "perform", "air patrol mission".

[0214] Next, label the part-of-speech of each segmented word, such as noun, verb, adjective, etc. This helps with subsequent syntactic analysis and semantic understanding. For example, "thunderstorm weather" is labeled as a noun, and "perform" is labeled as a verb. Clearly define the types of entities to be recognized, which mainly include flight mission type, flight area, and meteorological conditions in the flight training scenario.

[0215] Finally, the electronic device 6 trains a named entity recognition model using machine learning or deep learning methods. Common methods include conditional random field (CRF), long short-term memory network (LSTM) combined with conditional random field (LSTM-CRF), fine-tuning based on pre-trained language models (such as BERT), etc.

[0216] Input the preprocessed text into the trained named entity recognition model, and the model will output the types and positions of each entity in the text. For example, for the instruction "Perform an aerial mapping mission in the western region under cloudy weather", the model will recognize "cloudy weather" as the meteorological condition, "western region" as the flight area, and "aerial mapping mission" as the flight mission type.

[0217] In addition, the electronic device 6 analyzes the grammatical structure of the sentence to determine the dependency relationship between words. For example, using a dependency parsing tool, it is analyzed that there is an object-predicate relationship between "execute" and "air patrol mission". Combining the results of named entity recognition and syntactic analysis, the semantics of the sentence is further understood. For example, according to the words in the instruction and the relationships between them, it is determined that the core semantics of the instruction is to execute a specific flight mission under specific meteorological conditions and flight areas.

[0218] Finally, based on the above analysis results, the electronic device 6 filters out key information such as flight mission type, flight area, and meteorological conditions from the text. If certain types of key information are not explicitly mentioned in the instruction, the corresponding information is empty. For example, in the instruction "Conduct low-altitude flight mission", the flight mission type is "low-altitude flight mission", and the flight area and meteorological condition information are empty.

[0219] Step S2012: Using each key information as an initial node, the dependency relationship between each key information as an initial edge, and setting attribute information for each initial node, an initial graph structure is constructed.

[0220] Specifically, the key information such as flight mission type, flight area, and meteorological conditions extracted from the preset training instructions are the initial nodes. For example, the flight mission type may be "air patrol", "aerial mapping", etc.; the flight area may be "North China region", "Western region", etc.; the meteorological conditions may be "thunderstorm weather", "cloudy weather", etc. Each specific key information will be used as an independent initial node in the graph structure.

[0221] Then, the initial edges are constructed according to the dependency relationship between each key information. Exemplarily, different flight mission types may have specific requirements or restrictions on the flight area. For example, when executing a maritime patrol mission, it is usually carried out in coastal areas or specific sea areas. Therefore, there is a dependency relationship between the flight mission type node of "maritime patrol" and the corresponding coastal flight area node, and an edge can be set to connect them. Some flight missions can only be carried out under specific meteorological conditions, or certain meteorological conditions will affect the flight mission. For example, when the meteorological condition is "fog", it may limit some flight missions that require visual operation, such as "low-altitude flight". At this time, there is a dependency relationship between the meteorological condition node of "fog" and the flight mission type node of "low-altitude flight", and this relationship is represented by an edge. Different flight areas may have their common meteorological conditions. For example, when flying in tropical regions, it is more likely to encounter meteorological conditions such as heavy rain and typhoons. Therefore, there is a certain dependency relationship between the flight area node of "tropical region" and the meteorological condition nodes of "heavy rain", "typhoon", etc., and they can be connected by edges.

[0222] Next, attribute information is set for each initial node.

[0223] Among them, the node attributes of the flight mission type: Attributes such as the difficulty level of the mission, the required flight skills, and the mission objectives can be set. For example, for the "in-air refueling" mission, its difficulty level is relatively high, requiring the pilot to have precise operation skills and rich experience, and the mission objective is to refuel other aircraft.

[0224] The node attributes of the flight area: Attributes such as the geographical features of the area, airspace restrictions, and navigation facilities can be set. For example, for the "mountainous area" flight area, the geographical features are complex terrain, possibly including high mountains, canyons, etc.; airspace restrictions may include altitude restrictions, military restricted areas, etc.; navigation facilities may be relatively few, and the pilot needs to rely on other methods for navigation.

[0225] The node attributes of the meteorological conditions: Attributes such as the intensity, duration, and the degree of impact on flight of the meteorological conditions can be set. For example, for "thunderstorm weather", the intensity can be divided into different levels such as weak, medium, and strong, the duration may range from several hours to several days, and the degree of impact on flight is relatively large, seriously affecting flight safety, and may cause aircraft bumps, communication interruptions, etc.

[0226] Connect all the initial nodes (flight mission type, flight area, meteorological conditions) with edges according to their dependency relationships to form a graph structure. In this graph, the initial nodes represent key information, the edges represent the dependency relationships between the key information, and the attribute information of the nodes further enriches the description of the key information. For example, for "performing an air patrol mission in North China under thunderstorm weather", a graph structure can be constructed where there are edges connecting the "thunderstorm weather" node and the "North China" node, the "thunderstorm weather" node and the "air patrol" node, and the "North China" node and the "air patrol" node, and each node has corresponding attribute information to describe its own characteristics in detail. Through such an initial graph structure, the relationships between the key information in the preset training instructions can be represented more clearly, providing a basis for subsequent analysis and processing.

[0227] Step S2013, perform a convolution operation on the initial graph structure to obtain the initial node features corresponding to each initial node in the graph structure.

[0228] Specifically, before performing the convolution operation, it is necessary to clarify the mathematical representation of the initial graph structure. Generally speaking, the graph G=(V,E) consists of the initial node set V and the edge set E. For each initial node vi∈V, there is its corresponding feature vector xi. Combine the feature vectors of all initial nodes to form the initial node feature matrix X∈R N×D , where N is the number of initial nodes, and D is the dimension of the initial node features. At the same time, the connection relationship of the initial graph structure can be represented by the adjacency matrix A∈R N×NIt is represented that if there is an edge between the initial node i and the initial node j, then Aij = 1; otherwise, Aij = 0.

[0229] The operation steps of graph convolution include the following: 1. Preprocessing of the adjacency matrix: To better aggregate the information of adjacent nodes, some preprocessing is usually performed on the adjacency matrix. A common method is to normalize the adjacency matrix, such as the calculation of the symmetric normalized Laplacian matrix: ; ; .

[0230] where I N is the N×N identity matrix, is 's degree matrix, is the adjacency matrix after normalization processing.

[0231] Convolution operation process: Assume that we have a graph convolution layer whose input is the node feature matrix X, and the weight matrix of this layer is (D′ is the dimension of the output feature). The formula for graph convolution operation is as follows: . Among them, σ is the activation function, such as the ReLU (Rectified Linear Unit) function, and H ∈ R N×D′ is the new node feature matrix obtained after convolution operation. In this operation process, realizes the aggregation of node features, that is, each initial node comprehensively considers the feature information of its adjacent nodes; multiplying by the weight matrix W performs a linear transformation on the aggregated features; finally, introducing non-linearity through the activation function to obtain the updated node feature representation.

[0232] Optionally, in order to further extract higher-level node features, multiple graph convolution layers are usually stacked. The output of each layer is used as the input of the next layer. Through multiple layers of convolution operations, the nodes can capture the information of nodes at farther distances, thereby mining more complex spatial associations in the graph structure. For example, after the first layer of graph convolution operation, the node features initially aggregate the information of directly adjacent nodes; after the second layer of graph convolution operation, the node features fuse the information of the adjacent nodes of the adjacent nodes, and so on.

[0233] Step S2014, generating an initial virtual driving scene based on each initial node feature.

[0234] Specifically, the electronic device 6 can search the storage space for an initial virtual driving scene that matches the initial node features according to each initial node feature.

[0235] Step S202: Based on the initial virtual driving scenario, generate a backup virtual driving field based on operation action data, button action data, and parameter setting data.

[0236] Specifically, the above step S202 may include the following steps: Step S2021: Abstract operation action data, button action data, and parameter setting data into update nodes, and establish update edge connections with each initial node in the initial graph structure.

[0237] Specifically, the operation action data includes information such as the execution order corresponding to each operation action and the operation force corresponding to each operation action. For this data, first parse and classify it. For example, specific operation actions such as "pull the lever to take off", "adjust the flap angle", and "step on the brake" are regarded as independent operation action types. Then, create an update node for each operation action type, and record the relevant attributes of the operation action in the node, such as the name of the operation action, the operation stage it belongs to (takeoff, cruise, landing, etc.), and the range of operation force. For example, for the "pull the lever to take off" node, the name can be recorded as "pull the lever to take off", the operation stage as "takeoff", and the operation force range may be within a certain angle range, etc. Through this abstraction method, the specific operation actions are transformed into nodes in the graph structure, facilitating subsequent analysis and processing.

[0238] The button action data is mainly the button action information input by the target user on the button control console 3 based on preset training instructions. Similarly, classify and organize these button actions. For example, button actions such as "turn on the autopilot mode", "confirm the navigation system settings", and "operate the light switch" are abstracted into corresponding update nodes respectively. Each button action update node records attributes such as the name of the button, the function description, and the system or device operated. Taking the "turn on the autopilot mode" button action node as an example, the name can be recorded as "turn on the autopilot mode", the function description as turning on the autopilot system of the aircraft, and the device operated as the autopilot control system of the aircraft, etc., thus transforming the button action data into valid nodes in the graph structure.

[0239] The parameter setting data is the data set by the target user for flight parameters, such as the settings of parameters like flight altitude, speed, heading, engine power, etc. For these parameter setting data, they are classified according to the type of parameters, and each parameter type can be abstracted into an update node. For example, update nodes such as "flight altitude setting", "speed setting", "engine power setting", etc. Each node records attributes such as the name of the parameter, the set value range, and the impact on flight. For example, for the "flight altitude setting" node, its name can be recorded as "flight altitude setting", the value range depends on the actual flight situation, and the impacts on flight include information such as affecting the aerodynamic performance of the aircraft and fuel consumption, realizing the conversion from parameter setting data to update nodes.

[0240] When establishing the update edge connection, the electronic device 6 needs to determine the association relationships between the operation action data, button action data, parameter setting data and each initial node (flight task type, flight area, meteorological conditions, etc.) in the initial graph structure as the connection basis. For example, the operation action of "pulling the lever to take off" has a direct association with the flight task type of "takeoff training" because "pulling the lever to take off" is a specific operation action in the "takeoff training" task; the button action of "enabling the autopilot mode" may be related to the "cruise task" in the flight task type because the autopilot mode is often used during the cruise phase; the parameter setting of "flight altitude setting" may be related to the flight area and meteorological conditions, and different flight areas and meteorological conditions may have different requirements and restrictions on the flight altitude.

[0241] Then, the electronic device 6 establishes update edge connections between the update nodes and the initial nodes according to the determined association relationships. If an operation action is associated with a certain flight task type, an edge is added between the corresponding operation action update node and the flight task type initial node; if a button action is related to the flight operation under certain meteorological conditions, an edge connection is established between the button action update node and the corresponding meteorological condition initial node; for the parameter setting data, if it is related to both the flight area and the flight task type, edge connections are respectively established between the parameter setting update node and the flight area initial node and the flight task type initial node. The direction of the edge can be determined according to the specific association logic. For example, from the operation action node to the flight task type node, indicating that the operation action is for completing a specific flight task; from the parameter setting node to the flight area node, indicating that the parameter setting is for the requirements of a specific flight area, etc.

[0242] To more accurately describe the relationship between updated nodes and initial nodes, the electronic device 6 sets corresponding attributes for each updated edge. The attributes can include the strength of association, the degree of influence, etc. For example, for the edge between the "pull the lever to take off" operation action node and the "takeoff training" flight mission type node, the association strength can be set to high because "pull the lever to take off" is a very crucial operation action in the "takeoff training" mission; for the edge between the "autopilot mode on" button action node and the "cruise mission" flight mission type node, the degree of influence can be set to medium, indicating that the autopilot mode plays a certain role in the cruise mission but is not the only operation method. By setting the attributes of the edges, the graph structure can more clearly reflect the complex relationships between the nodes, providing richer information for subsequent analysis and processing.

[0243] Step S2022, according to the association relationships among the operation action data, button action data, parameter setting data, and preset training instructions, determine the weights and directions of each updated edge, and update the attribute information of each updated node to generate a target graph structure.

[0244] Specifically, the electronic device 6 determines the internal connections among the operation action data (such as operation sequence, force), button action data (button operation situation), parameter setting data (settings of flight parameters), and preset training instructions. For example, if the preset training instruction is to execute a takeoff mission, then operation action data such as pushing the throttle and pulling the lever is closely related to it.

[0245] Determine the weights of the updated edges based on the tightness of the above-mentioned association relationships. If an operation action is a key step to complete the training instruction and is frequently executed, then the weight of the updated edge between it and the relevant node is high. For example, in takeoff training, the weight of the edge between the "push the throttle" action and the "takeoff training" node is relatively large.

[0246] Determine the directions of the updated edges according to the causal or influence relationships among the data. Operation actions are usually for achieving the training instruction tasks, so the edges of the operation action updated nodes generally point to the initial nodes of the flight mission types. For example, the edge of the "pull the lever" action node points to the "takeoff training" node.

[0247] Update the attributes of the updated nodes such as operation actions, button actions, and parameter settings according to their associations with the preset training instructions. For example, the operation action nodes can update the execution conditions and expected effects; the button action nodes update the function descriptions and applicable scenarios; the parameter setting nodes update the value ranges and impacts on flight performance, etc.

[0248] Integrate the determined weights and directions of the updated edges, as well as the updated node attribute information, into the graph to form a target graph structure. This structure clearly presents the relationships between various types of data and the preset training instructions, providing a more accurate basis for subsequent flight training analysis and virtual scene generation.

[0249] Step S2023, perform a convolution operation on the target graph structure to obtain the target node features corresponding to each target node in the graph structure.

[0250] Among them, the target nodes include each initial node and each updated node.

[0251] Specifically, during the operation, for each target node, first preprocess the adjacency matrix, such as normalization and other operations, to better aggregate adjacent node information. Then, the electronic device 6 performs an operation on the target node feature matrix corresponding to each target node, the processed adjacency matrix, and the weight matrix, and introduces non-linearity through an activation function to obtain a new representation of the target node features. In this process, each target node comprehensively considers its own features and the feature influence of surrounding neighbor nodes.

[0252] After the convolution operation, each target node (including the initial node and the updated node) in the target graph structure has obtained an updated feature representation, that is, the target node features. These features integrate the original attribute information of the node itself and the relevant information transmitted by the nodes connected to it, and can more comprehensively reflect the role and context relationship of the node in the entire graph structure. For example, the target node features of the operation action update node not only contain the information of the operation action itself, but also integrate information such as the flight mission and area related to the operation, providing a richer and more valuable data basis for subsequent analysis and applications (such as generating virtual driving scenarios) based on these features.

[0253] Step S2024, input each target node feature matrix into a preset recurrent neural network at each time step to obtain the current target state features.

[0254] Specifically, the target node feature matrix is passed as input to the input layer of the preset recurrent neural network. At the same time, the state of the hidden layer is usually initialized to a zero vector or a random value.

[0255] For each time step t, the input layer passes the target node feature matrix X t of the current time step to the hidden layer. The hidden layer will calculate the hidden state h t of the current time step according to the current input X t . The specific calculation process is as follows: For a general RNN, the calculation method is h t =σ(W xh X t +W hh h t-1 +b h ), where σ is an activation function, such as tanh or ReLU, W xh and W hhis the weight matrix, b h is the bias term.

[0256] For LSTM, the calculation process is relatively complex, including the input gate i t , the forget gate f t , the output gate o t and the update of the cell state c t . The specific formulas are as follows: i t =σ(W xi X t +W hi h t-1 +b i ) f t =σ(W xf X t +W hf h t-1 +b f ) c t =f t ⊙c t-1 +i t ⊙tanh(W xc X t +W hc h t-1 +b c ) o t =σ(W xo X t +W ho h t-1 +b o ) h t =ot⊙tanh(c t ) The calculation method of GRU is similar to that of LSTM, but relatively simple. It combines the input gate and the forget gate into an update gate and directly updates the hidden state.

[0257] After the hidden layer calculates the hidden state ht at the current time step, it will be passed to the output layer. The output layer calculates the output yt according to the hidden state ht, the output weight matrix Why, and the bias term by, that is, yt = Whyht + by. In some cases, the output may be the current target state feature; in other cases, further processing or transformation may be required for the output to obtain the current target state feature.

[0258] Step S2025, based on the current target state feature, adjust the initial virtual driving scene to generate an alternative virtual driving scene.

[0259] Specifically, establish associations between the current target state features and various elements in the initial virtual driving scenario. For example, if the features show that the trainer's control of the aircraft speed is not stable during the operation, then associate this information with the aircraft speed-related elements in the scenario (such as the speed display on the instrument panel, speed control devices, etc.) and the flight environment elements (such as factors affecting speed like wind speed and air currents) for correlation analysis. Make adjustment decisions for the initial virtual driving scenario based on the analysis results. If it is found that the current operation actions do not match the requirements of the flight mission, it may be necessary to modify the task prompt information, target location, etc. in the scenario; if the current weather condition settings do not match the trainer's operation performance, appropriately adjust the weather conditions, such as changing from sunny to cloudy to increase a certain level of flight difficulty. For features related to parameter settings, if it is found that certain parameter settings are unreasonable, correspondingly adjust the parameter settings of the aircraft in the scenario, such as engine power, flap angle, etc. According to the adjustment decisions, make specific modifications and updates to the initial virtual driving scenario. This includes adjusting the position, attitude, and attributes of the 3D models (such as aircraft, buildings, etc.) in the scenario; changing the environmental effects (such as lighting, weather special effects, etc.); optimizing the interactive elements (such as instrument panel display, button feedback, etc.). Through these adjustments, generate a backup virtual driving scenario that can better reflect the current training state and meet the training requirements, providing a more targeted and effective training environment for the trainer.

[0260] The flight training system provided by the embodiments of the present application enables the electronic device 6 to perform semantic recognition on preset training instructions and extract key information from the preset training instructions, ensuring the accuracy of the extracted key information. These key information define the core direction for subsequent scenario construction, ensuring that the generated initial virtual driving scenario highly matches the training objectives and avoiding deviations and resource waste in scenario construction. Using each key information as an initial node, the dependency relationships between the key information as initial edges, and setting attribute information for each initial node, an initial graph structure is constructed. The initial graph structure can clearly display the connections between the key information. Through the initial graph structure, the electronic device 6 can systematically sort out these complex relationships and ensure that the elements in the scenario are coordinated with each other and conform to actual logic when generating the initial virtual driving scenario. Performing a convolution operation on the initial graph structure to obtain the initial node features corresponding to each initial node in the graph structure. Thus, it is possible to effectively mine the feature information of each initial node in the graph. The convolution operation generates more representative initial node features for each node by aggregating the information of the node and its adjacent nodes. These features not only contain the key information of the node itself but also incorporate the comprehensive information after association with other nodes. Based on the initial node features, an initial virtual driving scenario is generated. This feature-based scenario generation method makes the initial virtual driving scenario richer and more realistic in details, providing a more comprehensive and in-depth training environment for the driver and helping the driver better understand and master the operation key points of different flight tasks under various conditions. The electronic device 6 abstracts operation action data, button action data, and parameter setting data as updated nodes and establishes update edge connections with each initial node in the initial graph structure. This operation breaks the relatively static situation after the initial scenario construction and integrates the real-time operation data of the driver during the training process into the graph structure. This dynamic data fusion method enables the virtual driving scenario to no longer be limited to a fixed mode under preset instructions but can be adjusted in real time according to the driver's operations, enhancing the adaptability of the scenario to the training process, making the training closer to the constantly changing operation scenarios in actual flight, and improving the driver's response ability in actual operations. According to the association relationships between the operation action data, button action data, parameter setting data, and the preset training instructions, determine the weights and directions of each update edge and update the attribute information of each updated node to generate a target graph structure. The setting of the weights reflects the tightness of the association between the data, and the direction clarifies the information transmission logic. In this way, the target graph structure can more accurately sort out the logical relationships of various types of information during the training process and avoid unreasonable element combinations or operation contradictions in the scenario. When generating a backup virtual driving scenario, based on the optimized target graph structure, the elements and operation processes in the scenario will more conform to the actual flight logic, further enhancing the authenticity and credibility of the scenario and helping the driver form a correct flight operation logical thinking. Performing a convolution operation on the target graph structure to obtain the target node features corresponding to each target node in the graph structure.Convolution operation can fully explore the deep features of each node and its associated information in the target graph structure. Compared with the feature extraction of the initial graph structure, this operation integrates real-time operation data, making the extracted features more rich and comprehensive. Based on these more detailed and dynamic target node features, when generating alternative virtual driving scenarios, it is possible to depict in more detail the details such as aircraft movement, instrument display, and environmental changes in the scenario, making the scenario more realistically simulate the complex situations in actual flight, providing a more immersive training environment for the driver, and helping the driver to more deeply understand the relationship between operations and flight states. The feature matrices of each target node are input into a preset recurrent neural network at each time step to obtain the current target state features. Based on the current target state features, the initial virtual driving scenario is adjusted to generate an alternative virtual driving scenario. The recurrent neural network is good at processing time series data and can analyze the dynamic change trends and rules of data during the training process. Based on these analysis results, the system can adjust the initial virtual driving scenario targeted, such as generating scenario elements related to predicted operations in advance or adjusting the scenario difficulty. For drivers with proficient operations, complex meteorological conditions or equipment failures are added in a timely manner; for drivers with insufficient proficiency, more operation tips and auxiliary information are provided. This precise scenario adjustment mechanism enables the alternative virtual driving scenario to be dynamically adapted according to the driver's real-time performance and training needs, realizing personalized training, improving the pertinence and effectiveness of training, and helping the driver to improve flight skills faster.

[0261] In an optional implementation manner of the present application, as Figure 6 shown, the above "adjusting the complexity of the alternative initial virtual driving scenario according to the target cognitive load level to generate a target virtual driving scenario" may include the following steps: Step S301, inputting physiological time series data, electroencephalogram time series data, eye movement image data, facial image data, and the target cognitive load level into a meta-learner.

[0262] Specifically, the electronic device 6 inputs physiological time series data, electroencephalogram time series data, eye movement image data, facial image data, and the target cognitive load level into the meta-learner.

[0263] Step S302, the meta-learner obtains an initial cognitive load-scenario response model.

[0264] Among them, the initial cognitive load-scenario response model includes an input layer, a hidden layer, and an output layer; the input layer is responsible for receiving data, and the number of its neurons is set according to the data feature dimension; the hidden layer is composed of multiple layers of neurons and processes the data through complex connection weights; the output layer outputs the final result.

[0265] Specifically, the meta-learner searches for an initial cognitive load-scenario response model from the storage space, wherein the initial cognitive load-scenario response model is trained based on the training data.

[0266] Step S303: The meta-learner adjusts the initial cognitive load-scenario response model based on the physiological time series data, the EEG time series data, the eye movement image data, and the facial image data to generate a target cognitive load-scenario response model corresponding to the target user.

[0267] Specifically, the meta-learner first extracts features from the input physiological time series data, EEG time series data, eye movement image data, and facial image data, and fuses these features to generate a comprehensive feature vector. The fusion method can be simple splicing or weighted combination through a specific algorithm to form a comprehensive feature vector. This comprehensive feature vector contains information about the user in multiple aspects such as physiology, psychology, and behavior, and can more comprehensively reflect the user's cognitive state.

[0268] The meta-learner analyzes the unique features and patterns of the target user based on the comprehensive feature vector. Then, by adjusting parameters such as the connection weights of neurons in the model, the output of the model can better fit the relationship between the cognitive load of the target user and the scenario. The specific adjustment method can use the gradient descent algorithm to calculate the error between the model output and the true value, back propagate the gradient, and then update the connection weight. In addition, the meta-learner can also use some optimization techniques, such as adaptive learning rate adjustment and regularization, to improve the training efficiency and generalization ability of the model.

[0269] The meta-learner does not adjust the model in one go, but continuously optimizes the model through multiple iterations. In each iteration, the meta-learner adjusts the model's parameters based on the new input data and the model's output, so that the model gradually adapts to the characteristics of the target user. As the number of iterations increases, the model fits the target user's cognitive load and scene relationship better and better.

[0270] The target cognitive load-scenario response model generated after adjustment by the meta-learner can more accurately output appropriate results based on the state of the target user.

[0271] Step S304: input the target cognitive load level into the target cognitive load-scenario response model, and output the scenario complexity adjustment parameter.

[0272] Among them, the scenario complexity adjustment parameters include fault type adjustment, fault frequency adjustment, and environment complexity adjustment.

[0273] Specifically, the target cognitive load level is input into the target cognitive load - scenario response model. After receiving the target cognitive load level, each layer of neurons (including the input layer, hidden layer, and output layer) inside the target cognitive load - scenario response model will perform a series of calculations and analyses on the input data according to the learned parameters and connection weights. The hidden layer will extract the features of the input data and process these features through complex operations to explore the potential relationship between the cognitive load level and the scenario complexity.

[0274] After the processing and analysis of the model, the scenario complexity adjustment parameter is finally output by the output layer.

[0275] Step S305: Adjust the complexity of the standby initial virtual driving scenario based on the scenario complexity adjustment parameter to generate the target virtual driving scenario.

[0276] Specifically, the electronic device 6 can adjust the parameter according to the fault type and add or replace the corresponding fault elements in the standby initial virtual driving scenario. For example, if the parameter requires increasing the electrical system fault, fault manifestations will be set in the electrical system part of the virtual aircraft, such as the dashboard indicator lights flashing and some devices malfunctioning.

[0277] The electronic device 6 can adjust the parameter according to the fault occurrence frequency to control the rhythm of the fault occurrence. If the parameter is set with a high fault occurrence frequency, then during the virtual driving process, various faults will occur more frequently, increasing the difficulty and challenge of training; on the contrary, the number of fault occurrences will be reduced, lowering the complexity of the scenario.

[0278] The electronic device 6 can adjust the parameter according to the environmental complexity to change the environmental settings in the scenario. When the parameter indicates increasing the environmental complexity, the weather may be adjusted to a bad state, such as heavy rain, strong wind, etc., and at the same time, the terrain complexity is increased, such as more obstacles and rugged terrain; if the environmental complexity is to be reduced, the weather will be improved and the terrain will be more flat and open.

[0279] By adjusting the standby virtual driving scenario in terms of fault type, fault occurrence frequency, and environmental complexity, the target virtual driving scenario is finally generated. This scenario can closely match the current cognitive load level of the target user, providing a training environment that is both challenging and does not cause excessive cognitive pressure, which helps to improve the effect of virtual driving training and the user's learning experience.

[0280] The flight training system and the electronic device 6 provided in the embodiments of the present application are further configured to input the physiological time series data, electroencephalogram time series data, eye movement image data, facial image data, and the target cognitive load level into the meta-learner; the meta-learner obtains an initial cognitive load-scenario response model. The meta-learner adjusts the initial cognitive load-scenario response model based on the physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data to generate a target cognitive load-scenario response model corresponding to the target user, ensuring that the generated target cognitive load-scenario response model matches the target user. Input the target cognitive load level into the target cognitive load-scenario response model to output the scene complexity adjustment parameter. This ensures the accuracy of the output scene complexity adjustment parameter. These parameters are generated based on the precise analysis of the user's state and can effectively balance the training difficulty and the user's tolerance. Adjust the complexity of the standby initial virtual driving scene based on the scene complexity adjustment parameter to generate a target virtual driving scene. During the adjustment process, the system adjusts the parameters according to the type of fault, adds or replaces corresponding fault simulations in the scene, such as engine failure, instrument malfunction, etc.; adjusts the parameters according to the frequency of fault occurrence to control the interval time and number of fault occurrences, creating a reasonable training rhythm; adjusts the environmental elements in the scene through the environmental complexity adjustment parameter, such as changing the weather conditions, increasing the number of obstacles, etc. After such adjustment, the target virtual driving scene can closely match the user's current cognitive load and training needs, providing the most suitable training environment for the user, avoiding the user's frustration and fatigue caused by excessive training difficulty, and preventing the training from being too simple to achieve the improvement effect, thereby effectively improving the training quality and efficiency and helping the user better master the flight skills.

[0281] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A flight training system, characterized in that, The flight training system includes a multi-axis gravity seat, a flight joystick, a button control console, a parameter setting interface, a dynamic scene display device, and an electronic device; wherein, the multi-axis gravity seat, the flight joystick, and the dynamic scene display device are all communicatively connected to the electronic device; wherein: The flight joystick is used to receive operation action data input by a target user based on a preset training instruction, and the operation action data includes an execution sequence corresponding to each operation action and an operation force corresponding to each operation action; The button control console is used to receive button action data input by the target user based on the preset training instruction; The electronic device is used to receive parameter setting data input by the target user for setting flight parameters based on the preset training instruction; The multi-axis gravity seat is used to perform six-degree-of-freedom motion based on the operation action data, the button action data, and the parameter setting data, so as to simulate the flight attitude of an aircraft; The electronic device is further used to generate a target virtual driving scene based on the preset training instruction, the operation action data, the button action data, and the parameter setting data, and control the dynamic scene display device to display the target virtual driving scene.

2. The flight training system according to claim 1, wherein The flight training system further includes an air flow control system, and the air flow control system includes a plurality of air flow valves and air flow pressure sensors; the air flow control system is communicatively connected to the electronic device, wherein: The electronic device is used to determine external environment data corresponding to the preset training instruction based on the preset training instruction; and determine a target air flow direction and a target air flow speed corresponding to the aircraft based on the external environment data and the flight attitude; The air flow control system is used to detect the current air flow direction and the current air flow speed based on each of the air flow pressure sensors; and control the opening or closing of each of the air flow valves according to the difference between the target air flow direction and the target air flow speed and the current air flow direction and the current air flow speed.

3. The flight training system according to claim 1, wherein The multi-axis gravity seat includes a superconducting magnetic levitation device, a six-degree-of-freedom electric seat, and an adjustable seat belt array; wherein: The six-degree-of-freedom electric seat is installed on the superconducting magnetic levitation device, and the six-degree-of-freedom electric seat is suspended above the superconducting magnetic levitation device through the Meissner effect generated by superconducting materials in a preset temperature environment; The six-degree-of-freedom electric seat is configured with a linear drive unit composed of multiple permanent magnets and electromagnetic coils to complete six-degree-of-freedom motion; The adjustable seat belt array is connected to the six-degree-of-freedom electric seat through a plurality of independent electric tightening devices, and adjusts the pressure applied to the target user according to the flight attitude.

4. The flight training system according to claim 1, characterized in that The flight training system further includes a data acquisition device, and the data acquisition device is communicatively connected to the electronic device, wherein: The data acquisition device is used to acquire corresponding physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data of the target user during the training process; The electronic device is configured to input the physiological time-series data, the electroencephalogram time-series data, the eye movement image data, and the facial image data into a preset workload recognition model, and output the current workload level corresponding to the target user; The electronic device is further configured to generate a standby virtual driving scenario based on the preset training instruction, the operation action data, the button action data, and the parameter setting data; adjust the complexity of the standby virtual driving scenario according to the target cognitive load level to generate a target virtual driving scenario.

5. The flight training system according to claim 4, characterized in that, The preset workload recognition model includes a first feature recognition network and a second feature recognition network. The electronic device is configured to: Input the physiological time-series data and the electroencephalogram time-series data into the first feature extraction network to generate the time-series modal features; Input the facial image data and the eye movement image data into the second feature extraction network to generate visual modal features; Perform a fusion process on the time-series modal features and the visual modal features to generate target fusion features; Based on the target fusion features, output the current workload level corresponding to the target user.

6. The flight training system according to claim 5, characterized in that The electronic device is configured to use the visual modal features as a first query matrix, and the time-series modal features as a first key matrix and a first value matrix; Calculate the first dependence weight of the visual modal features on the time-series modal features; Use the time-series modal features as a second query matrix, and the visual modal features as a second key matrix and a second value matrix; Calculate the second dependence weight of the time-series modal features on the visual modal features; Multiply the time-series modal features by the first dependence weight to obtain target time-series features; Multiply the visual modal features by the second dependence weight to obtain target visual features; Merge the target time-series features and the target visual features to generate the target fusion features.

7. The flight training system according to claim 4, characterized in that The electronic device is configured to generate an initial virtual driving scenario based on the preset training instruction; Based on the initial virtual driving scenario, generate the standby virtual driving scenario based on the operation action data, the button action data, and the parameter setting data.

8. The flight training system according to claim 7, wherein, The electronic device is configured to perform semantic recognition on the preset training instruction and extract key information from the preset training instruction; the key information includes at least one of a flight mission type, a flight area, and meteorological conditions; Using each of the key information as an initial node, the dependence relationship between each of the key information as an initial edge, and setting attribute information for each of the initial nodes, construct an initial graph structure; Perform a convolution operation on the initial graph structure to obtain initial node features corresponding to each of the initial nodes in the graph structure; Based on each of the initial node features, generate the initial virtual driving scenario.

9. The flight training system according to claim 8, wherein The electronic device is configured to abstract the operation action data, the button action data, and the parameter setting data into update nodes, and establish update edge connections with each of the initial nodes in the initial graph structure; Determine the weights and directions of each of the updated edges according to the association relationships among the operation action data, the button action data, the parameter setting data, and the preset training instructions, and update the attribute information of each updated node to generate a target graph structure; Perform a convolution operation on the target graph structure to obtain target node features corresponding to each target node in the graph structure; The target nodes include each of the initial nodes and each of the updated nodes; Each of the target node feature matrices is input into a preset recurrent neural network at each time step to obtain a current target state feature; Based on the current target state feature, adjust the initial virtual driving scenario to generate the backup virtual driving scenario.

10. The flight training system according to claim 4, wherein, The electronic device is further configured to input the physiological time series data, the electroencephalogram time series data, the eye movement image data, the facial image data, and the target cognitive load level into a meta-learner; The meta-learner obtains an initial cognitive load-scenario response model; the initial cognitive load-scenario response model includes an input layer, a hidden layer, and an output layer; the input layer is responsible for receiving data, and the number of its neurons is set according to the data feature dimension; the hidden layer is composed of multiple layers of neurons and processes the data through complex connection weights; The output layer outputs the final result; The meta-learner adjusts the initial cognitive load-scenario response model based on the physiological time series data, the electroencephalogram time series data, the eye movement image data, and the facial image data to generate a target cognitive load-scenario response model corresponding to the target user; Input the target cognitive load level into the target cognitive load-scenario response model to output a scenario complexity adjustment parameter; the scenario complexity adjustment parameter includes fault type adjustment, fault occurrence frequency adjustment, and environmental complexity adjustment: Adjust the complexity of the backup initial virtual driving scenario based on the scenario complexity adjustment parameter to generate a target virtual driving scenario.

Images