Multi-mode man-machine interaction aircraft simulation training system

Abstract

The invention relates to the technical field of aircraft simulation training, and discloses a multi-mode man-machine interaction aircraft simulation training system. A multi-modal feature extraction module of the system generates a multi-modal feature map including a flight environment situation map and an equipment operation health map through a hierarchical feature extraction network; the incremental analysis module divides an incremental data set, updates flight state prediction model parameters through an incremental clustering algorithm and outputs an incremental state prediction result; the parameter optimization layer initializes a tabu search population according to a prediction result, and iteratively optimizes parameters of an operation rule base through a dynamic efficiency evaluation matrix; the decision generation module fuses the optimized rule base parameters and the real-time multi-modal feature map, and generates a flight operation instruction sequence based on dynamic confidence score; and the execution optimization layer analyzes the instruction sequence, satisfies constraint conditions through a constraint projection algorithm, and completes instruction sequence integer programming in combination with the operation portrait deviation. The system improves the interaction accuracy and scene adaptability of simulation training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aircraft simulation training, in particular to a multi-modal human-computer interaction aircraft simulation training system. BACKGROUND

[0002] In the field of aircraft driving training, the simulation training system is an important means to reduce training cost and improve training safety, and its performance directly affects the training effect. The traditional aircraft simulation training system relies on a single data source for training scene construction, usually only collects limited flight attitude parameters or simple device state information, and it is difficult to fully reflect the complex flight environment and cockpit interaction process. The limitation of such data acquisition leads to a large difference between the simulation scene and the actual flight scene, and the operation experience obtained by the training personnel in the simulation environment is difficult to effectively transfer to the actual flight. The feature extraction method of the existing simulation training system is relatively simple, and a single-dimensional feature analysis method is often used, which cannot deeply fuse and represent multi-source data. For example, some systems only analyze basic parameters such as flight altitude and speed, ignoring the correlation between the running state of airborne devices and cockpit interaction behavior, resulting in an incomplete description of the flight environment situation and device health state, which is difficult to support accurate training guidance. In terms of model updating, the flight state prediction model of the traditional system mostly uses a static training method, and the model parameters are not updated once they are determined, which cannot adapt to the dynamic changes of the flight scene. When encountering new flight conditions or abnormal device states, the model prediction accuracy decreases significantly, which cannot provide timely and accurate state feedback for training personnel, affecting the pertinence and effectiveness of training. The optimization mechanism of the operation rule library is also a weak link of the existing system. The parameters of the operation rule library of most systems are preset by artificial or optimized by simple algorithms, and lack dynamic adjustment capability. In complex flight scenarios, fixed rule parameters are difficult to match the real-time changes of flight state and interaction demand, resulting in a deviation between the generated operation instruction sequence and the actual operation requirement, reducing the authenticity and guidance of simulation training.

[0003] The existing system lacks fine processing of operation constraint conditions in the instruction execution optimization link, and only ensures the legality of the instruction through simple rule checking, without fully considering the operation habits and skill differences of the training personnel. This extensive instruction planning method is easy to cause the operation instruction to be unmatched with the actual operation ability of the training personnel, affecting the improvement of training effect. Therefore, it becomes a current problem to be solved to construct an aircraft simulation training system that can realize multi-modal data fusion, dynamic model updating, accurate rule optimization and adaptive instruction generation. SUMMARY

[0004] The present application aims to provide a multi-modal human-computer interaction aircraft simulation training system to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides a multi-modal human-computer interaction aircraft simulation training system, which comprises: A multi-modal data acquisition module acquires flight attitude parameters, airborne equipment state parameters and cockpit interaction data streams dynamically to generate multi-modal data streams; A multi-modal feature extraction module receives the multi-modal data streams and generates multi-modal feature maps through a hierarchical feature extraction network, wherein the multi-modal feature maps include flight environment situation maps and equipment operation health maps; An incremental analysis module divides the multi-modal feature maps into an incremental data set, updates the flight state prediction model parameters through an incremental clustering algorithm, and outputs incremental state prediction results; A parameter optimization layer initializes a tabu search population according to the incremental state prediction results, and iteratively optimizes the operation rule library parameters through a dynamic performance evaluation matrix; A decision generation module fuses the optimized operation rule library parameters and real-time multi-modal feature maps, and generates a flight operation instruction sequence based on a dynamic confidence score; An execution optimization layer analyzes the flight operation instruction sequence, satisfies the operation sequence constraint conditions through a constraint projection algorithm, and completes the instruction sequence integer programming in combination with the operation image deviation.

[0006] Preferably, the system further comprises: A feedback calibration module processes simulator response data through a multi-source filtering algorithm to generate calibrated flight environment situation deviation and cognitive behavior deviation vectors; An image management module triggers cognitive behavior image backtracking based on a preset management period, calculates the image deviation vector of the real-time cognitive behavior image and the backtracked cognitive behavior image; A data management engine retrieves the operation data deviation modulus set corresponding to the image deviation vector, and deletes historical operation data according to the variance threshold and the storage time ratio.

[0007] Preferably, the dynamic acquisition of flight attitude parameters, airborne equipment state parameters and cockpit interaction data streams comprises: According to the historical flight task type, a high-resolution focus area and a low-resolution peripheral area are divided; Based on the flight phase time sequence characteristics, the sensor sampling density is dynamically adjusted, and the high-resolution focus area adopts a dense timestamp sampling strategy; The cockpit interaction data stream is preprocessed to generate a time-aligned multi-modal data stream.

[0008] Preferably, the updating of the flight state prediction model parameters through the incremental clustering algorithm comprises: Initialize the incremental cluster center vector and the incremental clustering error threshold; Divide the incremental dataset into multiple data blocks, and calculate the cluster center offset of the current data block for each block. When the cluster center offset exceeds the incremental clustering error threshold, update the weight parameters of the flight state prediction model. Output the incremental state prediction results to the parameter optimization layer.

[0009] Preferably, the step of iteratively optimizing the operation rule base parameters through a dynamic performance evaluation matrix includes: Decode individuals in the taboo search population into combinations of operation rule base parameters; Inject the combination of operation rule base parameters into the decision generation module to obtain a dynamic performance evaluation matrix; Neighborhood solution generation and tabu table update are performed based on the dynamic performance evaluation matrix; When the maximum number of iterations is reached, the final operation rule base parameters are output.

[0010] Preferably, the generation of flight operation command sequence based on dynamic confidence score includes: The real-time multimodal feature map is input into the fuzzy inference engine to generate an initial instruction candidate set; Initialize the dynamic confidence score parameters and generate a new set of instruction candidates in each iteration; When the dynamic confidence score meets the convergence condition, the final flight operation command sequence is output to the execution optimization layer.

[0011] Preferably, satisfying the operation sequence constraints through the constraint projection algorithm includes: Set the set of operation sequence constraints and projection step size parameters; Calculate the subgradient projection direction of the flight operation command sequence; Iteratively update the instruction sequence value range within the set of operation sequence constraints; Output a sequence of flight operation commands that satisfies all operation sequence constraints.

[0012] Preferably, the process of processing the simulator response data using a multi-source filtering algorithm includes: Collect flight environment sensor feedback data and student physiological response data, construct multi-source filter state transition equations and observation equations, and fuse flight environment situational deviation and cognitive behavior deviation vectors to generate calibration data; The calibration data is fed back to the multimodal feature extraction module and the portrait management module.

[0013] Preferably, the calculation of the profile deviation vector between the real-time cognitive behavior profile and the retrospective cognitive behavior profile includes: extracting a historical task cycle parameter and a historical cognitive load parameter in the cognitive behavior image, comparing a current task cycle parameter and a current cognitive load parameter in a real-time cognitive behavior image, and generating an image deviation vector including a task cycle deviation component and a cognitive load deviation component; outputting the image deviation vector to a data management engine.

[0014] Preferably, the operation data deviation modulus set generation includes: defining a spatiotemporal feature map coordinate mapping relationship, aligning a flight environment situation map and a device operation health map; extracting an operation instruction execution delay feature value in the spatiotemporal alignment region, and calculating a deviation modulus value of the operation instruction execution delay feature value and a reference delay threshold value; aggregating the deviation modulus values of all spatiotemporal regions to form an operation data deviation modulus set.

[0015] Compared with the prior art, the present application has the following beneficial effects: The multi-modal human-computer interaction aircraft simulation training system brings many positive effects through the collaborative work of multiple modules. The multi-modal data acquisition module breaks through the limitation of traditional single data source, simultaneously acquires flight attitude parameters, airborne device state parameters and cockpit interaction data streams, and the generated multi-modal data stream can more comprehensively cover the key information in flight training, making the system's perception of the flight scene more complete, providing rich basic data for subsequent feature extraction and analysis, and making the simulation training scene more close to the actual flight environment. The multi-modal feature extraction module generates a flight environment situation map and a device operation health map using a hierarchical feature extraction network. This hierarchical processing method can deeply mine the internal features of different types of data and the correlation between data. The flight environment situation map clearly presents the environmental changes during flight, and the device operation health map intuitively reflects the operation state of the airborne device. The combination of the two feature maps makes the system's representation of the flight state more accurate, providing a clearer scene cognition basis for the training personnel, and helping the training personnel better understand the flight environment and device status. The incremental analysis module realizes the dynamic optimization of the model by dividing the incremental data set and updating the flight state prediction model parameters using an incremental clustering algorithm. As the training process progresses, the system can continuously absorb new flight data information, so that the prediction model can adapt to different flight working conditions and scene changes, and the output incremental state prediction result is more consistent with the real-time flight state, providing a reliable state reference for subsequent parameter optimization and instruction generation, so that the model always maintains effective perception ability for the flight state. The parameter optimization layer initializes the tabu search population based on the incremental state prediction result, and iteratively optimizes the operation rule library parameters through a dynamic performance evaluation matrix. The introduction of the tabu search algorithm improves the global search ability of parameter optimization, and the dynamic performance evaluation matrix ensures the adaptability of the optimization process to the real-time flight state. The operation rule library parameters optimized through iteration are more reasonable and can better match the operation requirements in different flight scenarios, laying a foundation for generating accurate operation instructions. The decision generation module fuses the optimized operation rule library parameters and real-time multi-modal feature maps, and generates a flight operation instruction sequence based on a dynamic confidence score. This fusion mechanism fully combines the optimized rule knowledge and real-time scene features, and the dynamic confidence score ensures the reliability and pertinence of instruction generation. The generated operation instruction sequence can accurately respond to the current flight state and training requirements, improving the accuracy and coordination of human-computer interaction. After analyzing the flight operation instruction sequence, the execution optimization layer satisfies the operation sequence constraint conditions through a constraint projection algorithm, and completes the integer programming of the instruction sequence in combination with the operation image deviation. The constraint projection algorithm ensures the feasibility of the operation instruction in actual execution, and the introduction of the operation image deviation considers the individual operation characteristics of the training personnel, making the final instruction sequence more adaptive to the operation habits and skill levels of the training personnel, improving the applicability and effectiveness of the simulation training, and making the training process smoother and more in line with the actual ability of the training personnel. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A timing diagram of the multi-modal human-computer interaction aircraft simulation training system described in the present application; Figure 2 A flowchart for updating the incremental clustering algorithm; Figure 3 A flowchart for tabu search and parameter optimization; Figure 4 A flowchart for optimization of the constraint projection algorithm. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0018] Please refer to Figure 1 The present application provides a multi-modal human-computer interaction aircraft simulation training system, which comprises: The multi-modal data acquisition module is responsible for collecting flight attitude parameters, airborne device state parameters and cockpit interaction data streams, and generating real-time multi-modal data streams; the multi-modal feature extraction module processes the data streams, and generates multi-modal feature maps through a hierarchical feature extraction network; the incremental analysis module divides the multi-modal feature maps into an incremental data set, applies an incremental clustering algorithm to update the flight state prediction model parameters, and outputs an incremental state prediction result; the parameter optimization layer initializes the tabu search population according to the incremental state prediction result, iteratively optimizes the operation rule library parameters through a dynamic performance evaluation matrix; the decision generation module fuses the optimized operation rule library parameters and the real-time multi-modal feature maps, and generates a flight operation instruction sequence based on a dynamic confidence score; the execution optimization layer analyzes the instruction sequence, uses a constraint projection algorithm to meet the operation sequence constraint conditions, and combines the operation image deviation to perform integer programming on the instruction sequence. Each module is interconnected through a data bus, flight attitude parameters are collected through an inertial measurement unit, airborne device state parameters are obtained through a CAN bus protocol, and cockpit interaction data streams include voice and gesture signals, which are captured in real time by multiple sensors such as gyroscopes, accelerometers and cameras. The hierarchical feature extraction network uses a convolutional neural network architecture, including convolutional layers, pooling layers and fully connected layers, which outputs multi-modal feature maps after inputting data streams. The incremental state prediction result is stored in the memory buffer area in the form of a vector for subsequent module calls. The tabu search population is generated based on a random number initialization algorithm, and the neighborhood solution is generated using a mutation operation rule. The dynamic confidence score uses fuzzy logic theory to calculate the probability distribution. The constraint projection algorithm optimizes the execution efficiency of the instruction sequence for integer programming problems. The system is deployed in a simulator hardware environment, the processor is a multi-core architecture with a main frequency of no less than 2.5 GHz to ensure real-time processing capability.

[0019] Example 1: refer to Figure 2 The multi-modal data acquisition module divides the collection area according to the historical flight mission type. The historical mission type is determined by the flight scene record obtained from the system database, which stores parameter configurations of emergency landing, cruising and take-off and landing scenes. The high-resolution focus area is defined as the key instrument area, including airspeed indicator, altitude table and attitude indicator; the low-resolution peripheral area covers the auxiliary device area, such as navigation light state display and cabin temperature monitoring panel. During the collection process, the high-resolution focus area uses a dense timestamp sampling strategy, and the low-resolution peripheral area uses a standard interval sampling strategy. The sensor sampling density is dynamically adjusted according to the time sequence characteristics of the flight phase: during take-off and landing, the three-axis accelerometer collects flight attitude parameters at a frequency of 100 frames per second; during cruising, the frequency is reduced to 50 frames per second. Airborne device state parameters are obtained through RS-232 communication interface or Ethernet data bus to get binary data streams, and a data parser extracts parameter values at a fixed period and converts them into a structured data format.

[0020] The cockpit interaction data stream is captured by multi-channel sensors: a microphone array collects voice commands, and an infrared camera captures gesture motion coordinates at a rate of 30 frames per second. The preprocessing unit performs multi-source data time alignment: after the voice signal is filtered to remove environmental noise, a timestamp labeling algorithm is applied to generate time-labeled audio frames; gesture coordinate data is generated as a continuous time series through a motion trajectory interpolation algorithm. The signal synchronization calibration unit uses the system clock as a reference to perform time domain interpolation on audio frames and video frames, ensuring that the time deviation of different modal data does not exceed ±5 milliseconds. The processed voice text data and gesture coordinate matrix are merged into a time-aligned multi-modal data stream, stored in a shared memory buffer and output to the multi-modal feature extraction module.

[0021] The input of the incremental analysis module is the above-mentioned multi-modal data stream. In the module initialization stage, the incremental clustering center vector dimension parameter (fixed at 32 dimensions) is loaded from the system configuration file, and a random initial clustering center vector is generated. The incremental clustering error threshold is preset to 0.01. The module divides the real-time input multi-modal feature map into an incremental data set, and the division mechanism uses an equal division algorithm: the feature map tensor is divided into continuous data blocks, each containing 100 feature vector samples. The data block processing unit sequentially reads the data blocks, calculates the Euclidean geometric distance between the new data block mean vector and the current clustering center, and this distance value is defined as the clustering center offset. The offset monitoring unit continuously compares the offset value with the error threshold: when the offset exceeds 0.01, the weight update unit activates the gradient descent algorithm of the flight state prediction model. The model uses a linear regression structure, and the weight matrix is adjusted by the gradient descent method with a step factor of 0.001; if the offset does not exceed the threshold, the original weight parameters remain unchanged. The updated model performs prediction operations on the current data block to generate incremental state prediction results: stored in the real-time database as a floating-point vector, with vector elements corresponding to the probability distribution of different flight states. The vector is pushed to the parameter optimization layer through a direct memory transfer channel, and the entire processing cycle is completed within 50 milliseconds, and the cycle trigger is realized through a hardware timer.

[0022] During the data flow process, the flight environment situation map and device operation health map output by the multi-modal feature extraction module are converted to tensor format. The feature map buffer uses a ring queue structure, and the newly generated incremental data set covers the earliest historical data. The implementation of the dynamic sampling strategy depends on the flight phase identifier: this identifier automatically switches the sampling frequency by analyzing the rate of change of attitude parameters (such as vertical speed exceeding ±5 meters / second to determine the take-off and landing phase). The sensor resource allocator adjusts the working mode of the data acquisition unit according to the current sampling strategy: the accelerometer in the high-resolution focus area is enabled in 100Hz sampling mode during take-off and landing, and switched to 50Hz during cruising; the temperature sensor in the peripheral area maintains a fixed sampling rate of 10Hz.

[0023] Implementation details of time alignment algorithm: The time labels of audio frames and video frames are generated by high-precision clock chips, and the synchronization calibration unit establishes a timestamp mapping table. The interpolation algorithm uses a cubic spline interpolation method to compensate for the missing data at the time point to generate a continuous sequence. The preprocessed data stream is packaged as a structured data packet, and the packet header includes the timestamp, data type identifier and data length field. The data packet is transmitted to the data receiving queue of the incremental analysis module through the high-speed serial bus.

[0024] In the incremental clustering process, the data block division algorithm uses a sliding window mechanism: the newly input feature vectors continuously fill the buffer, and when the number of vectors reaches 100, the block processing process is triggered. The offset calculation step involves feature vector normalization: after each feature vector is converted to a uniform dimension by Z-score standardization, the mean vector of the block sample is calculated. The weight update operation is performed on a dedicated mathematical coprocessor, and the matrix operation is accelerated in parallel using SIMD instructions. The prediction result vector contains 20 elements, each element corresponding to the probability value of a specific flight state label (such as "climb" or "dive"). The vector is transmitted to the input register of the parameter optimization layer through memory-mapped I / O.

[0025] At the hardware level, the data acquisition module implements sensor interface control through an FPGA chip, and the preprocessing unit is equipped with a dedicated digital signal processor. The incremental analysis module is deployed on a multi-core processor, with each core independently processing a data block. The timer interrupt service program triggers the processing flow at a 50ms period, ensuring real-time requirements. The system state monitoring unit records the timestamp and offset value of each clustering center update and writes it into the operation log for maintenance reference.

[0026] Example 2: refer to Figure 3 The implementation process of the parameter optimization layer starts with receiving the incremental state prediction result. The result is stored in the form of a 32-dimensional vector in the shared memory area, and the vector elements represent the probability distribution of different flight states. In the initialization stage of the parameter optimization layer, the basic parameters of the tabu search algorithm are loaded from the system configuration file: the population size is set to 100, the maximum number of iterations is limited to 200, and the length of the tabu table is fixed at 10. The population initialization uses a uniform distribution random number generator to generate 100 real number coded individuals, each individual being a 56-dimensional vector corresponding to the double parameter combination of 28 rules in the operation rule library (priority parameter and trigger threshold parameter). The individual decoding process converts the real number vector into executable rule parameters: the priority parameter is converted to the [0, 1] interval through linear mapping, and the trigger threshold parameter is normalized to the [0.2, 0.8] range using the Sigmoid function. The decoded parameter combination is injected into the rule engine interface of the decision generation module through the data bus.

[0027] The decision generation module initiates a simulation reasoning process upon receiving a new rule parameter combination. This process loads the real-time multi-modal feature map at the current time, including a 32x32 pixel flight environment situation map and a 16x16 pixel device operation health map. The rule engine evaluates all active conditions according to the injected parameter combination, generating a candidate instruction set. The construction of the dynamic performance evaluation matrix is based on the simulation execution results: the row dimension corresponds to the task type (such as "emergency obstacle avoidance", "flight path correction", etc. 20 preset tasks), and the column dimension records the resource consumption indicators (including time delay, energy consumption and system load three sub-items). The matrix element value is calculated by the following formula:

[0028] Wherein: represents the normalized performance score of the ith task on the jth resource, is the weight coefficient of the kth resource (preset value is 0.4, 0.3, 0.3), is the actual measured value, and represent the lower and upper limit reference values of the kth resource respectively. After the matrix is generated, the parameter optimization layer is returned through the cache.

[0029] The neighborhood solution generation operation uses improved genetic algorithm operators. The selection stage uses a tournament selection strategy to randomly select 5 individuals from the current population, and retains the top 2 individuals with the highest performance score to enter the mating pool. The crossover operation implements two-point crossover: randomly select two partition points in the 56-dimensional vector, and exchange the gene sequences of the intermediate segment of the parent individuals. The mutation operation uses a non-uniform mutation strategy, with the mutation rate decreasing linearly from the initial value of 0.1 to 0.01 as the iteration number increases. After decoding and verification, the newly generated offspring individuals replace the individuals with the lowest score in the population. The taboo table management unit maintains a first-in-first-out queue to record the parameter combination hash values used in the last 10 iterations, preventing repeated searches.

[0030] When the iteration counter reaches 200 times, or the global optimal solution does not improve for 30 consecutive iterations, the algorithm terminates. The final output of the optimized parameter combination is transmitted to the persistent storage area of the decision generation module through the DMA channel. This storage area uses a double buffering mechanism to ensure that the parameter update process does not affect real-time decision making. The optimized parameter combination is also backed up to the system configuration database as the initial parameters for subsequent task execution.

[0031] The core component of the decision generation module is the fuzzy inference engine. The engine input includes optimized rule parameters and real-time updated multi-modal feature maps. The feature map preprocessing unit down-samples the 32x32 flight environment situation map to an 8x8 feature matrix, and the device operational health map is converted into a 16-dimensional feature vector. The fuzzification process maps the input feature values to 7 linguistic variable levels ("very low", "low", "relatively low", "medium", "relatively high", "high", "very high"), and the membership function adopts a triangular distribution. The inference machine performs parallel rule evaluation, processing 8 rules per clock cycle, and the activation strength calculation uses the min-max composition method.

[0032] The implementation of the dynamic confidence scoring system is based on the Bayesian probability framework. The initial candidate instruction set is generated by the fuzzy inference engine, containing 15-20 candidate instructions. In the scoring parameter initialization stage, the basic confidence weight is set to 0.5 and the learning rate parameter is set to 0.05. Each time a new candidate instruction set is generated, the system retrieves the execution records in similar scenarios from the operation history database and updates the conditional probability distribution. The scoring update formula introduces a time decay factor, so that the weight of recent data is higher than that of historical data. The convergence determination unit monitors the score change rate of 5 consecutive iterations, and triggers the output latch when the maximum change rate is less than 0.001.

[0033] The final generated flight operation instruction sequence is encoded in a hierarchical structure. The top layer is the task type identifier (4 bytes), the middle layer contains the parameter list (8-byte floating-point number for each parameter), and the bottom layer adds the timestamp and priority marker. The sequence optimizer performs topological sorting on the instructions to ensure that instructions with dependencies are executed in the correct order. The output interface encapsulates the sequence into a TCP / IP data packet and writes it to the instruction buffer in shared memory. The hardware accelerator parses the data packet content and converts it into underlying control signals to drive the actuators of the flight simulator.

[0034] At the hardware architecture level, the parameter optimization layer is deployed on a dedicated computing node equipped with a 12-core processor with a 2.8 GHz clock frequency and 64 GB of memory. The decision generation module uses FPGA to accelerate fuzzy inference, and is equipped with 8 GB GDDR6 video memory for storing the rule base. The calculation task of the dynamic performance evaluation matrix is assigned to 4 CUDA cores, which communicate with the main processor through a PCIe 4.0x16 bus. The system uses redundant power supply design, and all critical data paths have ECC memory protection. The real-time performance monitoring unit records the time consumption and resource usage of each iteration, with a data sampling interval of 100 milliseconds.

[0035] The data persistence scheme adopts a log-structured merge tree storage engine. The optimized parameter combinations are stored in the form of key-value pairs, with the key being the MD5 hash value of the parameter combination and the value being the serialized parameter structure. The operation history database uses time-partitioned tables, with each partition storing the execution records within 24 hours. The system maintenance daemon periodically compresses the historical data and archives the records over 30 days to offline storage. The network communication module supports the breakpoint resume function and can resume the unfinished parameter transmission task after system restart.

[0036] The exception handling mechanism includes two levels of parameter checking and instruction verification. The parameter checking unit checks the value range and logical consistency before injecting new parameters each time. The instruction verifier simulates the execution of the generated instruction sequence to detect possible resource conflicts or constraint violations. When a serious error is detected, the system automatically rolls back to the last stable version of parameter configuration and notifies the maintenance personnel through the event bus. All abnormal events are recorded in the system log with a complete context state snapshot.

[0037] The clock synchronization system uses the PTP precision time protocol, and all computing nodes are equipped with rubidium atomic clocks as time references. The distributed lock manager coordinates the access of multiple modules to shared resources to avoid race conditions. Data consistency is guaranteed by the two-phase commit protocol, and critical operations require the confirmation of a majority of nodes to be committed. The system performs a self-check program when starting to verify the connection status of all sensors and actuators and loads the parameter configuration snapshot at the last normal shutdown.

[0038] Embodiment 3: refer to Figure 4 The implementation process of the execution optimization layer starts with receiving the flight operation instruction sequence output by the decision generation module. The sequence uses a hierarchical encoding structure, including an instruction type identifier, a parameter list, and timestamp information. The instruction parser first performs syntax analysis on the sequence, converting each instruction to an internal representation: the operation code field is mapped to the predefined 128 basic operations, the operand field stores floating-point parameter values, and the constraint condition field records the execution limitations of the instruction. The parsed instruction set is stored in a ring buffer with a capacity of 50 instructions, and the producer-consumer mode is used for read-write synchronization.

[0039] The constraint handling unit loads the set of operation sequence constraints defined in the system configuration file. This set contains three types of constraints: timing constraints specify the minimum / maximum time interval between instructions, resource constraints limit the number of concurrent operations for each subsystem (e.g. power, hydraulic, avionics), and logic constraints prohibit the occurrence of certain instruction combinations. Constraints are stored in predicate logic form, e.g. "the time interval between instruction A and instruction B ≥ 100ms" is represented as a timing constraint, and "no more than 3 operations are executed concurrently by the hydraulic system" as a resource constraint. The projection step parameter is initialized to 0.1, which is dynamically adjusted based on historical optimization results, with an adjustment range limited to ±0.02.

[0040] The implementation of the subgradient projection algorithm consists of three main stages. The initialization stage constructs the objective function: aiming to minimize the total execution delay, the cost function is defined as the weighted sum of the delays of each instruction. The weight coefficients are determined according to the instruction priority, with high-priority instructions set to a weight of 0.7 and normal instructions set to 0.3. In the projection direction calculation stage, the algorithm first solves the subgradient of the objective function:

[0041] where: denotes the subgradient vector at the t-th iteration, is the current number of active instructions, is the weight coefficient of the i-th instruction, represents the execution delay of the instruction, is the time adjustment variable, is the Lagrange multiplier (initial value 0.5), denotes the penalty term for violating constraints. The calculation process is performed on a mathematical coprocessor, utilizing SIMD instructions to parallel process the gradient components of multiple instructions.

[0042] The iteration update stage employs the projected subgradient descent method. In each iteration, the algorithm first updates the instruction time parameters along the negative direction of the subgradient: the new time value is equal to the original value minus the product of the step parameter and the subgradient component. Then, the projection operation is performed to map the updated time values into the feasible region defined by the constraint conditions. For timing constraints, the projection operation ensures that the adjusted time stamps satisfy all precedence relations; resource constraints are allocated by an integer programming algorithm, which uses the branch-and-price method for solution; logic constraints are verified using a rule engine, and prohibited instruction combinations are automatically rescheduled. After each projection operation, the algorithm checks the convergence conditions: the change rate of the objective function over the last 10 iterations is less than 0.0001, or the total number of iterations reaches the upper limit of 50.

[0043] After the optimization of the instruction sequence, the integer programming module processes the discretization problem. This module converts continuous time values into specific execution clock cycles, with a system base clock frequency of 10 MHz, thus achieving a time resolution of 100 ns. The conversion process takes into account the pipeline characteristics of the hardware execution unit: each instruction is decomposed into micro-operations, and the micro-operation duration must be an integer multiple of the clock period. The planning algorithm uses an improved constraint satisfaction method, first establishing a task graph containing all micro-operations and their dependencies, and then applying a list scheduling algorithm to generate the final execution sequence. During scheduling, the algorithm dynamically adjusts the load balancing of each functional unit (such as the floating-point operator, memory interface) to avoid resource conflicts.

[0044] The final output of the execution sequence uses a two-level buffer structure. The first level buffer stores the optimized abstract instruction sequence for the system monitoring module; the second level buffer contains the converted hardware microcode, which directly drives the execution unit. The two levels of buffer are connected through a dedicated bus, with a data transfer rate of up to 8 GB / s. The hardware interface module converts the microcode into specific control signals: digital output controls the state of the relay through the GPIO pin, and analog output adjusts the position or speed of the actuator after DAC conversion. Each output channel is equipped with an independent watchdog timer that automatically triggers a safety recovery process if no update signal is received within a specified time.

[0045] The exception handling mechanism runs throughout the execution optimization process. The constraint violation detector checks the projection results after each iteration and activates the recovery program when it finds that the constraints cannot be satisfied: it first attempts to relax the restrictions of secondary constraints, and if it still cannot be solved, it requests the decision generation module to re-plan the instruction sequence. The timing exception monitoring unit compares the actual execution time with the predicted value, and triggers online rescheduling when the deviation exceeds 20%. All abnormal events are recorded in the system log, accompanied by complete context state information, including the current instruction sequence, resource occupation, and system load indicators.

[0046] The system maintenance module provides a configuration interface for the execution optimization layer. Administrators can dynamically adjust the strictness of the constraint conditions, modify the projection step parameters, or switch to different optimization algorithm strategies. Configuration changes are guaranteed to be atomic through the transaction mechanism: the new configuration is verified in memory before being committed, and the old configuration is retained as a rollback point. The performance statistics module records key indicators of each optimization, including the number of iterations, the final objective function value, and the constraint satisfaction rate, which are used for subsequent analysis of the behavior characteristics of the optimization algorithm.

[0047] In hardware implementation, the execution optimization layer is deployed on computing nodes with real-time characteristics. The host processor adopts a multi-core architecture, in which two dedicated cores handle the subgradient computation, one core manages the constraint satisfaction, and the other core is responsible for exception handling. The mathematical acceleration unit is equipped with 128-bit floating-point operators, supporting parallel processing of multiple gradient components. The instruction buffer uses ECC memory protection, and all critical data paths have redundant designs. The system communicates with other modules through time-triggered Ethernet, ensuring deterministic latency for message transmission. The power management system provides multiple independent power supplies and can seamlessly switch to backup power when voltage anomalies are detected.

[0048] The data persistence scheme adopts a multi-version concurrency control mechanism. The intermediate state during optimization is regularly saved to non-volatile memory, and each checkpoint contains a complete snapshot of the system state. Historical execution sequences are archived in a time series database, supporting fast retrieval by time range or task type. The maintenance tool can replay the execution process at any point in time for analyzing optimization decisions in specific scenarios. All persistent operations are recorded in a write-ahead log, and the system recovers a consistent state by replaying the log upon restart.

[0049] The clock synchronization system uses a hybrid approach: the master clock source comes from a GPS disciplined rubidium clock, and each computing node synchronizes its local clock through the IEEE 1588 protocol. Critical timing operations are implemented using hardware timers, with an error of no more than 1 microsecond. The distributed lock manager coordinates access to shared resources by multiple execution units, using an optimistic concurrency control strategy to reduce waiting time. The system performs a self-check program upon startup, verifying the readiness status of all hardware components and loading the parameter configuration of the last successful optimization. The runtime integrity check module continuously monitors the checksum of critical data structures and automatically triggers a recovery program upon detecting memory errors.

[0050] Example 4: The feedback calibration module processes simulator response data through a multi-source filtering algorithm, with a workflow starting from the data acquisition stage. The flight environment sensor group includes a three-axis accelerometer, a barometric altimeter, and a GPS receiver, which output raw measurement values at frequencies of 100 Hz, 50 Hz, and 10 Hz, respectively. The student physiological monitoring equipment includes an electroencephalogram acquisition helmet (sampling rate 256 Hz) and a heart rate band (sampling rate 1 Hz), and all sensor data is sent to the preprocessing queue after timestamp alignment. The data synchronization unit uses a hardware-level time stamping scheme, based on the PTP protocol to control the time deviation of each data source within ±2 ms. The structure of the raw data packet includes a device identifier, a timestamp, and a data payload, for example, the payload field of the barometer data packet stores the current barometric pressure value (unit hPa) and the temperature compensation coefficient.

[0051] The core of the multi-source filtering algorithm is the construction of the state space model. The system defines two types of state variables: the flight environment state vector contains 12 parameters such as altitude, airspeed, pitch angle, etc.; the cognitive state vector is composed of electroencephalogram power spectrum features (δ, θ, α, β band energy proportion) and heart rate variability indicators. The state transition equation describes the dynamic evolution law of these parameters, for example, the relationship between altitude change and vertical speed is modeled by a first-order kinematics model. The observation equation establishes the mapping relationship between sensor data and state variables, and the nonlinear relationship between barometer readings and altitude is realized by table lookup method. The Kalman filter implementation adopts the square root covariance form, which has better numerical stability than the conventional implementation. The filtering process produces two key outputs: the calibrated flight environment situation deviation (a three-dimensional vector representing the difference between actual and expected pitch, roll, and yaw angles) and the cognitive behavior deviation vector (a four-dimensional vector quantifying the differences in attention concentration, workload, and other cognitive indicators).

[0052] The application of calibration data is reflected in the coordinated work of multiple modules of the system. The flight environment situation deviation is pushed in real time to the feature correction unit of the multi-modal feature extraction module, which maintains a dynamic compensation matrix to correct the angle-related parameters in the original feature map. For example, when the pitch angle deviation exceeds 0.5 degrees, the height channel data of the feature map will be compensated accordingly. The cognitive behavior deviation vector is transmitted to the behavior analyzer of the portrait management module, which combines the vector with the current task difficulty coefficient to generate a cognitive load assessment report. A priority marking mechanism is used in the data flow process, and high-priority flight safety-related data (such as sudden altitude deviation) can interrupt the transmission of low-priority data.

[0053] The cognitive behavior portrait backtracking function of the portrait management module is activated at a preset period. The system maintains a ring-shaped portrait database, each record containing a timestamp, a task identifier, and a cognitive state snapshot. When the management period (default 5 minutes) arrives, the backtracking engine performs the following operations: retrieves historical portraits from the database within the last three periods, extracts key parameters including task duration, operation error rate, and cognitive load score. The comparison between the real-time portrait and the historical portrait is completed by a difference analysis algorithm, which first aligns the time series data by dynamic time warping, and then calculates the root mean square error of each dimension. The generated portrait deviation vector contains four components: task period deviation (percentage difference between actual and expected time), cognitive load deviation (difference between current load and historical average), operation precision deviation (error rate change), and attention fluctuation deviation (standard deviation of electroencephalogram α band energy).

[0054] The operation data deviation modulus set generation process of the data management engine involves spatio-temporal feature alignment. The system defines a four-dimensional coordinate system: three-dimensional spatial position (X / Y / Z axes) plus a time dimension, and each coordinate point corresponds to a flight phase and an operation context. The feature alignment unit first maps the grid coordinates of the flight environment situation map to this four-dimensional space, and then matches the monitoring points of the equipment operation health map through nearest neighbor interpolation. In the aligned region, the system extracts the execution delay data of the operation instruction, and calculates its absolute deviation from the reference delay threshold. The reference threshold is defined according to the flight manual, for example, the standard delay of the throttle adjustment instruction in the cruise phase is 120ms. The storage structure of the deviation modulus set is shown in Table 1.

[0055] Table 1: Storage structure of deviation modulus set.

[0056] Coordinate block ID Spatial position Time window Instruction type Reference delay (ms) Measured delay (ms) Deviation modulus B-0127 X: 12 / Y: 45 / Z: 2800 00:05:21-00:05:23 Throttle adjustment 120 138 18 B-0128 X: 15 / Y: 47 / Z: 2850 00:05:24-00:05:26 Course correction 80 92 12 B-0129 X: 10 / Y: 40 / Z: 2750 00:05:27-00:05:29 Altitude hold 150 143 7 The data cleaning strategy is based on the dual standards of variance analysis and timeliness. The engine first calculates the statistical variance of each deviation modulus set, and when the variance is lower than the threshold of 0.05, it is determined as low-value data; secondly, it checks the data storage time length, and records exceeding 30 days are automatically marked as items to be cleaned. The deletion operation adopts a lazy strategy, and the actual space recovery is performed when the system is idle. Meta-data summaries are preserved during the cleaning process, including the maximum / minimum deviation values and the most frequently occurring instruction types for each block.

[0057] In terms of hardware architecture, the feedback calibration module is deployed on an embedded system with real-time processing capability, with a dual-core Cortex-A72 main processor and a dedicated DSP processor to handle filtering algorithms. Sensor data is collected through isolated ADC channels, and the sampling clock is provided by a constant temperature crystal oscillator for stable reference. The image database uses NVMe SSD storage, and data redundancy is ensured through RAID1 configuration. The anomaly handling unit monitors the integrity of each data stream, and when sensor failure or data verification error is detected, it automatically switches to a backup data source or enables a predictive compensation mode.

[0058] The system maintenance interface provides calibration parameter adjustment functions. Administrators can modify the process noise matrix of the Kalman filter, adjust the parameters of the state transition model, or redefine the cognitive behavior index weights. All configuration changes are recorded in the audit log, including modification time, operator ID, and parameter values before and after modification. The performance monitoring panel displays key indicators such as data processing delay, filter convergence status, and storage space usage in real time. The debugging mode supports raw data recording and playback, which facilitates the analysis of system behavior in specific scenarios.

[0059] The clock synchronization system adopts a hybrid scheme: the master clock source comes from the 1PPS signal of the GPS receiver, and each subsystem synchronizes the local clock through the IEEE1588v2 protocol. Key timing tasks such as sensor sampling and data packaging are triggered by hardware timers to ensure timing accuracy. The power management system monitors the power consumption status of each module and automatically isolates the faulty unit when abnormal power consumption is detected. The startup self-test program verifies the functional integrity of all hardware components, including memory testing, communication loop detection, and sensor calibration verification.

[0060] The data persistence scheme adopts a hierarchical storage structure. Raw sensor data is retained for 7 days, calibrated intermediate data is retained for 30 days, and portrait data and deviation analysis results are retained for 1 year. Archival data is transferred to the offline storage system after compression and encryption. The network transmission module supports breakpoint continuation and bandwidth adjustment to adapt to different network environment conditions. All communication channels are encrypted with AES-256, and the key is rotated every 24 hours. The system log records detailed operation traces, including data flow path, processing time consumption, and resource occupation, and the log file is updated daily and automatically uploaded to the central log server.

[0061] In embodiment 5, the portrait management module maintains a real-time storage structure of cognitive behavior portraits, each portrait object containing a timestamp field, a task period parameter, and a multi-dimensional cognitive load parameter. The task period parameter records the duration from task start to the current time (millisecond level accuracy), and the cognitive load parameter consists of four sub-items: electroencephalogram concentration index (0-100 scale), heart rate variation coefficient (standard deviation to mean percentage), operation error frequency count, and visual attention allocation index. The historical portrait database adopts a time-sharing storage architecture, generating a snapshot file every 5 minutes and storing it in non-volatile memory. When the preset management period (default 5 minutes) is triggered, the backtracking retrieval unit locates the snapshot files of the previous three periods, and the file naming contains an accurate time identifier. The file reading process uses direct memory mapping technology to avoid data copy overhead.

[0062] The decoding process of the cognitive behavior profile first generates a set of historical parameters. The task cycle parameter is extracted from a fixed offset in the file header (address 0x00-0x08) as a 64-bit integer value, which is converted into time length data. The four sub-items of the cognitive load parameter are stored in consecutive memory regions (addresses 0x10-0x27): the brain concentration index is 4 bytes of floating-point type, the heart rate variation coefficient is stored as 2 bytes of fixed-point number (resolution 0.01%), the operation error frequency is 2 bytes of unsigned integer, and the visual attention allocation index is encoded as 4 bytes of floating-point. The real-time cognitive behavior profile is obtained through a shared memory interface, and the data format is consistent with the historical file. The comparison algorithm performs field-by-field difference operation: the task cycle deviation component is calculated as (current cycle - historical cycle) / historical cycle percentage value; the cognitive load deviation component exists in vector form, and each dimension corresponds to the absolute value difference of the above four sub-items. The result is packaged as a deviation vector structure, containing four single-precision floating-point fields.

[0063] After receiving the vector, the data management engine starts the spatial mapping process. The spatiotemporal feature map coordinate system is defined as a three-dimensional space grid (grid size 0.5m x 0.5m x 100ft) superimposed with a time window (length 200ms). The engine maintains a coordinate mapping table to convert pixel coordinates in the flight environment situation map to spatial grid IDs. The monitoring point coordinates of the device running health map are aligned to the same grid system through a spatial transformation matrix: the gyroscope data point coordinates are converted using affine transformation, and the oil pressure sensor position uses a rigid body transformation model. The aligned spatiotemporal blocks generate unique identifiers in the format "Grid-X:Y:Z_Time Start Stamp", for example, block "G-1278_093045000" represents the data unit of space grid number 1278 at 09:30:45.000.

[0064] The collection of instruction delay feature values within the block relies on the precise timing system. The start time of instruction execution is recorded by the operation code analysis module (based on the system clock counter), and the end time is captured through the state register of the execution unit. Each block stores the execution delay records of the last 20 instructions, and the delay value is the difference between the end timestamp and the start timestamp (nanosecond-level resolution). The threshold value of the reference delay is dynamically adjusted according to the operation type: the threshold value of the regular operation is preset to 120ms, and the threshold value of the emergency operation is set to 80ms. The deviation module value calculation adopts a simplified implementation of the Euclidean distance formula: taking the absolute value of the difference between the execution delay value and the threshold value. The deviation module values of all instructions within the block form a temporary set, and the execution memory is compressed (each module value is represented by 2 bytes of fixed-point number).

[0065] The aggregation process of the set of bias modulus adopts a divide-and-conquer strategy. The system divides the space into 8x8 grid upper blocks (4m x 4m x 800ft range), and the bias modulus data of the same level block is transmitted through the tree collection protocol. After receiving the modulus list of each sub-block, the upper block performs sorting and truncation operations: the top 50 modulus values with the largest values are retained, and the remaining data is deleted to avoid storage overflow. After aggregation, the set of operation data bias modulus of the whole space is formed, and the storage format is a key-value pair database, the key is the block identifier, and the value is a variable length modulus array. This database supports fast query through hash index, and the response time is controlled within 5 milliseconds.

[0066] The data cleaning mechanism implements two-level screening standards. Time effectiveness screening: each storage unit is marked with the latest access timestamp, and the periodic task (started at 00:00 every day) deletes records over 30 days. Value screening: calculate the median and variance of the modulus array in the block, and when the median is less than 20 and the variance is less than 5, mark the entire block as a low-value area. The deletion operation adopts a log-structured merge strategy: new data is written to a separate partition, and when merging, skip the marked block. The deletion operation also removes the corresponding metadata records in the portrait database.

[0067] In terms of system hardware deployment, the portrait management module adopts a dual-redundant server architecture. The main node runs on a 2.4GHz processor, equipped with 64GB of memory and a 1TB NVMe solid state disk; the standby node remains in a hot standby state and automatically switches through heartbeat detection. The data management engine is deployed on an FPGA platform with hardware acceleration, integrated with a PCIe interface for communication with the main system. The dedicated processor for space-time coordinate mapping contains 1000 parallel computing units with a processing clock frequency of 400MHz. The power system is configured with dual-channel UPS backup to ensure the integrity of data storage.

[0068] The maintenance interface provides parameter adjustment functions. Administrators can modify the partitioning rules of the space-time grid: the grid size range can be adjusted from 0.2-1 meters, and the time window length can be configured in the 50-500 millisecond interval. The calculation weight of cognitive load parameters supports online update: the EEG concentration weight is set to 0.4 by default, and other parameter weights can be dynamically adjusted. The system monitoring module displays the portrait data processing delay, bias calculation efficiency, and storage space usage rate curves during system operation. The log recorder saves the input and output data of each comparison operation in detail, including timestamp, bias vector value, and involved historical portrait version number. The debugging mode allows importing external data sets to replay the processing flow and verify the system behavior in specific scenarios.

[0069] The abnormality processing unit monitors the data consistency status. When detecting the historical profile data damage, the interpolation compensation of adjacent period data is automatically triggered: taking the mean value of the corresponding parameters of the previous and next periods to replace. When the space-time block mapping error occurs, the system falls back to the basic grid division scheme (1 meter x 1 meter x 200 feet). All error events generate a unique identification code and report to the central monitoring system, while freezing the current operating environment data for subsequent analysis. The clock synchronization system maintains nanosecond-level time consistency through the IEEE1588 protocol, and the key timestamp generation uses the onboard atomic clock pulse. The communication protocol uses encrypted data transmission, and performs two-way authentication before each session establishment.

[0070] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0071] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, alternatives, and variations can be made in the embodiments without departing from the spirit and scope of the present application as defined by the appended claims and their equivalents.

Claims

1. A multi-modal human-in-the-loop aircraft simulation training system, characterized in that, The application relates to a flight operation optimization method based on multi-modal data fusion, which comprises the following steps: A multi-modal data acquisition module dynamically acquires flight attitude parameters, airborne equipment state parameters and cockpit interaction data streams, and generates multi-modal data streams; A multi-modal feature extraction module receives the multi-modal data streams, generates multi-modal feature maps through a hierarchical feature extraction network, and the multi-modal feature maps comprise a flight environment situation map and a device operation health map; An incremental analysis module divides the multi-modal feature maps into an incremental data set, updates flight state prediction model parameters through an incremental clustering algorithm, and outputs incremental state prediction results; A parameter optimization layer initializes a tabu search population according to the incremental state prediction results, iteratively optimizes operation rule library parameters through a dynamic performance evaluation matrix, and outputs the operation rule library parameters; A decision generation module fuses the optimized operation rule library parameters and real-time multi-modal feature maps, generates a flight operation instruction sequence based on a dynamic confidence score, and outputs the flight operation instruction sequence to an execution optimization layer. The application further comprises:

2. The multi-modal human interaction aircraft simulation training system of claim 1, wherein, A feedback calibration module processes simulator response data through a multi-source filtering algorithm, generates a calibrated flight environment situation deviation and a cognitive behavior deviation vector, and outputs the calibrated flight environment situation deviation and the cognitive behavior deviation vector to the execution optimization layer; An image management module triggers a cognitive behavior image backtracking based on a preset management period, calculates an image deviation vector of a real-time cognitive behavior image and a backtracked cognitive behavior image, and outputs the image deviation vector to the execution optimization layer; A data management engine retrieves an operation data deviation modulus set corresponding to the image deviation vector, deletes historical operation data according to a variance threshold and a storage time length ratio, and outputs the historical operation data to the execution optimization layer. The dynamic acquisition of flight attitude parameters, airborne equipment state parameters and cockpit interaction data streams comprises the following steps:

3. The multi-modal human interaction aircraft simulation training system of claim 1, wherein, According to historical flight task types, a high-resolution focus area and a low-resolution peripheral area are divided; Based on flight stage time sequence characteristics, a sensor sampling density is dynamically adjusted, and a dense timestamp sampling strategy is adopted in the high-resolution focus area; Cockpit interaction data streams are preprocessed, and time-aligned multi-modal data streams are generated. The updating of flight state prediction model parameters through an incremental clustering algorithm comprises the following steps:

4. The multi-modal human interaction aircraft simulation training system of claim 1, wherein, An incremental clustering center vector and an incremental clustering error threshold are initialized; An incremental data set is divided into multiple data blocks, and a current data block clustering center offset is calculated block by block; When the clustering center offset exceeds the incremental clustering error threshold, flight state prediction model weight parameters are updated; Incremental state prediction results are output to a parameter optimization layer. The iterative optimization of operation rule library parameters through a dynamic performance evaluation matrix comprises the following steps:

5. The multi-modal human interaction aircraft simulation training system of claim 1, wherein, Individual solutions in a tabu search population are decoded into operation rule library parameter combinations; The operation rule library parameter combinations are injected into a decision generation module, a dynamic performance evaluation matrix is obtained, neighborhood solution generation and tabu table updating are performed according to the dynamic performance evaluation matrix, and final operation rule library parameters are output when a maximum iteration number is reached. The generation of a flight operation instruction sequence based on a dynamic confidence score comprises the following steps: Real-time multi-modal feature maps are input into a fuzzy reasoning engine to generate an initial instruction candidate set; Dynamic confidence score parameters are initialized, and new instruction candidate sets are generated in each iteration; 6. The multi-modal human interaction aircraft simulation training system of claim 1, wherein, When the dynamic confidence score meets a convergence condition, a final flight operation instruction sequence is output to an execution optimization layer. The satisfaction of operation sequence constraint conditions through a constraint projection algorithm comprises the following steps: ​ ​ 7. The multi-modal human interaction aircraft simulation training system of claim 1, wherein, ​ Set the operation sequence constraint condition set and the projection step parameter; Calculate the sub-gradient projection direction of the flight operation instruction sequence; Iteratively update the instruction sequence value range within the operation sequence constraint condition set; Output the flight operation instruction sequence that meets all operation sequence constraint conditions.

8. The multi-modal human interaction aircraft simulation training system of claim 2, wherein, The processing of the simulator response data by the multi-source filtering algorithm comprises: Collecting flight environment sensor feedback data and student physiological response data, constructing multi-source filtering state transition equations and observation equations, and fusing flight environment situation deviation and cognitive behavior deviation vectors to generate calibration data; The calibration data is fed back to the multi-modal feature extraction module and the portrait management module.

9. The multi-modal human interaction aircraft simulation training system of claim 2, wherein, The calculation of the portrait deviation vector of the real-time cognitive behavior portrait and the backtracking cognitive behavior portrait comprises: Extracting historical task cycle parameters and historical cognitive load parameters in the backtracking cognitive behavior portrait, comparing current task cycle parameters and current cognitive load parameters in the real-time cognitive behavior portrait, and generating a portrait deviation vector containing task cycle deviation components and cognitive load deviation components; Output the portrait deviation vector to the data management engine.

10. The multi-modal human interaction aircraft simulation training system of claim 2, wherein, The generation of the operation data deviation module value set comprises: Defining the spatiotemporal feature map coordinate mapping relationship, aligning the flight environment situation map and the equipment operation health map; Extracting the operation instruction execution delay feature value in the spatiotemporal alignment area, calculating the deviation module value of the operation instruction execution delay feature value and the reference delay threshold value; Aggregate all spatiotemporal region deviation module values to form the operation data deviation module value set.

Images