Multimodal human-machine interaction aircraft simulation training system

Abstract

The application relates to the technical field of aircraft simulation training and discloses a multi-modal man-machine interaction aircraft simulation training system. A multi-modal feature extraction module of the system generates a multi-modal feature map containing a flight environment situation graph and a device operation health graph through a hierarchical feature extraction network; an incremental analysis module divides an incremental data set, updates flight state prediction model parameters through an incremental clustering algorithm, and outputs an incremental state prediction result; a parameter optimization layer initializes a tabu search population according to the prediction result, iteratively optimizes operation rule library parameters through a dynamic performance evaluation matrix, a decision generation module fuses the optimized rule library parameters and real-time multi-modal feature maps, generates a flight operation instruction sequence based on a dynamic confidence score, an execution optimization layer analyzes the instruction sequence, satisfies constraint conditions through a constraint projection algorithm, and completes integer programming of the instruction sequence in combination with operation image deviation. The system improves the interaction accuracy and scene adaptability of simulation training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aircraft simulation training, in particular to a multi-modal human-computer interaction aircraft simulation training system. BACKGROUND

[0002] In the field of aircraft driving training, the simulation training system is an important means to reduce training cost and improve training safety, and its performance directly affects the training effect. The traditional aircraft simulation training system relies on a single data source for training scene construction, usually only collects limited flight attitude parameters or simple device state information, and it is difficult to fully reflect the complex flight environment and cockpit interaction process. The limitation of such data acquisition leads to a large difference between the simulation scene and the actual flight scene, and the operation experience obtained by the training personnel in the simulation environment is difficult to effectively transfer to the actual flight.

[0003] The feature extraction method of the existing simulation training system is relatively simple, and a single-dimensional feature analysis method is often used, which cannot deeply fuse and represent multi-source data. For example, some systems only analyze basic parameters such as flight altitude and speed, ignoring the correlation between the running state of airborne devices and cockpit interaction behavior, resulting in incomplete description of the flight environment situation and device health state, which is difficult to support accurate training guidance.

[0004] In terms of model updating, the flight state prediction model of the traditional system mostly adopts a static training method, and the model parameters are not updated once determined, which cannot adapt to the dynamic changes of the flight scene. When encountering new flight conditions or abnormal device states, the model prediction accuracy decreases significantly, which cannot provide timely and accurate state feedback for the training personnel, affecting the pertinence and effectiveness of the training.

[0005] The optimization mechanism of the operation rule library is also a weak link of the existing system. The parameters of the operation rule library of most systems are preset by artificial or optimized by simple algorithms, and lack dynamic adjustment capability. In complex flight scenarios, fixed rule parameters are difficult to match the real-time changes of flight state and interaction demand, resulting in deviation between the generated operation instruction sequence and the actual operation requirement, reducing the authenticity and guidance of simulation training.

[0006] The existing system lacks fine processing of operation constraint conditions in the instruction execution optimization link, and only ensures the legality of the instruction through simple rule checking, without fully considering the operation habits and skill differences of the training personnel. This extensive instruction planning method is easy to cause the mismatch between the operation instruction and the actual operation ability of the training personnel, affecting the improvement of training effect. Therefore, it has become a problem to be solved to construct an aircraft simulation training system that can realize multi-modal data fusion, dynamic model updating, accurate rule optimization and adaptive instruction generation. SUMMARY

[0007] The present application aims to provide a multi-modal human-computer interaction aircraft simulation training system to solve the problems raised in the background art.

[0008] To achieve the above-mentioned purpose, the present application provides a multi-modal human-computer interaction aircraft simulation training system, which comprises:

[0009] A multi-modal data acquisition module acquires flight attitude parameters, airborne equipment state parameters and cockpit interaction data streams dynamically, and generates multi-modal data streams;

[0010] A multi-modal feature extraction module receives the multi-modal data streams and generates multi-modal feature maps through a hierarchical feature extraction network, wherein the multi-modal feature maps include flight environment situation maps and equipment operation health maps;

[0011] An incremental analysis module divides the multi-modal feature maps into incremental data sets, updates the flight state prediction model parameters through an incremental clustering algorithm, and outputs incremental state prediction results;

[0012] A parameter optimization layer initializes a tabu search population according to the incremental state prediction results, and iteratively optimizes the operation rule library parameters through a dynamic performance evaluation matrix;

[0013] A decision generation module fuses the optimized operation rule library parameters and real-time multi-modal feature maps, and generates a flight operation instruction sequence based on a dynamic confidence score;

[0014] An execution optimization layer analyzes the flight operation instruction sequence, satisfies the operation sequence constraint conditions through a constraint projection algorithm, and completes the instruction sequence integer programming in combination with the operation image deviation.

[0015] Preferably, the system further comprises:

[0016] A feedback calibration module processes simulator response data through a multi-source filtering algorithm to generate calibrated flight environment situation deviation and cognitive behavior deviation vectors;

[0017] An image management module triggers cognitive behavior image backtracking based on a preset management period, and calculates the image deviation vector of the real-time cognitive behavior image and the backtracked cognitive behavior image;

[0018] A data management engine retrieves the operation data deviation module value set corresponding to the image deviation vector, and deletes historical operation data according to the variance threshold and storage time length ratio.

[0019] Preferably, the dynamic acquisition of flight attitude parameters, airborne equipment state parameters and cockpit interaction data streams comprises:

[0020] According to the historical flight task type, a high-resolution focus area and a low-resolution peripheral area are divided;

[0021] The sensor sampling density is dynamically adjusted based on the flight phase timing characteristics, and a dense timestamp sampling strategy is adopted in the high-resolution focus area;

[0022] The cockpit interaction data stream is preprocessed to generate a time-aligned multi-modal data stream.

[0023] Preferably, the updating of the flight state prediction model parameters by the incremental clustering algorithm comprises:

[0024] Initializing the incremental clustering center vector and the incremental clustering error threshold;

[0025] Dividing the incremental data set into multiple data blocks, and calculating the current data block clustering center offset;

[0026] When the clustering center offset exceeds the incremental clustering error threshold, updating the flight state prediction model weight parameters;

[0027] Outputting the incremental state prediction result to the parameter optimization layer.

[0028] Preferably, the iterative optimization of the operation rule library parameters by the dynamic performance evaluation matrix comprises:

[0029] Decoding the individual solution in the tabu search population into an operation rule library parameter combination;

[0030] Injecting the operation rule library parameter combination into the decision generation module to obtain a dynamic performance evaluation matrix;

[0031] Generating a neighborhood solution and updating the tabu table according to the dynamic performance evaluation matrix;

[0032] When the maximum number of iterations is reached, output the final operation rule library parameters.

[0033] Preferably, the generation of the flight operation instruction sequence based on the dynamic confidence score comprises:

[0034] Inputting the real-time multi-modal feature map into a fuzzy reasoning engine to generate an initial instruction candidate set;

[0035] Initializing the dynamic confidence score parameters, and generating a new instruction candidate set in each iteration;

[0036] When the dynamic confidence score meets the convergence condition, output the final flight operation instruction sequence to the execution optimization layer.

[0037] Preferably, the satisfaction of the operation sequence constraint condition by the constraint projection algorithm comprises:

[0038] Setting the operation sequence constraint condition set and the projection step parameter;

[0039] Calculating the sub-gradient projection direction of the flight operation instruction sequence;

[0040] iterating the updated instruction sequence value domain within the operation sequence constraint condition set;

[0041] outputting the flight operation instruction sequence satisfying all operation sequence constraint conditions.

[0042] Preferably, the processing of the simulator response data by the multi-source filtering algorithm comprises:

[0043] collecting flight environment sensor feedback data and student physiological response data, constructing multi-source filtering state transition equations and observation equations, and fusing flight environment situation deviation and cognitive behavior deviation vectors to generate calibration data;

[0044] feeding the calibration data to the multi-modal feature extraction module and the portrait management module.

[0045] Preferably, the calculation of the portrait deviation vector of the real-time cognitive behavior portrait and the backtracking cognitive behavior portrait comprises:

[0046] extracting historical task cycle parameters and historical cognitive load parameters in the backtracking cognitive behavior portrait, comparing current task cycle parameters and current cognitive load parameters in the real-time cognitive behavior portrait, and generating a portrait deviation vector containing a task cycle deviation component and a cognitive load deviation component;

[0047] outputting the portrait deviation vector to the data management engine.

[0048] Preferably, the generation of the operation data deviation module value set comprises:

[0049] defining a spatiotemporal feature map coordinate mapping relationship, aligning a flight environment situation map and a device operation health map;

[0050] extracting operation instruction execution delay feature values in the spatiotemporally aligned region, and calculating deviation module values of the operation instruction execution delay feature values and the reference delay threshold value;

[0051] aggregating the deviation module values of all spatiotemporal regions to form an operation data deviation module value set.

[0052] Compared with the prior art, the beneficial effects of the present application are:

[0053] The multi-modal human-computer interaction aircraft simulation training system works collaboratively through multiple modules, bringing positive effects in multiple aspects. The multi-modal data acquisition module breaks through the limitations of traditional single data sources, simultaneously acquiring flight attitude parameters, airborne device state parameters, and cockpit interaction data streams, and the generated multi-modal data stream can more comprehensively cover key information in flight training, making the system's perception of the flight scene more complete, providing rich basic data for subsequent feature extraction and analysis, and making the simulation training scene more close to the actual flight environment.

[0054] The multi-modal feature extraction module generates a flight environment situation graph and a device operation health graph using a hierarchical feature extraction network. This hierarchical processing method can deeply mine the internal characteristics of different types of data and the correlation between data. The flight environment situation graph clearly presents the environmental changes during flight, and the device operation health graph intuitively reflects the running state of the airborne equipment. The combination of the two feature graphs makes the system's characterization of the flight state more accurate, providing a clearer scene cognition basis for the trainer, which helps the trainer better understand the flight environment and device status.

[0055] The incremental analysis module updates the flight state prediction model parameters by dividing the incremental data set and using an incremental clustering algorithm, achieving dynamic optimization of the model. As the training process progresses, the system can continuously absorb new flight data information, enabling the prediction model to adapt to different flight conditions and scene changes, and the output incremental state prediction results are more consistent with the real-time flight state, providing a reliable state reference for subsequent parameter optimization and instruction generation, so that the model always maintains effective perception ability for the flight state.

[0056] The parameter optimization layer initializes the tabu search population based on the incremental state prediction results, and iteratively optimizes the operation rule library parameters through a dynamic performance evaluation matrix. The introduction of the tabu search algorithm improves the global search ability of parameter optimization, and the dynamic performance evaluation matrix ensures the adaptability of the optimization process to the real-time flight state. The operation rule library parameters optimized through iteration are more reasonable and can better match the operation requirements in different flight scenarios, laying a foundation for generating accurate operation instructions.

[0057] The decision generation module fuses the optimized operation rule library parameters and real-time multi-modal feature graphs, and generates a flight operation instruction sequence based on dynamic confidence scoring. This fusion mechanism fully combines the optimized rule knowledge and real-time scene features, and the dynamic confidence scoring ensures the reliability and relevance of instruction generation. The generated operation instruction sequence can accurately respond to the current flight state and training requirements, improving the accuracy and coordination of human-machine interaction.

[0058] After analyzing the flight operation instruction sequence, the execution optimization layer uses a constraint projection algorithm to satisfy the operation sequence constraints and combines the operation image deviation to complete the integer programming of the instruction sequence. The constraint projection algorithm ensures the feasibility of the operation instruction in actual execution, and the introduction of the operation image deviation considers the individual operation characteristics of the trainer, making the final instruction sequence more suitable for the trainer's operation habits and skill levels, improving the applicability and effectiveness of the simulation training, and making the training process smoother and more in line with the actual ability of the trainer. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1A timing diagram for the multi-modal human-machine interaction aircraft simulation training system of the present invention;

[0060] Figure 2 A flowchart for updating the incremental clustering algorithm;

[0061] Figure 3 A flowchart for tabu search and parameter optimization;

[0062] Figure 4 A flowchart for constraint projection algorithm optimization. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0064] Please refer to Figure 1 The present application provides a multi-modal human-machine interaction aircraft simulation training system, which comprises:

[0065] The multi-modal data acquisition module is responsible for collecting flight attitude parameters, airborne device state parameters and cockpit interaction data streams, and generating real-time multi-modal data streams; the multi-modal feature extraction module processes the data streams, and generates multi-modal feature maps through a hierarchical feature extraction network; the incremental analysis module divides the multi-modal feature maps into an incremental data set, applies an incremental clustering algorithm to update the flight state prediction model parameters, and outputs an incremental state prediction result; the parameter optimization layer initializes the tabu search population according to the incremental state prediction result, iteratively optimizes the operation rule library parameters through a dynamic performance evaluation matrix; the decision generation module fuses the optimized operation rule library parameters and the real-time multi-modal feature maps, and generates a flight operation instruction sequence based on a dynamic confidence score; the execution optimization layer analyzes the instruction sequence, uses a constraint projection algorithm to meet the operation sequence constraint conditions, and combines the operation image deviation to perform integer programming on the instruction sequence. Each module is interconnected through a data bus, flight attitude parameters are collected through an inertial measurement unit, airborne device state parameters are obtained through a CAN bus protocol, and cockpit interaction data streams include voice and gesture signals, which are captured in real time by multiple sensors such as gyroscopes, accelerometers and cameras. The hierarchical feature extraction network uses a convolutional neural network architecture, including convolutional layers, pooling layers and fully connected layers, which outputs multi-modal feature maps after inputting data streams. The incremental state prediction result is stored in the memory buffer area in the form of a vector for subsequent module calls. The tabu search population is generated based on a random number initialization algorithm, and the neighborhood solution is generated using a mutation operation rule. The dynamic confidence score uses fuzzy logic theory to calculate the probability distribution. The constraint projection algorithm optimizes the execution efficiency of the instruction sequence for integer programming problems. The system is deployed in a simulator hardware environment, the processor is a multi-core architecture with a main frequency of no less than 2.5 GHz to ensure real-time processing capability.

[0066] Example 1: refer to Figure 2 The multi-modal data acquisition module divides the collection area according to the historical flight task type. The historical task type is determined by the flight scene record obtained from the system database, which stores parameter configurations of emergency landing, cruising and take-off and landing scenes. The high-resolution focus area is defined as the key instrument area, including airspeed indicator, altitude table and attitude indicator; the low-resolution peripheral area covers the auxiliary device area, such as navigation light state display and cabin temperature monitoring panel. During the collection process, the high-resolution focus area uses a dense timestamp sampling strategy, and the low-resolution peripheral area uses a standard interval sampling strategy. The sensor sampling density is dynamically adjusted according to the time sequence characteristics of the flight phase: in the take-off and landing phase, the three-axis accelerometer collects flight attitude parameters at a frequency of 100 frames per second; in the cruising phase, the frequency is reduced to 50 frames per second. Airborne device state parameters are obtained through RS-232 communication interface or Ethernet data bus to get binary data stream, and data parser extracts parameter values at fixed period and converts them into structured data format.

[0067] The cockpit interaction data stream is captured by multi-channel sensors: a microphone array collects voice commands, and an infrared camera captures gesture motion coordinates at a rate of 30 frames per second. The preprocessing unit performs multi-source data time alignment: after the voice signal is filtered to remove environmental noise, a timestamp labeling algorithm is applied to generate time-labeled audio frames; gesture coordinate data is generated as a continuous time series through a motion trajectory interpolation algorithm. The signal synchronization calibration unit uses the system clock as a reference to perform time domain interpolation on audio frames and video frames, ensuring that the time deviation of different modal data does not exceed ±5 milliseconds. The processed voice text data and gesture coordinate matrix are merged into a time-aligned multi-modal data stream, stored in a shared memory buffer and output to the multi-modal feature extraction module.

[0068] The input of the incremental analysis module is the above-mentioned multi-modal data stream. In the module initialization stage, the incremental clustering center vector dimension parameter (fixed at 32 dimensions) is loaded from the system configuration file, and a random initial clustering center vector is generated. The incremental clustering error threshold is preset to 0.01. The module divides the real-time input multi-modal feature map into an incremental data set, and the division mechanism uses an equal division algorithm: the feature map tensor is divided into continuous data blocks, each containing 100 feature vector samples. The data block processing unit sequentially reads the data blocks, calculates the Euclidean geometric distance between the new data block mean vector and the current clustering center, and this distance value is defined as the clustering center offset. The offset monitoring unit continuously compares the offset value with the error threshold: when the offset exceeds 0.01, the weight update unit activates the gradient descent algorithm of the flight state prediction model. The model uses a linear regression structure, and the weight matrix is adjusted by the gradient descent method with a step factor of 0.001; if the offset does not exceed the threshold, the original weight parameters remain unchanged. The updated model performs prediction operations on the current data block to generate incremental state prediction results: stored in the real-time database as a floating-point vector, with vector elements corresponding to the probability distribution of different flight states. The vector is pushed to the parameter optimization layer through a direct memory transfer channel, and the entire processing cycle is completed within 50 milliseconds, and the cycle trigger is realized through a hardware timer.

[0069] During the data flow process, the flight environment situation map and device operation health map output by the multi-modal feature extraction module are converted to tensor format. The feature map buffer uses a ring queue structure, and the newly generated incremental data set covers the earliest historical data. The implementation of the dynamic sampling strategy depends on the flight phase identifier: this identifier automatically switches the sampling frequency by analyzing the rate of change of attitude parameters (such as vertical speed exceeding ±5 meters / second to determine the take-off and landing phase). The sensor resource allocator adjusts the working mode of the data acquisition unit according to the current sampling strategy: the accelerometer in the high-resolution focus area is enabled in 100Hz sampling mode during take-off and landing, and switched to 50Hz during cruising; the temperature sensor in the peripheral area maintains a fixed sampling rate of 10Hz.

[0070] Implementation details of time alignment algorithm: The time labels of audio frames and video frames are generated by high-precision clock chips, and the synchronization calibration unit establishes a timestamp mapping table. The interpolation algorithm uses cubic spline interpolation to compensate for the missing data at the time point to generate a continuous sequence. The preprocessed data stream is packaged into a structured data packet, and the packet header includes the timestamp, data type identifier and data length field. The data packet is transmitted to the data receiving queue of the incremental analysis module through the high-speed serial bus.

[0071] In the incremental clustering process, the data block division algorithm uses a sliding window mechanism: the newly input feature vectors continuously fill the buffer, and when the number of vectors reaches 100, the block processing flow is triggered. The offset calculation step involves feature vector normalization: after each feature vector is converted to a uniform dimension by Z-score standardization, the mean vector of the block sample is calculated. The weight update operation is performed on a dedicated mathematical coprocessor, and the matrix operation is accelerated in parallel using SIMD instructions. The prediction result vector contains 20 elements, each element corresponding to the probability value of a specific flight state label (such as "climb" or "dive"). The vector is transmitted to the input register of the parameter optimization layer through memory-mapped I / O.

[0072] At the hardware level, the data acquisition module implements sensor interface control through an FPGA chip, and the preprocessing unit is equipped with a dedicated digital signal processor. The incremental analysis module is deployed on a multi-core processor, with each core independently processing a data block. The timer interrupt service program triggers the processing flow at a 50ms period, ensuring real-time requirements. The system state monitoring unit records the timestamp and offset value of each clustering center update and writes it into the operation log for maintenance reference.

[0073] Example 2: refer to Figure 3 The implementation process of the parameter optimization layer starts with receiving the incremental state prediction result. The result is stored in the form of a 32-dimensional vector in the shared memory area, and the vector elements represent the probability distribution of different flight states. In the initialization stage, the parameter optimization layer loads the basic parameters of the tabu search algorithm from the system configuration file: the population size is set to 100, the maximum number of iterations is limited to 200, and the length of the tabu table is fixed at 10. The population initialization uses a uniform distribution random number generator to generate 100 real number coded individuals, each individual being a 56-dimensional vector corresponding to the double parameter combination of 28 rules in the operation rule library (priority parameter and trigger threshold parameter). The individual decoding process converts the real number vector into executable rule parameters: the priority parameter is converted to the [0, 1] interval through linear mapping, and the trigger threshold parameter is normalized to the [0.2, 0.8] range using the Sigmoid function. The decoded parameter combination is injected into the rule engine interface of the decision generation module through the data bus.

[0074] The decision generation module initiates a simulation reasoning process upon receiving a new rule parameter combination. This process loads the real-time multi-modal feature map at the current time, including a 32x32 pixel flight environment situation map and a 16x16 pixel device operation health map. The rule engine evaluates all active conditions according to the injected parameter combination, generating a candidate instruction set. The construction of the dynamic performance evaluation matrix is based on the simulation execution results: the row dimension corresponds to the task type (such as "emergency obstacle avoidance", "flight path correction", etc. 20 preset tasks), and the column dimension records the resource consumption indicators (including time delay, energy consumption and system load three sub-items). The matrix element value is calculated by the following formula:

[0075]

[0076] Wherein: represents the normalized performance score of the ith task on the jth resource, is the weight coefficient of the kth resource (preset value is 0.4, 0.3, 0.3), is the actual measured value, and represent the lower and upper limit reference values of the kth resource respectively. After the matrix is generated, the parameter optimization layer is returned through the cache.

[0077] The neighborhood solution generation operation uses improved genetic algorithm operators. The selection stage uses a tournament selection strategy to randomly select 5 individuals from the current population, and the top 2 individuals with the highest performance scores are retained into the mating pool. The crossover operation implements two-point crossover: randomly select two partition points in the 56-dimensional vector, and exchange the gene sequences of the intermediate segments of the parent individuals. The mutation operation uses a non-uniform mutation strategy, with the mutation rate decreasing linearly from the initial value of 0.1 to 0.01 as the iteration number increases. After decoding and verification, the newly generated offspring individuals replace the individuals with the lowest scores in the population. The taboo table management unit maintains a first-in-first-out queue to record the parameter combination hash values used in the last 10 iterations, preventing repeated searches.

[0078] When the iteration counter reaches 200 or the global optimal solution does not improve for 30 consecutive iterations, the algorithm terminates. The final output of the optimized parameter combination is transmitted to the persistent storage area of the decision generation module through the DMA channel. This storage area uses a double buffering mechanism to ensure that the parameter update process does not affect real-time decision making. The optimized parameter combination is also backed up to the system configuration database as the initial parameters for subsequent task execution.

[0079] The core component of the decision generation module is the fuzzy inference engine. The engine input includes optimized rule parameters and real-time updated multi-modal feature maps. The feature map preprocessing unit down-samples the 32x32 flight environment situation map to an 8x8 feature matrix, and the device operational health map is converted into a 16-dimensional feature vector. The fuzzification process maps the input feature values to 7 linguistic variable levels ("very low", "low", "relatively low", "medium", "relatively high", "high", "very high"), and the membership function adopts a triangular distribution. The inference machine performs parallel rule evaluation, processing 8 rules per clock cycle, and the activation strength calculation uses the min-max composition method.

[0080] The implementation of the dynamic confidence scoring system is based on the Bayesian probability framework. The initial candidate instruction set is generated by the fuzzy inference engine, containing 15-20 instructions to be selected. In the scoring parameter initialization stage, the basic confidence weight is set to 0.5 and the learning rate parameter is set to 0.05. Each time a new candidate instruction set is generated, the system retrieves the execution records in similar scenarios from the operation history database and updates the conditional probability distribution. The scoring update formula introduces a time decay factor, so that the weight of recent data is higher than that of historical data. The convergence determination unit monitors the score change rate of 5 consecutive iterations, and triggers the output latch when the maximum change rate is less than 0.001.

[0081] The final generated flight operation instruction sequence is encoded in a hierarchical structure. The top layer is the task type identifier (4 bytes), the middle layer contains the parameter list (8-byte floating-point number for each parameter), and the bottom layer adds the timestamp and priority marker. The sequence optimizer performs topological sorting on the instructions to ensure that instructions with dependencies are executed in the correct order. The output interface encapsulates the sequence into a TCP / IP data packet and writes it to the instruction buffer in shared memory. The hardware accelerator parses the data packet content and converts it into bottom-level control signals to drive the actuators of the flight simulator.

[0082] At the hardware architecture level, the parameter optimization layer is deployed on a dedicated computing node equipped with a 12-core processor with a 2.8 GHz clock frequency and 64 GB of memory. The decision generation module uses FPGA to implement fuzzy inference acceleration, and is equipped with 8 GB GDDR6 video memory for storing the rule base. The calculation task of the dynamic performance evaluation matrix is assigned to 4 CUDA cores, which communicate with the main processor through a PCIe 4.0x16 bus. The system uses redundant power supply design, and all key data paths have ECC memory protection. The real-time performance monitoring unit records the time consumption and resource usage of each iteration, with a data sampling interval of 100 milliseconds.

[0083] The data persistence scheme adopts a log-structured merge tree storage engine. The optimized parameter combination is stored in the form of key-value pairs, with the key being the MD5 hash value of the parameter combination and the value being the serialized parameter structure. The operation history database uses time partition tables, with each partition storing the execution records within 24 hours. The system maintenance daemon periodically compresses the historical data and archives the records exceeding 30 days to offline storage. The network communication module supports the breakpoint resume function and can resume the unfinished parameter transmission task after system restart.

[0084] The exception handling mechanism includes two levels of parameter checking and instruction verification. The parameter checking unit checks the value range and logical consistency before injecting new parameters each time. The instruction verifier simulates the execution of the generated instruction sequence to detect possible resource conflicts or constraint violations. When a serious error is detected, the system automatically rolls back to the last stable version of parameter configuration and notifies the maintenance personnel through the event bus. All abnormal events are recorded in the system log with a complete context state snapshot.

[0085] The clock synchronization system uses the PTP precise time protocol, and all computing nodes are equipped with rubidium atomic clocks as time references. The distributed lock manager coordinates the access of multiple modules to shared resources to avoid race conditions. Data consistency is guaranteed by the two-phase commit protocol, and critical operations require the confirmation of a majority of nodes to be committed. The system performs a self-check program when starting to verify the connection status of all sensors and actuators and loads the parameter configuration snapshot at the last normal shutdown.

[0086] Embodiment 3: refer to Figure 4 The implementation process of the execution optimization layer starts with receiving the flight operation instruction sequence output by the decision generation module. The sequence uses a hierarchical encoding structure, including instruction type identifiers, parameter lists, and timestamp information. The instruction parser first performs syntax analysis on the sequence, converting each instruction to an internal representation: the operation code field is mapped to the predefined 128 basic operations, the operand field stores floating-point parameter values, and the constraint condition field records the execution limitations of the instruction. The parsed instruction set is stored in a ring buffer with a capacity of 50 instructions, and the producer-consumer mode is used for read-write synchronization.

[0087] The constraint handling unit loads the set of operation sequence constraints defined in the system configuration file. This set contains three types of constraints: timing constraints specify the minimum / maximum time interval between instructions, resource constraints limit the number of concurrent operations for each subsystem (e.g. power, hydraulic, avionics), and logic constraints prohibit the occurrence of certain instruction combinations. Constraints are stored in predicate logic form, e.g. "the time interval between instruction A and instruction B ≥ 100ms" is represented as a timing constraint, and "no more than 3 operations are executed concurrently by the hydraulic system" as a resource constraint. The projection step parameter is initialized to 0.1, which is dynamically adjusted based on historical optimization results, with an adjustment range limited to ±0.02.

[0088] The implementation of the subgradient projection algorithm consists of three main stages. The initialization stage constructs the objective function: aiming to minimize the total execution delay, the cost function is defined as the weighted sum of the delays of each instruction. The weight coefficients are determined according to the instruction priority, with high-priority instructions set to a weight of 0.7 and normal instructions set to 0.3. In the projection direction calculation stage, the algorithm first solves the subgradient of the objective function:

[0089]

[0090] where: is the subgradient vector at the t-th iteration, is the current number of active instructions, is the weight coefficient of the i-th instruction, represents the execution delay of the instruction, is the time adjustment variable, is the Lagrange multiplier (initial value 0.5), represents the penalty term for violating constraints. The calculation process is performed on a mathematical coprocessor, utilizing SIMD instructions to parallel process the gradient components of multiple instructions.

[0091] The iteration update stage uses the projected subgradient descent method. In each iteration, the algorithm first updates the instruction time parameters along the negative direction of the subgradient: the new time value is equal to the original value minus the product of the step parameter and the subgradient component. Then the projection operation is performed to map the updated time value to the feasible region defined by the constraint conditions. For timing constraints, the projection operation ensures that the adjusted time stamp satisfies all the precedence relations; resource constraints are allocated by an integer programming algorithm, using the branch and bound method for solution; logic constraints are verified using a rule engine, and prohibited instruction combinations are automatically rescheduled. After each projection operation, the algorithm checks the convergence condition: the change rate of the objective function for the last 10 iterations is less than 0.0001, or the total number of iterations reaches the upper limit of 50.

[0092] After the optimization of the instruction sequence, the integer programming module processes the discretization problem. The module converts continuous time values into specific execution clock cycles, with a system base clock frequency of 10 MHz, thus achieving a time resolution of 100 ns. The conversion process takes into account the pipeline characteristics of the hardware execution unit: each instruction is decomposed into micro-operations, and the micro-operation duration must be an integer multiple of the clock cycle. The planning algorithm uses an improved constraint satisfaction method, first establishing a task graph containing all micro-operations and their dependencies, and then applying a list scheduling algorithm to generate the final execution sequence. During scheduling, the algorithm dynamically adjusts the load balancing of each functional unit (such as the floating-point operator, memory interface) to avoid resource conflicts.

[0093] The final output of the execution sequence uses a two-level buffer structure. The first level buffer stores the optimized abstract instruction sequence for the system monitoring module; the second level buffer contains the converted hardware microcode, which directly drives the execution unit. The two levels of buffer are connected through a dedicated bus, with a data transfer rate of up to 8 GB / s. The hardware interface module converts the microcode into specific control signals: digital output controls the state of the relay through the GPIO pin, and analog output adjusts the position or speed of the actuator after DAC conversion. Each output channel is equipped with an independent watchdog timer that automatically triggers a safety recovery process if no update signal is received within a specified time.

[0094] The exception handling mechanism runs throughout the execution optimization process. The constraint violation detector checks the projection results after each iteration and activates the recovery program when it finds that the constraints cannot be met: it first attempts to relax the restrictions of secondary constraints, and if it still cannot be solved, it requests the decision generation module to re-plan the instruction sequence. The timing exception monitoring unit compares the actual execution time with the predicted value, and triggers online rescheduling when the deviation exceeds 20%. All exception events are recorded in the system log, accompanied by complete context state information, including the current instruction sequence, resource occupation, and system load indicators.

[0095] The system maintenance module provides a configuration interface for the execution optimization layer. Administrators can dynamically adjust the strictness of the constraint conditions, modify the projection step parameters, or switch to different optimization algorithm strategies. Configuration changes are guaranteed to be atomic through the transaction mechanism: the new configuration is verified in memory before being committed, and the old configuration is retained as a rollback point. The performance statistics module records key indicators of each optimization, including the number of iterations, the final objective function value, and the constraint satisfaction rate, which are used for subsequent analysis of the behavior characteristics of the optimization algorithm.

[0096] In hardware implementation, the execution optimization layer is deployed on computing nodes with real-time characteristics. The host processor adopts a multi-core architecture, in which two dedicated cores handle the subgradient computation, one core manages the constraint satisfaction, and the other core is responsible for exception handling. The mathematical acceleration unit is equipped with 128-bit floating-point operators, supporting parallel processing of multiple gradient components. The instruction buffer uses ECC memory protection, and all critical data paths have redundant designs. The system communicates with other modules through time-triggered Ethernet, ensuring deterministic latency for message transmission. The power management system provides multiple independent power supplies and can seamlessly switch to backup power when voltage anomalies are detected.

[0097] The data persistence scheme adopts a multi-version concurrency control mechanism. The intermediate state during optimization is regularly saved to non-volatile memory, and each checkpoint contains a complete snapshot of the system state. Historical execution sequences are archived in a time series database, supporting fast retrieval by time range or task type. The maintenance tool can replay the execution process at any point in time for analyzing optimization decisions in specific scenarios. All persistent operations are recorded in a write-ahead log, and the system recovers a consistent state by replaying the log upon restart.

[0098] The clock synchronization system adopts a hybrid scheme: the master clock source comes from a GPS disciplined rubidium clock, and each computing node synchronizes its local clock through the IEEE 1588 protocol. Critical timing operations are implemented using hardware timers, with an error of no more than 1 microsecond. The distributed lock manager coordinates access to shared resources by multiple execution units, using an optimistic concurrency control strategy to reduce waiting time. The system performs a self-check program upon startup, verifying the readiness status of all hardware components and loading the parameter configuration of the last successful optimization. The runtime integrity check module continuously monitors the checksum of critical data structures and automatically triggers a recovery program upon detecting memory errors.

[0099] Example 4: The feedback calibration module processes simulator response data through a multi-source filtering algorithm, with a workflow starting from the data acquisition stage. The flight environment sensor group includes a three-axis accelerometer, a barometric altimeter, and a GPS receiver, which output raw measurement values at frequencies of 100 Hz, 50 Hz, and 10 Hz, respectively. The student physiological monitoring equipment includes an electroencephalogram acquisition helmet (sampling rate 256 Hz) and a heart rate band (sampling rate 1 Hz), and all sensor data is sent to the preprocessing queue after timestamp alignment. The data synchronization unit uses a hardware-level time stamping scheme, based on the PTP protocol to control the time deviation of each data source within ±2 ms. The structure of the raw data packet includes a device identifier, a timestamp, and a data payload, for example, the payload field of the barometer data packet stores the current barometric pressure value (unit hPa) and the temperature compensation coefficient.

[0100] The core of the multi-source filtering algorithm is the construction of the state space model. The system defines two types of state variables: the flight environment state vector contains 12 parameters such as altitude, airspeed, pitch angle, etc.; the cognitive state vector is composed of electroencephalogram power spectrum features (δ, θ, α, β band energy proportion) and heart rate variability indicators. The state transition equation describes the dynamic evolution law of these parameters, for example, the relationship between altitude change and vertical speed is modeled by a first-order kinematics model. The observation equation establishes the mapping relationship between sensor data and state variables, and the nonlinear relationship between barometer readings and altitude is realized by table lookup method. The Kalman filter implementation adopts the square root covariance form, which has better numerical stability than the conventional implementation. The filtering process produces two key outputs: the calibrated flight environment situation deviation (a three-dimensional vector representing the difference between actual and expected pitch, roll, and yaw angles) and the cognitive behavior deviation vector (a four-dimensional vector quantifying the differences in attention concentration, workload, and other cognitive indicators).

[0101] The application of calibration data is reflected in the coordinated work of multiple modules of the system. The flight environment situation deviation is pushed in real time to the feature correction unit of the multi-modal feature extraction module, which maintains a dynamic compensation matrix to correct the angle-related parameters in the original feature map. For example, when the pitch angle deviation exceeds 0.5 degrees, the height channel data of the feature map will be compensated accordingly. The cognitive behavior deviation vector is transmitted to the behavior analyzer of the portrait management module, which combines the vector with the current task difficulty coefficient to generate a cognitive load assessment report. A priority marking mechanism is used in the data flow process, and high-priority flight safety-related data (such as sudden altitude deviation) can interrupt the transmission of low-priority data.

[0102] The cognitive behavior portrait backtracking function of the portrait management module is activated at a preset period. The system maintains a ring-shaped portrait database, each record containing a timestamp, a task identifier, and a cognitive state snapshot. When the management period (default 5 minutes) arrives, the backtracking engine performs the following operations: retrieves historical portraits from the database within the last three periods, extracts key parameters including task duration, operation error rate, and cognitive load score. The comparison between the real-time portrait and the historical portrait is completed by a difference analysis algorithm, which first aligns the time series data by dynamic time warping, and then calculates the root mean square error of each dimension. The generated portrait deviation vector contains four components: task period deviation (percentage difference between actual and expected time), cognitive load deviation (difference between current load and historical average), operation precision deviation (error rate change), and attention fluctuation deviation (standard deviation of electroencephalogram α band energy).

[0103] The operation data deviation modulus set generation process of the data management engine involves spatio-temporal feature alignment. The system defines a four-dimensional coordinate system: three-dimensional spatial position (X / Y / Z axes) plus a time dimension, and each coordinate point corresponds to a flight phase and an operation context. The feature alignment unit first maps the grid coordinates of the flight environment situation map to this four-dimensional space, and then matches the monitoring points of the equipment operation health map through nearest neighbor interpolation. In the aligned region, the system extracts the execution delay data of the operation instruction, and calculates the absolute deviation thereof from the reference delay threshold value. The reference threshold value is defined according to the flight manual, for example, the standard delay of the throttle adjustment instruction in the cruising phase is 120 ms. The storage structure of the deviation modulus set is shown in Table 1.

[0104] Table 1: Storage structure of the deviation modulus set.

[0105] Coordinate block ID Spatial position Time window Instruction type Reference delay (ms) Measured delay (ms) Deviation modulus B-0127 X: 12 / Y: 45 / Z: 2800 00:05:21-00:05:23 Throttle adjustment 120 138 18 B-0128 X: 15 / Y: 47 / Z: 2850 00:05:24-00:05:26 Course correction 80 92 12 B-0129 X: 10 / Y: 40 / Z: 2750 00:05:27-00:05:29 Altitude hold 150 143 7

[0106] The data cleaning strategy is based on the dual standards of variance analysis and timeliness. The engine first calculates the statistical variance of each deviation modulus set, and when the variance is lower than the threshold value 0.05, it is determined as low-value data; secondly, it checks the data storage time length, and records exceeding 30 days are automatically marked as items to be cleaned. The deletion operation adopts a lazy strategy, and the actual space recovery is performed when the system is idle. Meta-data summaries are retained during the cleaning process, including the maximum / minimum deviation values and the most frequently occurring instruction types for each block.

[0107] In terms of hardware architecture, the feedback calibration module is deployed on an embedded system with real-time processing capability, the main processor is a dual-core Cortex-A72, and a dedicated DSP is provided to process the filtering algorithm. Sensor data is collected through an isolated ADC channel, and the sampling clock is provided with a stable reference by a constant temperature crystal oscillator. The image database uses NVMe SSD storage, and data redundancy is ensured through RAID1 configuration. The abnormal processing unit monitors the integrity of each data stream, and when sensor failure or data verification error is detected, it automatically switches to a backup data source or enables a predictive compensation mode.

[0108] The system maintenance interface provides calibration parameter adjustment functions. Administrators can modify the process noise matrix of the Kalman filter, adjust the parameters of the state transition model, or redefine the cognitive behavior index weights. All configuration changes are recorded in the audit log, including modification time, operator ID, and parameter values before and after modification. The performance monitoring panel displays key indicators such as data processing delay, filter convergence state, and storage space usage in real time. The debugging mode supports raw data recording and playback, which facilitates the analysis of system behavior in specific scenarios.

[0109] The clock synchronization system adopts a hybrid scheme: the master clock source comes from the 1PPS signal of the GPS receiver, and each subsystem synchronizes the local clock through the IEEE1588v2 protocol. Key timing tasks such as sensor sampling and data packaging are triggered by hardware timers to ensure timing accuracy. The power management system monitors the power consumption status of each module and automatically isolates the faulty unit when abnormal power consumption is detected. The startup self-test program verifies the functional integrity of all hardware components, including memory testing, communication loop detection, and sensor calibration verification.

[0110] The data persistence scheme adopts a hierarchical storage structure. Raw sensor data is retained for 7 days, calibrated intermediate data is retained for 30 days, and portrait data and deviation analysis results are retained for 1 year. Archival data is transferred to an offline storage system after compression and encryption. The network transmission module supports breakpoint continuation and bandwidth adjustment to adapt to different network environment conditions. All communication channels are encrypted with AES-256, and the key is rotated every 24 hours. The system log records detailed operation traces, including data flow path, processing time consumption, and resource occupation, and the log file is updated daily and automatically uploaded to the central log server.

[0111] In embodiment 5, the portrait management module maintains a real-time storage structure of cognitive behavior portraits, each portrait object containing a timestamp field, a task period parameter, and a multi-dimensional cognitive load parameter. The task period parameter records the duration from task start to the current time (millisecond level accuracy), and the cognitive load parameter consists of four sub-items: electroencephalogram concentration index (0-100 scale), heart rate variation coefficient (standard deviation to mean percentage), operation error frequency count, and visual attention allocation index. The historical portrait database adopts a time-sharing storage architecture, generating a snapshot file every 5 minutes and storing it in non-volatile memory. When the preset management period (default 5 minutes) is triggered, the backtracking retrieval unit locates the snapshot files of the previous three periods, and the file naming contains an accurate time identifier. The file reading process uses direct memory mapping technology to avoid data copy overhead.

[0112] The decoding process of the cognitive behavior profile first generates a set of historical parameters. The task cycle parameter is extracted from a fixed offset in the file header (address 0x00-0x08) as a 64-bit integer value, which is converted into time length data. The four sub-items of the cognitive load parameter are stored in consecutive memory regions (addresses 0x10-0x27): the brain concentration index is 4 bytes of floating-point type, the heart rate variation coefficient is stored as 2 bytes of fixed-point number (resolution 0.01%), the operation error frequency is 2 bytes of unsigned integer, and the visual attention allocation index is encoded as 4 bytes of floating-point. The real-time cognitive behavior profile is obtained through a shared memory interface, and the data format is consistent with the historical file. The comparison algorithm performs field-by-field difference operation: the task cycle deviation component is calculated as (current cycle - historical cycle) / historical cycle percentage value; the cognitive load deviation component exists in vector form, and each dimension corresponds to the absolute value difference of the above four sub-items. The result is packaged as a deviation vector structure, containing four single-precision floating-point fields.

[0113] After receiving the vector, the data management engine starts the spatial mapping process. The spatiotemporal feature map coordinate system is defined as a three-dimensional space grid (grid size 0.5m x 0.5m x 100ft) superimposed with a time window (length 200ms). The engine maintains a coordinate mapping table to convert pixel coordinates in the flight environment situation map to spatial grid IDs. The monitoring point coordinates of the device running health map are aligned to the same grid system through a spatial transformation matrix: the gyroscope data point coordinates are converted using affine transformation, and the oil pressure sensor position uses a rigid body transformation model. The aligned spatiotemporal blocks generate unique identifiers in the format "Grid-X:Y:Z_Time Start Stamp", for example, block "G-1278_093045000" represents the data unit of spatial grid number 1278 at 09:30:45.000.

[0114] The collection of instruction delay feature values within the block relies on the precise timing system. The start time of instruction execution is recorded by the operation code analysis module (based on the system clock counter), and the end time is captured through the state register of the execution unit. Each block stores the execution delay records of the last 20 instructions, and the delay value is the difference between the end timestamp and the start timestamp (nanosecond-level resolution). The threshold value of the reference delay is dynamically adjusted according to the operation type: the threshold value of the regular operation is preset to 120ms, and the threshold value of the emergency operation is set to 80ms. The deviation module value calculation adopts a simplified implementation of the Euclidean distance formula: taking the absolute value of the difference between the execution delay value and the threshold value. The deviation module values of all instructions within the block form a temporary set, and the execution memory is compressed and stored (each module value is represented by 2 bytes of fixed-point number).

[0115] The aggregation process of the set of bias modulus adopts a divide-and-conquer strategy. The system divides the space into 8x8 grid upper blocks (4m x 4m x 800ft range), and the bias modulus data of the same level block is transmitted through the tree collection protocol. After receiving the modulus list of each sub-block, the upper block performs sorting and truncation operations: the top 50 modulus values with the largest values are retained, and the remaining data is deleted to avoid storage overflow. After aggregation, the set of operation data bias modulus of the whole space is formed, and the storage format is a key-value pair database, the key is the block identifier, and the value is a variable length modulus array. This database supports fast query through hash index, and the response time is controlled within 5 milliseconds.

[0116] The data cleaning mechanism implements two-level screening standards. Time effectiveness screening: each storage unit is marked with the latest access timestamp, and the periodic task (started at 00:00 every day) deletes records over 30 days. Value screening: calculate the median and variance of the modulus array in the block, and when the median is less than 20 and the variance is less than 5, mark the entire block as a low-value area. The deletion operation adopts a log-structured merge strategy: new data is written to a separate partition, and when merging, skip the marked block. The deletion operation also removes the corresponding metadata records in the portrait database.

[0117] In terms of system hardware deployment, the portrait management module adopts a dual-redundant server architecture. The main node runs on a 2.4GHz processor, equipped with 64GB of memory and a 1TB NVMe solid state disk; the standby node remains in a hot standby state and automatically switches through heartbeat detection. The data management engine is deployed on an FPGA platform with hardware acceleration, integrated with a PCIe interface for communication with the main system. The dedicated processor for space-time coordinate mapping contains 1000 parallel computing units with a processing clock frequency of 400MHz. The power system is configured with dual-channel UPS backup to ensure the integrity of data storage.

[0118] The maintenance interface provides parameter adjustment functions. Administrators can modify the partitioning rules of the space-time grid: the grid size range can be adjusted from 0.2-1 meters, and the time window length can be configured in the 50-500 millisecond interval. The calculation weight of cognitive load parameters supports online update: the brain electric concentration weight is set to 0.4 by default, and other parameter weights can be dynamically adjusted. The system monitoring module displays the portrait data processing delay, bias calculation efficiency, and storage space usage rate curves during system operation. The log recorder saves the input and output data of each comparison operation in detail, including timestamp, bias vector value, and involved historical portrait version number. The debugging mode allows importing external data sets to replay the processing flow and verify the system behavior in specific scenarios.

[0119] The abnormality processing unit monitors the data consistency status. When detecting the historical profile data damage, the interpolation compensation of adjacent period data is automatically triggered: taking the mean value of the corresponding parameters of the previous and next periods to replace. When the space-time block mapping error occurs, the system falls back to the basic grid division scheme (1 meter x 1 meter x 200 feet). All error events generate a unique identification code and report to the central monitoring system, while freezing the current operating environment data for subsequent analysis. The clock synchronization system maintains nanosecond-level time consistency through the IEEE1588 protocol, and the key timestamp generation uses the onboard atomic clock pulse. The communication protocol uses encrypted data transmission, and performs two-way authentication before each session establishment.

[0120] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0121] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, alternatives, and variations can be made in the embodiments without departing from the spirit and scope of the present application as defined by the appended claims and their equivalents.

Claims

1. A multimodal human-computer interaction aircraft simulation training system, characterized in that, include: The multimodal data acquisition module dynamically acquires flight attitude parameters, airborne equipment status parameters, and cockpit interaction data streams to generate multimodal data streams. A multimodal feature extraction module receives the multimodal data stream and generates a multimodal feature map through a hierarchical feature extraction network. The multimodal feature map includes a flight environment situation map and an equipment operation health map. The incremental analysis module divides the multimodal feature map into an incremental dataset, updates the parameters of the flight state prediction model through an incremental clustering algorithm, and outputs the incremental state prediction results. The parameter optimization layer initializes the tabu search population based on the incremental state prediction results and iteratively optimizes the operation rule base parameters through a dynamic performance evaluation matrix. The decision generation module integrates optimized operation rule base parameters with real-time multimodal feature maps to generate flight operation command sequences based on dynamic confidence scores. The execution optimization layer parses the flight operation command sequence, satisfies the operation sequence constraints through a constraint projection algorithm, and completes the integer programming of the command sequence by combining the operation profile deviation.

2. The multimodal human-computer interaction aircraft simulation training system according to claim 1, characterized in that, Also includes: The feedback calibration module processes simulator response data through a multi-source filtering algorithm to generate calibrated flight environment situational deviation and cognitive behavior deviation vectors. The profile management module triggers cognitive behavior profile retrospective based on a preset management cycle, and calculates the profile deviation vector between the real-time cognitive behavior profile and the retrospective cognitive behavior profile. The data management engine retrieves the set of operational data deviation modulus values ​​corresponding to the profile deviation vector, and deletes historical operational data based on the variance threshold and storage duration ratio.

3. The multimodal human-computer interaction aircraft simulation training system according to claim 1, characterized in that, The dynamic acquisition of flight attitude parameters, airborne equipment status parameters, and cockpit interaction data streams includes: The high-resolution focus area and the low-resolution peripheral area are divided according to the type of historical flight mission. The sensor sampling density is dynamically adjusted based on the temporal characteristics of the flight phase, and a dense timestamp sampling strategy is adopted in the high-resolution focusing area. The cockpit interaction data stream is preprocessed to generate a time-aligned multimodal data stream.

4. The multimodal human-computer interaction aircraft simulation training system according to claim 1, characterized in that, The step of updating the flight state prediction model parameters using the incremental clustering algorithm includes: Initialize the incremental cluster center vector and the incremental clustering error threshold; Divide the incremental dataset into multiple data blocks, and calculate the cluster center offset of the current data block for each block. When the cluster center offset exceeds the incremental clustering error threshold, update the weight parameters of the flight state prediction model. Output the incremental state prediction results to the parameter optimization layer.

5. The multimodal human-computer interaction aircraft simulation training system according to claim 1, characterized in that, The parameters of the operation rule base are iteratively optimized through the dynamic performance evaluation matrix, including: Decode individuals in the taboo search population into combinations of operation rule base parameters; Inject the combination of operation rule base parameters into the decision generation module to obtain a dynamic performance evaluation matrix; Neighborhood solution generation and tabu table update are performed based on the dynamic performance evaluation matrix; When the maximum number of iterations is reached, the final operation rule base parameters are output.

6. The multimodal human-computer interaction aircraft simulation training system according to claim 1, characterized in that, The generation of flight operation command sequences based on dynamic confidence scoring includes: The real-time multimodal feature map is input into the fuzzy inference engine to generate an initial instruction candidate set; Initialize the dynamic confidence score parameters and generate a new set of instruction candidates in each iteration; When the dynamic confidence score meets the convergence condition, the final flight operation command sequence is output to the execution optimization layer.

7. The multimodal human-computer interaction aircraft simulation training system according to claim 1, characterized in that, The constraint projection algorithm is used to satisfy the operation sequence constraints, including: Set the set of operation sequence constraints and projection step size parameters; Calculate the subgradient projection direction of the flight operation command sequence; Iteratively update the instruction sequence value range within the set of operation sequence constraints; Output a sequence of flight operation commands that satisfies all operation sequence constraints.

8. The multimodal human-computer interaction aircraft simulation training system according to claim 2, characterized in that, The process of processing simulator response data using a multi-source filtering algorithm includes: Collect flight environment sensor feedback data and student physiological response data, construct multi-source filter state transition equations and observation equations, and fuse flight environment situational deviation and cognitive behavior deviation vectors to generate calibration data; The calibration data is fed back to the multimodal feature extraction module and the portrait management module.

9. The multimodal human-computer interaction aircraft simulation training system according to claim 2, characterized in that, The calculation of the profile deviation vector between the real-time cognitive behavior profile and the retrospective cognitive behavior profile includes: Extract historical task cycle parameters and historical cognitive load parameters from the retrospective cognitive behavior profile, compare them with the current task cycle parameters and current cognitive load parameters in the real-time cognitive behavior profile, and generate a profile deviation vector containing task cycle deviation components and cognitive load deviation components. Output the image deviation vector to the data management engine.

10. The multimodal human-computer interaction aircraft simulation training system according to claim 2, characterized in that, The generation of the operational data deviation modulus set includes: Define the coordinate mapping relationship of the spatiotemporal feature map and align the flight environment situation map with the equipment operation health map; Extract the operation instruction execution delay feature value within the spatiotemporal alignment region, and calculate the deviation magnitude between the operation instruction execution delay feature value and the benchmark delay threshold; The deviation modulus values ​​of all spatiotemporal regions are aggregated to form a set of operational data deviation modulus values.

Images