Civil aviation flight situation real-time perception and display method and system based on multi-modal fusion

Abstract

This invention relates to a method and system for real-time perception and display of civil aviation flight situation based on multimodal fusion. The system includes: a data acquisition module, a data synchronization and processing module, a model training module, a real-time data acquisition module, a model inference module, a deviation calculation module, a situation analysis module, and a fusion display module. The system acquires historical aircraft flight attitude data and pilot perception data, performs time alignment and sampling synchronization, constructs aircraft state characteristics and pilot state characteristics, and trains an aircraft-pilot association model based on these to determine a theoretical perception threshold range. During flight, by acquiring aircraft attitude data and pilot perception data in real time, the system calculates the perception deviation vector, and combines the actual aircraft attitude change rate with the expected attitude change rate calculated from operational behavior to generate a flight situation analysis result, which is finally fused and displayed with the current aircraft attitude data.

Description

Technical Field

[0001] This invention relates to the field of flight safety monitoring and flight situation analysis, specifically to a method and system for real-time perception and display of civil aviation flight situation based on multimodal fusion. Background Technology

[0002] With the continuous expansion of civil aviation transport and the increasing complexity of the aviation operating environment, flight safety places higher demands on flight situation monitoring and information presentation. Flight situation generally refers to the aircraft's attitude, motion, and overall relationship with the external environment during flight. Accurate acquisition and effective presentation of this information are crucial for pilots' operational decisions. Currently, civil aircraft cockpits generally use flight instrument systems, integrated avionics systems, and flight data recording systems to collect and display flight data such as aircraft attitude parameters, speed, and altitude in real time, helping pilots understand the aircraft's current operating status. However, these systems primarily focus on displaying the aircraft's physical state information and lack effective means of analyzing the relationship between the pilot's perceived state and operational behavior.

[0003] In actual flight, factors such as the pilot's visual attention distribution, head posture changes, physiological state, and control inputs all significantly impact flight situation assessment. For example, when a pilot operates under high workload conditions, their visual attention area, reaction time, and control actions may change. If these changes are not promptly identified by the system and correlated with changes in aircraft attitude, it may lead to deviations in the pilot's perception of the flight situation. Furthermore, traditional flight display systems typically use fixed parameters or empirical thresholds to indicate flight status. This approach struggles to reflect perceptual differences between pilots and cannot provide real-time assessment of the perceived status based on dynamic changes during flight.

[0004] In recent years, with the development of sensing technology, data processing technology, and artificial intelligence technology, researchers have begun to explore the introduction of multi-source information, such as eye-tracking data, physiological signals, and operational behavior data, into the field of flight safety analysis, modeling pilot behavior through multi-source data fusion. However, most existing technologies focus on single-modal data analysis or simply fuse multi-modal data, lacking a systematic modeling method for the coupling relationship between aircraft state information and pilot perception information. Furthermore, existing methods still have limitations in data time synchronization, cross-pilot differential processing, and real-time flight situation analysis, making it difficult to achieve stable and reliable flight situation awareness and display in actual flight environments.

[0005] Therefore, it is necessary to propose a new flight situation awareness method that establishes the correlation between the aircraft state and the pilot's perceived state by collaboratively analyzing aircraft attitude data and pilot multimodal perception data, and comprehensively analyzes and visualizes the flight situation during real-time flight, thereby providing pilots with more complete and intuitive flight situation information. Summary of the Invention

[0006] In response, this application provides a method and system for real-time perception and display of civil aviation flight situation based on multimodal fusion, so as to at least partially solve the above-mentioned technical problems.

[0007] This application provides a method for real-time perception and display of civil aviation flight situation based on multimodal fusion, including the following steps: Acquire historical flight attitude data of the target aircraft and historical pilot perception data corresponding to the historical flight attitude data; The historical flight attitude data and the historical pilot perception data are time-aligned and sampled to synchronize the aircraft state characteristics and pilot state characteristics. The aircraft-pilot association model is trained based on the synchronized aircraft state features and pilot state features to output the pilot state feature distribution parameters corresponding to each aircraft state feature, and the theoretical perception threshold range is determined based on the distribution parameters. Real-time acquisition of current aircraft attitude data and current pilot perception data; inputting the current aircraft attitude data into the aircraft-pilot association model to obtain the theoretical perception threshold range corresponding to the current aircraft attitude. The perception deviation vector is calculated based on the current pilot perception data and the theoretical perception threshold range. The actual attitude change rate of the aircraft is calculated based on the current aircraft attitude data. The expected attitude change rate is calculated based on the operational behavior data in the current pilot perception data. Based on the perception deviation vector and the operational deviation between the actual rate of change of the aircraft attitude and the expected rate of change of attitude, a flight situation analysis result is generated, and the flight situation analysis result is fused and displayed with the current aircraft attitude data.

[0008] In one possible embodiment, the historical flight attitude data includes pitch angle, roll angle, yaw angle, ground speed, vertical speed, overload, and control surface deflection angle data. The historical pilot perception data includes visual attention data, head posture data, physiological stress data, and operational behavior data; wherein, the visual attention data includes at least fixation point distribution data, pupil diameter variation data, and blink frequency data; the head posture data includes at least head pitch angle, head yaw angle, and head roll angle; the physiological stress data includes at least heart rate variability data and skin conductance activity data; and the operational behavior data includes at least stick displacement data or stick force variation data.

[0009] In one possible embodiment, historical flight attitude data is time-aligned and sampled to synchronize with historical pilot perception data, including: Historical flight attitude data and historical pilot perception data are unified to the same timestamp reference; interpolation is performed on data with inconsistent sampling frequencies; statistics of each feature within a preset time window are extracted, and the aircraft state features and pilot state features are spliced ​​together to form training samples corresponding to flight event segments; wherein, the aircraft state features consist of pitch angle, roll angle, yaw angle, ground speed, vertical speed and overload, and the pilot state features consist of head attitude, pupil diameter change, blink frequency, heart rate variability, skin activity and control stick displacement.

[0010] In one possible embodiment, the training aircraft-pilot association model includes: Using aircraft state features as model input and pilot state features as model output, a deep probabilistic model based on variational autoencoder and Gaussian process regression is constructed. The encoder learns latent variable representations of aircraft state features and pilot state features, and the decoder reconstructs pilot state features based on latent variables and aircraft state features. Finally, Gaussian process regression is used to output the mean and variance parameters corresponding to each pilot state feature. The theoretical perception threshold range corresponding to each pilot's state characteristics is determined based on the mean and variance parameters.

[0011] In one possible embodiment, the training aircraft-pilot association model further includes: A cognitive time dynamics model is established to perform time-series modeling of the leading and lagging relationships of pilot perception data relative to changes in aircraft attitude, thereby obtaining the dynamic cognitive delay time. The cognitive load coefficient is calculated based on the dynamic cognitive delay time, the pilot's individual cognitive delay baseline, pupil diameter variation data, and the resting pupil diameter baseline value. The dynamic cognitive delay time or the cognitive load coefficient is used as a supplementary factor to the pilot state characteristics in the training or real-time analysis of the aircraft-pilot association model.

[0012] In one possible embodiment, the training aircraft-pilot association model further includes: To address the differences in physiological baselines and reaction patterns among different pilots, an adversarial domain adaptive network was established. Shared features are extracted from training samples by a feature extractor, and adversarial training is performed on the pilot identity domain information by connecting a domain discriminator through a gradient inversion layer. The aircraft-pilot association model is updated with a weighted combination of reconstruction loss, domain discrimination loss, and temporal consistency loss as the optimization objective to obtain a perceptual feature representation shared across pilots.

[0013] In one possible embodiment, the step of calculating the perception deviation vector based on the current pilot perception data and the theoretical perception threshold range includes: Input the current aircraft attitude data into the aircraft-pilot association model to obtain the mean and variance parameters of the pilot state characteristics corresponding to the current aircraft attitude. The difference between the current pilot's perceived data and each mean parameter is normalized according to the corresponding variance parameter to obtain the perception deviation components of each dimension; and the perception deviation components are combined to form a perception deviation vector; the operation deviation is determined by the difference between the actual rate of change of aircraft attitude and the expected rate of change of attitude calculated by the aircraft dynamics model from the control stick displacement data.

[0014] In one possible embodiment, generating flight situation analysis results includes: When an abnormal flight attitude is detected, a counterfactual query is constructed based on the current operational behavior data. The current operational behavior data is then input into the aircraft dynamics model to calculate the expected attitude under unbiased perception conditions. The operation-perception consistency coefficient is determined based on the difference between the expected attitude and the actual attitude; then, the flight deviation is classified according to the operation-perception consistency coefficient and the perception deviation vector. The classification includes at least three types: perception error-dominated, execution error or external disturbance-dominated, and composite factor-dominated.

[0015] In one possible embodiment, generating flight situation analysis results further includes: Based on the dwell time of the gaze posture indicator, the total time window, heart rate variability data, and the resting heart rate variability baseline value, the attention resource utilization rate is calculated; and the attention resource utilization rate, the cognitive load coefficient, the perceptual deviation vector, and the type classification result are written into the fused display data; wherein, the fused display data includes at least basic flight parameter display data, perceptual deviation graphic display data, deviation type identification data, and status prompt data.

[0016] In another aspect, this application also provides a real-time civil aviation flight situation awareness and display system based on multimodal fusion, comprising: The data acquisition module is used to acquire historical flight attitude data of the target aircraft and historical pilot perception data corresponding to the historical flight attitude data; The data synchronization processing module is used to time-align and sample the historical flight attitude data with the historical pilot perception data to construct aircraft state characteristics and pilot state characteristics. The model training module is used to train an aircraft-pilot association model based on the synchronized aircraft state features and pilot state features, so as to output the pilot state feature distribution parameters corresponding to each aircraft state feature, and determine the theoretical perception threshold range based on the distribution parameters. The real-time data acquisition module is used to acquire current aircraft attitude data and current pilot perception data in real time. The model inference module is used to input the current aircraft attitude data into the aircraft-pilot association model to obtain the theoretical perception threshold range corresponding to the current aircraft attitude. The deviation calculation module is used to calculate the perception deviation vector based on the current pilot perception data and the theoretical perception threshold range, calculate the actual attitude change rate of the aircraft based on the current aircraft attitude data, and calculate the expected attitude change rate based on the operational behavior data in the current pilot perception data. The situation analysis module is used to generate flight situation analysis results based on the perception deviation vector and the operational deviation between the actual rate of change of the aircraft attitude and the expected rate of change of attitude. The fusion display module is used to fuse the flight situation analysis results with the current aircraft attitude data for display.

[0017] This application constructs a correlation analysis mechanism between aircraft attitude data and pilot multimodal perception data to collaboratively model aircraft state information and pilot perception state during flight, and completes multi-source data synchronous processing under a unified time reference, thereby achieving comprehensive analysis and fusion display of flight situation. By establishing an aircraft-pilot correlation model, the distribution parameters of pilot perception characteristics are predicted based on aircraft state characteristics, and the theoretical perception threshold range is determined. Then, the perception deviation vector is calculated by combining real-time acquired pilot perception data, and jointly analyzed with the actual aircraft attitude change rate and the expected attitude change rate inferred from operational behavior, which can distinguish and identify the sources of flight deviation.

[0018] Simultaneously, by integrating flight situation analysis results with aircraft attitude parameters through a unified display method, pilots can simultaneously obtain aircraft operational status and related perception deviation information. Compared to traditional technologies that only display situational awareness based on aircraft status parameters, this invention introduces pilot perception and operational behavior information during flight situation analysis, establishing a correspondence between aircraft status and pilot perception status. This helps to more comprehensively reflect the human-machine interaction status during flight, providing a more complete data foundation for flight situation monitoring and information presentation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a real-time perception and display method for civil aviation flight situation based on multimodal fusion, provided as an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of the aircraft-pilot association model structure provided in an embodiment of this disclosure.

[0022] Figure 3 This is a schematic diagram of the structure of a real-time civil aviation flight situation perception and display system based on multimodal fusion, provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] It should be noted that all user information (including but not limited to user device information, user personal information, object information corresponding to device usage data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, device usage data, etc.) involved in all embodiments of this application are information and data authorized by the user or fully authorized by all parties.

[0025] This implementation method is applicable to civil aviation transport aircraft, flight training simulators, and aircraft cockpit environments with flight attitude acquisition capabilities. During system deployment, the cockpit side must at least have a flight data recording interface, eye-tracking acquisition device, head attitude acquisition device, physiological signal acquisition device, operational behavior acquisition interface, and a display terminal. Training samples are generated during the historical phase, and analysis and display are performed during the real-time phase. The aircraft data and pilot data collected during the historical phase correspond to the same flight mission or the same flight event segment. The data input during the real-time phase uses the same feature definitions, the same time base, and the same normalization rules as the historical phase to ensure consistency in the technical links between model input, model output, bias calculation, and display content.

[0026] For the sake of consistency in the following text, the following terminology is explained: Aircraft state characteristics, as referred to in this text, are the state representation consisting of pitch angle, roll angle, yaw angle, ground speed, vertical speed, and G-force; pilot state characteristics, as referred to in this text, are the perception and operational representation consisting of head attitude, pupil diameter changes, blink rate, heart rate variability, skin activity, and stick displacement; theoretical perception threshold range, as referred to in this text, is the allowable fluctuation range determined by the mean and variance parameters of each pilot state characteristic output by the correlation model, given the aircraft state characteristics.

[0027] The flight event segment referred to in this paper refers to a data segment obtained from continuous flight data according to a preset time window, which can maintain the correspondence between attitude changes and perception changes. The dynamic cognitive delay time referred to in this paper refers to the delay in the time-series response of the pilot's perceived variables relative to the aircraft's attitude variables; this parameter is output by the cognitive time dynamics model and is used to reflect the time lag in the perception, cognition, and operation link. The cognitive load coefficient referred to in this paper is a quantitative index formed by the dynamic cognitive delay time, the pilot's individual cognitive delay baseline, pupil diameter changes, and the resting pupil diameter baseline. The perception bias vector referred to in this paper is a multi-dimensional vector formed by normalizing the difference between the real-time pilot state characteristics and the theoretical mean parameters according to the corresponding variance parameters. The operation bias referred to in this paper is the difference between the actual rate of change of aircraft attitude and the expected rate of change of attitude calculated by the aircraft dynamics model based on the control stick displacement. The operation-perception consistency coefficient referred to in this paper is a quantitative index reflecting the degree of consistency between the expected attitude and the actual attitude corresponding to the current operation input. The attention resource occupancy rate mentioned in the article refers to the degree of cognitive resource occupancy characterized by the pilot's visual persistence on the attitude indicator and changes in physiological stress within a given time window.

[0028] The following detailed description, in conjunction with specific embodiments, illustrates the implementation process of the real-time civil aviation flight situation perception and display method based on multimodal fusion described in this application. It should be noted that these embodiments are merely for explaining this application and not for limiting its scope of protection. Any conventional adjustments or substitutions made by those skilled in the art to the steps without departing from the concept of this application should be included within the scope of protection of this application.

[0029] like Figure 1 As shown in the figure, this application discloses a schematic diagram of a real-time perception and display method for civil aviation flight situation based on multimodal fusion, including the following method steps: Acquire historical flight attitude data of the target aircraft and historical pilot perception data corresponding to the historical flight attitude data; The historical flight attitude data and the historical pilot perception data are time-aligned and sampled to synchronize the aircraft state characteristics and pilot state characteristics. The aircraft-pilot association model is trained based on the synchronized aircraft state features and pilot state features to output the pilot state feature distribution parameters corresponding to each aircraft state feature, and the theoretical perception threshold range is determined based on the distribution parameters. Real-time acquisition of current aircraft attitude data and current pilot perception data; inputting the current aircraft attitude data into the aircraft-pilot association model to obtain the theoretical perception threshold range corresponding to the current aircraft attitude. The perception deviation vector is calculated based on the current pilot perception data and the theoretical perception threshold range. The actual attitude change rate of the aircraft is calculated based on the current aircraft attitude data. The expected attitude change rate is calculated based on the operational behavior data in the current pilot perception data. Based on the perception deviation vector and the operational deviation between the actual rate of change of the aircraft attitude and the expected rate of change of attitude, a flight situation analysis result is generated, and the flight situation analysis result is fused and displayed with the current aircraft attitude data.

[0030] In step S1, the historical flight attitude data of the target aircraft and the historical pilot perception data corresponding to the historical flight attitude data are obtained.

[0031] In some embodiments, historical flight attitude data is read from a fast access recorder, flight management system, or integrated avionics bus, and includes at least pitch angle, roll angle, yaw angle, ground speed, vertical speed, G-force, and control surface deflection angle. Historical pilot perception data is generated through a non-contact or low-intrusion multimodal acquisition link, wherein visual attention data is output from an eye-tracking device, including at least fixation point distribution, pupil diameter variation, and blink frequency; head attitude data is output from a depth camera, including at least head pitch angle, head yaw angle, and head roll angle; physiological stress data is output from millimeter-wave radar or wearable acquisition devices, including at least heart rate variability and skin conductance; and operational behavior data is output from a stick sensor, force feedback interface, or flight parameter interface, including at least stick displacement and stick force rate of change. For each sampling source in the same flight mission, the system records a unified mission identifier, a unified flight segment identifier, and a unified timestamp label for subsequent establishment of event-level samples.

[0032] In step S2, the historical flight attitude data and the historical pilot perception data are time-aligned and sampled to synchronize, thereby constructing aircraft state features and pilot state features.

[0033] According to embodiments of this disclosure, the system first performs time alignment on historical flight attitude data and historical pilot perception data. Specifically, each data stream is first mapped to a unified timestamp reference, and then interpolation processing is performed for cases where sampling frequencies are inconsistent. For attitude quantities that change continuously and have a high sampling frequency, such as pitch angle, roll angle, yaw angle, ground speed, and vertical speed, linear interpolation can be optionally used; for operational behavior quantities with short-term abrupt changes, such as control stick displacement and control stick force rate of change, a combination of piecewise preservation and local linear interpolation can be configured; for low-frequency physiological quantities such as heart rate variability and skin conductance, missing point imputation can be performed first, and then mapped to a unified time grid. Through this processing, a one-to-one correspondence can be formed between historical aircraft state sequences and historical pilot state sequences at the same time, avoiding misalignment problems such as input preceding label or label preceding input in subsequent training phases.

[0034] In one example, the unified time reference is not limited to a single frequency; it only needs to ensure that both types of data fall on the same time axis. If the original frequency of historical aircraft attitude data is lower than that of eye-tracking data, the aircraft attitude data is interpolated first, and then the pilot's perception data is downsampled. If the original frequency of historical aircraft attitude data is higher than that of physiological data, the physiological data is interpolated first, and then a unified window extraction is performed. The resulting synchronization sequence serves as the sole data source for constructing subsequent samples. The aircraft state features and pilot state features discussed later are all generated from this synchronization sequence.

[0035] Specifically, the system segments the synchronized continuous data according to a preset time window. The length of the time window can be configured based on the flight phase and data frequency; for example, it can be configured as a short window to reflect rapid changes during the approach phase, or as a long window to reflect the stable trend during the cruise phase. The window step size and window length together determine the degree of overlap between adjacent segments. Within each time window, statistics are extracted for aircraft state characteristics and pilot state characteristics. The statistics include at least central trend and discrete trend; for attitude, visual, and physiological signals, rates of change, kurtosis, or local fluctuation intensity can also be further extracted. If a statistic is only used for intermediate judgment and not included in subsequent models, it is not written into the final training samples, thus avoiding the generation of intermediate variables that are not used later.

[0036] Aircraft state features are extracted from pitch angle, roll angle, yaw angle, ground speed, vertical speed, and G-force within the window. Control surface deflection angle data, in this implementation, primarily serves as the basis for training sample selection and dynamic model calibration, and is not directly incorporated into the uniformly defined aircraft state feature vector. Pilot state features are extracted from head pitch angle, head yaw angle, head roll angle, pupil diameter variation, blink rate, heart rate variability, skin activity, and stick displacement within the window. This definition is maintained because the input and output of subsequent correlation models, the calculation objects of perception bias vectors, and data extraction in the real-time phase all depend on this feature caliber. If additional observations are introduced during implementation, such as gaze point thermal distribution or stick force change rate, they are used as auxiliary features in feature engineering, but the names and dependencies of the aforementioned core features remain unchanged. Training samples corresponding to flight event segments are formed by splicing statistics within the window.

[0037] In some embodiments, baseline parameters related to each individual need to be established after sample construction. The baseline value for resting pupil diameter is derived from eye-tracking sequences under stable lighting and operational conditions; the baseline value for individual pilot cognitive delay is derived from the average delay between changes in aircraft state and changes in pilot perception during stable flight; the baseline value for resting heart rate variability is derived from statistical values ​​of heart rate variability during stable flight. These baseline parameters do not directly participate in the classification of historical samples, but rather serve as reference values ​​for subsequent cognitive load coefficients, attention resource utilization, and online fine-tuning. Normalization rules are fixed in the historical phase and saved to the parameter configuration file, and are invoked according to the same rules during the real-time phase, avoiding deviation drift caused by using different scaling calibers at different stages.

[0038] Understandably, baseline parameters possess individual pilot attributes; therefore, common parameters and individual parameters should be distinguished during cross-pilot training. Common parameters include a unified time reference definition, window length definition, feature names, and loss function structure; individual parameters include baseline values ​​for resting pupil diameter, resting heart rate variability, and individual pilot cognitive delay. This hierarchical division provides boundary conditions for subsequent adversarial domain adaptive networks and online fine-tuning.

[0039] In step S3, an aircraft-pilot association model is trained based on the synchronized aircraft state features and pilot state features to output the pilot state feature distribution parameters corresponding to each aircraft state feature, and the theoretical perception threshold range is determined based on the distribution parameters.

[0040] According to embodiments of this disclosure, please refer to Figure 2 , Figure 2 This is a schematic diagram of the aircraft-pilot association model structure provided in an embodiment of this disclosure. Figure 2 As shown, the aircraft-pilot association model employs a deep probabilistic model combining variational autoencoders and Gaussian process regression. The model's input is the aircraft state features, and its output is the distribution parameters of the pilot state features, including mean and variance parameters. To maintain a clear input-output link, the training phase receives two sets of data: one part is the aircraft state features, and the other part is the pilot state features aligned with their time. The encoder receives the concatenation result of the aircraft and pilot state features and outputs the latent variable distribution parameters; the latent variables are reparameterized and input to the decoder; the decoder combines the latent variables with the aircraft state features to reconstruct the pilot state features; the Gaussian process regression module receives the latent or intermediate representation from the decoder and outputs the mean and variance parameters corresponding to each pilot state feature. Thus, the training phase forms a complete path from historical samples to distribution parameters, while the real-time phase can obtain the corresponding theoretical perception threshold range with only the current aircraft state features as input.

[0041] According to embodiments of this disclosure, the latent variable is a compressed representation of the pilot's implicit state. This implicit state is not directly measured, but is obtained by the encoder based on synchronized samples. Since the latent variable only serves distribution reconstruction and probability prediction, it is not directly displayed in the real-time phase, nor is it output separately to the display layer.

[0042] Specifically, the encoder can be configured as a multi-layer fully connected network or as a structure combining a one-dimensional temporal convolutional network and a fully connected network. If the input samples have already undergone statistical compression at the window level, a multi-layer fully connected network is sufficient; if the local temporal distribution within the window is preserved, it can be configured to first extract the local temporal pattern through a one-dimensional convolutional layer, and then map it to the latent variable parameters through a fully connected layer. The encoder output includes the latent variable mean and latent variable variance, which are reparameterized to obtain the latent variable samples. The decoder receives the latent variable samples and aircraft state features, and outputs the reconstructed pilot state features. The decoder output can be used directly as the reconstructed value or as the input feature of the Gaussian process regression module.

[0043] The Gaussian process regression module serves to probabilistically model the conditional distribution of various pilot state characteristics. In practical applications, the Gaussian process regression module can be configured to create a separate regression head for each pilot state characteristic, or it can be configured to create multi-output regression heads grouped according to visual, head posture, physiological, and operational categories. The input to each regression head remains consistent: a joint representation of the intermediate representation output by the decoder and the aircraft state characteristics. The output of each regression head corresponds to the mean and variance parameters of a single or group of pilot state characteristics, respectively. Therefore, the theoretical perception threshold range is explicitly limited to the output of the Gaussian process regression module, rather than directly from the reconstructed values ​​themselves. This design ensures that the threshold inference in the real-time stage has distributional significance, rather than single-point prediction significance.

[0044] In one example, if a pilot's state feature has high noise in historical samples, its corresponding regression head can be configured to use a higher kernel width or a higher noise term; if a pilot's state feature fluctuates relatively smoothly, it can be configured with a lower noise term. The noise term here is an internal model fitting parameter and is not directly used in subsequent display layers; therefore, it only participates in the training process and is not used as an external operational parameter.

[0045] The training process uses joint loss for parameter updates. The joint loss includes at least reconstruction loss and divergence loss; after introducing an adversarial domain adaptive network, it also includes domain discrimination loss and temporal consistency loss. Configurably, the total loss is defined as...

[0046] in, Represents the reconstruction loss, used to measure the difference between the reconstructed values ​​of the pilot state features output by the decoder and the historical true values; This represents the divergence loss, used to constrain the distance between the latent variable distribution and the prior distribution; The domain discrimination loss is used to constrain the separability between different pilot identity domains; This represents the timing consistency loss, used to constrain the temporal smoothness and continuity of outputs from adjacent time windows. and These are the weight parameters.

[0047] Regarding the method for determining the weight parameters, in one implementation, and Pre-set before training; in another implementation, and Obtained by searching the validation set; in a further implementation, and The weights are adaptively adjusted based on the magnitude of change in each loss term during training. If an adaptive adjustment method is used, the rate of decrease in each loss term within a given training epoch can be used as a reference. Loss terms that decrease too quickly will have their weights gradually decreased, while those that decrease too slowly will have their weights gradually increased, thus maintaining the contribution of each loss term to the total loss at the same order of magnitude. These weight parameters are updated during the training phase and are fixed in real-time, no longer changing dynamically.

[0048] After model convergence, for any given aircraft state characteristics, the system can obtain the mean and variance parameters corresponding to each pilot's state characteristic. The theoretical perception threshold interval is determined by the following formula:

[0049] in, Indicates the first The theoretical perception threshold range of individual pilot state characteristics; The mean parameter represents this feature; The standard deviation form corresponding to the variance parameter of this feature; This represents the confidence interval coefficient. Regarding... The determination can be configured as an empirical preset in one example; in another example, it can be configured to be calibrated based on historical false alarm and false alarm rate statistics; if it is necessary to adapt to different flight phases, the corresponding values ​​can be saved for takeoff, cruise, and approach phases respectively. Configuration.

[0050] In some embodiments, for real-time flight situation analysis, static distribution relationships alone are insufficient to characterize the time lag in the pilot's perception link; therefore, a cognitive temporal dynamics model is further established. This model receives aircraft state feature sequences and pilot state feature sequences within a continuous time window and outputs the dynamic cognitive delay time. The model structure can be configured as a temporal convolutional network connected in series with a gated recurrent unit. The temporal convolutional network is responsible for extracting multi-scale change patterns within a local time period, while the gated recurrent unit is responsible for maintaining long-short-term dependencies based on the local patterns. The connection relationship between the two is as follows: the multi-scale temporal features output by the temporal convolutional network serve as the input to the gated recurrent unit; the gated recurrent unit outputs the latent temporal state; and the latent temporal state is then mapped by a regression head to obtain the dynamic cognitive delay time.

[0051] The dynamic cognitive delay time referred to here is a quantification of the time lag between the pilot's perceived response and the change in aircraft attitude at a given moment. Since this quantity needs to be used in the calculation of the cognitive load coefficient later, its generation order must precede the cognitive load coefficient. The system trains the model during the historical phase by minimizing the error between the predicted delay and the calibration delay. The calibration delay can be obtained statistically from the corresponding time difference between the peak of the aircraft attitude change and the peak of the pilot's perceived change. If a segment of data in the historical samples has a sensor outage, that segment of data is not included in the dynamic cognitive delay time calibration to avoid erroneous delay propagation to subsequent load calculations.

[0052] In one example, if the model detects a significant deviation of the dynamic cognitive delay time from the pilot's individual cognitive delay baseline, this deviation is used as an important basis for subsequent load assessment. This significant deviation does not directly constitute an early warning result, but rather serves as one of the inputs for generating the cognitive load coefficient. This avoids directly using a single time delay uncoupled from other physiological quantities for the final judgment.

[0053] In some embodiments, the cognitive load coefficient is determined by a combination of dynamic cognitive delay time, pilot-specific cognitive delay baseline, pupil diameter variation, and resting pupil diameter baseline. Optionally, the cognitive load coefficient is defined as:

[0054] in, Indicates the cognitive load coefficient; Indicates the dynamic cognitive delay time; This represents the baseline of individual cognitive delay in pilots; Indicates changes in pupil diameter; This represents the baseline value for the resting pupil diameter; and This represents the weighting parameter. To maintain dimensional consistency, both parts of the fraction are expressed in relative form. Regarding the determination of the weighting parameter, in one implementation method, and The results are obtained statistically from the ability to distinguish between abnormal and normal segments in historical samples. In another implementation, the calibration is performed based on the sensitivity of the two input quantities to the error contribution. If there are no separate calibration conditions, it can be configured as a preset and updated in a small range during the online fine-tuning phase. This update is only performed during the stable flight calibration period and not during abnormal segments to prevent abnormal states from contaminating the individual baseline.

[0055] The significance of the cognitive load coefficient lies in placing temporal lag changes and visual implicit changes within the same computational framework, enabling a unified measure to reflect the current level of cognitive stress during subsequent fusion display. This coefficient can be incorporated into the fusion display data in real-time or used as a supplementary feature in the correlation model for real-time analysis.

[0056] Considering the differences in physiological baselines and reaction patterns among different pilots, an adversarial domain adaptive network is introduced when training the aircraft-pilot association model. This network includes a feature extractor, a gradient inversion layer, and a domain discriminator. The feature extractor receives training samples and outputs a shared feature representation; the gradient inversion layer connects the feature extractor and the domain discriminator, changing the domain discrimination gradient direction during backpropagation; the domain discriminator receives the shared feature representation and outputs the pilot identity domain discrimination result. Through this adversarial training, the feature extractor is forced to learn shared features that are weakly correlated with pilot identity but strongly correlated with attitude-perception correspondence.

[0057] The shared feature constraint mentioned in the text refers to imposing restrictions on the distribution stability of shared feature representations during online fine-tuning and offline training, ensuring that the new pilot data remains aligned with the learned common representations in the feature space. This constraint can be reflected by both domain discrimination loss and temporal consistency loss. In practice, the feature extractor can share the front-end layer with the aforementioned encoder, or it can have an independent front-end layer concatenated with the encoder's intermediate layer. If the front-end layer is shared, the common representation is directly fed into the domain discriminator and the latter half of the encoder; if the front-end layer is set independently, the common representation first passes through a mapping layer before participating in decoding and domain discrimination.

[0058] In step S4, the current aircraft attitude data and the current pilot perception data are acquired in real time. The current aircraft attitude data is input into the aircraft-pilot association model to obtain the theoretical perception threshold range corresponding to the current aircraft attitude. In step S5, a perception deviation vector is calculated based on the current pilot perception data and the theoretical perception threshold range. The actual aircraft attitude change rate is calculated based on the current aircraft attitude data, and the expected attitude change rate is calculated based on the operational behavior data in the current pilot perception data.

[0059] Specifically, the real-time phase corresponds to the cockpit online processing link. The inputs to this phase include current aircraft attitude data and current pilot perception data. The system first generates current aircraft state features and current pilot state features according to the same time base and normalization rules as the historical phase. Then, the current aircraft state features are input into the trained aircraft-pilot association model to obtain the mean and variance parameters corresponding to the current moment, thereby forming the theoretical perception threshold range. At this point, the objects for subsequent deviation vector calculations, operational deviation calculations, and the objects written to the display layer all have clear sources.

[0060] The perceptual bias vector can optionally be calculated in the following form:

[0061] in, Indicates time The perceptual bias vector; Indicates time Current pilot status characteristics; This represents the mean parameter vector of pilot state features obtained by mapping the current aircraft state features; This represents the corresponding standard deviation parameter vector. This formula indicates that for each dimension of pilot state characteristics, difference and standard deviation normalization are performed separately, thereby transforming visual, physiological, and operational quantities of different dimensions into a unified deviation scale. If the standard deviation of a certain dimension is too small, it is first truncated using a lower bound to avoid numerical amplification. The truncation lower bound is obtained statistically from historical training phases and is fixedly applied during real-time phases.

[0062] In practical implementation, the system can assign different priorities to the various deviation components in the perception deviation vector. For example, changes in pupil diameter, the degree of coordination between head posture and aircraft posture, and deviations related to control stick displacement are often prioritized for inspection; heart rate variability and skin conductance can be used as supporting evidence. The priority here only affects the display order and auxiliary judgment, and does not change the definition of the perception deviation vector or the subsequent calculation method.

[0063] The calculation of the actual and expected rates of change of aircraft attitude has a clear input source. The actual rate of change of aircraft attitude is obtained from the change of current aircraft attitude data over time; the expected rate of change of attitude is obtained by inputting the stick displacement from the current pilot state characteristics into the aircraft dynamics model. Optionally, the actual rate of change of aircraft attitude is defined as:

[0064] in, This represents the actual rate of change of the aircraft's attitude. Considering... As a multi-dimensional aircraft state feature vector, the rates of change in the pitch, roll, and yaw channels can be calculated separately during implementation, and then combined according to the flight phase configuration to form a unified representation of the actual attitude change rate. The expected attitude change rate is denoted as... It is calculated by the aircraft dynamics model based on the control stick displacement, current aircraft state, and necessary aircraft parameters. The control deviation is defined as:

[0065] in, This indicates operational deviation. The aircraft dynamics model here does not need to be a high-fidelity six-DOF full model; it can also be configured as a simplified handling and stability response model corresponding to the aircraft type. However, it must ensure that the input includes the control stick displacement and the output includes the expected rate of attitude change comparable to the actual rate of attitude change.

[0066] When the system detects an abnormal flight attitude, it further performs counterfactual analysis. Counterfactual analysis, while keeping the current operational data constant, calculates the expected attitude that the current operational input should correspond to if the pilot's perception were without bias. An abnormal flight attitude can be triggered by any of the following conditions: the aircraft attitude deviates from the flight envelope, the operational deviation exceeds a threshold, or the critical component of the perception deviation vector exceeds a threshold. After triggering, the system inputs the current stick displacement, the current aircraft state, and the aircraft's dynamic parameters into the aircraft dynamics model, calculates the expected attitude, and then compares it with the actual attitude to form an operational-perception consistency coefficient.

[0067] Optionally, the operational-perception consistency coefficient is defined as:

[0068] in, Indicates the operational-perceptual consistency coefficient; Indicates the actual posture; This represents the expected attitude obtained from the aircraft dynamics model under the current operational input; This represents a stable term that prevents the denominator from being zero. The closer it is to 1, the more consistent the expected posture derived from the current operation is with the actual posture; The closer a value is to 0, the lower the consistency. (Stability term) This is an internal numerical parameter, preset by the system during implementation and not displayed on the business side. This coefficient, together with the perception deviation vector, is used for flight deviation type classification.

[0069] When classifying deviation types, the system first checks the overall magnitude of the perceived deviation vector, and then checks the operation-perception consistency coefficient. If the operation-perception consistency coefficient is small and the perceived deviation vector is large, it is classified as a perception error-dominated type; if the operation-perception consistency coefficient is large and the perceived deviation vector is small, it is classified as an execution error or external disturbance-dominated type; if both are in a high deviation state, it is classified as a composite factor-dominated type. The overall magnitude referred to here can be the norm of the perceived deviation vector, or it can be the weighted sum of the deviation components of the key dimensions. If a weighted sum is used, the weights of each dimension are obtained from historical sample statistics or expert calibration, and are saved as configuration during the device adaptation stage. Since the deviation type is directly written to the display layer later, this classification result is the terminal analysis quantity in the real-time processing flow and will not enter the next round of model training.

[0070] Regarding attention resource occupancy, in some embodiments it can be defined as:

[0071] in, Indicates the utilization rate of attention resources; Indicates the dwell time of the gaze attitude indicator; Indicates the total time window; Indicates current heart rate variability; This represents the baseline value for resting heart rate variability; and This represents the weighting parameter. The method for determining the weighting parameter is similar to that of the cognitive load coefficient mentioned above; it can be obtained through pre-setting, calibration, or statistical methods. If... If the threshold is exceeded continuously across multiple consecutive windows, the system will write this status into the status prompt field of the fused display data. It should be noted that the attention resource utilization rate is based on the defined dwell time, total time window, and resting heart rate variability baseline values, which are respectively derived from eye-tracking sequence statistical results, window configuration results, and individual baseline results.

[0072] In step S6, a flight situation analysis result is generated based on the perception deviation vector and the operational deviation between the actual aircraft attitude change rate and the expected attitude change rate, and the flight situation analysis result is fused and displayed with the current aircraft attitude data.

[0073] The fused display data consists of basic flight parameter display data, perceived deviation graphic display data, deviation type identification data, and status prompt data. Basic flight parameter display data comes directly from the current aircraft attitude data, including pitch, roll, heading, altitude, and speed. Perceived deviation graphic display data comes from the perceived deviation vector and its key dimensional components. Deviation type identification data comes from deviation classification results. Status prompt data comes from cognitive load coefficients, attention resource utilization, and necessary anomaly trigger results. To ensure consistency between the display layer and the analysis layer, the display layer does not recalculate the above quantities; it only performs graphic mapping and layout rendering on the existing results. If the perceived deviation graphic display data uses a radar chart or bar chart, each axis or bar corresponds to a predefined pilot status characteristic deviation component. If the deviation type identification uses color coding, the color-to-type mapping relationship is pre-defined in the configuration table.

[0074] In practical applications, basic flight parameters can be optionally displayed in the central area of ​​the main flight display, perceived deviation graphics in the side areas, and deviation type indicators and status prompts in prominent areas. If prompts need to be combined with the head-up display (HUD), the HUD only receives status prompt levels and directional guidance information, and does not directly display all graphic elements. If guidance needs to be output in conjunction with the seat haptic unit, the haptic trigger conditions also come from status prompt data, rather than directly from raw sensor values. Therefore, the output of the display layer is always based on the results determined by the analysis layer, avoiding the introduction of new judgment logic at the display end.

[0075] In some embodiments, due to differences in resting physiological baselines and reaction patterns among different pilots, the system performs adaptive calibration during the smooth flight phase after the start of the flight mission. The smooth flight phase can be defined by altitude change rate, roll change rate, control stick displacement fluctuation, and external disturbance indicators. Upon entering this phase, the system collects the current pilot's multimodal perception data and forms a current calibration sample according to rules consistent with historical phases. The calibration sample is first used to update the resting pupil diameter baseline, resting heart rate variability baseline, and individual pilot cognitive delay baseline; then it is used to fine-tune the aircraft-pilot correlation model online.

[0076] During online fine-tuning, to prevent the model from shifting towards local noise due to insufficient individual samples, the shared feature constraints in the adversarial domain adaptive network are kept unchanged. Specifically, the front-end layer parameters of the feature extractor can be set to frozen or updated with a low learning rate, while the back-end layer parameters of the decoder and the regression head parameters can be updated with a medium learning rate. The domain discriminator remains enabled to maintain alignment between new individual samples and existing common features. If the overall distribution of the perceptual bias vector within the validation window is closer to zero mean and the fluctuation of operational bias decreases after online fine-tuning, the theoretical perceptual threshold range and real-time analysis parameters after this update are saved; if the bias fluctuation increases after online fine-tuning, the previous version of parameters is rolled back. The validation basis here is entirely derived from data during the stable flight phase, and abnormal segments are not used as the basis for fine-tuning, thereby reducing the risk of abnormal behavior entering individual models.

[0077] In another implementation, if the system is deployed in a simulator training environment, a short-term calibration can be performed before each training session to form training session-level parameters. If deployed in a real-world flight operation environment, confirmed stable flight segments from historical flights can be overlaid to form a long-term baseline for the pilot, which can then be slightly adjusted in each mission. Regardless of the method used, baseline parameter updates follow the same principle: individual baseline parameters are updated first, and then fine-tuned online using individual baseline parameter constraints, without reversing the order. This ensures that the calculation basis for cognitive load coefficient, attention resource utilization rate, and theoretical perception threshold range remains consistent.

[0078] Regarding the aircraft dynamics model, in this embodiment, its core function is to determine the expected rate of attitude change and the expected attitude under counterfactual conditions based on the control stick displacement. Therefore, the model should at least include a mapping between control inputs and attitude responses. Different parameter sets can be configured for different aircraft models; these parameter sets can be derived from aircraft manuals, ground test calibrations, or historical flight parameter statistical fitting. If the aircraft parameters are updated, only the dynamics model parameter set is updated, without changing the definitions of the perception bias vector, cognitive load coefficient, and attention resource utilization rate. If a certain aircraft model does not provide complete aerodynamic parameters, a simplified response model can be used as a substitute, but the substitute model should still maintain the input as control stick displacement and the output as a comparable attitude response quantity. In this way, the calculation logic of control bias and control-perception consistency coefficient remains unchanged.

[0079] Another alternative association modeling structure can be based on a multimodal converter model. This structure can serve as an alternative implementation, using a sequence of aircraft state features over a past period as the query term and a sequence of pilot state features over a past period as the key and value. Attention calculations are then used to obtain the predicted pilot state features for the next moment. If this structure is adopted, its output still needs to pass through a probabilistic header to generate mean and variance parameters, and still needs to conform to the same processing chain for subsequent theoretical perception threshold intervals, perception bias vectors, bias type classification, and fusion display.

[0080] In actual deployment, if the eye-tracking or physiological devices experience a brief interruption, the system can mask the missing window, not outputting the complete perceptual bias vector for that window, but instead maintaining the most recent valid threshold range and marking the status as incomplete data. For short-term loss of head posture data, it can be configured to use smooth estimation with preceding and following windows; for short-term loss of heart rate variability or skin conductance activity, it can be configured to retain only visual and operational bias calculations and pause cognitive load coefficient updates. This fault-tolerant handling does not change the definition of each parameter, only the computable range of certain windows. Therefore, the system can maintain the continuity of the analysis chain during real-time operation, while avoiding directly incorporating missing values ​​into subsequent calculations.

[0081] In summary, this method constructs a probabilistic correlation between aircraft state features and pilot state features through multimodal synchronous samples during the historical phase. In the real-time phase, it generates a theoretical perception threshold range based on the current aircraft state. Then, it combines the current pilot state, the actual aircraft attitude change rate, the expected attitude change rate calculated based on the control stick displacement, and the operation-perception consistency coefficient under counterfactual conditions to perform attribution analysis on flight deviations. Furthermore, it incorporates the cognitive load coefficient, attention resource occupancy rate, perception deviation vector, and deviation type into the fused display data. This forms a complete implementation chain from historical samples, model training, real-time analysis to display output, enabling multimodal data and flight attitude analysis to operate within the same technical framework.

[0082] Please see Figure 3 , Figure 3 This is a schematic diagram of a real-time civil aviation flight situation awareness and display system based on multimodal fusion, provided in an embodiment of this application. As shown in the figure, the system includes: Data acquisition module 301 is used to acquire historical flight attitude data of the target aircraft and historical pilot perception data corresponding to the historical flight attitude data; The data synchronization processing module 302 is used to perform time alignment and sampling synchronization between the historical flight attitude data and the historical pilot perception data in order to construct aircraft state characteristics and pilot state characteristics. The model training module 303 is used to train an aircraft-pilot association model based on the synchronized aircraft state features and pilot state features, so as to output the pilot state feature distribution parameters corresponding to each aircraft state feature, and determine the theoretical perception threshold range based on the distribution parameters. The real-time data acquisition module 304 is used to acquire the current aircraft attitude data and the current pilot perception data in real time. The model inference module 305 is used to input the current aircraft attitude data into the aircraft-pilot association model to obtain the theoretical perception threshold range corresponding to the current aircraft attitude. The deviation calculation module 306 is used to calculate the perception deviation vector based on the current pilot perception data and the theoretical perception threshold range, calculate the actual attitude change rate of the aircraft based on the current aircraft attitude data, and calculate the expected attitude change rate based on the operational behavior data in the current pilot perception data. Situation analysis module 307 is used to generate flight situation analysis results based on the perception deviation vector and the operational deviation between the actual rate of change of the aircraft attitude and the expected rate of change of attitude. The fusion display module 308 is used to fuse and display the flight situation analysis results with the current aircraft attitude data.

[0083] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0084] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0085] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0086] The embodiments disclosed herein are preferred embodiments, but are not limited thereto. Those skilled in the art can readily grasp the spirit of the present invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. A multi-modal fusion-based civil aviation flight situation real-time perception and display method, characterized in that, include: Acquire historical flight attitude data of the target aircraft and historical pilot perception data corresponding to the historical flight attitude data; The historical flight attitude data and the historical pilot perception data are time-aligned and sampled to synchronize the aircraft state characteristics and pilot state characteristics. The aircraft-pilot association model is trained based on the synchronized aircraft state features and pilot state features to output the pilot state feature distribution parameters corresponding to each aircraft state feature, and the theoretical perception threshold range is determined based on the distribution parameters. Real-time acquisition of current aircraft attitude data and current pilot perception data; inputting the current aircraft attitude data into the aircraft-pilot association model to obtain the theoretical perception threshold range corresponding to the current aircraft attitude. The perception deviation vector is calculated based on the current pilot perception data and the theoretical perception threshold range. The actual attitude change rate of the aircraft is calculated based on the current aircraft attitude data. The expected attitude change rate is calculated based on the operational behavior data in the current pilot perception data. Based on the perception deviation vector and the operational deviation between the actual rate of change of the aircraft attitude and the expected rate of change of attitude, a flight situation analysis result is generated, and the flight situation analysis result is fused and displayed with the current aircraft attitude data.

2. The civil aviation flight situation real-time perception and display method based on multi-modal fusion according to claim 1, characterized in that, The historical flight attitude data includes pitch angle, roll angle, yaw angle, ground speed, vertical speed, overload, and control surface deflection angle data. The historical pilot perception data includes visual attention data, head posture data, physiological stress data, and operational behavior data; wherein, the visual attention data includes at least fixation point distribution data, pupil diameter variation data, and blink frequency data; the head posture data includes at least head pitch angle, head yaw angle, and head roll angle; the physiological stress data includes at least heart rate variability data and skin conductance activity data; and the operational behavior data includes at least stick displacement data or stick force variation data.

3. The multi-modal fusion-based civil aviation flight situation real-time perception and display method according to claim 1, characterized in that, The historical flight attitude data is time-aligned and sampled and synchronized with the historical pilot perception data, including: Historical flight attitude data and historical pilot perception data are unified to the same timestamp reference; interpolation is performed on data with inconsistent sampling frequencies; statistics of each feature within a preset time window are extracted, and the aircraft state features and pilot state features are spliced ​​together to form training samples corresponding to flight event segments; wherein, the aircraft state features consist of pitch angle, roll angle, yaw angle, ground speed, vertical speed and overload, and the pilot state features consist of head attitude, pupil diameter change, blink frequency, heart rate variability, skin activity and control stick displacement.

4. The method for real-time perception and display of civil aviation flight situation based on multimodal fusion according to claim 1 or 3, characterized in that, The training aircraft-pilot association model includes: Using aircraft state features as model input and pilot state features as model output, a deep probabilistic model based on variational autoencoder and Gaussian process regression is constructed. The encoder learns latent variable representations of aircraft state features and pilot state features, and the decoder reconstructs pilot state features based on latent variables and aircraft state features. Finally, Gaussian process regression is used to output the mean and variance parameters corresponding to each pilot state feature. The theoretical perception threshold range corresponding to each pilot's state characteristics is determined based on the mean and variance parameters.

5. The method for real-time perception and display of civil aviation flight situation based on multimodal fusion according to claim 1 or 3, characterized in that, The training aircraft-pilot association model also includes: A cognitive time dynamics model is established to perform time-series modeling of the leading and lagging relationships of pilot perception data relative to changes in aircraft attitude, thereby obtaining the dynamic cognitive delay time. The cognitive load coefficient is calculated based on the dynamic cognitive delay time, the pilot's individual cognitive delay baseline, pupil diameter variation data, and the resting pupil diameter baseline value. The dynamic cognitive delay time or the cognitive load coefficient is used as a supplementary factor to the pilot state characteristics in the training or real-time analysis of the aircraft-pilot association model.

6. The method for real-time perception and display of civil aviation flight situation based on multimodal fusion according to claim 5, characterized in that, The training aircraft-pilot association model also includes: To address the differences in physiological baselines and reaction patterns among different pilots, an adversarial adaptive network was established. Shared features are extracted from training samples by a feature extractor, and adversarial training is performed on the pilot identity domain information by connecting a domain discriminator through a gradient inversion layer. The aircraft-pilot association model is updated with a weighted combination of reconstruction loss, domain discrimination loss, and temporal consistency loss as the optimization objective to obtain a perceptual feature representation shared across pilots.

7. The method for real-time perception and display of civil aviation flight situation based on multimodal fusion according to claim 1, characterized in that, The calculation of the perception deviation vector based on the current pilot perception data and the theoretical perception threshold range includes: Input the current aircraft attitude data into the aircraft-pilot association model to obtain the mean and variance parameters of the pilot state characteristics corresponding to the current aircraft attitude. The difference between the current pilot's perceived data and each mean parameter is normalized according to the corresponding variance parameter to obtain the perception deviation components of each dimension; and the perception deviation components are combined to form a perception deviation vector; the operation deviation is determined by the difference between the actual rate of change of aircraft attitude and the expected rate of change of attitude calculated by the aircraft dynamics model from the control stick displacement data.

8. The method for real-time perception and display of civil aviation flight situation based on multimodal fusion according to claim 7, characterized in that, The generated flight situation analysis results include: When an abnormal flight attitude is detected, a counterfactual query is constructed based on the current operational behavior data. The current operational behavior data is then input into the aircraft dynamics model to calculate the expected attitude under unbiased perception conditions. The operation-perception consistency coefficient is determined based on the difference between the expected attitude and the actual attitude; then, the flight deviation is classified according to the operation-perception consistency coefficient and the perception deviation vector. The classification includes at least three types: perception error-dominated, execution error or external disturbance-dominated, and composite factor-dominated.

9. The method for real-time perception and display of civil aviation flight situation based on multimodal fusion according to claim 8, characterized in that, The generated flight situation analysis results also include: Based on the dwell time of the gaze posture indicator, the total time window, heart rate variability data, and the resting heart rate variability baseline value, the attention resource utilization rate is calculated; and the attention resource utilization rate, the cognitive load coefficient, the perceptual deviation vector, and the type classification result are written into the fused display data; wherein, the fused display data includes at least basic flight parameter display data, perceptual deviation graphic display data, deviation type identification data, and status prompt data.

10. A real-time civil aviation flight situation awareness and display system based on multimodal fusion, characterized in that, include: The data acquisition module is used to acquire historical flight attitude data of the target aircraft and historical pilot perception data corresponding to the historical flight attitude data; The data synchronization processing module is used to time-align and sample-synchronize the historical flight attitude data with the historical pilot perception data to construct aircraft state characteristics and pilot state characteristics. The model training module is used to train an aircraft-pilot association model based on the synchronized aircraft state features and pilot state features, so as to output the pilot state feature distribution parameters corresponding to each aircraft state feature, and determine the theoretical perception threshold range based on the distribution parameters. The real-time data acquisition module is used to acquire current aircraft attitude data and current pilot perception data in real time. The model inference module is used to input the current aircraft attitude data into the aircraft-pilot association model to obtain the theoretical perception threshold range corresponding to the current aircraft attitude. The deviation calculation module is used to calculate the perception deviation vector based on the current pilot perception data and the theoretical perception threshold range, calculate the actual attitude change rate of the aircraft based on the current aircraft attitude data, and calculate the expected attitude change rate based on the operational behavior data in the current pilot perception data. The situation analysis module is used to generate flight situation analysis results based on the perception deviation vector and the operational deviation between the actual rate of change of the aircraft attitude and the expected rate of change of attitude. The fusion display module is used to fuse the flight situation analysis results with the current aircraft attitude data for display.

Images