A flight crew operation state recognition method and system based on multi-modal data fusion

Abstract

The application relates to the technical field of aviation safety monitoring, and provides a flight personnel operation state recognition method and system based on multi-modal data fusion, which solves the problems of high misjudgment rate and poor environmental adaptability in the prior art. The method comprises the following steps: synchronously collecting electroencephalogram signals of flight personnel, head inertial measurement unit angular velocity, eye movement trajectories, and pitch angle and roll angle parameters of an aircraft; constructing an electroencephalogram dynamic topology graph through dynamic segmentation of the electroencephalogram signals, and generating a head stability index; extracting wave duration features of the frontal lobe brain region, and generating a cognitive conflict feature vector through weighted fusion with the head stability index; converting eye movement trajectory coordinates to a cockpit three-dimensional coordinate system to generate spatial encoding, fusing spatial distribution and trajectory density gradient through a space-time convolution kernel, and constructing a space-time feature body; and recognizing the operation state as normal, pre-warning or out-of-control in three levels. The application realizes accurate grading and pre-warning of the flight personnel operation state, and significantly improves the real-time performance and accuracy of flight safety monitoring.

Description

Technical Field

[0001] The present application relates to the technical field of aviation safety monitoring based on the fusion of multimodal physiological and aircraft data, and in particular to a method and system for identifying the operational status of flight personnel based on multimodal data fusion. Background Art

[0002] During high-maneuverability missions, pilots face extreme attitude changes, high G-forces, and complex airspace. The real-time balance between cognitive load and situational awareness directly determines flight safety. These scenarios require the system to synchronously capture the aircraft's pitch and roll dynamic parameters and the pilot's physiological and behavioral responses within milliseconds.

[0003] Currently, mainstream solutions focus on analyzing the correlation between eye movement data and flight instrumentation. Specifically, an infrared eye tracker captures the coordinates of the pilot's gaze point and simultaneously records key instrument parameters such as airspeed and altitude. This quantifies the concentration and frequency of visual attention, generating a situational awareness score. When the score falls below a preset threshold, the system triggers a buzzer or light warning, indicating that cognitive load is exceeding the limit.

[0004] However, this solution exposed bottlenecks in high-maneuver scenarios. First, the unimodal data relies solely on eye movement data, unable to distinguish physiological compensation from actual cognitive distraction, leading to frequent misjudgments. Second, it lacks dynamic adaptability. Furthermore, the entropy model threshold is fixed and not linked to the rate of change of the aircraft's roll angle. During rapid rolls, the demand for instrument tracking increases dramatically, but the static threshold can easily misinterpret normal visual search as a failure of situational awareness. Summary of the Invention

[0005] The present application provides a flight crew operating status identification method and system based on multimodal data fusion, which is used to solve the problems of high misjudgment rate and poor environmental adaptability in the existing technology.

[0006] In a first aspect, the present application provides a method for identifying flight crew operating status based on multimodal data fusion, comprising:

[0007] Synchronously collect the flight crew's EEG signals, head inertial measurement unit angular velocity, eye movement trajectory coordinates, and the aircraft's pitch and roll angle parameters;

[0008] Using the change rates of the pitch angle parameter and the roll angle parameter as dynamic constraints on the time series boundaries, a dynamic segmentation operation is performed on the EEG signal to construct an EEG dynamic topology map, and based on the variance value of the angular velocity of the head inertial measurement unit, the damping negative feedback strength of the head motion control device is dynamically adjusted to generate a head stability index that is inversely correlated with the head motion interference strength;

[0009] extracting a wave persistence feature of the prefrontal brain region from the EEG dynamic topology map, and performing a weighted fusion operation on the wave persistence feature and the head stability index to generate a feature vector representing cognitive conflict;

[0010] The eye movement trajectory coordinates are converted into the three-dimensional coordinate system of the aircraft cockpit to generate an eye movement spatial code, the feature vector representing the cognitive conflict and the eye movement spatial code are input into the situation awareness entropy model, and the spatial distribution of the cognitive conflict feature and the spatial density gradient of the eye movement trajectory are fused through the spatiotemporal convolution kernel to construct a spatiotemporal feature volume;

[0011] Based on the entropy change amplitude of the spatiotemporal feature body and in combination with the value of the roll angle parameter, the entropy change threshold is dynamically adjusted to determine the operation state level as a normal state, a warning state, or an out-of-control state.

[0012] Optionally, the feature vector representing the cognitive conflict and the eye movement spatial encoding are input into a situation awareness entropy model, and the spatial distribution of the cognitive conflict feature and the spatial density gradient of the eye movement trajectory are fused through a spatiotemporal convolution kernel to construct a spatiotemporal feature body, including:

[0013] Combining the feature vector representing cognitive conflict with the eye movement spatial encoding into an input combination, and feeding the input combination into a situation awareness entropy model;

[0014] Loading a preset convolution kernel weight set, wherein the convolution kernel weight set slides synchronously in the spatial dimension and the temporal dimension, and extracting the spatial distribution value of the cognitive conflict feature during the spatial sliding process;

[0015] During the time sliding process, extracting the spatial density difference ratio value of the eye movement trajectory, multiplying the spatial distribution value by the spatial density difference ratio value to generate a fusion feature value;

[0016] Through the convolution kernel weight set, a weighted superposition operation in the spatial dimension and a recursive integration operation in the temporal dimension are performed on the fused feature values ​​to output a fusion result body. Based on the fusion result body, a timestamp index is added to construct a spatiotemporal feature body.

[0017] Optionally, dynamically adjusting the entropy change threshold based on the entropy value change amplitude of the spatiotemporal feature body and in combination with the value of the roll angle parameter to determine whether the operating state level is a normal state, a warning state, or an out-of-control state includes:

[0018] Extracting a numerical sequence of the spatiotemporal feature body within a continuous time period, calculating the degree of disorder at each time point in the numerical sequence, and obtaining an entropy value sequence;

[0019] Calculate the absolute value sequence of differences between adjacent time points from the entropy value sequence, and take the result of subtracting the minimum value from the maximum value of the absolute value sequence of differences as the entropy value change amplitude;

[0020] Setting a critical value range for the roll angle parameter, extracting an absolute value of the roll angle parameter within the critical value range as a scaling factor, multiplying a baseline entropy change threshold by the scaling factor, and outputting an adjusted entropy change threshold;

[0021] Comparing the magnitude relationship between the entropy value change amplitude and the adjusted entropy change threshold, and when the entropy value change amplitude is less than or equal to the adjusted entropy change threshold, determining that the operating state level is normal;

[0022] When the amplitude of the entropy value change is greater than the adjusted entropy change threshold, it is determined whether the absolute value of the roll angle parameter exceeds the preset threshold value. If it exceeds, the operating status level is determined to be an out-of-control state; if it does not exceed, the operating status level is determined to be a warning state.

[0023] Optionally, converting the eye movement trajectory coordinates into a three-dimensional coordinate system of an aircraft cockpit to generate an eye movement spatial code includes:

[0024] Obtaining the horizontal and vertical position values ​​of the eye movement trajectory coordinates in the original plane coordinate system, combining them with the pre-stored aircraft cockpit coordinate system, executing coordinate conversion equation calculations, and outputting the converted three-dimensional space position values;

[0025] The aircraft cockpit coordinate system is divided into spatial grid areas, the number of occurrences of the eye movement trajectory coordinates in each grid area is counted as spatial density data, and the spatial density difference ratio value is calculated based on the spatial density data to obtain the eye movement spatial coding.

[0026] Optionally, using the change rates of the pitch angle parameter and the roll angle parameter as dynamic constraints of the time series boundary, performing a dynamic segmentation operation on the EEG signal to construct an EEG dynamic topology map includes:

[0027] Calculating a first instantaneous derivative of the pitch angle parameter in a time series and a second instantaneous derivative of the roll angle parameter in a time series, and taking the sum of the absolute values ​​of the first instantaneous derivative and the second instantaneous derivative as a total dynamic constraint value;

[0028] At a position where the total dynamic constraint value exceeds a preset segmentation threshold, a segmentation point is marked, and the EEG signal is separated from the time series at the segmentation point as a boundary to form a plurality of signal segments;

[0029] For each signal segment, the data sequence of the EEG electrode channel is extracted, each of the EEG electrode channels is used as a node of the graph, and based on the data sequence correlation between the nodes, the connection weight value is calculated to construct an EEG dynamic topology graph.

[0030] Optionally, dynamically adjusting the damping negative feedback strength of the head motion control device based on the variance value of the angular velocity of the head inertial measurement unit to generate a head stability index inversely correlated with the head motion interference strength includes:

[0031] Extracting a continuous sequence of angular velocity of the head inertial measurement unit within a window of fixed time length, and calculating a variance value of the continuous sequence as a motion disturbance;

[0032] Comparing the motion interference amount with a preset interference threshold, if the motion interference amount is greater than the preset interference threshold, setting the reciprocal of the motion interference amount as a gain adjustment factor according to empirical rules, otherwise setting a fixed constant as the gain adjustment factor according to empirical rules;

[0033] Multiplying the set gain adjustment factor by the basic damping coefficient to obtain an adjusted damping parameter value, and inputting the adjusted damping parameter value into the negative feedback controller of the head motion control device;

[0034] An error calculation is performed by a negative feedback controller, which takes the difference between the motion disturbance amount and the zero error reference as input, outputs an inverse proportional coefficient inversely proportional to the motion disturbance amount, and divides the inverse proportional coefficient by the unit time length to generate a head stability index.

[0035] Optionally, extracting a wave persistence feature of the prefrontal brain region from the EEG dynamic topology map, and performing a weighted fusion operation on the wave persistence feature and the head stability index to generate a feature vector representing cognitive conflict, including:

[0036] Selecting an electrode data sequence corresponding to the frontal lobe brain region from the nodes of the EEG dynamic topology map, performing signal waveform screening in a specific frequency band range, and calculating the length of time the screened frequency band waveform appears continuously as a wave duration feature;

[0037] Setting a first fusion weighting factor for the wave duration feature and a second fusion weighting factor for the head stability index, wherein the first fusion weighting factor is set to be greater than the second fusion weighting factor;

[0038] Multiplying the wave duration feature by the first fusion weight factor to obtain a first component, and multiplying the head stability index by the second fusion weight factor to obtain a second component;

[0039] The sum of the first component and the second component is taken as a single element, and the time series average value of the single element is extracted and filled into a vector container of a fixed length to form a feature vector representing cognitive conflict.

[0040] In a second aspect, the present application provides a flight crew operating status recognition system based on multimodal data fusion, comprising:

[0041] The acquisition module is used to synchronously collect the pilot's EEG signals, head inertial measurement unit angular velocity, eye movement trajectory coordinates, and the aircraft's pitch and roll angle parameters;

[0042] a generation module configured to perform a dynamic segmentation operation on the EEG signal using the change rates of the pitch angle parameter and the roll angle parameter as dynamic constraints of the time series boundary, construct an EEG dynamic topology map, dynamically adjust the damping negative feedback strength of the head motion control device based on the variance value of the angular velocity of the head inertial measurement unit, and generate a head stability index that is inversely correlated with the head motion interference strength;

[0043] A fusion module is configured to extract a wave persistence feature of the prefrontal lobe brain region from the EEG dynamic topology map, and perform a weighted fusion operation on the wave persistence feature and the head stability index to generate a feature vector representing cognitive conflict;

[0044] a construction module, configured to convert the eye movement trajectory coordinates into a three-dimensional coordinate system of the aircraft cockpit, generate an eye movement spatial code, input the feature vector representing the cognitive conflict and the eye movement spatial code into a situational awareness entropy model, and fuse the spatial distribution of the cognitive conflict feature with the spatial density gradient of the eye movement trajectory through a spatiotemporal convolution kernel to construct a spatiotemporal feature volume;

[0045] The identification module is used to dynamically adjust the entropy change threshold based on the entropy value change amplitude of the spatiotemporal feature body and in combination with the value of the roll angle parameter to determine the operating state level as normal state, warning state or out-of-control state.

[0046] In a third aspect, the present application provides a computing device comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a method for identifying flight personnel operating status based on multimodal data fusion as described in the first aspect above.

[0047] In a fourth aspect, the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements a method for identifying flight crew operating status based on multimodal data fusion as described in the first aspect.

[0048] This application synchronously collects the EEG signals, angular velocity of the head inertial measurement unit, eye movement trajectory coordinates, and pitch angle parameters and roll angle parameters of the aircraft of the flight personnel, establishes a physiological-behavioral-environmental full-dimensional monitoring foundation, and solves the problem of multimodal data time series alignment; uses the change rate of the pitch angle parameters and roll angle parameters as the dynamic constraint of the time series boundary, performs dynamic segmentation operations on the EEG signals, constructs an EEG dynamic topology map, filters the motion artifacts caused by high-maneuver posture changes, and improves the robustness of prefrontal feature extraction; based on the variance value of the angular velocity of the head inertial measurement unit, dynamically adjusts the damping negative feedback strength of the head motion control device, generates a head stability index that is inversely correlated with the head motion interference intensity, and quantifies the interference of physiological compensatory actions on the cognitive state; extracts the wave persistence characteristics of the prefrontal brain area in the EEG dynamic topology map, and The wave persistence feature is weightedly fused with the head stability index to generate a feature vector representing cognitive conflict, capturing the contradictory signals between neural resource occupancy and posture stability; the eye movement trajectory coordinates are converted into the three-dimensional coordinate system of the aircraft cockpit to generate eye movement spatial coding to solve the problem of gaze point offset caused by visual field jitter under high G force; the feature vector representing cognitive conflict and the eye movement spatial coding are input into the situation awareness entropy model, and the spatial distribution of cognitive conflict features and the spatial density gradient of eye movement trajectory are fused through the spatiotemporal convolution kernel to construct a spatiotemporal feature body, realizing geometric association modeling of cross-modal features; based on the entropy value change amplitude of the spatiotemporal feature body and in combination with the value of the roll angle parameter, the entropy change threshold is dynamically adjusted to determine the operation status level as normal, warning or out-of-control, and the false alarm rate is reduced by the environment adaptive threshold.

[0049] Beneficial effects of this application:

[0050] This application combines the feature vector representing cognitive conflict and the eye movement spatial encoding into an input combination, feeds the input combination into a situational awareness entropy model, constructs a structured input channel for cognitive-visual data, and avoids dimensionality explosion caused by feature stacking; loads a preset convolution kernel weight set, and the convolution kernel weight set slides synchronously in the spatial dimension and the temporal dimension. During the spatial sliding process, the spatial distribution value of the cognitive conflict feature is extracted to locate the abnormal activation site of the brain area; during the temporal sliding process, the spatial density difference ratio value of the eye movement trajectory is extracted, and the spatial distribution value is multiplied by the spatial density difference ratio value to generate a fusion feature value, thereby amplifying the cross-modal conflict signal; through the convolution kernel weight set, the fusion feature value is subjected to a weighted superposition operation in the spatial dimension and a recursive integration operation in the temporal dimension, and a fusion result body is output to realize feature importance weighting and long-term dependency modeling; based on the fusion result body, a timestamp index is added to construct a spatiotemporal feature body to support millisecond-level state backtracking and root cause analysis.

[0051] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 A flowchart of a method for identifying flight crew operating status based on multimodal data fusion provided by the present application is shown;

[0054] Figure 2 A scenario diagram showing a method for identifying flight crew operating status based on multimodal data fusion provided by the present application is shown;

[0055] Figure 3 A schematic diagram of the structure of a flight crew operating status recognition system based on multimodal data fusion provided by the present application is shown;

[0056] Figure 4 A schematic structural diagram of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0058] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0059] Research has found that existing pilot status monitoring technology has significant shortcomings during high-maneuverability flight. On the one hand, relying solely on a single type of data makes it difficult to distinguish physiological compensatory movements such as head shaking from true distraction, leading to frequent false alarms. On the other hand, the evaluation model uses fixed judgment criteria and cannot adapt to real-time changes in flight attitude. Especially during rapid rolls, pilots need to frequently check their instruments, but traditional static thresholds often misinterpret these necessary visual movements as "decreased situational awareness," resulting in false alarms. Therefore, there is an urgent need to develop a recognition method that can dynamically adapt to flight status and integrate multi-dimensional physiological and operational data.

[0060] To address these issues, the present invention proposes a multimodal dynamic fusion recognition method. Its core approach is to dynamically segment physiological signals using aircraft attitude changes and integrate head stability and eye movement spatial features to achieve adaptive decision-making. Specifically, by synchronously collecting EEG signals, head angular velocity, eye movement trajectories, and aircraft pitch / roll parameters, the instantaneous rate of change of the roll angle is used as a temporal segmentation benchmark to dynamically segment the EEG signals and construct a topological map. This avoids feature distortion caused by fixed time windows in high-maneuverability scenarios. Simultaneously, a motion suppression device is dynamically adjusted based on head angular velocity fluctuations to generate an index quantifying head stability. This index is then fused with prefrontal EEG wave features to effectively isolate physiological interference and extract true cognitive conflict features. Eye movement trajectories are further mapped into the three-dimensional space of the cockpit to generate a positional code, which is then input into a situational awareness assessment model along with the cognitive conflict features. A spatiotemporal fusion feature is constructed by jointly analyzing the spatial correlation between the eye movement spatial distribution density and cognitive features. The entropy change threshold is dynamically adjusted based on the real-time roll angle value, ultimately outputting a graded operational status as normal, warning, or out-of-control. This method innovatively drives signal segmentation and decision adaptation through flight attitude parameters, which not only solves the problem of single-modal misjudgment, but also breaks through the limitations of static models in high-maneuverability scenarios, thereby reducing the false alarm rate.

[0061] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0062] Figure 1 The present invention provides a flowchart of a method for identifying the operating status of flight personnel based on multimodal data fusion, as shown in FIG. Figure 1 As shown, the method includes:

[0063] 101. Synchronously collect the flight crew's EEG signals, head inertial measurement unit angular velocity, eye movement trajectory coordinates, and aircraft pitch and roll angle parameters;

[0064] In the above steps, EEG signals refer to the flight crew's EEG activity data, which is generated by the EEG equipment monitoring changes in brain potential; the head inertial measurement unit angular velocity is the head rotation speed data obtained by the sensor worn on the head; the eye movement trajectory coordinates are the eye gaze point position coordinates recorded by the eye tracking device; the pitch angle parameter is the angle value of the aircraft's up and down tilt in the vertical direction; and the roll angle parameter is the angle value of the aircraft's left and right tilt in the horizontal direction.

[0065] In an embodiment of the present application, the EEG signals of the flight crew, the angular velocity of the head inertial measurement unit, the eye movement trajectory coordinates, and the pitch angle parameters and roll angle parameters of the aircraft are first synchronously collected; this process uses a multi-sensor system, including a central control computer connected to the EEG cap device, the head-mounted inertial measurement unit sensor, the eye tracking camera and the aircraft attitude sensor, and performs device calibration before starting the collection, such as checking whether the signals of each sensor are normal and setting the time synchronization protocol, and then sends a synchronization trigger signal through the central control software, such as sending a unified pulse command to let all devices start recording at the same time point, and the EEG cap device generates brain waves The head inertial measurement unit sensor captures the angular velocity value, the eye tracking camera calculates the coordinate data, and the aircraft sensor reads the angle parameters. During the acquisition period, these data with millisecond timestamps are stored in the database file in real time for subsequent analysis. For example, in a pilot simulation flight scenario, time t = 0 seconds triggers all equipment recording, the EEG signal intensity is 10 microvolts, the head inertial measurement unit angular velocity is recorded as 0.5 radians per second in the x direction and 0.2 radians per second in the y direction, the eye movement trajectory coordinates are recorded as 100 pixel 200 pixel positions, the aircraft pitch angle is recorded as 10 degrees, and the roll angle is recorded as 5 degrees. All data are stored in the computer at the same time.

[0066] In a practical application, during a flight simulation training session, during Task 101, data collection for Pilot A was performed simultaneously. We collected EEG activity signals (e.g., brain wave changes), head movement angular velocity data (e.g., rotational velocity data recorded by a wearable sensor), eye movement coordinates (e.g., pupil position), and the flight simulator's pitch and roll angle parameters. For example, at a critical moment in the test, the aircraft started from an initial level position with a pitch and roll angle of 0 degrees. The system then quickly adjusted to a set position, with the pitch angle increased to 15 degrees and the roll angle to 10 degrees. The pilot's response was then observed. The angle change was calculated by subtracting the initial 0 degree pitch angle change (15 degrees) from the initial 0 degree pitch angle, and then subtracting the initial 0 degree roll angle change (10 degrees) from the initial 0 degree pitch angle change (10 degrees). The total angular offset was calculated by adding 15 degrees plus 10 degrees, equaling 25 degrees. At the same time, head angular velocity data is recorded as changing by 3 degrees per second, starting at 0 degrees per second. The change is 3 degrees minus 0 degrees, which equals 3 degrees per second. Eye movement coordinates also shift from their original position, with horizontal movement calculated as 5 mm minus 0 mm, equaling 5 mm. This method facilitates analysis of pilots' response patterns under stress. It can improve flight training safety and learning efficiency, helping instructors optimize instructional plans.

[0067] In the overall solution of step 101 above, high-precision time-synchronized acquisition of flight personnel's EEG signals, head posture dynamic data, visual attention distribution characteristics, and aircraft pitch and roll attitude parameters is achieved. Through millisecond-level aligned multimodal data streams, a comprehensive status monitoring system is constructed that integrates human physiological responses, movement intentions, visual focus, and aircraft spatial posture. This provides complete data closed-loop support with temporal and spatial consistency for human-machine collaborative cognitive load assessment, flight control efficiency analysis, and abnormal operating condition tracing. Ultimately, a data acquisition and analysis capability is formed that can dynamically track the human-machine-environment coupling status in flight scenarios, significantly improving the multi-dimensional analysis accuracy of personnel situational awareness and operational behavior in complex aviation environments, and providing a scientific basis for optimizing flight training strategies and system ergonomics design.

[0068] 102. Using the change rates of the pitch angle parameter and the roll angle parameter as dynamic constraints on the time series boundaries, perform a dynamic segmentation operation on the EEG signal to construct an EEG dynamic topology map. Based on the variance value of the angular velocity of the head inertial measurement unit, dynamically adjust the damping negative feedback strength of the head motion control device to generate a head stability index that is inversely correlated with the head motion interference strength.

[0069] Optionally, step 102 may specifically include the following steps:

[0070] 1021. Calculate a first instantaneous derivative of the pitch angle parameter in a time series and a second instantaneous derivative of the roll angle parameter in a time series, and take the sum of the absolute values ​​of the first instantaneous derivative and the second instantaneous derivative as a total dynamic constraint value;

[0071] 1022. Mark a segmentation point at a position where the total dynamic constraint value exceeds a preset segmentation threshold, and separate the EEG signal from the time series using the segmentation point as a boundary to form multiple signal segments;

[0072] 1023. For each signal segment, extract a data sequence of an EEG electrode channel, use each EEG electrode channel as a node of a graph, calculate a connection weight value based on the data sequence correlation between the nodes, and construct an EEG dynamic topology graph;

[0073] 1024. Extract a continuous sequence of angular velocities of the head inertial measurement unit within a window of a fixed time length, and calculate a variance value of the continuous sequence as a motion interference value;

[0074] 1025. Compare the motion interference amount with a preset interference threshold. If the motion interference amount is greater than the preset interference threshold, set the reciprocal of the motion interference amount as a gain adjustment factor according to empirical rules; otherwise, set a fixed constant as the gain adjustment factor according to empirical rules.

[0075] 1026. Multiply the set gain adjustment factor by the basic damping coefficient to obtain an adjusted damping parameter value, and input the adjusted damping parameter value into the negative feedback controller of the head motion control device;

[0076] 1027. Perform error calculation through a negative feedback controller, take the difference between the motion disturbance amount and the zero error reference as input, output an inverse proportional coefficient inversely proportional to the motion disturbance amount, divide the inverse proportional coefficient by the unit time length, and generate a head stability index.

[0077] In the above steps, the EEG signal is the electrical signal data of the flight crew's brain activity, recorded and generated by multiple electrode channels; the head inertial measurement unit angular velocity is the head rotation speed data measured by the sensor; the eye track coordinates are the coordinates of the eye gaze point position recorded by the eye tracking device; the pitch angle parameter is the angle value of the aircraft's vertical tilt; the roll angle parameter is the angle value of the aircraft's horizontal tilt; the first instantaneous derivative is the rate of change of the pitch angle parameter in the time series; the second instantaneous derivative is the rate of change of the roll angle parameter in the time series; the total dynamic constraint value is a value obtained by calculating the sum of the absolute values ​​of the first instantaneous derivative and the second instantaneous derivative, which is used to quantify the dynamic fluctuation of the flight attitude; the split point is the position marked on the time series; the signal segment is the segment of the EEG signal after it is broken by the split point as the boundary; the EEG electrode channel is the data sequence of each electrode position recording the EEG signal; the node is the basic unit in the EEG dynamic topology map, each corresponding to an EEG electrode channel; the connection weight value is the relationship strength calculated based on the correlation of the data sequence between nodes; and the EEG dynamic topology map is a graphical structure composed of nodes and connection weight values.

[0078] Among them, the window of fixed time length is a predefined continuous time interval; the continuous sequence is a sequence of angular velocity values ​​of the head inertial measurement unit extracted within the window; the variance value is a value for calculating the degree of discreteness of the continuous sequence; the motion interference amount is the variance value, which represents the intensity of head motion interference; the preset interference threshold is a predefined value used to compare the motion interference amount; the gain adjustment factor is an adjustment coefficient set according to the motion interference amount; the fixed constant is a preset constant value; the basic damping coefficient is a preset basic parameter of damping control; the damping parameter value is the result of multiplying the gain adjustment factor by the basic damping coefficient; the negative feedback controller is a unit in the control device that generates reverse control based on the input; the zero error reference is a reference value preset to zero; the difference is the difference between the motion interference amount and the zero error reference; the inverse proportional coefficient is a parameter inversely proportional to the motion interference amount; the unit time length is a preset time unit value; the head stability index is a stability index generated by calculating the inverse proportional coefficient divided by the unit time length, and the larger the value, the more stable it is.

[0079] In the embodiment of the present application, first, step 1021 is used to calculate the first instantaneous derivative of the pitch angle parameter in the time series, i.e., the rate of change, and the second instantaneous derivative of the roll angle parameter in the time series, i.e., the rate of change. The sum of the absolute values ​​of the first instantaneous derivative and the second instantaneous derivative is taken as the total dynamic constraint value. The process uses a derivative algorithm to calculate the slope of the parameter value over time. For example, the pitch angle change value and the roll angle change value at two time intervals in the aircraft attitude sensor data are taken and divided by the time difference to obtain the derivative. , and then take the absolute value and add them together. The specific calculation time point is time t=1 second, the pitch angle parameter changes from 10 degrees to 15 degrees, the time interval is 0.5 seconds, the first instantaneous derivative is calculated as 15-10 divided by 0.5, which is equal to 10 degrees per second, and the absolute value is 10. The roll angle parameter changes from 5 degrees to 10 degrees, with an interval of 0.5 seconds. The second instantaneous derivative is calculated as 10-5 divided by 0.5, which is equal to 10 degrees per second, and the absolute value is 10. The total dynamic constraint value is calculated as 10+10=20 degrees per second, which is used in subsequent steps.

[0080] Secondly, through step 1022, a segmentation point is marked at the position where the total dynamic constraint value exceeds the preset segmentation threshold. The position corresponds to the boundary point of the signal change in the time series. Then, the EEG signal is disconnected with the segmentation point as the boundary to form multiple signal segments. This process uses a threshold comparison algorithm to detect the position during the sequence scanning process. When the total dynamic constraint value is greater than the preset segmentation threshold, the position is marked. For example, in the flight simulation, the total dynamic constraint value is calculated to be 20 degrees per second, and the preset segmentation threshold is 10 degrees per second. Since 20 is greater than 10, the segmentation point is marked at the corresponding time t=1 second. The data of the original EEG signal sequence from time 0 second to 2 seconds is disconnected with t=1 second as the boundary, forming signal segment 1 from 0 second to 1 second and signal segment 2 from 1 second to 2 seconds. Each signal segment is used for subsequent analysis.

[0081] Then, step 1023 is used to extract the data sequence of the EEG electrode channel for each segmented signal segment, and the EEG signal value of the channel corresponding to the electrode position is used. Each EEG electrode channel is used as a node, i.e., a vertex of the graph. Based on the correlation of the data sequences between the nodes, the connection weight value, i.e., the similarity measure, is calculated to construct an EEG dynamic topology graph, i.e., a relationship network. This process uses the correlation calculation formula to calculate the covariance and standard deviation between sequences. The connection weight is equal to the covariance divided by the product of the standard deviations of the nodes. The covariance is the average value of the product of the deviations of the values ​​between the two sequences from the mean value, and the standard deviation is the degree of discreteness of the sequence itself. For example, two electrode channels are taken in signal segment 1, such as channel A sequence values ​​of 8 microvolts, 10 microvolts, and 12 microvolts, and channel B sequence values ​​of 9 microvolts, 11 microvolts, and 13 microvolts. The average value of channel A is 10 microvolts, and the average value of channel B is 11 microvolts. According to the covariance formula, the covariance is calculated as the average of the product of the deviations of each point, that is, ((8-10)(9-11)+(10-10)(11-11)+(12-10)(13-11)) / 3=((-2)(-2)+(0)(0)+(2)(2)) / 3=(4+0+4) / 3=8 / 3≈2.67. The standard deviation of channel A is calculated as the square root of the average of the squares of the deviations of each value, that is, , the standard deviation of channel B is similarly calculated as 1.63, and the connection weight is calculated as the covariance , the weight value is used as the value of the connecting edge to construct the topological graph.

[0082] Next, step 1024 is used to extract a continuous sequence of the angular velocity of the head inertial measurement unit, i.e., a sequence of head rotation speed values ​​measured by the sensor, within a window of fixed time length, and calculate the variance value of the continuous sequence as a motion interference quantity to represent the interference intensity. The process uses a variance algorithm to calculate the square average of the deviations of the sequence values ​​from the mean value. The variance is equal to the sum of the squares of the differences between each value in the sequence and the mean value divided by the sequence length. For example, within a window length of 1 second, the sequence contains three points: time point t1=0.5 second, angular velocity 0.4 radians per second, t2=1.0 second, angular velocity 0.5 radians per second, t3=1.5 seconds, angular velocity 0.6 radians per second, and an average value of 0.5 radians per second. The deviations of each value are -0.10 and +0.10, respectively, and the squares of the deviations are 0.01, 0.00, and 0.01, respectively. The sum of the squares is 0.02, which is equal to 0.00667 radians per second squared, divided by the number of points 3, as the variance value, i.e., the motion interference quantity.

[0083] Next, step 1025 is used to compare the motion interference amount, i.e., the variance value, with a preset interference threshold. The threshold is a preset reference value. If the motion interference amount is greater than the preset interference threshold, the inverse of the motion interference amount is set as the gain adjustment factor. Otherwise, a fixed constant is set as the gain adjustment factor. The process is set using conditional judgment rules. For example, the preset interference threshold is 0.005 radians squared per second squared, and the calculated motion interference amount of 0.00667 is greater than 0.005. Therefore, the gain adjustment factor is set to 1 divided by 0.00667, which is approximately equal to 150. If the motion interference amount is less than or equal to the threshold, such as the interference amount of another scene is 0.004, the fixed constant is set to 1.0.

[0084] Then, through step 1026, the set gain adjustment factor is multiplied by the basic damping coefficient, that is, the preset basic control parameter value, to obtain the adjusted damping parameter value, that is, the intensity value of the control force, and then the value is input into the negative feedback controller of the head movement control device, that is, the control core unit. This process uses multiplication operation for proportional adjustment. For example, if the basic damping coefficient is set to 0.2 and the gain adjustment factor is 150, the damping parameter value is calculated as 150×0.2=30, which is directly input into the negative feedback controller for controlling the output.

[0085] Finally, step 1027 is used to perform an error operation through a negative feedback controller, that is, to calculate the difference between the input value and the reference value, and to use the difference between the motion disturbance amount and the zero error reference preset to zero as input, and to output an inverse proportional coefficient that is inversely proportional to the motion disturbance amount, that is, an inverse force intensity coefficient, and then divide the inverse proportional coefficient by the unit time length preset to 1 second to generate a head stability index. The larger the value, the higher the stability. This process uses an error feedback mechanism, and the difference is equal to the motion disturbance amount minus 0, which is the motion disturbance amount itself. The inverse proportional coefficient calculation formula is the negative preset gain multiplied by the input difference. For example, the motion disturbance amount is 0.00667, the unit time length is 1 second, the preset gain is 1000, and the inverse proportional coefficient is calculated as -1000×0.00667=-6.67, and then divided by 1 to equal negative 6.67 as the head stability index. The change in value reflects the degree of disturbance.

[0086] In practical applications, during a flight simulation training experiment, when monitoring Pilot B performing complex maneuvers, the system first acquired pitch and roll angle parameters in real time through the flight simulator. For example, during a climb-to-circle maneuver, the pitch angle increased from 5 to 20 degrees within 2 seconds, a change of 15 degrees at a rate of 7.5 degrees per second. Simultaneously, the roll angle changed from 0 to 15 degrees over the same period, at a rate of 7.5 degrees per second. The system then added the absolute values ​​of these two rates: 7.5 degrees per second plus 7.5 degrees per second, yielding a total dynamic constraint value of 15 degrees per second. When this value exceeded the preset safety threshold of 10 degrees per second, a split point was marked at the second second of the time series, segmenting the continuously acquired EEG signal into two separate signal segments: the first two seconds of the climb phase and the last three seconds of the circle phase. For the three seconds of the circle phase, the system extracted data from the Fp1 channel of the prefrontal cortex and the C3 channel of the motor cortex. By calculating the synchronization of signal fluctuations between the two within a time window, the correlation coefficient reached 0.8, exceeding the set threshold of 0.6. Therefore, a strongly connected node was established in the EEG dynamic topology map, with a connection weight of 0.8. This topology map intuitively shows the coordinated activation pattern of the frontal area and motor cortex during the pilot's intense maneuvers.

[0087] During the same hovering phase, the head motion sensor recorded a series of angular velocity data: 12, 18, and 10 degrees per second, respectively. The degree of dispersion of this data set from the mean was calculated using a variance formula: first finding the mean value equal to 13.3 degrees per second, then calculating the sum of the squared differences between each value and the mean value, which equaled 38.9, resulting in a final variance of approximately 12.97 degrees per second. When this value exceeded the preset interference threshold of 8 degrees per second, the system adjusted the damping gain coefficient to its reciprocal: 1 divided by 12.97 degrees per second, which is approximately 0.077. Multiplying this gain coefficient by the base damping value of 100 Newtons per second per meter yields an adjusted damping parameter of approximately 7.7 Newtons per second per meter. The negative feedback controller continuously compares the actual angular velocity with a zero reference. For example, when the instantaneous angular velocity reaches 18 degrees per second, the difference between it and the zero reference is proportionally calculated to output a reverse control coefficient of -0.055 seconds. Dividing this coefficient by the unit time of 1 second yields a head stability index of -0.055. This negative value indicates that the device is effectively damping head movement. This entire process enables the system to construct an EEG topology map reflecting cognitive state and dynamically enhance the head stabilization device's anti-interference capabilities during drastic changes in flight posture.

[0088] In the overall solution of the above-mentioned step 102, a dynamic constraint value is formed by extracting the change rate of the aircraft's pitch angle and roll angle in real time, and based on this, the segmentation point is automatically located on the EEG signal time axis to realize time series division, thereby forming an EEG signal segment sequence corresponding to the flight state transition event; based on the data of each segment, an association graph model reflecting the dynamic topological characteristics of the brain functional network is constructed; at the same time, the head movement angular velocity variance is used to quantify the movement interference intensity, and the negative feedback gain coefficient of the head stabilization control device is dynamically adjusted based on this; the closed-loop control system outputs a head stability index that is inversely proportional to the interference intensity, effectively suppressing the head posture oscillation during the flight maneuver, improving the quality of physiological signal acquisition and the stability of spatial state perception, thereby establishing a precise mapping relationship between the dynamic changes of flight operations and human physiological responses, and providing a quantitative evaluation method for analyzing the cognitive load transfer law and operation stability in special situation handling.

[0089] 103. Extracting the wave persistence feature of the prefrontal lobe brain region from the EEG dynamic topology map, and performing a weighted fusion operation on the wave persistence feature and the head stability index to generate a feature vector representing cognitive conflict;

[0090] Optionally, step 103 may specifically include the following steps:

[0091] 1031. Selecting an electrode data sequence corresponding to the frontal lobe from the node of the EEG dynamic topology map, filtering the signal waveform within a specific frequency band, and calculating the duration of the continuous appearance of the filtered frequency band waveform as a wave duration feature;

[0092] 1032. Set a first fusion weighting factor for the wave duration feature, and set a second fusion weighting factor for the head stability index, wherein the first fusion weighting factor is set to be greater than the second fusion weighting factor;

[0093] 1033. Multiply the wave duration feature by the first fusion weight factor to obtain a first component, and multiply the head stability index by the second fusion weight factor to obtain a second component;

[0094] 1034. Take the sum of the first component and the second component as a single element, extract the time series average value of the single element and fill it into a vector container of fixed length to form a feature vector representing cognitive conflict.

[0095] In the above steps, the prefrontal cortex is the area in the brain responsible for high-level cognitive functions such as decision-making and attention regulation, and its electrode data sequence refers to the time-series voltage signal collected by the EEG sensor in this area; the wave duration feature is obtained by screening a specific frequency band such as the θ wave 4-7Hz, and calculating the length of time that the waveform in this frequency band appears continuously; the first fusion weight factor is the weighting coefficient assigned to the wave duration feature, which is usually greater than 0.5, and the second fusion weight factor is the weighting coefficient assigned to the head stability index, which is usually less than 0.5; the first component is the result of multiplying the wave duration feature by the first weight, and the second component is the result of multiplying the head stability index by the second weight; the single element is the sum of the first and second components, which represents the intensity of cognitive conflict at a single time point; the feature vector is a mathematical vector formed by filling the time series average value of multiple single elements to a fixed length, which is used to quantify the overall pattern of cognitive conflict.

[0096] In the embodiment of the present application, first, in step 1031, the electrodes corresponding to the frontal lobe brain region, such as Fp1 and Fp2 channels, are located from the nodes of the EEG dynamic topology map, and their data sequences are extracted. A bandpass filter is used to filter a specific frequency band, such as the θ band 4-7Hz, and the length of time that the waveform of this frequency band appears continuously is calculated as the wave duration feature. The specific process includes: first, performing a frequency domain transformation such as a fast Fourier transform on the electrode data sequence to identify the period when the energy of the target frequency band exceeds the threshold; then, the length of the continuous period is counted, for example, in the time series, the θ wave energy continuously exceeds the threshold. The period is from t=2.0 seconds to t=5.5 seconds, so the wave duration characteristic is 3.5 seconds.

[0097] Next, in step 1032, a first fusion weighting factor, such as 0.7, is set for the wave duration feature, and a second fusion weighting factor, such as 0.3, is set for the head stability index. Because prefrontal activity contributes more directly to cognitive conflict, the first weighting factor is higher than the second weighting factor. The weighting is set according to an empirical rule: if the correlation coefficient between the prefrontal feature and behavioral error in the cognitive conflict task reaches 0.6, while the correlation coefficient between the head stability index is 0.3, then the weighting ratio is allocated based on the correlation coefficient ratio: 0.6: (0.6 + 0.3) = 0.67 ≈ 0.7 and 0.3.

[0098] Next, in step 1033, the wave duration feature is multiplied by the first fusion weighting factor to obtain the first component, and the head stability index is multiplied by the second fusion weighting factor to obtain the second component. For example, if the wave duration feature is 3.5 seconds and the first weighting factor is 0.7, the first component is 3.5 × 0.7 = 2.45; if the head stability index is -6.67 (where a larger absolute value indicates greater stability), and the second weighting factor is 0.3, the second component is -6.67 × 0.3 = -2.00. The calculation process must retain the sign to reflect the inverse correlation of stability.

[0099] Finally, in step 1034, the sum of the first component and the second component is taken as a single element, such as 2.45+(-2.00)=0.45, and the average value of the element in the time series is extracted, such as the single element sequence [0.45, 0.38, 0.52, ..., 0.41] at 10 time points. , fill a fixed-length vector container, such as a vector of length 128, with 0.43 placed in the first 10% of the positions to form a feature vector representing cognitive conflict. This vector can be used in a machine learning classifier such as an SVM to identify high-conflict states.

[0100] In actual applications, during the special situation handling phase of a flight simulation training, the system first processes the previously segmented EEG dynamic topology map of the pilot's C-circling phase, extracts the data sequence of the FP1 channel of the prefrontal brain area, filters the EEG signals in a specific frequency band range, such as 4 Hz to 8 Hz, and analyzes the maximum time length of continuous appearance of these signal waveforms as the wave duration feature; for example, within a 3-second window, the system detects a waveform starting from 90 milliseconds and ending at 120 milliseconds, with a duration of 120-90=30 milliseconds, and another waveform starting from 120 milliseconds and ending at 150 milliseconds, with a duration of 150-120=30 milliseconds, so the maximum wave duration feature value is determined to be 30 milliseconds. At the same time, the system obtains the head stability index in the same time window. The index comes from the negative feedback control output, and the numerical records are such as -0.04, -0.06, and -0.03. In order to fuse these two features to generate a vector representing cognitive conflict, the system sets the fusion weight factor of the wave duration feature to 0.7 and the fusion weight factor of the head stability index to 0.3. When performing the weighted operation, the wave duration feature 30 milliseconds × 0.7 is first obtained as 21 milliseconds as the first component, and then the average value of the head stability index is divided by 3 by the sum of -0.04 + (-0.066) + (-0.03). The result is negative 0.043, which is then multiplied by a weight of 0.3 to approximately -0.0129 as the second component; the first component and the second component are added together, 21+(-0.0129) is approximately equal to 20.987 milliseconds, as a single element value; the system repeats this process at different time points, for example, in three consecutive treatments, element values ​​of 18.2 milliseconds, 20.9 milliseconds, and 15.7 milliseconds are generated respectively, and these values ​​are filled into a feature vector container of a preset length; when the element value in the vector exceeds the threshold of 18 milliseconds, the system can automatically determine the cognitive conflict state for optimizing training decisions.

[0101] In the overall solution of step 103 above, by extracting the duration characteristics of waveforms in specific frequency bands of the prefrontal brain region in the EEG dynamic topology map and setting a fusion weight ratio higher than the head stability index for dominant weighting, combined with a stability index dynamically generated by the head angular velocity variance and negatively correlated with the intensity of motion interference, differentiated fusion processing of cognitive physiological core characteristics and motion compensation signals is achieved; by calculating the average value of the fusion component time series to construct a fixed-dimensional feature vector, the cognitive resource allocation state reflected by the persistence of prefrontal neural oscillations and the motion stability parameters after the head movement interference is suppressed are effectively integrated, significantly enhancing the feature vector's sensitivity to the neural representation of high-order cognitive states such as attention conflict and decision hesitation during flight control. At the same time, by suppressing the influence of head movement noise on EEG characteristics, the recognition robustness in a dynamic flight environment is improved, forming a spatiotemporal standardized data structure that can be directly applied to real-time cognitive monitoring systems, providing a key quantitative means for neuroergonomic evaluation in the field of human factors engineering.

[0102] 104. Convert the eye movement trajectory coordinates to the three-dimensional coordinate system of the aircraft cockpit to generate an eye movement spatial code, input the feature vector representing the cognitive conflict and the eye movement spatial code into a situation awareness entropy model, and fuse the spatial distribution of the cognitive conflict feature with the spatial density gradient of the eye movement trajectory through a spatiotemporal convolution kernel to construct a spatiotemporal feature volume;

[0103] Optionally, step 104 may specifically include the following steps:

[0104] 1041. Obtain the horizontal and vertical position values ​​of the eye movement trajectory coordinates in the original plane coordinate system, combine them with the pre-stored aircraft cockpit coordinate system, execute coordinate conversion equation calculation, and output the converted three-dimensional space position value;

[0105] 1042. Divide the aircraft cockpit coordinate system into spatial grid areas, count the number of occurrences of the eye movement trajectory coordinates in each grid area as spatial density data, and calculate a spatial density difference ratio value based on the spatial density data to obtain an eye movement spatial code.

[0106] 1043. Combine the feature vector representing cognitive conflict and the eye movement spatial encoding into an input combination, and feed the input combination into a situation awareness entropy model;

[0107] 1044. Load a preset convolution kernel weight set, and slide the convolution kernel weight set synchronously in the spatial dimension and the temporal dimension. During the spatial sliding process, extract the spatial distribution value of the cognitive conflict feature;

[0108] 1045. During the time sliding process, extract the spatial density difference ratio value of the eye movement trajectory, multiply the spatial distribution value by the spatial density difference ratio value, and generate a fusion feature value;

[0109] 1046. Perform a weighted superposition operation in the spatial dimension and a recursive integration operation in the temporal dimension on the fused feature values ​​through the convolution kernel weight set, output a fusion result body, add a timestamp index based on the fusion result body, and construct a spatiotemporal feature body.

[0110] In the above steps, the eye movement trajectory coordinates are the position values ​​of the eye gaze point in the two-dimensional plane recorded by the eye tracking device, including the horizontal position X axis and the vertical position Y axis; the three-dimensional coordinate system of the aircraft cockpit is a predefined three-dimensional space reference system, including the three axes of X left and right, Y up and down, and Z front and back, which is used to locate the position of objects in the cockpit; the coordinate conversion equation is a mathematical operation that maps plane coordinates to three-dimensional space based on geometric transformation rules; the spatial grid area is a three-dimensional unit that divides the three-dimensional coordinate system of the cockpit at a fixed interval; the spatial density data is the result of counting the frequency of eye movement trajectory coordinates in each grid unit; the spatial density difference ratio value is the comparison value of the density of adjacent grids, reflecting the uneven distribution of eye movements. Uniformity; eye movement spatial coding is the digital spatial distribution feature of the combination of spatial density data and difference ratio value; input combination is the merged data of feature vector and eye movement spatial coding; situational awareness entropy model is a machine learning model for quantifying cognitive state; convolution kernel weight set is a preset filter parameter group used to extract spatiotemporal features; spatial distribution value is the intensity distribution of cognitive conflict features in the three-dimensional grid; fusion feature value is the product of spatial distribution value and density difference ratio value; fusion result body is a multidimensional data set after weighted superposition and recursive integration; timestamp index is a time series label that marks the data moment; spatiotemporal feature body is a four-dimensional data structure with three spatial dimensions + one temporal dimension after adding time dimension.

[0111] In the embodiment of the present application, first, the horizontal position X and vertical position Y of the eye movement trajectory coordinates in the original plane coordinate system are obtained through step 1041, and the pre-stored conversion parameters of the aircraft cockpit three-dimensional coordinate system, such as the camera angle of view and the cockpit size, are called to perform the coordinate conversion equation operation: first, the plane coordinates are multiplied by the rotation matrix to adjust the direction, and then the translation vector is added to align the cockpit origin, and the three-dimensional space position value is output ( For example, the raw eye movement coordinates ( Pixels, pixels) through the rotation matrix and translation vectors Calculation: New coordinates , , Fixed to 1.2, the three-dimensional position is (2.5, 3.3, 1.2).

[0112] Next, in step 1042, the aircraft cockpit's three-dimensional coordinate system is divided into 1-cubic-decimeter spatial grids. The three-dimensional coordinate points of all eye movement trajectories are traversed, and the number of occurrences of coordinate points within each grid is counted as spatial density data. The density ratio of adjacent grids is then calculated as the spatial density difference ratio. For example, if grid A contains five eye movement points and adjacent grid B contains one eye movement point, the difference ratio of A to B is 5 / 1=5. Finally, the density data and difference ratios of all grids are encoded into a matrix to generate the eye movement spatial code.

[0113] Next, in step 1043, the feature vector representing cognitive conflict (e.g., the vector generated in step 1024) and the eye movement spatial encoding are concatenated according to the data dimensions to form an input combination. For example, if the feature vector is 128-dimensional and the eye movement encoding is 64-dimensional, then the combination generates 192-dimensional input data, which is directly input into the fully connected layer of the situational awareness model.

[0114] Next, in step 1044, a preset convolution kernel weight set is loaded, such as a 5×5×5 three-dimensional filter, and the convolution kernel is slid along the spatial dimensions (X / Y / Z axis) to extract the spatial distribution value of the cognitive conflict feature. For example, the convolution kernel calculates the weighted sum of the feature vectors in the grid at the cockpit coordinate system position (2, 3, 1): If the weight is , the eigenvalues ​​in the grid are , then the spatial distribution value = 0.1×0.5+0.2×0.6+0.3×0.7+0.4×0.8=0.05+0.12+0.21+0.32=0.7.

[0115] Then, in step 1045, a sliding convolution kernel is applied along the time dimension to extract the difference ratio between the eye movement spatial encoding at the current moment and the previous moment. This difference ratio is then multiplied by the spatial distribution value to generate a fused feature value. For example, if the spatial distribution value is 0.7 and the time dimension eye movement density difference ratio (density at time t / density at time t-1) is 1.2, then the fused feature value = 0.7 × 1.2 = 0.84.

[0116] Finally, in step 1046, the fused feature values ​​are subjected to weighted convolution in the spatial dimension, with the kernel moving along the X / Y / Z axes for accumulation, and recursive integration in the temporal dimension, resulting in a linear combination of the current result and the previous time series result. This outputs a three-dimensional fused result volume, and then adds a millisecond timestamp index to each data unit to construct a four-dimensional spatiotemporal feature volume. For example, at t = 1.5 seconds, the fused result volume has a size of 20 × 20 × 10. After adding the timestamp, a spatiotemporal feature volume of 20 × 20 × 10 × T (T is the number of time points) is formed.

[0117] In practical applications, in a certain aviation simulator training scenario, the system needs to integrate pilot cognitive data with visual behavior. For example, when pilot D performs an emergency climb, the original coordinates recorded by his eye tracking device are 80% horizontal and 60% vertical on the screen. Combined with the pre-stored cockpit three-dimensional coordinate system conversion rules, 100% horizontal corresponds to an actual cockpit distance of 0.25 meters. 80% horizontal is converted to 0.2 meters, and 60% vertical is converted to 0.15 meters. A vertical height value of 0 meters is assigned, resulting in the three-dimensional coordinate point (0.2, 0.15, 0). During the five-second operation, the system divides the cockpit instrument area into eight spatial grid regions. Statistics show that the eye movement point appears four times in grid 1, three times in grid 2, and only once in the peripheral area, grid 8. Based on this, regional density differences are calculated, such that the ratio of appearances in grid 1 to grid 8 reaches 4:1. Furthermore, the value of the cognitive conflict feature vector in the current time window is 23 milliseconds, reflecting the level of load on the pilot's prefrontal cortex. When the two sets of data are input into the situational awareness model, the model simultaneously performs spatiotemporal feature fusion. The specific process is as follows: First, in the spatial dimension, the 23 millisecond feature value, representing cognitive load intensity, is loaded onto the three-dimensional grid coordinate point (0.2, 0.15, 0). Simultaneously, during the time sliding process, the density data for the four occurrences of grid 1 and the one occurrence of grid 8 within the current operation period are extracted, resulting in a spatial density gradient ratio of 4. The spatial distribution value of 23 milliseconds interacts with the density gradient value of 4, and feature fusion is performed using the model's built-in intensity adjustment coefficient. This spatiotemporal convolution process is continuously applied to all grid regions. For example, over the next two seconds, the model performs feature enhancement on five key grid regions, ultimately generating a feature fusion containing three-dimensional spatial coordinates and timestamps. If the model detects a sudden increase in fusion intensity exceeding 40 milliseconds in the instrument core area, combined with the recorded altitude deviation, it determines that the pilot's attention has shifted away from the heading indicator area. This immediately triggers the warning system to assist the pilot in correcting his or her visual focus.

[0118] In the overall solution of step 104 above, the eye movement trajectory coordinates are converted from the original plane coordinate system to the three-dimensional coordinate system of the aircraft cockpit. After calculating the spatial position value, the cockpit coordinate system is further divided into spatial grid areas. The number of occurrences of the eye movement trajectory coordinates within the grid areas is counted to obtain spatial density data, and the spatial density difference ratio value is calculated to generate the eye movement spatial code. The eye movement spatial code is then combined with the feature vector representing cognitive conflict to form an input combination and input into the situation awareness entropy model. The model loads a preset convolution kernel weight set, extracts the spatial distribution value of the cognitive conflict feature during the sliding process in the spatial dimension, and simultaneously extracts the spatial density difference ratio value of the eye movement trajectory during the sliding process in the temporal dimension. The two are multiplied to generate a fused feature value. Then, a weighted superposition operation is performed in the spatial dimension and a recursive integration operation is performed in the temporal dimension using the convolution kernel weights. The fusion result volume is output and a timestamp index is added to construct a spatiotemporal feature volume. This effectively integrates the spatial distribution of the cognitive conflict feature with the dynamic changes in the spatial density gradient of the eye movement trajectory in time and space, realizes feature integration of the situation awareness entropy model, and improves the accuracy and real-time performance of situation awareness in the aircraft cockpit.

[0119] 105. Based on the entropy change amplitude of the spatiotemporal feature body and in combination with the value of the roll angle parameter, dynamically adjust the entropy change threshold to determine whether the operation state level is a normal state, a warning state, or an out-of-control state.

[0120] Optionally, step 105 may specifically include the following steps:

[0121] 1051. Extract the numerical sequence of the spatiotemporal feature body within a continuous time period, calculate the degree of disorder at each time point in the numerical sequence, and obtain an entropy value sequence;

[0122] 1052. Calculate a sequence of absolute values ​​of differences between adjacent time points from the entropy value sequence, and take the result of subtracting the minimum value from the maximum value of the absolute value of the difference sequence as the entropy value change amplitude;

[0123] 1053. Set a critical value range for the roll angle parameter, extract the absolute value of the roll angle parameter within the critical value range as a scaling factor, multiply a baseline entropy change threshold by the scaling factor, and output an adjusted entropy change threshold;

[0124] 1054. Compare the magnitude relationship between the entropy value change amplitude and the adjusted entropy change threshold value; when the entropy value change amplitude is less than or equal to the adjusted entropy change threshold value, determine that the operating state level is normal;

[0125] 1055. When the amplitude of the entropy value change is greater than the adjusted entropy change threshold, determine whether the absolute value of the roll angle parameter exceeds the preset threshold value; if so, determine that the operating status level is an out-of-control state; if not, determine that the operating status level is a warning state.

[0126] In the above steps, the spatiotemporal feature body is a four-dimensional data structure with three dimensions of space and one dimension of time that integrates cognitive conflict features and eye movement spatial encoding; the entropy value sequence is a sequence composed of the degree of chaos of the spatiotemporal feature body at each time point in a continuous time period, which is obtained by probability distribution calculation; the degree of chaos is a quantitative value of the disorder of the local data of the spatiotemporal feature body, calculated based on the Shannon entropy formula; the entropy value change amplitude is the fluctuation range of the absolute value of the difference between adjacent time points in the entropy value sequence; the critical value range is the preset safety interval of the roll angle parameter; the scaling factor is the absolute value of the roll angle parameter, which is used to dynamically adjust the threshold; the baseline entropy change threshold is the preset entropy change reference standard; the adjusted entropy change threshold is the result of multiplying the baseline entropy change threshold by the scaling factor; the preset threshold value is the roll angle boundary value that distinguishes between warning and out-of-control states; the operating state levels include normal state entropy change is stable, warning state entropy change is abnormal but controllable, and out-of-control state entropy change is abnormal and the posture is out of control.

[0127] In the embodiment of the present application, the first instantaneous derivative of the aircraft pitch angle parameter in the time series, i.e., the rate of change of the angle in the vertical direction, and the second instantaneous derivative of the roll angle parameter in the time series, i.e., the rate of change of the angle in the horizontal direction, are calculated in step 1021. The specific process is as follows: for the pitch angle and roll angle time series data collected by the sensor, a differential algorithm is used to calculate the angle difference between adjacent time points and divide it by the time interval to obtain the instantaneous derivative value. For example, at time point When the pitch angle changes from becomes , time interval , the first instantaneous derivative is ; Roll angle from becomes , the second instantaneous derivative is Then take the sum of the absolute values ​​of the two as the total dynamic constraint value, that is, , which quantifies the intensity of dynamic fluctuations in flight attitude.

[0128] Next, the total dynamic constraint value is compared with the preset segmentation threshold value in step 1022. When the total dynamic constraint value exceeds the threshold value, a segmentation point is marked at the corresponding time point. For example, if t= 1s, the total dynamic constraint value is 20° / s (greater than the threshold of 10), so a split point is marked there. The continuous EEG signal is then broken into multiple independent signal segments using the split points as boundaries. For example, if the original EEG signal has a time range of 0–2s, segmentation will generate signal segments 1 (0–1s) and 2 (1–2s). This process uses a threshold scanning algorithm to achieve dynamic segmentation, ensuring that subsequent analysis focuses on stable and highly dynamic periods.

[0129] Next, the data sequence of the EEG electrode channel (such as Fp1, Fp2, etc.) is extracted for each signal segment through step 1023, and each channel is defined as a node in the topology graph. The connection weight is calculated based on the covariance and standard deviation of the data sequence between nodes, and the formula is: For example, the sequence value of channel A in signal segment 1 is , the sequence value of channel B is . First calculate the covariance:

[0130] , and then calculate the standard deviation 、 , the final weight is This process is repeated to generate connection weights for all node pairs, and a complete EEG dynamic topology map is constructed to reflect the coordinated activation pattern of brain regions.

[0131] Then, step 1024 extracts the continuous sequence of angular velocity of the head inertial measurement unit (IMU) within a fixed time window (e.g., 1 s). The variance of the sequence is calculated as the motion disturbance, and the formula is: ,in is the mean of the sequence. For example, the angular velocity sequence The mean , the variance is calculated as:

[0132] This value quantifies the interference intensity of head movement, with larger values ​​indicating more severe head shaking.

[0133] Finally, in step 1025, the motion disturbance is compared with a preset disturbance threshold, such as 0.005 rad² / s². If the motion disturbance exceeds the threshold, the gain adjustment factor is set to its inverse; otherwise, it is set to a fixed constant of 1.0. For example, if the motion disturbance is 0.00667 > 0.005, the gain adjustment factor is 1 / 0.00667 ≈ 150; if the disturbance is 0.004 and is less than the threshold, the factor is 1.0. This design enhances the sensitivity of subsequent damping control during periods of high disturbance through a dynamic scaling mechanism.

[0134] In practical applications, during the emergency response monitoring phase of a certain flight training exercise, the system first performs state analysis on previously generated spatiotemporal feature volumes. For example, during an 8-second roll maneuver, the extracted feature volume values ​​are as follows: 0.23 in the first second, 0.41 in the second, 0.85 in the third second, and 0.72 in the fourth second. The entropy chaos metric is calculated using the absolute difference method: 0.18 is calculated by subtracting the first second from the second second, 0.44 from the second second, and 0.13 from the third second. The maximum value is 0.44, the minimum is 0.13, and the amplitude of change is 0.31, calculated by subtracting the minimum value of 0.13 from the maximum value of 0.44. Simultaneously, the aircraft's real-time roll angle parameters are monitored, reaching 28 degrees during this period. When the roll angle parameter falls within the preset critical range of 25 to 30 degrees, the system directly uses the actual roll angle value of 28 as a scaling factor. The baseline entropy change threshold of 0.3 is multiplied by the scaling factor of 28, resulting in a calculation of 0.3 times 28 equaling 8.4, generating the adjusted entropy change threshold of 8.4. The entropy change amplitude of 0.31 is then compared with the threshold of 8.4. Since 0.31 is less than 8.4, the system determines that the current state is normal. During another extreme maneuver, the entropy sequence jumps from 0.35 to 1.12 within 2 seconds, with a change amplitude of 0.82. At this point, the roll angle is 32 degrees, exceeding the critical upper limit of 30 degrees. The system multiplies the absolute value of the roll angle, 32, by the baseline threshold of 0.3, resulting in 9.6. If the change amplitude of 0.82 does not exceed the threshold of 9.6, the system checks whether the absolute value of the roll angle, 32, exceeds the preset threshold of 30. If this condition is met, the system immediately determines that the state is out of control and initiates the emergency protocol. If the entropy change amplitude reaches 1.25, exceeding the adjusted threshold of 0.9, but the roll angle of 29 degrees does not exceed the threshold of 30, the warning status indicator is triggered. This dynamic adjustment mechanism enables the state judgment to adapt to the complexity of the flight attitude change characteristic value sequence of 0.55, 0.61, 0.89, and 1.24, with the maximum difference of 0.35, the minimum difference of 0.06, and the entropy change amplitude of 0.29, effectively distinguishing the three state levels of routine operation, risk warning, and emergency intervention.

[0135] In the overall scheme of the above-mentioned step 105, an entropy value sequence is formed by extracting the numerical sequence of the spatiotemporal feature body in a continuous time period and calculating the degree of chaos at each time point, then calculating the absolute value sequence of the difference between adjacent time points and taking the difference between the maximum and minimum values ​​as the entropy value change amplitude, and at the same time dynamically generating a scaling factor within the critical value range according to the absolute value of the roll angle parameter, multiplying the baseline entropy change threshold by the scaling factor to obtain an adjusted entropy change threshold, and by comparing the magnitude relationship between the entropy value change amplitude and the threshold, when the entropy value change amplitude is less than or equal to the adjusted threshold, it is determined that the operation state is normal, and when the amplitude is greater than the threshold, it is further determined whether the absolute value of the roll angle exceeds the preset threshold value. When it does not exceed the threshold, it is determined to be a warning state, and when it exceeds the threshold, it is determined to be an out-of-control state, thereby realizing a dual dynamic judgment mechanism combining physical flight parameters and cognitive entropy change characteristics, and accurately and graded warning of the operational risk state of the aircraft cockpit.

[0136] The following is a complete embodiment of steps 101 to 105:

[0137] like Figure 2 As shown in the figure, during a flight simulation training test, the system simultaneously acquired physiological and instrument data streams from pilot F. When the aircraft performed a rapid turning maneuver, the pitch angle change rate reached 18 degrees per second and the roll angle change rate reached 22 degrees per second. The dynamic constraints of these two values ​​combined reached 40 degrees per second. This value exceeded the preset segmentation threshold of 30 degrees per second, triggering the EEG signal segmentation point at 1.5 seconds, forming two independent analysis phases.

[0138] For the turning phase, the EEG dynamic topology showed a connection weight of 0.75 between the FP1 and FP2 nodes in the prefrontal cortex. The variance of the head motion disturbance was 7.56, and the damping parameter was adjusted to 13.2 Newton-seconds per meter. This generated a head stability index sequence of -0.05, -0.08, and -0.06. A 120-millisecond feature of the 4-8 Hz frequency band in the prefrontal cortex was extracted and fused with the mean head stability index of -0.063, yielding a cognitive conflict feature value of 63.981 milliseconds.

[0139] The eye movement trajectory coordinates were converted to the cockpit's three-dimensional spatial position (0.175 m, 0.137 m, 0 m), and the gaze density gradient ratio in the instrument core area was 4. This spatial encoding and cognitive feature value were input into the model, and fused through the spatiotemporal convolution kernel to generate a fusion strength value of 127.962, forming a spatiotemporal feature volume sequence of 0.45, 0.82, and 1.27.

[0140] Entropy analysis shows a maximum change of 0.45. When the aircraft's roll angle reaches 32 degrees, the entropy change threshold is dynamically adjusted to 9.6. Based on the coupling relationship between the 0.08 change and the 32-degree roll angle parameter, the system directly determines a loss of control and activates the emergency protocol. If the entropy change reaches 0.2 at a roll angle of 25 degrees and the threshold is adjusted to 7.5, a warning state signal is triggered. This method achieves three-level precise identification of operational status through the linkage of quantitative parameters.

[0141] Figure 3 The present invention provides a schematic diagram of a system for identifying the flight crew's operating status based on multimodal data fusion, as shown in FIG. Figure 3 As shown, the system includes:

[0142] The acquisition module 31 is used to synchronously acquire the flight crew's EEG signals, head inertial measurement unit angular velocity, eye movement trajectory coordinates, and the pitch angle parameters and roll angle parameters of the aircraft;

[0143] a generating module 32 for performing a dynamic segmentation operation on the EEG signal using the change rates of the pitch angle parameter and the roll angle parameter as dynamic constraints of the time series boundary, constructing an EEG dynamic topology map, and dynamically adjusting the damping negative feedback strength of the head motion control device based on the variance value of the angular velocity of the head inertial measurement unit to generate a head stability index that is inversely correlated with the head motion interference strength;

[0144] A fusion module 33 is configured to extract a wave persistence feature of the frontal lobe from the EEG dynamic topology map, and perform a weighted fusion operation on the wave persistence feature and the head stability index to generate a feature vector representing cognitive conflict;

[0145] a construction module 34 for converting the eye movement trajectory coordinates into the three-dimensional coordinate system of the aircraft cockpit, generating an eye movement spatial code, inputting the feature vector representing the cognitive conflict and the eye movement spatial code into a situational awareness entropy model, and fusing the spatial distribution of the cognitive conflict feature with the spatial density gradient of the eye movement trajectory through a spatiotemporal convolution kernel to construct a spatiotemporal feature volume;

[0146] The identification module 35 is used to dynamically adjust the entropy change threshold based on the entropy value change amplitude of the spatiotemporal feature body and the value of the roll angle parameter to determine whether the operating state level is normal, warning or out-of-control.

[0147] Figure 3 The flight crew operation status recognition system based on multimodal data fusion can perform Figure 1The implementation principles and technical effects of the flight crew operating status identification method based on multimodal data fusion described in the illustrated embodiment will not be elaborated upon. The specific manner in which the various modules and units of the flight crew operating status identification system based on multimodal data fusion in the aforementioned embodiment perform their operations has been described in detail in the related embodiments of the method and will not be elaborated upon here.

[0148] In one possible design, Figure 3 The flight crew operation status recognition system based on multimodal data fusion of the embodiment shown can be implemented as a computing device, such as Figure 4 As shown, the computing device may include a storage component 41 and a processing component 42;

[0149] The storage component 41 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 42 .

[0150] The processing component 42 is used for the above Figure 1 The embodiment provides a method for identifying flight crew operating status based on multimodal data fusion.

[0151] The processing component 42 may include one or more processors to execute computer instructions to perform all or part of the steps in the above method. Of course, the processing component may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0152] The storage component 41 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0153] Of course, a computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0154] The input / output interface provides an interface between the processing component and the peripheral interface module, which can be an output device, an input device, etc.

[0155] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0156] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0157] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The illustrated embodiment provides a method for identifying flight crew operating status based on multimodal data fusion.

[0158] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0160] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer or server) to execute the methods described in each embodiment or certain portions of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying flight crew operating status based on multimodal data fusion, characterized in that: include: Synchronously collect the flight crew's EEG signals, head inertial measurement unit angular velocity, eye movement trajectory coordinates, and the aircraft's pitch and roll angle parameters; Using the change rates of the pitch angle parameter and the roll angle parameter as dynamic constraints on the time series boundaries, a dynamic segmentation operation is performed on the EEG signal to construct an EEG dynamic topology map, and based on the variance value of the angular velocity of the head inertial measurement unit, the damping negative feedback strength of the head motion control device is dynamically adjusted to generate a head stability index that is inversely correlated with the head motion interference strength; extracting a wave persistence feature of the prefrontal brain region from the EEG dynamic topology map, and performing a weighted fusion operation on the wave persistence feature and the head stability index to generate a feature vector representing cognitive conflict; The eye movement trajectory coordinates are converted into the three-dimensional coordinate system of the aircraft cockpit to generate an eye movement spatial code, the feature vector representing the cognitive conflict and the eye movement spatial code are input into the situation awareness entropy model, and the spatial distribution of the cognitive conflict feature and the spatial density gradient of the eye movement trajectory are fused through the spatiotemporal convolution kernel to construct a spatiotemporal feature volume; Based on the entropy change amplitude of the spatiotemporal feature body and in combination with the value of the roll angle parameter, the entropy change threshold is dynamically adjusted to determine the operation state level as a normal state, a warning state, or an out-of-control state.

2. The method for identifying flight crew operating status based on multimodal data fusion according to claim 1, characterized in that: The feature vector representing the cognitive conflict and the eye movement spatial encoding are input into the situation awareness entropy model, and the spatial distribution of the cognitive conflict feature and the spatial density gradient of the eye movement trajectory are fused through the spatiotemporal convolution kernel to construct a spatiotemporal feature body, including: Combining the feature vector representing cognitive conflict with the eye movement spatial encoding into an input combination, and feeding the input combination into a situation awareness entropy model; Loading a preset convolution kernel weight set, wherein the convolution kernel weight set slides synchronously in the spatial dimension and the temporal dimension, and extracting the spatial distribution value of the cognitive conflict feature during the spatial sliding process; During the time sliding process, extracting the spatial density difference ratio value of the eye movement trajectory, multiplying the spatial distribution value by the spatial density difference ratio value to generate a fusion feature value; Through the convolution kernel weight set, a weighted superposition operation in the spatial dimension and a recursive integration operation in the temporal dimension are performed on the fused feature values ​​to output a fusion result body. Based on the fusion result body, a timestamp index is added to construct a spatiotemporal feature body.

3. The method for identifying flight crew operating status based on multimodal data fusion according to claim 1, characterized in that: Based on the entropy change amplitude of the spatiotemporal feature body and in combination with the value of the roll angle parameter, the entropy change threshold is dynamically adjusted to determine the operation state level as a normal state, a warning state, or an out-of-control state, including: Extracting a numerical sequence of the spatiotemporal feature body within a continuous time period, calculating the degree of disorder at each time point in the numerical sequence, and obtaining an entropy value sequence; Calculate the absolute value sequence of differences between adjacent time points from the entropy value sequence, and take the result of subtracting the minimum value from the maximum value of the absolute value sequence of differences as the entropy value change amplitude; Setting a critical value range for the roll angle parameter, extracting an absolute value of the roll angle parameter within the critical value range as a scaling factor, multiplying a baseline entropy change threshold by the scaling factor, and outputting an adjusted entropy change threshold; Comparing the magnitude relationship between the entropy value change amplitude and the adjusted entropy change threshold, and when the entropy value change amplitude is less than or equal to the adjusted entropy change threshold, determining that the operating state level is normal; When the amplitude of the entropy value change is greater than the adjusted entropy change threshold, it is determined whether the absolute value of the roll angle parameter exceeds the preset threshold value. If it exceeds, the operating status level is determined to be an out-of-control state; if it does not exceed, the operating status level is determined to be a warning state.

4. The method for identifying flight crew operating status based on multimodal data fusion according to claim 1, characterized in that: The eye movement trajectory coordinates are converted into the three-dimensional coordinate system of the aircraft cockpit to generate the eye movement spatial coding, including: Obtaining the horizontal and vertical position values ​​of the eye movement trajectory coordinates in the original plane coordinate system, combining them with the pre-stored aircraft cockpit coordinate system, executing coordinate conversion equation calculations, and outputting the converted three-dimensional space position values; The aircraft cockpit coordinate system is divided into spatial grid areas, the number of occurrences of the eye movement trajectory coordinates in each grid area is counted as spatial density data, and the spatial density difference ratio value is calculated based on the spatial density data to obtain the eye movement spatial coding.

5. The method for identifying flight crew operating status based on multimodal data fusion according to claim 1, characterized in that: Using the change rates of the pitch angle parameter and the roll angle parameter as dynamic constraints of the time series boundary, performing a dynamic segmentation operation on the EEG signal to construct an EEG dynamic topology map, including: Calculating a first instantaneous derivative of the pitch angle parameter in a time series and a second instantaneous derivative of the roll angle parameter in a time series, and taking the sum of the absolute values ​​of the first instantaneous derivative and the second instantaneous derivative as a total dynamic constraint value; At a position where the total dynamic constraint value exceeds a preset segmentation threshold, a segmentation point is marked, and the EEG signal is separated from the time series at the segmentation point as a boundary to form a plurality of signal segments; For each signal segment, the data sequence of the EEG electrode channel is extracted, each of the EEG electrode channels is used as a node of the graph, and based on the data sequence correlation between the nodes, the connection weight value is calculated to construct an EEG dynamic topology graph.

6. The method for identifying flight crew operating status based on multimodal data fusion according to claim 1, characterized in that: Dynamically adjusting the damping negative feedback strength of the head motion control device based on the variance value of the angular velocity of the head inertial measurement unit to generate a head stability index inversely correlated with the head motion interference strength, including: Extracting a continuous sequence of angular velocity of the head inertial measurement unit within a window of fixed time length, and calculating a variance value of the continuous sequence as a motion disturbance; Comparing the motion interference amount with a preset interference threshold, if the motion interference amount is greater than the preset interference threshold, setting the reciprocal of the motion interference amount as a gain adjustment factor according to empirical rules, otherwise setting a fixed constant as the gain adjustment factor according to empirical rules; Multiplying the set gain adjustment factor by the basic damping coefficient to obtain an adjusted damping parameter value, and inputting the adjusted damping parameter value into the negative feedback controller of the head motion control device; An error calculation is performed by a negative feedback controller, which takes the difference between the motion disturbance amount and the zero error reference as input, outputs an inverse proportional coefficient inversely proportional to the motion disturbance amount, and divides the inverse proportional coefficient by the unit time length to generate a head stability index.

7. The method for identifying flight crew operating status based on multimodal data fusion according to claim 1, characterized in that: Extracting the wave persistence feature of the frontal lobe brain area from the EEG dynamic topology map, and performing a weighted fusion operation on the wave persistence feature and the head stability index to generate a feature vector representing cognitive conflict, including: Selecting an electrode data sequence corresponding to the frontal lobe brain region from the nodes of the EEG dynamic topology map, performing signal waveform screening in a specific frequency band range, and calculating the length of time the screened frequency band waveform appears continuously as a wave duration feature; Setting a first fusion weighting factor for the wave duration feature and a second fusion weighting factor for the head stability index, wherein the first fusion weighting factor is set to be greater than the second fusion weighting factor; Multiplying the wave duration feature by the first fusion weight factor to obtain a first component, and multiplying the head stability index by the second fusion weight factor to obtain a second component; The sum of the first component and the second component is taken as a single element, and the time series average value of the single element is extracted and filled into a vector container of a fixed length to form a feature vector representing cognitive conflict.

8. A flight crew operating status recognition system based on multimodal data fusion, characterized in that: include: The acquisition module is used to synchronously collect the pilot's EEG signals, head inertial measurement unit angular velocity, eye movement trajectory coordinates, and the aircraft's pitch and roll angle parameters; a generation module configured to perform a dynamic segmentation operation on the EEG signal using the change rates of the pitch angle parameter and the roll angle parameter as dynamic constraints of the time series boundary, construct an EEG dynamic topology map, dynamically adjust the damping negative feedback strength of the head motion control device based on the variance value of the angular velocity of the head inertial measurement unit, and generate a head stability index that is inversely correlated with the head motion interference strength; A fusion module is configured to extract a wave persistence feature of the prefrontal lobe brain region from the EEG dynamic topology map, and perform a weighted fusion operation on the wave persistence feature and the head stability index to generate a feature vector representing cognitive conflict; a construction module, configured to convert the eye movement trajectory coordinates into a three-dimensional coordinate system of the aircraft cockpit, generate an eye movement spatial code, input the feature vector representing the cognitive conflict and the eye movement spatial code into a situational awareness entropy model, and fuse the spatial distribution of the cognitive conflict feature with the spatial density gradient of the eye movement trajectory through a spatiotemporal convolution kernel to construct a spatiotemporal feature volume; The identification module is used to dynamically adjust the entropy change threshold based on the entropy value change amplitude of the spatiotemporal feature body and in combination with the value of the roll angle parameter to determine the operating state level as normal state, warning state or out-of-control state.

Images