Unmanned aerial vehicle pilot operation error real-time prediction method fusing multi-dimensional feature analysis

Abstract

This invention relates to the field of unmanned aerial vehicle (UAV) control safety technology, and discloses a real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis. The method includes: acquiring multi-source control data; generating a multi-layered attention transfer directed graph and calculating the hierarchical graph structure features; calculating the perception delay of mode change and determining the perception blind zone; inferring the pilot's implicit cognitive pattern and calculating the mode cognitive bias degree during the perception blind zone through multi-mode parallel forward simulation; extracting post-perception cognitive adaptation features; calculating the delay-adaptation coupling risk index and the attention fragmentation-information omission correlation index; fusing the multi-dimensional features into a time-series matrix and inputting it into a GRU network for sequence prediction, outputting the operational error probability value and the error outbreak time window; and generating differentiated early warning signals based on the prediction results.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control safety technology, and more specifically, to a method for real-time prediction of UAV pilot operational errors by integrating multi-dimensional feature analysis. Background Technology

[0002] In the multi-screen operation scenario of the UAV ground control station, the pilot faces a heterogeneous control interface composed of multiple physical displays. The UAV flight control system supports multiple flight control modes and automatically switches modes during flight based on preset conditions. The mode change notification is displayed in the mode status indicator area of ​​the main flight display screen, and the pilot needs to adjust the control strategy according to the new mode.

[0003] Existing methods for predicting operational errors mainly employ approaches that analyze pilot cognitive confusion based on control commands and dynamic simulation residuals, or methods that analyze attention allocation bias based on eye-tracking data.

[0004] Existing methods have the following technical problems: First, methods based on control commands and dynamic simulation residual analysis assume that the pilot is immediately aware of the mode change at the moment of mode switching, without considering the possibility that the pilot may be looking at other screens in a multi-screen control environment and is completely unaware of the mode change. This method mixes the deterministic cognitive state misalignment during the perception blind spot period with the probabilistic cognitive adaptation difficulty during the cognitive adaptation period, resulting in inaccurate prediction results. Second, methods based on eye-tracking data treat the mode change notification as a general information event, without considering the global impact of the mode change on all subsequent control behaviors, and cannot assess the chain risks at the control level caused by ignoring the mode change notification. Third, the above two types of methods analyze from a single dimension and cannot capture the coupling relationship between cross-screen attention fragmentation and flight control mode cognitive confusion in a multi-screen control environment, resulting in a lack of effective predictive ability for operational errors caused by the combined effects of perception omission and cognitive bias in multi-screen control scenarios. Summary of the Invention

[0005] This invention provides a real-time prediction method for unmanned aerial vehicle (UAV) pilot operational errors that integrates multi-dimensional feature analysis, solving the technical problem in related technologies that cannot effectively predict operational errors caused by pilots switching modes in a multi-screen control environment.

[0006] This invention discloses a real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis, comprising the following steps: acquiring the mode switching event log of the flight control system, the control permission mapping table corresponding to each flight control mode, the physical layout information and region of interest definition information of the multi-screen control interface, and the pilot's eye-tracking data; generating a gaze sequence based on the eye-tracking data and physical layout information, generating a multi-layered attention transfer directed graph containing in-screen transfer edges and cross-screen transfer edges according to the gaze sequence, and calculating the layered graph structure features; retrieving the timestamp of the pilot's first gaze at the region of interest where the mode status indicator is located after the mode switching timestamp from the gaze sequence, calculating the time difference between the two as the mode change perception delay, and defining the time interval between the two as the perception blind zone; within the perception blind zone, transferring the control command sequence according to the control channel-flight response transfer function of each candidate flight control mode. Forward simulation is performed, and the candidate mode with the smallest cumulative norm of residuals between the predicted flight state and the actual flight state is selected as the implicit cognitive mode. The Euclidean distance between the control gain vector of the implicit cognitive mode and the actual effective mode is calculated as the mode cognitive bias. After the perception blind zone period ends, the control command feature vector is extracted using a sliding window, and the Mahalanobis distance change sequence between the control command and the behavior reference baseline is calculated. An exponential decay curve is fitted to obtain the post-perception mode adaptation rate. Based on the mode change perception delay, the mode cognitive bias, and the post-perception mode adaptation rate, a delay-adaptation coupling risk index is calculated. The multi-dimensional fusion feature vector containing the hierarchical graph structure features, the mode change perception delay, the mode cognitive bias, and the delay-adaptation coupling risk index is arranged in chronological order to form a time series matrix. The input sequence prediction model outputs the operation error probability value and the error outbreak time window.

[0007] Furthermore, the multi-layered attention transfer directed graph includes an in-screen transfer layer and a cross-screen transfer layer; each directed edge in the in-screen transfer layer records the number of transfers from the source region to the target region and the average transfer time; each directed edge in the cross-screen transfer layer records the number of transfers and the average transfer time, and is weighted by a cross-screen distance, which is the Euclidean distance between the center position coordinate vectors of the source screen and the target screen.

[0008] Furthermore, the calculation of the layered graph structure features includes: calculating the ratio of the number of cross-screen transfer edges to the total number of transfer edges to obtain the cross-screen transfer frequency; calculating the ratio of the total time cost of all cross-screen transfers to the total time cost of all attention transfers to obtain the cross-screen transfer time cost ratio; calculating the Gini coefficient of the pilot's attention dwell time on each screen, wherein the Gini coefficient is calculated by: arranging the attention dwell time of each screen in ascending order, calculating the sum of the products of each screen's dwell time and the difference between twice the screen's sorting number and the total number of screens minus one, and then dividing by the product of the total number of screens and the total dwell time of all screens; extracting transfer path subsequences of a preset length from the cross-screen transfer edge sequence, calculating the frequency of each subsequence, and calculating the ratio of the sum of the frequencies of the subsequences with the highest frequency of occurrence to the sum of the frequencies of all subsequences to obtain the concentration of common patterns of cross-screen transfer paths.

[0009] Furthermore, it also includes information response delay analysis based on information update event logs and the gaze sequence: for each information update event, the timestamp of the pilot's gaze first reaching the region of interest where the information update event is located after the timestamp of the information update event is retrieved, and the time difference between the two is calculated as the gaze response delay time of the information update event; the gaze response delay time is weighted according to the change importance level of each information update event, and the weighted mean is calculated to obtain the weighted mean of key information response delay; the number of high-importance information change events that have not been gazed upon by the pilot within a preset time limit is accumulated; the Jensen-Shannon divergence between the attention distribution composed of the proportion of pilot attention dwell time on each screen and the information demand distribution composed of the proportion of information update frequency on each screen is calculated as an indicator of attention-information demand mismatch.

[0010] Furthermore, during the perception blind zone period, the following auxiliary indicators are extracted: The set of invalid channels is identified based on the control authority mapping table of the current actual effective mode; the ratio of the number of attempts by the pilot to issue commands to invalid channels to the total number of commands during the perception blind zone period is calculated to obtain the invalid channel command attempt frequency; for valid channels in the current actual effective mode, the standard control amplitude range of each valid channel is determined by the product of the gain coefficient of the channel in the current actual effective mode and the physical input range allowed by the flight control system; the proportion of commands issued by the pilot to the channel during the perception blind zone period that fall within the standard control amplitude range is calculated as the gain fit of the channel; the average of the gain fit of all valid channels is taken to obtain the effective channel gain fit.

[0011] Furthermore, the step of extracting control command feature vectors using a sliding window and calculating the Mahalanobis distance change sequence between the control command and the behavioral reference baseline, and fitting an exponential decay curve to obtain the post-perception mode adaptation rate, includes: extracting control command feature vectors, including the command mean, command variance, and command change rate of each effective channel, for each control command sequence within the sliding window; obtaining the mean vector and covariance matrix of the behavioral reference baseline under normal control conditions for the current actual effective mode, wherein the behavioral reference baseline is obtained from control command feature vector samples from historical flight data of pilots completing the stable control phase in the current actual effective mode. The data is obtained through set statistics; the Mahalanobis distance between the control command feature vector of each sliding window and the behavior reference baseline is calculated. The Mahalanobis distance is obtained by performing a quadratic operation on the difference between the control command feature vector and the mean vector, the inverse of the covariance matrix, and taking the square root, to obtain the Mahalanobis distance change sequence; the exponential decay curve is fitted to the Mahalanobis distance change sequence using the nonlinear least squares method, and the decay coefficient is used as the post-sensory mode adaptation rate; the segments in the Mahalanobis distance change sequence where the Mahalanobis distance does not show a decreasing trend within a preset number of consecutive sliding windows are detected, marked as adaptation stagnation events, and the start and end times are recorded.

[0012] Furthermore, the calculation of the delay-adaptation coupling risk index includes: applying mean normalization based on the range to the pattern change perception delay, the pattern cognition deviation, and the post-perception pattern adaptation rate; using the product of the normalized pattern change perception delay and the normalized pattern cognition deviation as the cumulative manipulation inertia strength; using the ratio of the cumulative manipulation inertia strength to the normalized post-perception pattern adaptation rate as the delay-adaptation coupling risk index; and further includes calculating the attention fragmentation-information omission correlation index: calculating the time change sequence of cross-screen transfer frequency and the time change sequence of the cumulative number of unresponsive high-importance information using a sliding time window, and calculating the cross-correlation coefficient between the two time change sequences as the attention fragmentation-information omission correlation index.

[0013] Furthermore, the multidimensional fusion feature vector also includes the invalid channel instruction attempt frequency, effective channel gain adaptation, post-perception mode adaptation rate, adaptation stagnation event marker, attention fragmentation-information omission correlation index, and attention-information demand mismatch index; the sequence prediction model is a sequence prediction model based on a gated recurrent unit network, and the output layer contains two branches: the first branch outputs the operation error probability value through a sigmoid activation function, and the second branch outputs the predicted value of the start time of the error outbreak time window and the width of the time window through a linear mapping layer; during training, the probability prediction branch adopts the binary cross-entropy loss function, and the time prediction branch adopts the mean squared error loss function. The parameters are updated after the two loss functions are weighted and summed.

[0014] Furthermore, when the probability of operational error exceeds a preset threshold, a real-time warning signal is generated. This warning signal is differentiated based on the pilot's current cognitive stage: if the mode change perception delay continues, a mandatory mode change notification pops up on the screen the pilot is currently looking at. This mandatory mode change notification includes the currently active mode identifier and a list of effective control channels under the currently active mode; if the perception blind spot period has ended but an adaptation stagnation event is detected, key control points for the currently active mode and a recommended control amplitude range reference are output; simultaneously, high-risk screen numbers and summaries of key information accumulated on high-risk screens that have not been noticed by the pilot are output.

[0015] This invention provides a real-time prediction system for unmanned aerial vehicle (UAV) pilot operational errors that integrates multi-dimensional feature analysis. The system includes: a multi-source data acquisition module for acquiring mode switching event logs of the flight control system, control permission mapping tables corresponding to each flight control mode, physical layout information and region of interest (ROI) definition information of the multi-screen control interface, and pilot eye-tracking data; an attention transfer graph generation module for generating a gaze sequence based on the eye-tracking data and the physical layout information, generating a multi-layered directed attention transfer graph containing in-screen and cross-screen transfer edges according to the gaze sequence, and calculating the layered graph structure features; a perception delay calculation module for retrieving the timestamp of the pilot's first gaze at the ROI of the mode status indicator after the mode switching timestamp from the gaze sequence, calculating the mode change perception delay, and determining the perception blind spot period; and a cognitive mode inference module for processing the control command sequence according to each ROI during the perception blind spot period. Forward simulation is performed on the control channel-flight response transfer function of the selected flight control mode. The candidate mode with the smallest residual cumulative norm is selected as the implicit cognitive mode, and the mode cognitive bias is calculated. The cognitive adaptation feature extraction module is used to extract the control command feature vector with a sliding window after the perception blind zone period ends and calculate the Mahalanobis distance change sequence between the vector and the behavior reference baseline. The exponential decay curve is fitted to obtain the post-perception mode adaptation rate. The coupling risk calculation module is used to calculate the delay-adaptation coupling risk index based on the mode change perception delay, the mode cognitive bias, and the post-perception mode adaptation rate. The fusion prediction module is used to form a time series matrix by arranging the multi-dimensional fusion feature vector containing the hierarchical graph structure features, the mode change perception delay, the mode cognitive bias, and the delay-adaptation coupling risk index in chronological order. The input sequence prediction model outputs the operation error probability value and the error outbreak time window.

[0016] This invention separates the actual time point of the pilot's perception mode change from the time point of the mode switching event, determines the time boundary of the perception blind zone, and deterministically infers the pilot's implicit cognitive mode through multi-mode parallel forward simulation within the perception blind zone. It quantifies the additional cognitive overhead and attention fragmentation caused by cross-screen switching by generating a multi-layered directed graph that distinguishes between intra-screen and cross-screen transfers, quantifies the cumulative impact of perception delay on the subsequent cognitive adaptation process by using a delay-adaptation coupling risk index, and quantifies the dynamic correlation between the degree of cross-screen attention fragmentation and the number of key information omissions by using an attention fragmentation-information omission correlation index. It integrates multi-dimensional information such as cross-screen attention characteristics, mode cognitive bias characteristics, and delay-adaptation coupling characteristics into a GRU network for time-series prediction. This solves the technical problems of existing methods that cannot accurately distinguish between the perception blind zone and the cognitive adaptation period, and cannot capture the coupling relationship between perception omission and cognitive confusion, and achieves the technical effect of real-time prediction of operational errors in flight control mode switching scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart of a real-time prediction method for UAV pilot operational errors that integrates multi-dimensional feature analysis, provided in an embodiment of the present invention. Figure 2 It is a schematic diagram of the multi-source control data acquisition and processing flowchart; Figure 3 This is a schematic diagram illustrating the distribution of gaze points before and after mode switching; Figure 4 This is a schematic diagram of a multi-layered directed graph structure for attention transfer; Figure 5 This is a schematic diagram illustrating the comparative analysis of information response delay times; Figure 6 This is a schematic diagram of the comparison of residuals in the forward simulation of the model; Figure 7 This is a schematic diagram of the temporal variation curve of the Mahalanobis distance after perception; Figure 8 This is a schematic diagram of the overall architecture of the operational error prediction system. Detailed Implementation

[0018] In the operation scenario of an unmanned aerial vehicle (UAV) ground control station, pilots typically face a heterogeneous control interface consisting of multiple physical displays, such as a main flight display, a tactical situation display, and a payload monitoring display. The UAV flight control system supports multiple flight control modes (such as fully autonomous mode, semi-autonomous mode, and manual mode) and automatically switches modes during flight based on preset conditions. When a mode switch occurs, a mode change notification is usually displayed in the mode status indicator area on the main flight display, and the pilot needs to adjust their control strategy according to the new mode.

[0019] Existing methods for predicting operational errors have the following technical problems: First, current methods for analyzing pilot cognitive confusion based on control commands and dynamic simulation residuals assume that pilots are immediately aware of the mode change at the moment of mode switching and assess the rationality of their control behavior accordingly. However, in a multi-screen control environment, pilots may be looking at other screens and completely unaware that the mode has changed. During this period of cognitive blind spot, the pilot continues to operate in the old mode. The deviation between their implicit cognitive mode and the actual effective mode is a deterministic cognitive misalignment, rather than a probabilistic difficulty in cognitive adaptation. Existing methods analyze control behavior during the cognitive blind spot period together with control behavior during the cognitive adaptation period, leading to inaccurate prediction results.

[0020] Second, the method of analyzing attention allocation bias based on eye-tracking data treats pattern change notifications as general information events, without considering the global impact of pattern changes on all subsequent manipulative behaviors, and cannot assess the chain of risks at the manipulative level caused by neglecting pattern change notifications.

[0021] Third, the two methods mentioned above analyze from a single dimension and cannot capture the coupling relationship between cross-screen attention fragmentation and flight control mode cognitive confusion in a multi-screen control environment. This results in a lack of effective predictive ability for operational errors caused by the combined effects of perceptual omissions and cognitive biases in multi-screen control scenarios.

[0022] Therefore, it is necessary to integrate multi-dimensional information such as cross-screen attention perception delay characteristics and pattern cognition bias characteristics, separate the actual time point of pilot's perception of pattern change from the time point of pattern switching event, and conduct multi-dimensional feature fusion analysis covering the perception blind spot period and the cognitive adaptation period to achieve real-time prediction of operational errors.

[0023] According to an embodiment of this invention, a real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis is provided. It should be understood that this real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis is executed by a data processing device at the UAV ground control station, which is communicatively connected to the flight control system, eye-tracking equipment, and multi-screen control interface.

[0024] At least one embodiment of the present invention discloses a real-time prediction method for unmanned aerial vehicle (UAV) pilot errors based on multi-dimensional feature analysis, such as... Figure 1 As shown, it includes the following steps: Step 1: Acquire multi-source control data; The system acquires the mode switching event logs of the flight control system, the control permission mapping table corresponding to each flight control mode, the physical layout information and region of interest definition information of the multi-screen control interface, the pilot's eye tracking data, and the information update event logs of each region of interest on each screen, as multi-source control data.

[0025] The mode switching event log includes the timestamp of each mode switch. Source pattern identifier and target pattern identifier The control access mapping table defines the set of channels that the pilot can control in each flight control mode, as well as the control gain coefficient and control channel-flight response transfer function for each channel. The physical layout information of the multi-screen control interface includes the spatial coordinates and dimensions of each screen. The region of interest (ROI) definition information includes the screen number and ROI number where the mode status indicator is located. Eye-tracking data includes a sequence of timestamped gaze coordinates. The information update event log includes the timestamp, change type, and change importance level for each information change.

[0026] It should be noted that the control gain coefficient in the aforementioned control authority mapping table refers to the proportional coefficient of the flight control system's response when the pilot applies a unit input to a certain control channel in a specific flight control mode. For example, in manual mode, the control gain coefficient for the aileron channel might be 1.0, indicating that the pilot's input is directly mapped to the control surface deflection command; while in semi-autonomous mode, the control gain coefficient for this channel might decrease to 0.3, indicating that the pilot's input is only used as a correction amount for the flight control system's autonomous control. The difference in the control gain coefficient of the same channel in different modes constitutes the basis for adjusting the control strategy when switching modes.

[0027] It should be noted that the aforementioned control channel-flight response transfer function refers to the input-output mapping function between the pilot's control commands and the UAV's flight state response under a specific flight control mode. Different control channels correspond to different control channel-flight response transfer functions under different flight control modes. This control channel-flight response transfer function is used in subsequent steps to perform forward simulation of the pilot's control commands to predict the flight state.

[0028] During a border reconnaissance mission, pilot OP-Alpha conducted a drone patrol mission on the morning of March 15, 20XX. The ground control station was equipped with three displays: Screen A was the main flight display, showing core flight parameters such as altitude, speed, and attitude, as well as mode status indicators; Screen B was the tactical situation display, showing maps, flight paths, and target information; and Screen C was the payload monitoring display, showing images from the electro-optical pods and sensor status. During the mission, the flight control system detected a change in weather conditions and automatically switched from fully autonomous mode to semi-autonomous mode at 10:23:45. The data processing unit retrieved this mode switch event and the switch timestamp from the flight control system. =10:23:45.000, Source Pattern Identifier =Fully autonomous mode, target mode identifier =Semi-autonomous mode.

[0029] Table 1 Mode switching event log data: Meanwhile, the data processing unit extracts the control channel configuration information corresponding to each flight control mode from the control authority mapping table. In fully autonomous mode, the pilot can only control the heading and altitude channels, while the aileron and elevator channels are completely taken over by the flight control system; in semi-autonomous mode, the pilot can control four channels: heading, altitude, aileron, and elevator, but the control gain coefficient of each channel is reduced compared to manual mode.

[0030] Table 2 Control Permission Mapping Table (Partial Channels): The data processing device acquired the pilot's gaze coordinate sequence from 10:23:30 to 10:24:30 from the eye-tracking device, with a sampling frequency of 60Hz. The physical layout information of the multi-screen control interface shows that screen A is located directly in front of the pilot, with a center coordinate of (0, 800) mm and dimensions of 600mm × 400mm; screen B is located to the pilot's left front, with a center coordinate of (-500, 750) mm and dimensions of 500mm × 350mm; and screen C is located to the pilot's right front, with a center coordinate of (500, 750) mm and dimensions of 500mm × 350mm. The areas of interest (ROIs) on screen A include: area A1 as a mode status indicator, area A2 as an altitude display area, area A3 as a speed display area, and area A4 as an attitude indicator.

[0031] Table 3 Physical layout information of the multi-screen control interface: The data processing device also retrieves information change records for each area of ​​interest on each screen from the information update event log. At 10:23:45.000, when the mode switch occurred, area A1 (mode status indicator) on screen A underwent an information update, with the change type being a mode identifier change and the change importance level being high. In addition, between 10:23:42.150 and 10:23:58.320, area B2 (target tracking status area) on screen B and area C1 (electro-optical pod field of view indicator area) on screen C also underwent multiple information updates.

[0032] Table 4 Information Update Event Log (Partial Records): Step 2: Generate a multi-layered directed graph of attention transfer; Based on the gaze coordinates in the eye-tracking data and the physical layout information of the multi-screen control interface, each gaze point is mapped to the corresponding screen number and the in-screen region of interest number to obtain a gaze sequence with screen and region identifiers.

[0033] Based on the order in which the gaze point jumps between regions of interest in the gaze sequence, a multi-layered directed attention transfer graph is generated. .in, The set of nodes that constitute all regions of interest; This is the set of in-screen transition edges, representing the attention transfer between different regions of interest within the same screen; This is a set of cross-screen transfer edges, representing the attention transfer between regions of interest on different screens.

[0034] It should be noted that the aforementioned multi-layered directed attention transfer graph includes an intra-screen transfer layer and a cross-screen transfer layer. Each directed edge in the intra-screen transfer layer records the number of transfers from the source region to the target region and the average transfer time. In addition to recording the number of transfers and the average transfer time, each directed edge in the cross-screen transfer layer also has a cross-screen distance weight calculated based on the spatial coordinates of the source and target screens. ,in and Source screen and target screen The center position coordinate vector, This represents the Euclidean norm.

[0035] The data processing unit performed screen and region mapping on 3,600 gaze point samples from 10:23:30 to 10:24:30. Analysis showed that before and after the mode switch, the pilot's attention was primarily focused on frequent switching between screens B and C to monitor target tracking status and electro-optical pod images. The gaze sequence showed that from 10:23:40 to 10:23:50, the pilot's gaze shifted seven times between region B2 on screen B and region C1 on screen C, with an average shift time of 420ms per shift. During the same period, the pilot only briefly gazed at region A3 (speed display area) on screen A at 10:23:38.560 for 180ms, and did not gaze at screen A again until 10:23:51.240.

[0036] The data processing device generates a multi-layered directed graph of attention transfer based on the gaze sequence. (Node set) Includes all nodes in the region of interest: {A1,A2,A3,A4,B1,B2,B3,C1,C2}. Set of in-screen transition edges. This includes transition edges within the same screen. For example, within screen A, a transition from region A2 to region A3 occurs 3 times, with an average time of 150ms. The set of cross-screen transition edges... It includes transition edges between different screens. For example, the transition from region B2 of screen B to region C1 of screen C occurs 7 times, with an average time of 420ms.

[0037] Table 5. Partial data of the directed graph edges of multi-layer attention transfer: The weighting of the cross-screen distance between screens B and C is calculated based on the coordinates of their center positions: In this embodiment of the application, in order to quantify the structured features of attention allocation in a multi-screen control environment, the hierarchical graph structure features of the multi-layer attention transfer directed graph are calculated, specifically including the following sub-steps: Step 201: Count the set of cross-screen transition edges. The cross-screen transfer frequency is obtained by comparing the number of edges in the current frame to the total number of transfer edges. .

[0038] Step 202: Calculate the ratio of the total time cost of all cross-screen transfers to the total time cost of all attention transfers, and obtain the cross-screen transfer time cost ratio. .

[0039] Step 203: Calculate the Gini coefficient of the pilot's attention dwell time on each screen. The Gini coefficient characterizes the degree of uneven distribution of attention across screens. The larger the value, the more focused attention is on a few screens.

[0040] Furthermore, the Gini coefficient is calculated as follows: Assume the attention dwell time series for each screen, arranged in ascending order, is... ,in Total number of screens Sort by length of stay in ascending order. The Gini coefficient is calculated based on the attention dwell time on each screen: in, The screen number is sorted in ascending order by dwell time.

[0041] Step 204: Extract subsequences of a preset length of transition path from the cross-screen transition edge sequence, count the frequency of occurrence of each subsequence, and calculate the ranking of the most frequent subsequences. The ratio of the sum of frequencies of a given subsequence to the sum of frequencies of all subsequences is used to obtain the concentration of common patterns in cross-screen transfer paths. ,in This is the preset number of ranking cutoffs.

[0042] The data processing device statistically analyzed attention shift data from 10:23:30 to 10:24:30. Among a total of 52 attention shifts, the cross-screen shift edge set... The number of edges in the image is 18, therefore the cross-screen transfer frequency is... The total time cost of all cross-screen transfers is 7830ms, and the total time cost of all attention transfers is 12450ms. Therefore, the time cost of cross-screen transfers is higher than that of attention transfers. .

[0043] The data processing device calculated the attention dwell time for each screen: screen A had a dwell time of 8.2 seconds, screen B had a dwell time of 28.5 seconds, and screen C had a dwell time of 23.3 seconds. These are then sorted in ascending order. Second, Second, Seconds. Calculate the Gini coefficient: The data processing device extracts a 3-length subsequence of the transition path from the cross-screen transition edge sequence and sets... Statistical results show that the three most frequent subsequences are: B2→C1→B2 (occurring 5 times), C1→B2→C1 (occurring 4 times), and A4→B2→C1 (occurring 2 times). The sum of the frequencies of these three subsequences is 11 times, and the sum of the frequencies of all subsequences is 16 times. Therefore, the concentration of common patterns in cross-screen transfer paths... .

[0044] Table 6: Calculation results of the hierarchical diagram structural features: In this embodiment of the application, in order to evaluate the timeliness of the pilot's response to changes in key information on the screen, information response delay analysis is performed based on information update event logs and gaze sequences, specifically including the following sub-steps: Step 211: For each information update event, retrieve the timestamp after the timestamp of the information update event when the pilot's gaze first reaches the region of interest where the information update event is located, and calculate the time difference between the two timestamps as the gaze response delay time for the information update event. .

[0045] Step 212: Weight the gaze response delay time according to the importance level of each information update event, and calculate the weighted mean to obtain the weighted mean of key information response delay. .

[0046] Step 213, accumulate within the preset time limit Number of high-importance information changes that went unnoticed by pilots .

[0047] The data processing unit performs gaze response delay analysis on each event in the information update event log. For event INFO-U-1123 (10:23:42.150, screen area B2), the pilot first gazed at this area at 10:23:43.280, and the gaze response delay time is... Seconds. For event INFO-U-1124 (10:23:45.000, screen area A1, mode identifier change), the pilot did not first look at the area until 10:23:51.240, gaze response delay time. Seconds. For event INFO-U-1125 (10:23:47.680, screen area C1), the pilot first looked at this area at 10:23:48.150, gaze response delay time. Second.

[0048] The data processing device weights events according to their importance: high-importance events have a weight of 3.0, medium-importance events have a weight of 2.0, and low-importance events have a weight of 1.0. The total weighted delay time for the four information update events analyzed is... seconds, with a total weight of Therefore, the weighted average of the response delay of key information Second.

[0049] Data processing device set preset time limit Seconds. Within this time limit, the gaze response delay for event INFO-U-1124 (pattern identifier change, high importance) was 6.240 seconds. Since it was not observed within this time limit, the cumulative number of unresponsive high importance information events is [number missing]. .

[0050] Table 7. Information Response Delay Analysis Results (Partial List of Events): Step 3: Calculate the latency of mode change perception; For each mode switch event, retrieve the mode switch timestamp from the gaze sequence. The time stamp of the pilot's first gaze at the region of interest where the mode status indicator was located. Delay in perceiving changes in computing mode: in, Mode change perception delay is the time interval between the pilot's actual switch from mode to visual perception and the change in mode. Time interval Defined as a perception blind spot, during which the pilot does not look at the mode status indicator and their control behavior is based on the old mode cognition before the mode switch.

[0051] It should be noted that the calculation of the above-mentioned mode change perception delay uses the arrival of the pilot's gaze point at the region of interest where the mode status indicator is located as the criterion for the occurrence of perception. This is based on a preset maximum waiting time after the mode switch event. If the pilot does not consistently monitor the area, the mode change perception delay will be set to [value missing]. This indicates that the pilot did not perceive a mode change within that timeframe.

[0052] The data processing unit performs perception delay calculations for the mode switching event EVT-M-0347. (Mode switching timestamp) =10:23:45.000, the mode status indicator is located in area A1 of screen A. Retrieval from the gaze sequence reveals the timestamp of the pilot's first gaze at area A1 after the mode switch. =10:23:51.240. Calculation mode change perception latency: Therefore, within the time interval [10:23:45.000, 10:23:51.240), the pilot is in a perception blind spot and is unaware that the mode has switched from fully autonomous mode to semi-autonomous mode. His control behavior is still executed according to the perception of fully autonomous mode.

[0053] In this embodiment, to quantify the deviation between pilot attention allocation and the actual information update needs of each screen, an attention-information demand mismatch index is calculated. Specifically, the proportion of pilot attention dwell time on each screen to the total dwell time constitutes the attention distribution. The information demand distribution is determined by the proportion of each screen's information update frequency to the total update frequency. ,in For the number of screens, The first To the The percentage of time attention is spent on each screen. The first To the The proportion of information update frequency for each screen. Calculation and Jensen-Shannon divergence between: in, , The divergence is Kullback-Leibler. A higher value indicates a greater degree of mismatch between the pilot's attention allocation and information needs. This attention-information demand mismatch index serves as an indicator of attention-information demand mismatch. .

[0054] The data processing device statistically analyzed the attention dwell time and information update frequency of each screen from 10:23:30 to 10:24:30. Screen A had a dwell time of 8.2 seconds and 2 information update events; Screen B had a dwell time of 28.5 seconds and 5 information update events; Screen C had a dwell time of 23.3 seconds and 3 information update events. The total dwell time was 60.0 seconds, and the total number of information update events was 10.

[0055] Construct attention distribution Information demand distribution Calculate the intermediate distribution. .

[0056] Calculate the Kullback-Leibler divergence and Then calculate the Jensen-Shannon divergence. This value serves as an indicator of attention-information mismatch. .

[0057] Step 4: Infer the pilot's implicit cognitive patterns and calculate the pattern cognitive bias. During the period of perception blind spot Inside, it acquires the sequence of control commands issued by the pilot. and the actual flight status sequence within the corresponding time period .

[0058] The control command sequence is divided into each candidate flight control mode. ( , The total number of candidate patterns. For the first Forward simulation is performed on the control channel-flight response transfer function corresponding to the control authority mapping table of each candidate flight control mode to obtain the predicted flight state sequence under each candidate mode. Calculate the cumulative norm of the residual vector between the predicted and actual flight states for each candidate mode: in, Candidate mode The cumulative norm of the residual vector, for The actual flight status at any given moment, By candidate mode Simulation results Predict flight status at all times. Represents the Euclidean norm, and the summation range covers the period of sensory blindness. All sampling times within.

[0059] The candidate pattern with the smallest cumulative residual norm was selected as the inference result of the pilot's implicit cognitive pattern. Extract the inferred implicit cognitive patterns from the control authority mapping table. Corresponding control gain vector and the current actual effective mode Corresponding control gain vector The Euclidean distance was used to calculate the pattern perception bias between the two: in, For pattern recognition bias, It represents the Euclidean norm, which quantifies the degree of difference between the pilot's implicit cognitive mode and the actual effective mode at the control level during the perception blind spot period.

[0060] Furthermore, the aforementioned control gain vector and The control gain vector is composed of the gain coefficients of each control channel in the corresponding flight control mode, arranged in order of channel number. and The dimension is equal to the total number of control channels. When a pilot is in a perception blind spot, the control gain vector implied by their control behavior... Control gain vector in actual effective mode The greater the Euclidean distance between them, the greater the deviation between the pilot's current control force distribution and the actual response ratio of each channel, and the higher the risk of control error.

[0061] It should be noted that in the aforementioned forward simulation process, the control channel-flight response transfer function for each candidate mode defines the mapping relationship from control commands to flight state responses under that candidate mode. When the pilot follows the old mode (i.e., the source mode before mode switching)... When issuing commands based on the user's control habits, The residual between the predicted state and the actual state obtained by forward simulation using the control channel-flight response transfer function is usually the smallest, thus the inference results... Usually The significance of this inference lies in definitively identifying the pilot's state of operation in the old mode during the perception blind spot period, rather than estimating the probability of the pilot adapting to the new mode.

[0062] The data processing unit acquires the sequence of control commands and the sequence of actual flight states during the perception blind spot period [10:23:45.000, 10:23:51.240] from the flight control system. During this period, the pilot issued 12 control commands, involving the heading and altitude channels, with a sampling frequency of 2Hz. The actual flight state includes state variables in four dimensions: heading angle, pitch angle, roll angle, and altitude.

[0063] The data processing unit sets the candidate flight control modes as: fully autonomous mode, semi-autonomous mode, and manual mode, corresponding to... The control command sequence was forward simulated according to the control channel-flight response transfer function of the three candidate modes. For fully autonomous mode, the directional channel gain coefficient was 0.5, the altitude channel gain coefficient was 0.5, and the aileron and elevator channels were taken over by the flight control system; for semi-autonomous mode, the directional channel gain coefficient was 0.8, the altitude channel gain coefficient was 0.7, the aileron channel gain coefficient was 0.4, and the elevator channel gain coefficient was 0.6; for manual mode, the gain coefficient of all channels was 1.0.

[0064] Forward simulation results show that the residual cumulative norm of the fully autonomous mode The cumulative norm of residuals in semi-autonomous mode The cumulative norm of residuals in manual mode The candidate pattern with the smallest residual is the fully autonomous pattern, therefore the inferred implicit cognitive pattern... =Fully autonomous mode, compared to the original mode before the mode switch Consistency indicates that the pilot continued to operate the controls according to the fully autonomous mode during the perception blind spot period.

[0065] The data processing unit extracts the control gain vector from the control authority mapping table. (Control gain vector in fully autonomous mode) (In order of heading, altitude, aileron, elevator channel), the control gain vector in semi-autonomous mode Calculation of cognitive bias in patterns: In this embodiment of the application, to further refine the deviation characteristics between the pilot's control behavior and the actual effective mode during the perception blind spot period, the following two auxiliary indicators are also extracted: Step 401, based on the current actual effective mode The control permission mapping table identifies the current actual effective mode. The set of uncontrollable invalid channels is analyzed, and the ratio of the number of pilot attempts to issue commands to invalid channels during the perception blind spot period to the total number of commands is calculated to obtain the invalid channel command attempt frequency. .

[0066] Step 402, for the current actual effective mode The effective channel is used to calculate the amplitude distribution of the pilot's commands during the perception blind spot period and the current actual effective mode. The effective channel gain fit is obtained by measuring the overlap between the standard control amplitude ranges corresponding to the gain coefficients. Specifically, let the effective channel set contain the first... The standard control amplitude range for each channel is: ,in For the first The lower limit of the standard control amplitude for each channel For the first The standard control amplitude limit for each channel, which constitutes the set of amplitudes of all commands issued by the pilot to that channel during the perception blind spot period. Then the gain fit of this channel is the percentage of commands falling within the standard amplitude range. The proportion of the total number. The effective channel gain fit is obtained by averaging the gain fit of all effective channels. Effective channel gain fit The lower the value, the greater the difference between the pilot's control effort in the effective channel and the requirements of the current mode.

[0067] Furthermore, the aforementioned standard control amplitude range The channel is in its current active mode as determined by the control permission mapping table. The gain coefficient is determined jointly by the physical input range allowed by the flight control system: let the upper limit of the physical input of this channel be... The physical input lower limit is The gain coefficient is The upper limit of the standard control amplitude range. lower limit This reflects the current actual effective model. The effective response amplitude range of the flight control system to the input of this channel under the gain.

[0068] The data processing unit, based on the semi-autonomous mode control authority mapping table, identified the aileron and elevator channels as controllable in semi-autonomous mode but inaccessible to the pilot in fully autonomous mode. However, during the perception blind spot, the pilot continued to operate in fully autonomous mode, without issuing commands to the aileron and elevator channels. Statistics show that out of 12 control commands, there were 0 attempts to issue commands to invalid channels (those perceived as invalid by the pilot but actually active). Therefore, the frequency of invalid channel command attempts was low. .

[0069] The data processing unit calculates the gain adaptation for the effective channels in semi-autonomous mode. The gain coefficient for the heading channel in semi-autonomous mode is 0.8, and the physical input range is... The standard control amplitude range is During the perception blind spot period, the amplitudes of the six commands issued by the pilot to the heading channel were -12°, -8°, -5°, -3°, 0°, and +2°, all falling within the standard amplitude range. The gain adaptability of this channel was [value missing]. The gain coefficient of the height channel in semi-autonomous mode is 0.7, and the physical input range is... The standard control amplitude range is During the perception blind spot, the amplitudes of the six commands issued by the pilot to the altitude channel were +3m / s, +2m / s, +1m / s, 0m / s, -1m / s, and -2m / s, all falling within the standard amplitude range. Therefore, the gain adaptation of this channel was [value missing]. Effective channel gain fit .

[0070] Step 5: Extract post-perceptual cognitive adaptation features; In this embodiment of the application, in order to evaluate the pilot's cognitive adaptation process to the new mode after perceiving a mode change, after the mode change perception delay ends (i.e. (Then), the control command features are continuously extracted using a preset short-interval sliding window. Based on step 4, the following sub-steps are also included: Step 501: Extract the control command feature vector for the control command sequence within each sliding window. This includes the instruction mean, instruction variance, and instruction rate of change for each effective channel.

[0071] Step 502, Obtain the current effective mode A behavioral reference baseline under normal operating conditions, which includes the mean vector of command characteristics. Covariance Matrix Calculate the Mahalanobis distance between the feature vector of the control command for each sliding window and the behavioral reference baseline: in, For a moment The Mahalanobis distance corresponding to the sliding window, This is the feature vector of the control commands for the sliding window. and These are the mean vector and covariance matrix of the behavioral reference baseline, respectively. Covariance matrix The inverse matrix, Indicates transpose. Obtains the Mahalanobis distance variation sequence. .

[0072] Furthermore, the above behavior refers to the mean vector of the baseline. Covariance Matrix Based on historical flight data, the pilot is in the current actual effective mode. The control command feature vector sample set is obtained by statistical analysis of the control command after the stable control phase (i.e., the period after mode switching when the control behavior tends to be stable). This reflects the current actual effective mode. Distribution characteristics of normal pilot control behavior.

[0073] Step 503: Fit an exponential decay curve to the Mahalanobis distance variation sequence. ,in The initial value of the Mahalanobis distance at the sensing moment. The attenuation coefficient is... To help pilots sense when a mode change occurs. Represents an exponential function. The decay coefficient is... As the post-perception mode adaptation rate Post-perception mode adaptation rate A higher value indicates that the pilot adapts to the new mode more quickly.

[0074] Furthermore, the above-mentioned exponential decay curve was fitted using a nonlinear least squares method, with the Mahalanobis distance variation sequence as the basis. The objective is to minimize the sum of squared residuals between the measured values ​​and the predicted values ​​of the exponential decay curve at each time point, and to solve for the parameters. and .

[0075] Step 504: Detect a preset number of consecutive values ​​in the Mahalanobis distance change sequence. The segment in which the Mahalanobis distance does not show a decreasing trend within a sliding window is marked as an adaptive stagnation event. And record adaptation stagnation events. The start and end times.

[0076] The data processing device processed data from 10:23:51.240 (sensing time). Starting with a 2-second wide sliding window, control command features are continuously extracted. Each sliding window contains 4 control command samples. For each sliding window, the mean, variance, and rate of change of commands for the heading, altitude, aileron, and elevator channels are extracted to form a 12-dimensional control command feature vector. .

[0077] The data processing unit extracts characteristic samples of pilot control commands during the stable control phase in semi-autonomous mode from the historical flight database and statistically obtains the mean vector of the behavioral reference baseline. Covariance Matrix For the 12 sliding windows from 10:23:51.240 to 10:24:15.240, the Mahalanobis distance between the control command feature vector and the behavior reference baseline is calculated.

[0078] The calculation results show the Mahalanobis distance for the first sliding window (10:23:51.240-10:23:53.240) corresponding to the perception time. The Mahalanobis distance of the second sliding window (10:23:53.240-10:23:55.240) Mahalanobis distance of the third sliding window Mahalanobis distance of the fourth sliding window Subsequently, the Mahalanobis distance of each sliding window gradually decreased to about 1.5 and then stabilized.

[0079] Table 8. Sequence of Mahalanobis distance changes after perception (partial data): The data processing unit fits the Mahalanobis distance variation sequence to an exponential decay curve. A nonlinear least squares method is used to obtain the fitting parameters. attenuation coefficient The attenuation coefficient is used as the post-sensing mode adaptation rate. .

[0080] Data processing device settings The study examined segments in the Mahalanobis distance variation sequence where the Mahalanobis distance did not show a decreasing trend within three consecutive sliding windows. The analysis results showed that during windows 8 to 10 (10:24:05.240-10:24:11.240), the Mahalanobis distances were 1.52, 1.54, and 1.53, respectively, showing no significant decrease and were marked as adaptation stagnation events. The start and end times are [10:24:05.240, 10:24:11.240].

[0081] Step 6: Calculate the cross-dimensional coupling risk index; In this embodiment of the application, in order to quantify the interactive effect between perceptual delay and cognitive adaptation, as well as the dynamic correlation between cross-screen attention fragmentation and key information omission, the following sub-steps are included in addition to steps 3 to 5: Step 601: Calculate the delay-adaptation coupling risk index. Before calculation, assess the perceived delay of mode change. Pattern recognition bias and post-perception mode adaptation rate Normalization based on range was applied to eliminate the influence of differences in the dimensions and numerical ranges of the variables on the calculation results. The corresponding normalized quantities are denoted as follows: , and The product of the normalized pattern change perception delay and the pattern recognition bias is quantified as the cumulative maneuvering inertia intensity during the perception blind spot period: Furthermore, the aforementioned cumulative control inertia strength The physical meaning is that, This reflects the duration during which pilots operate using the old mode of control during periods of visual blind spot. This reflects the difference in control gain between the old and new modes. The product of the two modes represents the cumulative control inertia intensity accumulated by the pilot due to using the old mode during the perception blind spot period. .

[0082] Accumulated control inertia strength With normalized post-perceptual mode adaptation rate The ratio is used as an indicator of delay-adaptive coupling risk: Among them, cumulative control inertia strength It reflects the strength of the control inertia accumulated by the pilot when operating in the old mode during the perception blind spot period. This reflects the rate at which a pilot corrects their control strategy after perceiving a mode change. Delay-adaptation coupling risk indicator. A higher value indicates that the accumulated control inertia due to perception delay is more relevant to the pilot's adaptability, and the greater the risk of operational errors.

[0083] Step 602: Calculate the correlation index between attention fragmentation and information omission. Calculate the cross-screen transition frequency using a sliding time window. Time-varying sequence and cumulative number of unresponsive high-importance information Calculate the cross-correlation coefficient between two time-varying series. The cross-correlation coefficient represents the dynamic correlation strength between changes in the degree of cross-screen attention fragmentation and changes in the amount of key information omitted.

[0084] Data processing device delay in sensing mode change Seconds, pattern recognition bias and post-perception mode adaptation rate Perform range-based normalization. Set the ranges for each variable in the historical data statistics as follows: scope Second, scope , scope .

[0085] Normalized calculation: Calculate the cumulative maneuvering inertia intensity: Computational latency-adaptive coupling risk index: The data processing device uses a 10-second sliding time window to count the cross-screen transfer frequency between 10:23:30 and 10:24:30. The cumulative number of times high-importance information was not responded to The time-varying sequence of the data. Within six time windows, the cross-screen transition frequency sequence is {0.28, 0.35, 0.42, 0.38, 0.31, 0.25}, and the cumulative number of unresponsive events for high-importance information is {0, 0, 1, 1, 1, 1}. Calculate the Pearson cross-correlation coefficient between the two sequences. .

[0086] Step 7: Perform multi-dimensional feature fusion and temporal prediction; Before concatenating the multidimensional fused feature vectors, each feature component is standardized using Z-scores to eliminate the impact of differences in units and numerical ranges between different features on model training. This standardization process reduces the perceived latency of mode changes. Pattern recognition bias The hierarchical graph structure features are concatenated to form a multidimensional fused feature vector. The multidimensional fused feature vectors from multiple consecutive time windows are arranged in chronological order to form a time-series matrix. ,in Number of time windows The first To the The multidimensional fused feature vector corresponding to each time window Indicates transpose. Transpose the time series matrix. The input is fed into a sequence prediction model based on a gated recurrent unit (GRU) network, and the output is the probability value of an operational error occurring after the current mode switching event. and the predicted window of error outbreak ,in The moment when the error began. This represents the width of the time window.

[0087] Furthermore, the input layer received timing matrix of the sequence prediction model based on the gated recurrent unit (GRU) network. The multidimensional fusion feature vectors are input sequentially according to time steps. The output layer contains two branches: the first branch outputs the operation error probability value after passing through the Sigmoid activation function. The second branch outputs the predicted start time of the error outbreak time window via a linear mapping layer. and time window width The sequence prediction model based on a gated recurrent unit (GRU) network selectively retains and forgets the hidden states of historical time steps through update and reset gates, capturing the evolution trend of multi-dimensional fused features over time. During training, labeled historical flight data is used, with binary labels indicating whether an operational error occurred and the corresponding error timestamps serving as supervision signals. The probabilistic prediction branch employs a binary cross-entropy loss function, while the temporal prediction branch uses a mean squared error loss function. The two loss functions are weighted and summed, and the parameters are updated using the Adam optimization algorithm.

[0088] In this embodiment of the application, in order to include more comprehensive information in the multidimensional fused feature vector, in addition to mode change perception delay, when concatenating the multidimensional fused feature vector, Pattern recognition bias In addition to the hierarchical graph structure features, it also splices the frequency of invalid channel command attempts. Effective channel gain adaptation Post-sensory pattern adaptation rate Adapting to Stagnation Event Marking Delay-adaptive coupling risk indicators Attention fragmentation-information omission correlation index Attention-Information Mismatch Index This enables sequence prediction models based on gated recurrent unit (GRU) networks to comprehensively utilize cross-screen attention features, pattern recognition bias features, and delay-adaptation coupling features to predict operational errors.

[0089] Furthermore, the aforementioned feature components have already experienced a delay in sensing mode change in step 601. Pattern recognition bias and post-perception mode adaptation rate After completing the range-based normalization process, the remaining feature components are uniformly standardized using Z-score before concatenation, ensuring that all concatenated features are included in the multi-dimensional fusion feature vector. All feature components are within a comparable numerical range to ensure the stability of the training process of the sequence prediction model based on the Gated Recurrent Unit (GRU) network.

[0090] The data processing unit performs Z-score normalization on each extracted feature component. This includes the cross-screen transition frequency in the hierarchical graph structure features. According to the average value of historical data and standard deviation The standardized value is After similar processing of other feature components, they are concatenated to form a 13-dimensional multidimensional fused feature vector. .

[0091] The data processing device arranges the multidimensional fusion feature vectors corresponding to the six time windows divided into 10-second intervals between 10:23:30 and 10:24:30 in chronological order to form a time series matrix. , dimension The time series matrix The input is fed into a pre-trained sequence prediction model based on a gated recurrent unit (GRU) network.

[0092] The model output shows the probability value of operational errors. The predicted start time of the error outbreak window =10:24:05, Time window width The prediction error outbreak time window is [10:24:05, 10:24:13].

[0093] Step 8: Generate early warning signals and adaptive auxiliary information; When the probability of operational error is Exceeding the preset threshold At that time, a real-time early warning signal is generated.

[0094] It should be noted that the specific content of the above warning signals is generated differently based on the pilot's current cognitive stage, including the following two situations: In the first scenario, if there is a delay in the perception of mode change... If the process is still ongoing (i.e., the pilot is not yet looking at the mode status indicator), a mandatory mode change notification will pop up on the screen the pilot is currently looking at. This mandatory mode change notification contains the currently active mode identifier. and the current actual effective mode List of effective control channels below.

[0095] The second scenario is if the perception delay has ended but the adaptation stagnation event has occurred. If detected (i.e., the pilot has perceived the mode change but failed to adapt effectively to the new mode), the currently effective mode will be output. The instructions provide key control points and recommended control amplitude ranges for reference.

[0096] In this embodiment of the application, while generating the warning signal, the high-risk screen number and the summary of key information accumulated on the high-risk screen that has not been noticed by the pilot are also output simultaneously, so as to guide the pilot to pay attention to the important information that has been missed while dealing with the mode adaptation problem.

[0097] Data processing device sets preset threshold Due to the probability value of operational errors If the threshold is exceeded, the system generates a real-time warning signal. At this time, it is 10:24:03. The pilot had already sensed the mode change at 10:23:51.240, and the perception delay had ended, but an adaptation stagnation event was detected between 10:24:05.240 and 10:24:11.240. This falls under the second category.

[0098] The data processing device generates a warning signal: Key control points are highlighted in the mode status indicator area of ​​screen A: "Currently in semi-autonomous mode, aileron and elevator channels are activated. Recommended control amplitude: aileron ±8°, elevator ±9°." Simultaneously, an auxiliary information bar pops up at the bottom of screen A, displaying: "Screen A contains unattended high-importance information: Mode switching notification."

[0099] The data processing unit also outputs a high-risk screen number: Screen A. This screen contains a summary of key information that was not being paid attention to by the pilots, including: Event INFO-U-1124 (Mode identifier change, high importance level, gaze response delay 6.240 seconds).

[0100] This implementation separates the pilot's actual perception of the mode change from the mode switching event time, thus defining the time boundary of the perception blind spot. This overcomes the erroneous assumption in existing methods that the pilot is immediately aware of the mode change at the moment of switching. Within the perception blind spot, multi-mode parallel forward simulation is used to simulate control commands according to the control channel-flight response transfer function of each candidate mode, and the residuals are compared to deterministically infer the pilot's implicit cognitive mode. This allows for the identification of deterministic cognitive misalignment in cases where the pilot continues to operate in the old mode without perceiving the mode change, rather than merely estimating probabilistic cognitive adaptation difficulties.

[0101] This implementation generates a multi-layered directed attention transfer graph that distinguishes between in-screen and cross-screen transfers, enabling the attention analysis dimension to reflect the additional cognitive overhead and attention fragmentation caused by cross-screen switching in multi-screen heterogeneous control interfaces. This provides a structured representation of attention allocation for gaze response delay analysis and attention-information demand mismatch index evaluation.

[0102] This implementation uses a delay-adaptive coupling risk index to measure the accumulated control inertia intensity during the perception blind spot period. Post-perception mode adaptation rate The correlation was established to quantify the cumulative impact of perceptual delay on subsequent cognitive adaptation. Simultaneously, the dynamic correlation between the degree of cross-screen attention fragmentation and the amount of key information omission was quantified using the attention fragmentation-information omission correlation index, enabling operational error prediction to capture the coupling relationship between attention allocation bias and information omission.

[0103] Therefore, this implementation integrates multi-dimensional information such as cross-screen attention features, pattern recognition bias features, and delay-adaptation coupling features, and uses a time-series matrix. The input is fed into a sequence prediction model based on a gated recurrent unit (GRU) network, which overcomes the limitation of single-dimensional feature analysis in failing to capture the coupling error relationship between perceptual omissions and cognitive confusion, and realizes real-time prediction of operational errors in flight control mode switching scenarios.

[0104] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A real-time prediction method for unmanned aerial vehicle (UAV) pilot operational errors based on multi-dimensional feature analysis, characterized in that, Includes the following steps: Acquire the flight control system's mode switching event logs, the control permission mapping table corresponding to each flight control mode, the physical layout information and area of ​​interest definition information of the multi-screen control interface, and the pilot's eye tracking data. Based on the eye-tracking data and physical layout information, a gaze sequence is generated. A multi-layered attention transfer directed graph containing in-screen transfer edges and cross-screen transfer edges is generated according to the gaze sequence, and the hierarchical graph structure features are calculated. The pilot first gazes at the region of interest where the mode status indicator is located after the mode switch timestamp is retrieved from the gaze sequence. The time difference between the two timestamps is calculated as the mode change perception delay, and the time interval between them is defined as the perception blind zone. During the perception blind zone, the control command sequence is forward simulated according to the control channel-flight response transfer function of each candidate flight control mode. The candidate mode with the smallest cumulative norm of residuals between the predicted flight state and the actual flight state is selected as the implicit cognitive mode. The Euclidean distance between the control gain vector of the implicit cognitive mode and the actual effective mode is calculated as the mode cognitive bias. After the perception blind zone ends, the control command feature vector is extracted using a sliding window, and the Mahalanobis distance change sequence between the pilot and the behavior reference baseline is calculated. An exponential decay curve is fitted to obtain the post-perception mode adaptation rate. The delay-adaptation coupling risk index is calculated based on the mode change perception delay, the mode cognitive bias, and the post-perception mode adaptation rate. The multi-dimensional fusion feature vector containing the hierarchical graph structure features, the mode change perception delay, the mode cognitive bias, and the delay-adaptation coupling risk index is arranged in chronological order to form a time series matrix. This matrix is ​​input into the sequence prediction model to output the operation error probability value and the error outbreak time window.

2. The real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis according to claim 1, characterized in that, The multi-layered attention transfer directed graph includes an in-screen transfer layer and a cross-screen transfer layer. Each directed edge in the in-screen transfer layer records the number of transfers from the source region to the target region and the average transfer time. Each directed edge in the cross-screen transfer layer records the number of transfers and the average transfer time, and is weighted by a cross-screen distance, which is the Euclidean distance between the center position coordinate vectors of the source screen and the target screen.

3. The real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis according to claim 1, characterized in that, The calculation of the layered graph structure features includes: calculating the ratio of the number of cross-screen transfer edges to the total number of transfer edges to obtain the cross-screen transfer frequency; calculating the ratio of the total time cost of all cross-screen transfers to the total time cost of all attention transfers to obtain the cross-screen transfer time cost ratio; calculating the Gini coefficient of the pilot's attention dwell time on each screen, wherein the Gini coefficient is calculated by: arranging the attention dwell time of each screen in ascending order, calculating the sum of the products of each screen's dwell time and the difference between twice the screen's sorting number and the total number of screens minus one, and then dividing by the product of the total number of screens and the total dwell time of all screens; extracting transfer path subsequences of a preset length from the cross-screen transfer edge sequence, calculating the frequency of each subsequence, and calculating the ratio of the sum of the frequencies of the subsequences with the highest frequency of occurrence to the sum of the frequencies of all subsequences to obtain the concentration of common patterns of cross-screen transfer paths.

4. The real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis according to claim 1, characterized in that, It also includes information response delay analysis based on information update event logs and the gaze sequence: for each information update event, the timestamp of the pilot's gaze first reaching the region of interest where the information update event is located after the timestamp of the information update event is retrieved, and the time difference between the two is calculated as the gaze response delay time of the information update event; the gaze response delay time is weighted according to the change importance level of each information update event, and the weighted mean is calculated to obtain the weighted mean of the key information response delay; the number of high-importance information change events that have not been gazed upon by the pilot within a preset time limit is accumulated; The Jensen-Shannon divergence between the attention distribution (composed of the proportion of pilot attention dwell time on each screen) and the information demand distribution (composed of the proportion of information update frequency on each screen) is calculated as an indicator of attention-information demand mismatch.

5. The real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis according to claim 1, characterized in that, During the perception blind zone, the following auxiliary indicators are also extracted: Invalid channel sets are identified based on the control authority mapping table of the current effective mode; the ratio of the number of attempts by the pilot to issue commands to invalid channels to the total number of commands during the perception blind zone is calculated to obtain the invalid channel command attempt frequency; for effective channels in the current effective mode, the standard control amplitude range of each effective channel is determined by the product of the gain coefficient of the channel in the current effective mode and the physical input range allowed by the flight control system; the proportion of commands issued by the pilot to the channel during the perception blind zone that fall within the standard control amplitude range is calculated as the gain fit of the channel; the average gain fit of all effective channels is taken to obtain the effective channel gain fit.

6. The real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis according to claim 1, characterized in that, The process of extracting control command feature vectors using a sliding window and calculating the Mahalanobis distance change sequence between the control command and the behavioral reference baseline, and fitting an exponential decay curve to obtain the post-perception mode adaptation rate, includes: extracting control command feature vectors, including the mean, variance, and rate of change of each effective channel, for each control command sequence within a sliding window; obtaining the mean vector and covariance matrix of the behavioral reference baseline under normal control conditions for the current actual effective mode, wherein the behavioral reference baseline is statistically obtained from a set of control command feature vector samples obtained by the pilot in the current actual effective mode during the stable control phase in historical flight data; calculating the Mahalanobis distance between the control command feature vector of each sliding window and the behavioral reference baseline, wherein the Mahalanobis distance is obtained by performing a quadratic operation on the difference between the control command feature vector and the mean vector, the inverse of the covariance matrix, and taking the square root, thereby obtaining the Mahalanobis distance change sequence; fitting an exponential decay curve to the Mahalanobis distance change sequence using a nonlinear least squares method, and using the decay coefficient as the post-perception mode adaptation rate; detecting segments in the Mahalanobis distance change sequence where the Mahalanobis distance does not show a decreasing trend within a consecutive preset number of sliding windows, marking them as adaptation stagnation events, and recording the start and end times.

7. The real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis according to claim 1, characterized in that, The calculation of the delay-adaptation coupling risk index includes: applying mean normalization based on the range to the pattern change perception delay, the pattern cognition deviation, and the post-perception pattern adaptation rate; using the product of the normalized pattern change perception delay and the normalized pattern cognition deviation as the cumulative manipulation inertia strength; using the ratio of the cumulative manipulation inertia strength to the normalized post-perception pattern adaptation rate as the delay-adaptation coupling risk index; and further including calculating the attention fragmentation-information omission correlation index: calculating the time change sequence of cross-screen transfer frequency and the time change sequence of the cumulative number of unresponsive high-importance information using a sliding time window, and calculating the cross-correlation coefficient between the two time change sequences as the attention fragmentation-information omission correlation index.

8. The real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis according to claim 7, characterized in that, The multidimensional fusion feature vector also includes the invalid channel instruction attempt frequency, effective channel gain adaptation, post-perception mode adaptation rate, adaptation stagnation event marker, attention fragmentation-information omission correlation index, and attention-information demand mismatch index; the sequence prediction model is a sequence prediction model based on a gated recurrent unit network, and the output layer contains two branches: the first branch outputs the operation error probability value through a sigmoid activation function, and the second branch outputs the predicted value of the start time of the error outbreak time window and the width of the time window through a linear mapping layer; during training, the probability prediction branch adopts the binary cross-entropy loss function, and the time prediction branch adopts the mean squared error loss function. The parameters are updated after the two loss functions are weighted and summed.

9. The real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis according to claim 1, characterized in that, When the probability value of the operational error exceeds a preset threshold, a real-time warning signal is generated. The warning signal is generated differently according to the pilot's current cognitive stage. If the mode change perception delay is still ongoing, a mode change forced notification will pop up on the screen that the pilot is currently looking at. The mode change forced notification includes the currently effective mode identifier and a list of effective control channels under the current effective mode. If the perception blind spot period has ended but the adaptation stagnation event is detected, output the key control points of the current effective mode and the recommended control amplitude range reference. Simultaneously output the high-risk screen number and a summary of key information accumulated on the high-risk screen that has not been noticed by the pilot.

10. A real-time prediction system for unmanned aerial vehicle (UAV) pilot operational errors based on multi-dimensional feature analysis, used to execute the real-time prediction method for UAV pilot operational errors based on multi-dimensional feature analysis as described in any one of claims 1 to 9, characterized in that, include: The multi-source data acquisition module is used to acquire the flight control system's mode switching event logs, the control permission mapping table corresponding to each flight control mode, the physical layout information and region of interest definition information of the multi-screen control interface, and the pilot's eye tracking data. The attention transfer graph generation module is used to generate a gaze sequence based on the eye tracking data and the physical layout information, generate a multi-layered attention transfer directed graph containing in-screen transfer edges and cross-screen transfer edges according to the gaze sequence, and calculate the layered graph structure features. The perception delay calculation module is used to retrieve the timestamp of the pilot's first gaze at the region of interest of the mode status indicator after the mode switching timestamp from the gaze sequence, calculate the mode change perception delay, and determine the perception blind zone period; the cognitive mode inference module is used to perform forward simulation of the control command sequence according to the control channel-flight response transfer function of each candidate flight control mode during the perception blind zone period, select the candidate mode with the smallest residual cumulative norm as the implicit cognitive mode, and calculate the mode cognitive bias degree; the cognitive adaptation feature extraction module is used to extract the control command feature vector with a sliding window after the perception blind zone period ends, calculate the Mahalanobis distance change sequence between the pilot and the behavioral reference baseline, and fit an exponential decay curve to obtain the post-perception mode adaptation rate; The coupling risk calculation module is used to calculate the delay-adaptation coupling risk index based on the pattern change perception delay, the pattern cognition deviation degree, and the post-perception pattern adaptation rate. The fusion prediction module is used to assemble a time series matrix by combining the multi-dimensional fusion feature vector containing the hierarchical graph structure features, the pattern change perception delay, the pattern cognition bias, and the delay-adaptation coupling risk index in chronological order. The input sequence prediction model outputs the operation error probability value and the error outbreak time window.

Images