Method for recognizing changes in pilot state in flight simulation experiments

Abstract

The application discloses a flight simulation experiment pilot state change identification method. The method comprises the following steps: acquiring eye movement data of a pilot in a flight simulation experiment; constructing an eye movement AOI sequence according to the eye movement data; using a time sliding window on the eye movement AOI sequence; for the eye movement AOI sequence in each window, constructing a visual scanning mode model based on CTMC; for the visual scanning mode model of each window, calculating a Pearson correlation coefficient between two windows, and forming a correlation matrix of the Pearson correlation coefficients between all windows; finding a window with the lowest correlation with the windows before the alarm in the window after the alarm, and determining the window as a conversion time point; comparing the conversion time point with a window of a marked alarm time point to obtain a time difference between the conversion time point and the window of the marked alarm time point, and determining the time difference as a pilot state change time difference. The pilot state change identification method can accurately determine a time point of a pilot state change.

Description

Technical Field

[0001] This invention relates to a pilot state analysis technology, and more particularly to a method for identifying changes in the state of pilots in flight simulation experiments. Background Technology

[0002] Most aviation accidents are caused by human factors. A crucial aspect of aviation human factors research is the pilot's information processing. During flight, pilots acquire over 80% of their information through the visual channel. Therefore, the pilot's visual scanning patterns are a vital component of cockpit human factors research.

[0003] Identifying the timing of pilot visual scanning mode transitions can reflect changes in the pilot's information processing and state, providing a reference for detecting whether pilots make mistakes during alarm mission execution, assessing pilot performance levels, and thus reducing the possibility of human error.

[0004] The current mainstream method for studying pilots' visual scanning patterns involves analyzing various eye movement parameters, including fixation duration, fixation frequency, saccade rate, saccade count, scan path, blink frequency, and entropy value. These eye movement parameters allow for the study of differences and variations in pilots' visual scanning patterns across different mission time periods, as well as the correlation between visual scanning patterns and flight missions.

[0005] The current problem is:

[0006] The eye-tracking indicators mentioned above can only describe a pilot's visual scanning pattern from a specific perspective and cannot comprehensively reflect the characteristics of the pilot's visual scanning pattern. Their ability to distinguish the pilot's visual scanning pattern under different flight missions is limited. Furthermore, existing studies and analyses of various eye-tracking indicators do not consider temporal sequence. However, most applications of eye tracking in aviation focus on independent flight mission segments. Actual flight is a complete, continuous process containing multiple flight missions. Therefore, analyzing the pilot's visual scanning pattern in terms of temporal sequence is crucial. Summary of the Invention

[0007] The purpose of this invention is to provide a method for identifying changes in the pilot's state in flight simulation experiments, which can accurately determine the time point of the pilot's state change.

[0008] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0009] A method for recognizing changes in pilot state in a flight simulation experiment includes:

[0010] Acquire pilots' eye-tracking data during flight simulation experiments and mark alarm timestamps;

[0011] Construct pilot eye-tracking AOI sequences based on eye-tracking data;

[0012] A time-sliding window was used on the pilot's eye-tracking AOI sequence;

[0013] For each eye-tracking AOI sequence within a window, a visual scanning pattern model is constructed based on CTMC;

[0014] For each window's visual scanning pattern model, calculate the Pearson correlation coefficient between each pair of windows, and form a correlation matrix by combining the Pearson correlation coefficients between all pairs of windows.

[0015] Based on the pre-marked alarm time points and the Pearson correlation coefficient between all pairs of windows, find the window with the lowest correlation with all windows before the alarm in the windows after the alarm, and determine the window as the transition time point.

[0016] The time difference between the transition time point and the marked alarm time point is determined by comparison, and this time difference is identified as the pilot state change time difference.

[0017] Furthermore, the specific method for acquiring the pilot's eye-tracking data during the flight simulation experiment includes:

[0018] The collected pilot gaze points are mapped onto the cockpit foreground map of the flight simulation experiment, thereby generating two-dimensional coordinates of each gaze point in the cockpit foreground map;

[0019] For each gaze point on the cockpit foreground map, the cockpit foreground map associated with that gaze point will be... Figure 2 Eye movement data is composed of a combination of dimensional coordinates, timestamps, and eye movement event types. All eye movement data are aggregated to form the aforementioned eye movement data.

[0020] Furthermore, the specific method for mapping the collected pilot's gaze point onto the cockpit foreground image of the flight simulation experiment includes:

[0021] Import the cockpit foreground image from the flight simulation experiment into the eye-tracking software that comes with the eye tracker;

[0022] In the eye-tracking software that comes with the eye tracker, the pilot's gaze points collected by the eye tracker are mapped onto the cockpit foreground image of the flight simulation experiment, generating two-dimensional coordinates of each gaze point in the cockpit foreground image.

[0023] Furthermore, the specific method for constructing the pilot's eye-tracking AOI sequence based on eye-tracking data includes:

[0024] Let the pilot's eye-tracking AOI sequence be X = {x1, x2, x3, ..., x...} n}, where n is the timestamp length, x k Let x be the AOI where the gaze point of the k-th sampling point falls. k ∈[0,11],x k ∈Z;

[0025] For each sampling point, it is first determined whether it is a fixation point based on the eye movement type in the eye movement data. If not, the sampling point is marked as 0 in the eye movement AOI sequence. If it is a fixation point, the two-dimensional coordinates of the fixation point on the cockpit foreground map are used to determine which AOI the fixation point falls in, and the sampling point is set as the element corresponding to the AOI in the eye movement AOI sequence.

[0026] Furthermore, the method for constructing a visual scanning pattern model based on CTMC for each eye-tracking AOI sequence within a window includes:

[0027] Calculate the conditional transition matrix between each pair of AOIs in each eye-tracking AOI sequence;

[0028] For each element p in the corresponding conditional transition matrix ij (t) Set two parameters, namely the delay time t d and the upward trend λ;

[0029] For each element p in the corresponding conditional transition matrix ij (t), calculate the delay time t d and the upward trend λ;

[0030] The delay time t of all elements in the conditional transition matrix. d Combined with the upward trend λ in a fixed order and Then combine the two The parameters are configured as a group to define the conditional transition matrix, thus completing the establishment of the visual scanning mode model within each window.

[0031] Furthermore, for the correlation matrix, the correlation between the visual scanning pattern within each window and the visual scanning pattern within the window before the alarm is calculated, and the average value is taken to obtain the mean correlation value V for each window. i ;

[0032] The formula for calculating the mean correlation is:

[0033]

[0034] In the formula, ρ ijThe correlation between the visual scanning modes of the i-th window and the j-th window is given by wb, where wb is the number of windows before the alarm.

[0035] Find the mean correlation V within the window after the alarm. i The lowest window i is the transition time point of the scanning mode identified by the visual scanning mode model.

[0036] Furthermore, the width of the time sliding window is set to 1 minute, and the sliding step size of the time sliding window is set to 2 seconds.

[0037] Furthermore, the AOI includes: airspeed, attitude, altitude, main flight display, navigation display, backup board, electronic centralized monitoring system, control display components, flight control components, and window view.

[0038] In the pilot state change recognition method of the present invention, the pilot's eye movement data is acquired, and the pilot's eye movement AOI sequence is constructed based on the eye movement data. A time sliding window is used on the eye movement AOI sequence. For the eye movement AOI sequence within each window, a visual scanning pattern model is constructed based on CTMC, and then the time point of the pilot state change is determined.

[0039] In the pilot state change recognition method of the present invention, the pilot's eye movement process is regarded as a continuous-time Markov process. In this way, compared with traditional eye movement indicators, it can more comprehensively reflect the characteristics of the pilot's visual scanning pattern from various angles, better distinguish different visual scanning patterns, and more accurately determine the time point of the pilot's state change. Attached Figure Description

[0040] Figure 1 This is a flowchart of the pilot state change recognition method based on the present invention;

[0041] Figure 2 This is a flowchart of constructing a visual scanning pattern model based on CTMC;

[0042] Figure 3 This is a schematic diagram of fitting the curve of each element in the conditional transition matrix. Detailed Implementation

[0043] The present invention will be further illustrated below with specific embodiments:

[0044] This embodiment provides a method for recognizing changes in the pilot's state in flight simulation experiments.

[0045] The flight simulation experiment involved in this embodiment refers to: a pilot conducting a control experiment on a flight simulator. At the start of each mission, the simulator unfreezes and sets the aircraft's initial position, altitude, and airspeed. The pilot then begins the approach mission. When a malfunction occurs, an audible alarm is triggered and corresponding warning lights illuminate. The pilot then follows the checklist procedure to handle the malfunction. After handling the malfunction, the pilot continues the approach mission until the simulator freezes. During the experiment, simulated alarm signals are issued to test the pilot's response to alarms and their ability to handle malfunctions.

[0046] During flight simulation experiments, pilots wear head-mounted eye trackers to scan and collect their eye movement data in real time. These eye trackers are existing technology devices capable of calculating the center position of the pilot's pupils and corneal reflex points to obtain data on eye movements.

[0047] Those skilled in the art will understand that the “CTMC” mentioned in this embodiment is the existing continuous-time Markov chain theory. In this embodiment, the pilot’s eye movement process is regarded as a continuous-time Markov process.

[0048] See Figure 1 The pilot status change recognition method of this embodiment includes the following steps S1 to S7.

[0049] S1 acquires the pilot's eye movement data during the flight simulation experiment and marks the alarm time points.

[0050] The eye-tracking data includes a series of eye-tracking behavior data, each of which includes the cockpit foreground. Figure 2 The dimensional coordinates, timestamps, and eye-tracking event types can be obtained by exporting data from the eye tracker.

[0051] Specifically

[0052] The phrase "acquiring eye-tracking data of pilots in flight simulation experiments" specifically includes:

[0053] S11 maps the pilot's gaze points collected by the eye tracker onto the cockpit foreground image of the flight simulation experiment (that is, the cockpit foreground image of the flight simulator), thereby generating two-dimensional coordinates of each gaze point in the cockpit foreground image.

[0054] In this embodiment, step S11 specifically includes the following steps:

[0055] S111, import the cockpit foreground image from the flight simulation experiment into the eye-tracking software that comes with the eye tracker.

[0056] S112, in the eye-tracking software accompanying the eye tracker, the pilot's gaze points collected by the eye tracker are mapped onto the cockpit foreground image of the flight simulation experiment, thereby generating two-dimensional coordinates of each gaze point in the cockpit foreground image.

[0057] S12, corresponding to each gaze point on the cockpit foreground map, will display the cockpit foreground map associated with that gaze point. Figure 2 Eye movement data is composed of a combination of dimensional coordinates, timestamps, and eye movement event types. All eye movement data are aggregated to form the aforementioned eye movement data.

[0058] In this embodiment, eye-tracking data can be exported from the eye tracker, and the exported eye-tracking data is shown in Table 1.

[0059] The final generated eye-tracking data is shown in Table 1, including: timestamp, X coordinate of the gaze point mapped on the cockpit foreground image, Y coordinate of the gaze point mapped on the cockpit foreground image, and the type of the mapped eye-tracking event.

[0060] The eye-tracking parameters exported in Table 1

[0061]

[0062] It should be noted that the eye movement data was collected by an eye tracker worn by the pilot.

[0063] S2, construct the pilot's eye-tracking AOI sequence based on the eye-tracking data obtained in step S1.

[0064] Specifically, it includes,

[0065] S21. Let the pilot's eye-tracking AOI sequence be X = {x1, x2, x3, ..., x...} n}, where n is the timestamp length, x k Let x be the AOI where the gaze point of the k-th sampling point falls. k ∈[0,11],x k ∈Z.

[0066] S22. For each sampling point, first determine whether it is a fixation point based on the eye movement type in the eye movement data. If not, the sampling point is marked as 0 in the eye movement AOI sequence. If it is a fixation point, determine which AOI the fixation point falls in by using the two-dimensional coordinates of the fixation point on the cockpit foreground image. The sampling point is then set as the element corresponding to the AOI in the eye movement AOI sequence.

[0067] Those skilled in the art will understand that the sampling points mentioned here correspond to the timestamps mentioned earlier.

[0068] It should be noted that the AOI mentioned in this embodiment includes: airspeed, attitude, altitude, primary flight display (PFD, the PFD area other than airspeed, attitude, and altitude), navigation display (MCP), standby board, electronic centralized surveillance system (ECAM), control display unit (CDU), flight control unit (FCP), outside the window (OTW), and 11 other regions of interest, which are represented by elements 1-11 in the sequence.

[0069] The correspondence between the various regions of interest and elements 1-11 in the cockpit is shown in Table 2.

[0070] Table 2. Pilot's Eye Area of ​​Interest (AOI) and Element Correspondence Table

[0071] panel airspeed attitude altitude mcp standby ecam cud fcp otw pfd others element 1 2 3 4 5 6 7 8 9 10 11

[0072] S3, using a time-sliding window on the pilot's eye-tracking AOI sequence obtained in step S2.

[0073] In this embodiment, the width of the time sliding window is set to 1 minute, and the sliding step size of the time sliding window is set to 2 seconds.

[0074] S4. For the eye-tracking AOI sequence within each window, a visual scanning pattern model is constructed based on CTMC.

[0075] See Figure 2 The specific process is as follows:

[0076] S41, calculate the conditional transition matrix for each "transition between pairs of AOIs in the eye-tracking AOI sequence".

[0077] Specifically

[0078] The conditional transition matrix is ​​defined over a time range of width win. n is the total number of AOIs within the window, and each element p in the conditional transition matrix is... ij The formula for calculating (t) is:

[0079] p ij (t)=N ij (t) / N ij (win)

[0080] In the formula, N ij (t) represents the number of times N moves from AOI i to AOI j within a time interval t after the start time of the win time range. ij(win) represents the total number of transitions from AOI i to AOI j within the time range win. Considering that there will be a certain amount of fixation time on each AOI, the transition time point is selected as the starting time of the transition from the fixation point leaving AOI i to AOI j;

[0081] It should be noted that the conditional transition matrix composed of n AOIs has a total of n... 2 There are 10 elements, each corresponding to a transition between any two AOIs.

[0082] S42, corresponding to each element p in the conditional transition matrix ij (t) Set two parameters, namely the delay time t d And the upward trend λ, the delay time t d The significance of the upward trend λ lies in reflecting "the moment when the transition corresponding to the element in the delayed time conditional transition matrix first appears within the window, and the upward trend represents the overall distribution of the transition corresponding to the element in the conditional transition matrix throughout the entire window time".

[0083] y = -e -λx +c

[0084] Specifically

[0085] Each element p in the conditional transition matrix ij (t) corresponds to a curve, such as Figure 3 The solid line in the diagram represents an example element of the conditional transition matrix, p. ij (t) is at its minimum of 0, after a delay time t. d It then begins to rise, and the upward trend is represented by the parameter λ. After a period of rising, its final value is 1. Therefore, the delay time t is used. d And the upward trend λ is used to represent an element p ij (t), the delay time t of all elements d Combined with the upward trend λ in a fixed order and It can be used to describe the conditional transition matrix p win (t). In this embodiment, This is called the definition parameter set of the conditional transition matrix.

[0086] S43, corresponding to each element p in the conditional transition matrix ij (t), calculate the delay time t d And the upward trend λ.

[0087] Specifically, the calculation process includes:

[0088] S431, the time when the transition corresponding to an element in the conditional transition matrix first appears within the window is denoted as the delay time t. d .like Figure 3 In the example, the time point at which the first non-zero value appears in the solid line portion is the delay time t. d ;

[0089] S432, Apply an exponential function to the curves corresponding to the elements in the conditional transition matrix.

[0090] y = -e -λx +c

[0091] By fitting the data, the upward trend λ is obtained.

[0092] The fitted curve is as follows Figure 3 As shown by the dashed line.

[0093] S44, the delay time t of all elements in the conditional transition matrix. d Combined with the upward trend λ in a fixed order and Then combine the two The parameters are configured as a group for the conditional transition matrix, thereby completing the establishment of the visual scanning pattern model within each window.

[0094] S5, for the visual scanning mode model of each window, uses the formula

[0095]

[0096] We calculate the Pearson correlation coefficient between each pair of windows to quantify the correlation between them.

[0097] The correlation matrix R is constructed by combining the Pearson correlation coefficients between all pairs of windows.

[0098]

[0099] Where m is the total number of windows, and ρ ij Let be the Pearson correlation coefficient between the i-th window and the j-th window.

[0100] It should be noted that the closer the absolute value of the Pearson correlation coefficient is to 1, the greater the correlation between the visual scanning patterns of the two corresponding windows, and the more similar the patterns are; the closer the Pearson correlation coefficient is to 0, the smaller the correlation between the visual scanning patterns of the two corresponding windows, and the greater the difference in patterns.

[0101] S6. Based on the pre-marked alarm time point and the "Pearson correlation coefficient between all windows" obtained in step S5, find the window with the lowest correlation with each window before the alarm (i.e., the absolute value of the Pearson correlation coefficient between the two is the smallest) in the window after the alarm, and thus determine that the pilot's visual scanning mode has changed significantly within the window time, that is, the pilot's state has changed.

[0102] For the correlation matrix, the correlation between the visual scanning pattern within each window and the visual scanning pattern within the window before the alarm is calculated, and the average value is taken to obtain the mean correlation value V for each window. i .

[0103] The formula for calculating the mean correlation is as follows:

[0104]

[0105] In the formula, ρ ij represents the correlation between the visual scanning modes of the i-th window and the j-th window, where wb is the number of windows before the alarm.

[0106] Find the mean correlation V within the window after the alarm. i The lowest window i represents the transition point of the scanning pattern identified by the visual scanning pattern model. This transition point can be considered as the point at which the pilot's state changes.

[0107] S7. By comparing the "transition time point obtained in step S6" with the "window marking the alarm time point", the time difference between the two is obtained. This time difference can be confirmed as the time difference between the transition time point of the visual scanning mode and the alarm time. In layman's terms, this is the pilot's reaction time to the alarm event.

[0108] The identification process is now complete.

[0109] In the pilot state change recognition method of this embodiment, pilot eye-tracking data is acquired, and an eye-tracking AOI sequence is constructed based on the eye-tracking data. A time-sliding window is used on the eye-tracking AOI sequence. For the eye-tracking AOI sequence within each window, a visual scanning pattern model is constructed based on CTMC, and then the time point of pilot state change is determined. That is, in this embodiment, the pilot's eye-tracking process is regarded as a continuous-time Markov process. In this way, compared with traditional eye-tracking indicators, it can more comprehensively reflect the characteristics of the pilot's visual scanning pattern from various angles, better distinguish different visual scanning patterns, and more accurately determine the time point of pilot state change.

[0110] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for recognizing changes in pilot state during flight simulation experiments, characterized in that: include: Acquire pilots' eye-tracking data during flight simulation experiments and mark alarm timestamps; Construct pilot eye-tracking AOI sequences based on eye-tracking data; A time-sliding window was used on the pilot's eye-tracking AOI sequence; For each eye-tracking AOI sequence within a window, a visual scanning pattern model is constructed based on CTMC; The method for constructing a visual scanning pattern model based on CTMC for each eye-tracking AOI sequence within a window includes: Calculate the conditional transition matrix between each pair of AOIs in each eye-tracking AOI sequence; Each element in the corresponding conditional transition matrix Set two parameters, namely the delay time. and upward trend ; Each element in the corresponding conditional transition matrix Calculate the delay time and upward trend ; Delay time of all elements in the conditional transition matrix and upward trend Combining in a fixed order and Then combine the two. The parameter group is configured as a conditional transition matrix to complete the establishment of the visual scanning pattern model within each window; For each window's visual scanning pattern model, calculate the Pearson correlation coefficient between each pair of windows, and form a correlation matrix by combining the Pearson correlation coefficients between all pairs of windows. Based on the pre-marked alarm time points and the Pearson correlation coefficient between all pairs of windows, find the window with the lowest correlation with all windows before the alarm in the windows after the alarm, and determine the window as the transition time point. The time difference between the transition time point and the marked alarm time point is determined by comparison, and this time difference is identified as the pilot state change time difference.

2. The method for recognizing changes in pilot state in flight simulation experiments according to claim 1, characterized in that: The specific methods for acquiring pilot eye-tracking data during flight simulation experiments include: The collected pilot gaze points are mapped onto the cockpit foreground map of the flight simulation experiment, thereby generating two-dimensional coordinates of each gaze point in the cockpit foreground map; For each gaze point on the cockpit foreground map, the two-dimensional coordinates, timestamps, and eye movement event types associated with that gaze point are combined to form eye movement behavior data. All eye movement behavior data are then aggregated to form the aforementioned eye movement data.

3. The method for recognizing changes in pilot state in flight simulation experiments according to claim 2, characterized in that: The method for mapping the pilot's gaze point to the cockpit foreground image in the flight simulation experiment includes: Import the cockpit foreground image from the flight simulation experiment into the eye-tracking software that comes with the eye tracker; In the eye-tracking software that comes with the eye tracker, the pilot's gaze points collected by the eye tracker are mapped onto the cockpit foreground image of the flight simulation experiment, generating two-dimensional coordinates of each gaze point in the cockpit foreground image.

4. The method for recognizing changes in pilot state in flight simulation experiments according to claim 1, characterized in that: The method for constructing the pilot's eye-tracking AOI sequence based on eye-tracking data includes: Let the pilot's eye-tracking AOI sequence be... , The length of the timestamp. For the AOI where the gaze point of the k-th sampling point falls, ; For each sampling point, it is first determined whether it is a fixation point based on the eye movement type in the eye movement data. If not, the sampling point is marked as 0 in the eye movement AOI sequence. If it is a fixation point, the two-dimensional coordinates of the fixation point on the cockpit foreground map are used to determine which AOI the fixation point falls in, and the sampling point is set as the element corresponding to the AOI in the eye movement AOI sequence.

5. The method for recognizing changes in pilot state in flight simulation experiments according to claim 1, characterized in that: For the correlation matrix, the correlation between the visual scanning pattern within each window and the visual scanning pattern within the window before the alarm is calculated, and the average value is taken to obtain the mean correlation value for each window. ; The formula for calculating the mean correlation is: ， In the formula, For the first The visual scanning mode of the first window and the first Correlation of visual scanning patterns of individual windows The number of windows before the alarm; Find the mean correlation within the window after the alarm. Lowest window This refers to the time point at which the scanning pattern is converted, as identified by the visual scanning pattern model.

6. The method for recognizing changes in pilot state in flight simulation experiments according to claim 1, characterized in that: The width of the time sliding window is set to 1 minute, and the sliding step size of the time sliding window is set to 2 seconds.

7. The method for recognizing changes in pilot state in flight simulation experiments according to claim 1, characterized in that: The AOI includes: airspeed, attitude, altitude, main flight display, navigation display, backup panel, electronic centralized monitoring system, control display components, flight control components, and window view.

Images