A system and method for identifying the type of mission performed by a pilot

Abstract

The present invention discloses a system and method for identifying the type of mission a pilot is performing. A system for identifying the type of mission a pilot is performing relates to the field of aviation safety technology, and includes an eye movement feature acquisition module, a sliding window processor module, a mission classification module, a sliding window adaptive module, and a sequence analyzer module. A method for identifying the type of mission a pilot is performing includes S100, collecting eye movement data; S200, obtaining an eye movement feature sequence; S300, predicting the type of mission to be performed; S400, dynamically adjusting the sliding window; S500, identifying the type of mission to be performed. The present invention can accurately analyze the changes in the type of mission a pilot is performing, and accurately describe the time when the pilot enters the type of mission to be performed.

Description

Technical Field

[0001] The present invention relates to the field of aviation safety technology, and in particular to a system and method for identifying the type of mission performed by a pilot. Background Art

[0002] When encountering emergencies, pilots need to perform a quick reference handbook (QRH). Different emergencies correspond to different QRH task processes, so the pilot's visual attention allocation will also change accordingly. Generally, the pilot's main task is to maintain the normal operation of the aircraft. When encountering emergencies, the pilot needs to perform both the original stable flight task and the check task according to the steps of the quick checklist. This means that the pilot's behavior at this time is a superposition of these two behaviors. This superposition causes the pilot's visual attention allocation to show two characteristics in the process of executing the quick checklist. The first is incompleteness. The change process of visual attention allocation will not be completely consistent with the type of checklist executed; the second is interference. For the pilot, the process of obtaining information through the eyes is a disturbed process. These characteristics make the change in visual attention allocation contain rich information that can be used to identify the type of checklist task performed by the pilot, but there are certain challenges in actual identification.

[0003] It is of great significance to describe when the pilot begins to enter the specific mission procedure to deal with emergencies. It provides an important signal basis for analyzing the interference that the pilot may encounter in the process of performing these specific tasks. It is very important information for crew management, training optimization, and accident analysis and prevention. However, at present, there is a lack of models specifically used to analyze the changes in the types of tasks performed by pilots, and there is even less effective method to accurately determine when the pilot enters a specific mission procedure.

[0004] The sliding window method can be used to understand the changes in the pilot's task execution during the entire process. However, the traditional methods for determining the length of the sliding window have some shortcomings, such as requiring a lot of prior knowledge to debug different window lengths and determining the final length based on experience; for example, the window length is generally fixed, rather than dynamically adjusted. During the flight, the pilot may need to perform many different QRH task procedures, each of which has a different execution time, and even for the same QRH task procedure, the execution time of different personnel may vary greatly. Therefore, using the traditional single fixed window length will lead to a large deviation in the analysis of changes in the pilot's task execution.

[0005] Therefore, those skilled in the art are committed to developing a system and method for identifying the type of mission performed by a pilot. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to truly and accurately reflect the process of switching the execution task type and accurately locate the time of entering a specific execution task type.

[0007] In order to better analyze the changes in the types of tasks performed by the pilot's QRH and to more accurately describe the time when the pilot enters a specific QRH task program, this application proposes a task type prediction model based on changes in the pilot's visual attention allocation. The pilot's eye movement data during the performance of a specific task is collected, and an eye movement feature sequence is generated as input data, and the task type is used as output data. A task classification module for the pilot's task type is trained based on the changes in visual attention allocation. Then, the sliding window method is used to analyze the type of task that the pilot is most likely to perform at a specific moment. According to the movement of the task type with the sliding window (time), the time when the pilot enters the most likely type of task is determined. This application uses a posterior algorithm that adaptively and dynamically adjusts the sliding window, and through an iterative method, the sliding window length can be dynamically adjusted over time, thereby obtaining a more accurate analysis result.

[0008] In one embodiment of the present application, a system for identifying the type of mission performed by a pilot is provided, comprising:

[0009] Eye movement feature acquisition module, which collects the pilot's eye movement data and generates an eye movement feature sequence;

[0010] The sliding window processor module sets a sliding window sequence for the eye movement feature sequence during the entire flight process, and allows the sliding window in the sliding window sequence to translate over time to obtain the eye movement feature sequence within the sliding window;

[0011] The task classification module inputs the eye movement feature sequence in the sliding window, performs feature extraction and fusion, identifies the task type, and obtains the most likely task type in the sliding window;

[0012] The sliding window adaptive module uses a posterior algorithm that dynamically adjusts the sliding window adaptively according to the most likely execution task type in the sliding window, automatically and iteratively calculates the optimal sliding window length, and the sliding window length is dynamically adjusted over time. The dynamically adjusted sliding windows form a new sliding window sequence.

[0013] The sequence analyzer module inputs the most likely mission type within the sliding window, accurately calculates the time point when the pilot enters the most likely mission type, and describes the switching of mission types and the time of a single mission type during the entire flight.

[0014] The eye movement feature acquisition module, the sliding window processor module, the execution task classification module and the sequence analyzer module are communicatively connected in sequence, and the execution task classification module and the sliding window adaptive module are communicatively connected.

[0015] Optionally, in the system for identifying the type of task performed by a pilot in the above embodiment, the eye movement feature acquisition module uses an eye tracker.

[0016] Furthermore, in the system for identifying the type of task performed by a pilot in the above embodiment, the eye tracker uses Tobii Glasses 3.

[0017] Optionally, in the system for identifying the type of task performed by a pilot in any of the above embodiments, the eye movement data includes sampling point information, and the sampling point information includes: a timestamp, an eye movement type, and a position of the instrument panel being looked at.

[0018] Furthermore, in the system for identifying the type of task performed by a pilot in the above embodiment, the eye movement types include fixation and saccade.

[0019] Optionally, in the system for identifying the type of mission performed by a pilot in any of the above embodiments, the mission classification module includes:

[0020] A feature extraction unit, which takes the eye movement feature sequence of a task type as input and extracts a feature vector representing the distribution of visual attention under the task type;

[0021] The feature screening unit reduces the dimension and screens the feature vector representing the distribution of visual attention to remove redundant information in the feature;

[0022] The classifier unit, which resides in the execution task type prediction model, inputs the filtered feature vector and outputs the most likely execution task type within the sliding window.

[0023] Further, in the system for identifying the type of mission performed by a pilot in the above embodiment, the feature extraction unit includes:

[0024] The statistical calculation subunit calculates the statistical index representing the change of visual attention allocation according to the eye movement feature sequence in the sliding window and concatenates them into a vector to obtain a feature vector of the statistical index;

[0025] The AOI conversion subunit divides the visible area in the cockpit into different areas of interest (AOIs), divides the point where the pilot is looking at each moment into the corresponding AOI, and converts the eye movement feature sequence in the sliding window into an AOI state transition sequence. A Markov model is established for the AOI state transition sequence to extract the transfer rate feature vector and the time distribution and exponentially distributed distance feature vector in the AOI to describe the changes in the pilot's visual attention allocation.

[0026] Furthermore, in the system for identifying the type of task performed by a pilot in the above embodiment, the characteristic vector of the statistical indicator, the characteristic vector of the transfer rate, and the time distribution of the stay in the AOI and the distance characteristic vector of the exponential distribution are weighted and spliced ​​to obtain a characteristic vector representing the distribution of visual attention.

[0027] Furthermore, in the system for identifying the type of mission performed by a pilot in the above embodiment, the statistical indicators include average gaze duration, average glance duration, average glance span, blinking frequency, and average pupil diameter.

[0028] Further, in the system for identifying the type of mission performed by the pilot in the above embodiment, different areas of interest (AOIs) include a primary flight display (PFD), a navigation display (MCP), a standby board (STANDBY), an electronic centralized monitoring system (ECAM), a control display unit (CDU), a flight control unit (FCP), and outside the window (OTW).

[0029] Optionally, in the system for identifying the type of task performed by a pilot in any of the above embodiments, the feature vector characterizing the allocation of visual attention is a high-dimensional feature vector.

[0030] Optionally, in the system for identifying the type of mission performed by a pilot in any of the above embodiments, the length of the sliding window ranges from 400s to 1400s.

[0031] Furthermore, in the system for identifying the type of mission performed by a pilot in the above embodiment, the feature screening unit tests each dimensional feature of the high-dimensional feature vector through the method of analysis of variance (ANOVA), and analyzes whether there is a significant difference between different types of missions performed in this dimension. If it is not higher than the significance level, it is determined that there is no significant difference and the dimensional feature is discarded.

[0032] Preferably, in the system for identifying the type of mission performed by a pilot in the above embodiment, the significance level a= 0.05 .

[0033] Optionally, in the system for identifying the type of mission performed by a pilot in any of the above embodiments, the mission type prediction model is based on a support vector machine (Support Vector Machine, SVM).

[0034] Furthermore, in the system for identifying the type of mission performed by a pilot in the above embodiment, the mission type prediction model training process includes:

[0035] S1000, collect test data, design and execute mission type tests, and collect pilots’ eye movement data in multiple mission scenarios;

[0036] S2000, extracting task scene slice data. After the test, the process of each pilot performing each task scene is extracted separately as task scene slice data. The task scene slice data includes the pilot number, the task scene number (i.e., the execution task type number) and the corresponding eye movement data.

[0037] S3000, extracting a feature vector representing visual attention allocation, wherein the feature extraction unit obtains the feature vector representing visual attention allocation from the eye movement data corresponding to the task scene slice data;

[0038] S4000, screening feature vectors, wherein the feature screening unit tests each dimension feature of the feature vector representing the visual attention allocation by means of variance analysis, and analyzes whether there is a significant difference between different types of execution tasks in the dimension; if the difference is not higher than the significance level, it is determined that there is no significant difference, and the dimension feature is discarded;

[0039] S5000, training is completed, the filtered feature vector is used as input, the execution task type is used as output, and a support vector machine (SVM) is used to establish a mapping from input to output to obtain the execution task type prediction model.

[0040] Furthermore, in the system for identifying the type of mission performed by pilots in the above embodiment, the mission type test includes 10-30 experienced pilots aged between 30 and 50 years old; all pilots are combined into 10-30 crews, each crew includes a primary pilot (PM) and a co-pilot (PF).

[0041] Furthermore, in the system for identifying the type of mission performed by a pilot in the above embodiment, the multiple mission scenarios include mission scenarios with fixed procedures in normal flight and fault mission scenarios that occur during flight.

[0042] Furthermore, in the system for identifying the type of mission performed by a pilot in the above embodiment, mission scenarios with fixed procedures in normal flight include manual flight and standard instrument departure.

[0043] Furthermore, in the system for identifying the type of mission performed by a pilot in the above embodiment, the fault mission scenarios occurring during the flight include fuel imbalance and loss of normal lighting function in the cockpit.

[0044] Optionally, in the system for identifying the type of mission performed by a pilot in any of the above embodiments, each of the multiple mission scenarios lasts for 20-30 minutes, a round of tests lasts for approximately 90 minutes, and the crew rests for 30-50 minutes after a round of tests.

[0045] Based on any of the above embodiments, in another embodiment of the present application, a method for identifying the type of mission performed by a pilot is provided, comprising the following steps:

[0046] S100, collecting eye movement data, collecting the pilot's eye movement data, and generating an eye movement feature sequence;

[0047] S200, obtaining an eye movement feature sequence, for the eye movement feature sequence in the entire flight process, setting a sliding window sequence, allowing the sliding window in the sliding window sequence to shift over time, and obtaining the eye movement feature sequence in the sliding window;

[0048] S300, predicting the type of task to be performed, inputting the eye movement feature sequence in the sliding window, performing feature extraction and fusion, identifying the type of task to be performed, and predicting the most likely type of task to be performed in the sliding window;

[0049] S400, dynamically adjusting the sliding window, according to the most likely execution task type in the sliding window, using a posterior algorithm for adaptively dynamically adjusting the sliding window, automatically and iteratively calculating the optimal sliding window length, the sliding window length is dynamically adjusted over time, and the dynamically adjusted sliding windows form a new sliding window sequence;

[0050] S500, identifying the mission type, inputting the most likely mission type in the sliding window, accurately calculating the time point when the pilot enters the most likely mission type, and describing the switching of mission types during the entire flight and the time of a single mission type.

[0051] Optionally, in the method for identifying the type of mission performed by a pilot in the above embodiment, step S100 includes:

[0052] S110, collecting eye movement data of the pilot, using an eye tracker to collect eye movement data of the pilot, including sampling point information, the sampling point information including: timestamp, eye movement type, and gaze position of the instrument panel;

[0053] S120, generating an eye movement feature sequence, preprocessing the eye movement data, including removing noise, correcting deviation, and generating an eye movement feature sequence.

[0054] Furthermore, in the method for identifying the type of task performed by a pilot in the above embodiment, the eye movement types include fixation and saccade.

[0055] Optionally, in the method for identifying the type of mission performed by a pilot in the above embodiment, step S200 includes:

[0056] S210, setting a sliding window sequence, for the eye movement feature sequence in the entire flight process, setting a sliding window sequence, and setting an initial value of the sliding window length;

[0057] S220: Obtain an eye movement feature sequence in a sliding window, wherein the sliding window in the sliding window sequence is translated over time to obtain an eye movement feature sequence in the sliding window.

[0058] Further, in the method for identifying the type of mission performed by a pilot in the above embodiment, the initial value of the sliding window length is w* The average execution time of all tasks obtained during training, where w is the coefficient.

[0059] Preferably, in the method for identifying the type of mission performed by a pilot in the above embodiment, w=0.8 .

[0060] Optionally, in the method for identifying the type of mission performed by a pilot in any of the above embodiments, step S300 includes:

[0061] S310, extracting feature vectors, taking the eye movement feature sequence in the sliding window as input, and extracting feature vectors representing the distribution of visual attention in the case of the execution task type;

[0062] S320, dimensionality reduction and screening, reducing the dimensionality and screening the feature vector representing the distribution of visual attention, and removing redundant information in the feature;

[0063] S330, predicting the execution task type, inputting the screened feature vector into the trained execution task type prediction model, and outputting the most likely execution task type within the sliding window.

[0064] Further, in the method for identifying the type of mission performed by a pilot in the above embodiment, step S310 includes:

[0065] S311, calculating the characteristic vector of the statistical index, calculating the statistical index representing the change of visual attention allocation according to the eye movement feature sequence in the sliding window and splicing them into a vector to obtain the characteristic vector of the statistical index;

[0066] S312, extract the transfer rate feature vector and the distance feature vector, divide the visible area in the cockpit into different areas of interest (AOI), divide the point that the pilot is looking at at each moment into the corresponding AOI, and convert the eye movement feature sequence in the sliding window into an AOI state transition sequence; establish a Markov model for the AOI state transition sequence, extract the transfer rate feature vector and the time distribution of the stay in the AOI and the distance feature vector of the exponential distribution, and describe the changes in the pilot's visual attention allocation.

[0067] Furthermore, in the method for identifying the type of mission performed by a pilot in the above embodiment, the statistical indicators include average gaze duration, average glance duration, average glance span, blinking frequency, and average pupil diameter.

[0068] Optionally, in the method for identifying the type of mission performed by a pilot in any of the above embodiments, the mission type prediction model is based on a support vector machine (Support Vector Machine, SVM).

[0069] Furthermore, in the method for identifying the type of mission to be performed by a pilot in the above embodiment, the task type prediction model training process includes:

[0070] S1000, collect test data, design and execute mission type tests, and collect pilots’ eye movement data in multiple mission scenarios;

[0071] S2000, extracting task scene slice data. After the test, the process of each pilot performing each task scene is extracted separately as task scene slice data. The task scene slice data includes the pilot number, the task scene number (i.e., the execution task type number) and the corresponding eye movement data.

[0072] S3000, extracting a feature vector representing visual attention allocation, and obtaining a feature vector representing visual attention allocation from the eye movement data corresponding to the task scene slice data;

[0073] S4000, screening feature vectors, testing each dimension of the feature vectors representing the distribution of visual attention by means of variance analysis, analyzing whether there is a significant difference between different types of execution tasks in the dimension, and if it is not higher than the significance level, determining that there is no significant difference, and discarding the dimension feature;

[0074] S5000, training is completed, the filtered feature vector is used as input, the execution task type is used as output, and a support vector machine (SVM) is used to establish a mapping from input to output to obtain the execution task type prediction model.

[0075] Optionally, in the method for identifying the type of mission performed by a pilot in the above embodiment, step S400 includes:

[0076] S410, adaptively and dynamically adjusting the sliding window, according to the most likely execution task type in the sliding window, using an adaptively and dynamically adjusting the sliding window a posteriori algorithm, automatically and iteratively calculating the optimal sliding window length;

[0077] S420, generating a new sliding window sequence, wherein the sliding window length is dynamically adjusted as time changes, the dynamically adjusted sliding windows form a new sliding window sequence, and a Gaussian smoothing function is used for smoothing;

[0078] S430, iteratively predict the execution task type, use the sliding window sequence after Gaussian smoothing to predict the most likely execution task type in the sliding window, until the end condition is met, and obtain the most likely execution task type in the final sliding window, otherwise iteratively execute steps S410-S430.

[0079] Further, in the method for identifying the type of mission performed by a pilot in the above embodiment, the Gaussian smoothing function in step S420 is σ The selection range is greater than or equal to the number of sampling points / 50 and less than or equal to the number of sampling points / 30.

[0080] Preferably, in the method for identifying the type of mission performed by a pilot in the above embodiment, the Gaussian smoothing function in step S420 is σ Equal to the number of sampling points / 40.

[0081] Furthermore, in the method for identifying the type of mission performed by a pilot in the above embodiment, the termination condition in step S430 is that the sum of the squares of the differences between the sliding window length in the current sliding window sequence and the corresponding sliding window length in the previous iterative sliding window sequence is not greater than the set threshold value. threshold , the formula is as follows:

[0082]

[0083] in, is the sliding window length in the current sliding window sequence, is the length of the sliding window in the last sliding window sequence, | | represents the modulus of the vector, is the set threshold.

[0084] Preferably, in the method for identifying the type of mission performed by a pilot in the above embodiment, the threshold value is set threshold=5 .

[0085] Further, in the method for identifying the type of mission performed by a pilot in the above embodiment, step S500 includes:

[0086] S510, analyzing the final prediction result, and obtaining the most likely execution task type at each moment according to the final prediction result of the execution task type;

[0087] S520, calculating the time point for executing the mission type, calculating the time point when the pilot enters the most likely mission type, describing the switching of the mission type during the entire flight process and the time for executing a single mission type.

[0088] The present invention is based on the changes in the pilot's visual attention allocation and uses a sliding window method to analyze the switching process of the pilot's task type. The sliding window length can be dynamically adjusted over time in an iterative manner, and the changes in the pilot's task type can be accurately analyzed, as well as the time when the pilot enters the task type can be accurately described.

[0089] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 is a schematic diagram illustrating the structure of a system for identifying a type of mission performed by a pilot according to an exemplary embodiment;

[0091] Figure 2 is a flow chart illustrating a process of performing task type prediction model training according to an exemplary embodiment;

[0092] Figure 3 is a flow chart illustrating a method of identifying a type of mission performed by a pilot according to an exemplary embodiment;

[0093] Figure 4 is an experimental effect diagram illustrating a method for identifying a type of mission performed by a pilot according to an exemplary embodiment. DETAILED DESCRIPTION

[0094] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0095] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is schematically exaggerated in some places in the drawings.

[0096] This application designs a system for identifying the type of mission a pilot is performing, such as Figure 1 As shown, including:

[0097] The eye movement feature acquisition module uses the Tobii Glasses 3 eye tracker to collect the pilot's eye movement data, including sampling point information. The sampling point information includes: timestamp, eye movement type, and the position of the instrument panel. The eye movement types include fixation and saccade, and generate an eye movement feature sequence;

[0098] The sliding window processor module sets a sliding window sequence for the eye movement feature sequence during the entire flight process, and allows the sliding window in the sliding window sequence to translate over time to obtain the eye movement feature sequence within the sliding window.

[0099] The task classification module inputs the eye movement feature sequence in the sliding window, performs feature extraction and fusion, identifies the task type, and obtains the most likely task type in the sliding window. The task classification module includes:

[0100] The feature extraction unit takes the eye movement feature sequence of a task type as input and extracts the feature vector representing the visual attention allocation under the task type; specifically includes:

[0101] The statistical calculation subunit calculates the statistical indicators representing the change of visual attention allocation according to the eye movement feature sequence in the sliding window and splices them into vectors to obtain the characteristic vectors of the statistical indicators. The statistical indicators include the average gaze duration, the average saccade duration, the average saccade span, the blinking frequency, and the average pupil diameter.

[0102] The AOI conversion subunit divides the visible area in the cockpit into different areas of interest (AOI), including the primary flight display (PFD), navigation display (MCP), standby board (STANDBY), electronic centralized monitoring system (ECAM), control display unit (CDU), flight control unit (FCP), and outside the window (OTW). The pilot's gaze point at each moment is divided into the corresponding AOI, and the eye movement feature sequence in the sliding window is converted into an AOI state transition sequence; a Markov model is established for the AOI state transition sequence, and the transfer rate feature vector and the time distribution and exponential distribution distance feature vector of the AOI are extracted to describe the changes in the pilot's visual attention allocation;

[0103] The characteristic vectors of statistical indicators, transfer rate, time distribution within the AOI, and distance characteristic vectors of exponential distribution are weighted and concatenated to obtain a characteristic vector representing the distribution of visual attention. The characteristic vector representing the distribution of visual attention is a high-dimensional characteristic vector.

[0104] The feature screening unit reduces the dimension of the feature vector representing the distribution of visual attention and screens it to remove redundant information in the feature. The feature screening unit tests each dimension feature of the high-dimensional feature vector through the analysis of variance (ANOVA) method to analyze whether there is a significant difference between different types of execution tasks in this dimension, which is not higher than the set significance level. a=0.05 , it is determined that there is no significant difference and the dimension feature is discarded.

[0105] The classifier unit resides in a prediction model of the execution task type based on a support vector machine (SVM), which inputs the filtered feature vector and outputs the most likely execution task type within the sliding window; Figure 2 As shown in the figure, the task type prediction model training process includes:

[0106] S1000, collect test data, design and perform mission type tests, including 10-30 experienced pilots aged between 30 and 50 years old; all pilots are combined into 10-30 crews, each crew includes a primary pilot (PM) and a co-pilot (PF), collect pilots' eye movement data in multiple mission scenarios, multiple mission scenarios include mission scenarios with fixed procedures in normal flight and fault mission scenarios that occur during flight. Mission scenarios with fixed procedures in normal flight include manual flight and standard instrument departure, and fault mission scenarios include fuel imbalance and loss of normal lighting function in the cockpit. Each mission scenario in multiple mission scenarios lasts for 20-30 minutes, and a round of tests lasts for about 90 minutes. After a round of tests, the crew rests for 30-50 minutes.

[0107] S2000, extracting task scene slice data. After the test, the process of each pilot performing each task scene is extracted separately as task scene slice data (slice of task-scene). The task scene slice data includes the pilot number, the task scene number (i.e., the number of the type of task to be performed) and the corresponding eye movement data;

[0108] S3000, extracting a feature vector representing the distribution of visual attention, wherein the feature extraction unit obtains the feature vector representing the distribution of visual attention from the eye movement data corresponding to each task scene slice data;

[0109] S4000, screening feature vectors. The feature screening unit tests each dimension of the feature vector representing the distribution of visual attention by using an analysis of variance (ANOVA) method to analyze whether there is a significant difference between different types of execution tasks on the dimension. If the difference is not higher than the significance level, it is determined that there is no significant difference and the dimension feature is discarded.

[0110] S5000, training is completed, the filtered feature vector is used as input, the execution task type is used as output, and the support vector machine (SVM) is used to establish the input-output mapping to obtain the execution task type prediction model.

[0111] The sliding window adaptive module uses a posterior algorithm that adaptively and dynamically adjusts the sliding window according to the most likely execution task type in the sliding window, and automatically iterates to calculate the optimal sliding window length in the range of 400s-1400s. The sliding window length is dynamically adjusted over time, and the dynamically adjusted sliding windows form a new sliding window sequence.

[0112] The sequence analyzer module inputs the most likely mission type within the sliding window, accurately calculates the time point when the pilot enters the most likely mission type, and describes the switching of mission types and the time of a single mission type during the entire flight.

[0113] The eye movement feature acquisition module, the sliding window processor module, the execution task classification module and the sequence analyzer module are communicatively connected in sequence, and the execution task classification module and the sliding window adaptive module are communicatively connected.

[0114] Based on the above embodiments, the present application provides a method for identifying the type of mission performed by a pilot, such as Figure 3 As shown, the following steps are included:

[0115] S100, collecting eye movement data, collecting the pilot's eye movement data, and generating an eye movement feature sequence; specifically including:

[0116] S110. Collect the pilot's eye movement data using an eye tracker, including sampling point information. The sampling point information includes: timestamp, eye movement type, and the position of the instrument panel being looked at. The eye movement types include fixation and saccade.

[0117] S120, generating an eye movement feature sequence, preprocessing the eye movement data, including removing noise, correcting deviation, and generating an eye movement feature sequence.

[0118] S200, obtaining an eye movement feature sequence, for the eye movement feature sequence in the entire flight process, setting a sliding window sequence, allowing the sliding window in the sliding window sequence to shift over time, and obtaining the eye movement feature sequence in the sliding window; specifically including:

[0119] S210, set a sliding window sequence. For the eye movement feature sequence in the entire flight process, set a sliding window sequence and set an initial value of the sliding window length. The initial value of the sliding window length = w* The average execution time of all tasks obtained during training, where w is the coefficient, w=0.8 ;

[0120] S220: Obtain an eye movement feature sequence in a sliding window, wherein the sliding window in the sliding window sequence is translated over time to obtain an eye movement feature sequence in the sliding window.

[0121] S300, predicting the type of task to be performed, inputting the eye movement feature sequence in the sliding window, performing feature extraction and fusion, identifying the type of task to be performed, and predicting the most likely type of task to be performed in the sliding window; specifically including:

[0122] S310, extracting feature vectors, taking the eye movement feature sequence in the sliding window as input, and extracting feature vectors representing the distribution of visual attention in the case of the task type being performed; specifically including:

[0123] S311, calculating the characteristic vector of the statistical index, according to the eye movement feature sequence in the sliding window, calculating the statistical index representing the change of visual attention allocation and splicing it into a vector, to obtain the characteristic vector of the statistical index, the statistical index includes the average gaze duration, the average saccade duration, the average saccade span, the blinking frequency, and the average pupil diameter;

[0124] S312, extract the transfer rate feature vector and the distance feature vector, divide the visible area in the cockpit into different areas of interest (AOI), divide the point that the pilot is looking at at each moment into the corresponding AOI, and convert the eye movement feature sequence in the sliding window into an AOI state transition sequence; establish a Markov model for the AOI state transition sequence, extract the transfer rate feature vector and the time distribution of the stay in the AOI and the distance feature vector of the exponential distribution, and describe the changes in the pilot's visual attention allocation.

[0125] S320, dimensionality reduction and screening, reducing the dimensionality and screening the feature vector representing the distribution of visual attention, and removing redundant information in the feature;

[0126] S330, predicting the execution task type, inputting the filtered feature vector into the trained execution task type prediction model, the execution task type prediction model is based on the support vector machine (SVM), and outputs the most likely execution task type in the sliding window; Figure 2 As shown in the figure, the task type prediction model training process includes:

[0127] S1000, collect test data, design and execute mission type tests, and collect pilots’ eye movement data in multiple mission scenarios;

[0128] S2000, extracting task scene slice data. After the test, the process of each pilot performing each task scene is extracted separately as task scene slice data (slice of task-scene). The task scene slice data includes the pilot number, the task scene number (i.e., the number of the type of task to be performed) and the corresponding eye movement data;

[0129] S3000, extracting a feature vector representing the distribution of visual attention, and obtaining a feature vector representing the distribution of visual attention from the eye movement data corresponding to each task scene slice data;

[0130] S4000, screening feature vectors, testing each dimension of the feature vectors representing the distribution of visual attention by means of analysis of variance (ANOVA), analyzing whether there is a significant difference between different types of execution tasks on this dimension; if it is not higher than the significance level, it is determined that there is no significant difference, and the dimension feature is discarded;

[0131] S5000, training is completed, the filtered feature vector is used as input, the execution task type is used as output, and the support vector machine (SVM) is used to establish the input-output mapping to obtain the execution task type prediction model.

[0132] S400, dynamically adjusting the sliding window, according to the most likely execution task type in the sliding window, using a posterior algorithm for adaptively and dynamically adjusting the sliding window, automatically and iteratively calculating the optimal sliding window length, the sliding window length is dynamically adjusted over time, and the dynamically adjusted sliding windows form a new sliding window sequence; specifically including:

[0133] S410, adaptively and dynamically adjusting the sliding window, according to the most likely execution task type in the sliding window, using an adaptively and dynamically adjusting the sliding window a posteriori algorithm, automatically and iteratively calculating the optimal sliding window length;

[0134] S420, generating a new sliding window sequence, wherein the sliding window length is dynamically adjusted over time, and the dynamically adjusted sliding windows form a new sliding window sequence, and a Gaussian smoothing function is used for smoothing. σ Equal to the number of sampling points / 40;

[0135] S430, iteratively predict the execution task type, use the sliding window sequence after Gaussian smoothing to predict the most likely execution task type in the sliding window, until the end condition is met, and the most likely execution task type in the sliding window in the final sliding window is obtained, otherwise iteratively execute steps S410-S430; the end condition is that the sum of the squares of the differences between the sliding window length in the current sliding window sequence and the corresponding sliding window length in the previous iterative sliding window sequence is not greater than the set threshold threshold , the formula is as follows:

[0136]

[0137] in, is the sliding window length in the current sliding window sequence, is the length of the sliding window in the last sliding window sequence, | | represents the modulus of the vector, is the set threshold, threshold=5 .

[0138] S500, identifying the mission type, inputting the most likely mission type in the sliding window, accurately calculating the time point when the pilot enters the most likely mission type, describing the switching of mission types during the entire flight and the time of a single mission type; specifically including:

[0139] S510, analyzing the final prediction result, and obtaining the most likely execution task type at each moment according to the final prediction result of the execution task type;

[0140] S520, calculating the time point for executing the mission type, calculating the time point when the pilot enters the most likely mission type, describing the switching of the mission type during the entire flight process and the time for executing a single mission type.

[0141] The above embodiment predicts the time point when the pilot enters a specific task type and the task type switching process from the perspective of changes in visual attention distribution. This application uses the above embodiment to verify the prediction effect of a complete flight, such as Figure 4 As shown, the background box-shaped area represents the time period of the scene that actually occurs during a certain flight, and the curve represents the probability of the pilot's possible task type at a certain moment predicted by this embodiment. Specifically, the curve Standard Instrument Departure (SID) represents a standard instrument departure, the curve Go-Around in Crosswind represents a go-around under crosswind conditions, the curve Standard Instrument Arrival (STAR) represents a standard instrument arrival, the curve Rejected Takeoff represents an aborted takeoff, the curve Manual Flight represents manual flight, the curve Provision of Erroneous Position Information represents the provision of erroneous position information, and the curve Trim Air System Malfunction represents a thermal air system failure. For example, in the time period of 300s-1000s, the type of task performed by the pilot is manual flight, and this embodiment predicts that in this time period, the probability of the pilot performing manual flight is the highest. It can be seen from the test results that in the scenarios of the task types of go-around under crosswind conditions, standard instrument arrival, aborted takeoff, manual flight, provision of erroneous position information, and thermal air system failure, the prediction results of this application are consistent with the type of task performed by the pilot. By Figure 4It is possible to visually see the time point when the pilot enters a specific mission type and the time point when the mission type is switched. Therefore, the prediction result of the present application has a high accuracy rate and can visually and accurately observe the changes in the pilot's mission type.

[0142] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A system for identifying the type of mission a pilot is to perform, characterized in that: include: Eye movement feature acquisition module, which collects the pilot's eye movement data and generates an eye movement feature sequence; A sliding window processor module sets a sliding window sequence for the eye movement feature sequence during the entire flight process, and allows the sliding window in the sliding window sequence to translate over time to obtain the eye movement feature sequence within the sliding window; The task classification module inputs the eye movement feature sequence in the sliding window, performs feature extraction and fusion, identifies the type of task to be performed, and obtains the most likely type of task to be performed in the sliding window; A sliding window adaptive module, which uses a posterior algorithm for adaptively and dynamically adjusting the sliding window according to the most likely execution task type in the sliding window, automatically and iteratively calculates the optimal sliding window length, and the sliding window length is dynamically adjusted over time, and the dynamically adjusted sliding windows form a new sliding window sequence; A sequence analyzer module, which inputs the most likely mission type within the sliding window, accurately calculates the time point when the pilot enters the most likely mission type, and describes the switching of mission types and the time of a single mission type during the entire flight process; The eye movement feature acquisition module, the sliding window processor module, the execution task classification module and the sequence analyzer module are communicatively connected in sequence, and the execution task classification module and the sliding window adaptive module are communicatively connected.

2. The system for identifying the type of mission performed by a pilot according to claim 1, characterized in that: The task classification module includes: A feature extraction unit, taking the eye movement feature sequence in the sliding window as input, extracting a feature vector representing the distribution of visual attention under the execution task type; A feature screening unit is used to reduce the dimension of the feature vector representing the distribution of visual attention and screen it to remove redundant information in the feature; The classifier unit resides in the execution task type prediction model, inputs the filtered feature vector, and outputs the most likely execution task type within the sliding window.

3. The system for identifying the type of mission performed by a pilot according to claim 2, characterized in that: The feature extraction unit comprises: A statistical calculation subunit calculates statistical indicators representing changes in visual attention allocation based on the eye movement feature sequence in the sliding window and concatenates the statistical indicators into vectors to obtain a feature vector of the statistical indicators; The AOI conversion subunit divides the visible area in the cockpit into different areas of interest, divides the point that the pilot is looking at at each moment into the corresponding AOI, and converts the eye movement feature sequence in the sliding window into an AOI state transition sequence; a Markov model is established for the AOI state transition sequence, and a transfer rate feature vector and a time distribution of the stay in the AOI and a distance feature vector of the exponential distribution are extracted to describe the changes in the pilot's visual attention allocation.

4. The system for identifying the type of mission performed by a pilot according to claim 3, characterized in that: The execution task type prediction model is based on a support vector machine.

5. The system for identifying the type of mission performed by a pilot according to claim 4, characterized in that: The task type prediction model training process includes: S1000, collect test data, design and execute mission type tests, and collect pilots’ eye movement data in multiple mission scenarios; S2000, extracting task scene slice data. After the test, the process of each pilot performing each task scene is extracted separately as task scene slice data, and the task scene slice data includes the pilot number, the task scene number and the corresponding eye movement data; S3000, extracting a feature vector representing visual attention allocation, wherein the feature extraction unit obtains the feature vector representing visual attention allocation from the eye movement data corresponding to the task scene slice data; S4000, screening feature vectors, wherein the feature screening unit tests each dimension feature of the feature vector representing the visual attention allocation by a variance analysis method, and analyzes whether there is a significant difference between different types of execution tasks in the dimension; if the difference is not higher than the significance level, it is determined that there is no significant difference, and the dimension feature is discarded; S5000, training is completed, the filtered feature vector is used as input, the execution task type is used as output, and the support vector machine is used to establish a mapping from input to output to obtain the execution task type prediction model.

6. A method for identifying the type of mission a pilot is performing, using the system for identifying the type of mission a pilot is performing as claimed in claim 5, characterized in that: The steps include: S100, collecting eye movement data, collecting the pilot's eye movement data, and generating an eye movement feature sequence; S200, obtaining an eye movement feature sequence, and for the eye movement feature sequence in the entire flight process, setting a sliding window sequence, allowing the sliding windows in the sliding window sequence to shift over time, and obtaining the eye movement feature sequence in the sliding window; S300, predicting the type of task to be performed, inputting the eye movement feature sequence in the sliding window, performing feature extraction and fusion, identifying the type of task to be performed, and predicting the most likely type of task to be performed in the sliding window; S400, dynamically adjusting the sliding window, according to the most likely execution task type in the sliding window, using a posterior algorithm for adaptively dynamically adjusting the sliding window, automatically iteratively calculating the optimal sliding window length, the sliding window length is dynamically adjusted over time, and the dynamically adjusted sliding windows form a new sliding window sequence; S500, identifying the mission type, inputting the most likely mission type in the sliding window, accurately calculating the time point when the pilot enters the most likely mission type, and describing the switching of mission types during the entire flight process and the time of a single mission type.

7. The method for identifying the type of mission performed by a pilot according to claim 6, characterized in that: The step S100 includes: S110, collecting eye movement data of the pilot, using an eye tracker to collect eye movement data of the pilot, including sampling point information, the sampling point information including: timestamp, eye movement type, and gaze position of the instrument panel; S120, generating an eye movement feature sequence, preprocessing the eye movement data, including removing noise and correcting deviation, to generate the eye movement feature sequence.

8. The method for identifying the type of mission performed by a pilot according to claim 6, characterized in that: The step S200 includes: S210, setting a sliding window sequence, for the eye movement feature sequence in the entire flight process, setting the sliding window sequence, and setting an initial value of the sliding window length; S220: Obtain an eye movement feature sequence in a sliding window, wherein the sliding window in the sliding window sequence is translated over time to obtain the eye movement feature sequence in the sliding window.

9. The method for identifying the type of mission performed by a pilot according to claim 6, characterized in that: The step S300 includes: S310, extracting feature vectors, taking the eye movement feature sequence in the sliding window as input, and extracting feature vectors representing the distribution of visual attention in the case of the execution task type; S320, dimensionality reduction and screening, reducing the dimensionality and screening the feature vector representing the visual attention allocation to remove redundant information in the feature; S330, predicting the execution task type, inputting the screened feature vector into the trained execution task type prediction model, and outputting the most likely execution task type within the sliding window.

10. The method for identifying the type of mission performed by a pilot according to claim 6, characterized in that: The step S400 includes: S410, adaptively and dynamically adjusting the sliding window, according to the most likely execution task type in the sliding window, using a posteriori algorithm for adaptively and dynamically adjusting the sliding window, and automatically and iteratively calculating the optimal sliding window length; S420, generating a new sliding window sequence, wherein the length of the sliding window is dynamically adjusted as time changes, the dynamically adjusted sliding windows form a new sliding window sequence, and a Gaussian smoothing function is used for smoothing; S430, iteratively predict the execution task type, use the sliding window sequence after Gaussian smoothing to predict the most likely execution task type in the sliding window, until the end condition is met, and obtain the most likely execution task type in the final sliding window, otherwise iteratively execute the steps S410-S430.

Images