A flight trainee fatigue assessment system based on eye movement data

Through the flight cadet fatigue assessment system based on eye movement data, deep convolution is used to generate adversarial networks and long-term memory neural networks to analyze the eye movement data of flight cadets, solving the problem of difficulty in evaluating the fatigue status of flight cadets in the existing technology, reducing the problem of reducing the Data acquisition costs have been improved and flight safety awareness has been improved.

CN118614929BActive Publication Date: 2024-11-05CIVIL AVIATION FLIGHT UNIV OF CHINA
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411088325.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-05
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The prior art is difficult to effectively evaluate and early warning of the fatigue status of flight cadets, especially during the training stage, and the cost of data collection and processing is high, which affects flight safety.

Method used

Design a flight attendant fatigue assessment system based on eye movement data, including data acquisition, data processing and flight fatigue state analysis modules, use deep convolution to generate adversarial networks and long and short-term memory neural networks, generate simulated databases, analyze eye movement data and The correlation of fatigue degree predicts the fatigue status and flight results of flight students.

Benefits of technology

It reduces the cost of data acquisition, improves the efficiency of evaluating the fatigue status of flight students, realizes accurate assessment of the fatigue status of flight students during long-term flight processes, and improves flight safety awareness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118614929B_ABST
    Figure CN118614929B_ABST
Patent Text Reader

Abstract

The present invention discloses a flight trainee fatigue assessment system based on eye movement data, which relates to the field of electronic digital data processing technology, and specifically includes: a data acquisition module for building an eye movement data acquisition and storage platform to collect flight data of trainees under different fatigue levels; a data processing module for processing a hotspot map, and then simulating the hotspot map to generate a simulation database; a flight fatigue state analysis module for assigning influence proportions of different fatigue levels to the eye movement data of the flight trainee at different stages to be tested, predicting the later flight state of the assigned data, and using the predicted results as the flight results of the flight trainee under fatigue state; a result output module for outputting the flight results of the flight trainee under normal state and simulated fatigue state. The present invention realizes the flight effect of evaluating the fatigue state of the flight trainee during the entire flight process during teaching, and achieves the role of early warning of flight trainee teaching.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of electronic digital data processing, and in particular relates to a flight trainee fatigue assessment system based on eye movement data. Background Art

[0002] Pilot error has always been considered a major factor in aviation accidents and incidents. Factors that have a negative impact on pilot flight operations mainly include fatigue, stress, and distraction, which are usually difficult to detect during training. However, during the pilot training process, the duration of the pilot's flight ability test is shorter than the actual flight duration of the flight. Therefore, pilots tend to ignore the impact of these aspects on flight safety when performing actual flight work after training.

[0003] Long periods of wakefulness, lack of sleep and / or sleep disorders are the main causes of pilot fatigue, but they may also come from long-distance flights and long-term operational tasks. In the existing technology, fatigue warning can be initially carried out by analyzing the fatigue level of pilots who have mastered the flying ability during the flight through eye movement data, but there is still a lot of room for research on how to conduct fatigue warning and flight ability assessment for pilots who are still in the training stage. In addition, due to the noise caused by eye movement, the obtained data requires a lot of preprocessing, and pilot fatigue data requires a lot of time to monitor multiple pilots performing long-term flight missions (usually more than 2 hours of flight) multiple times before it can be collected. Collecting and processing this data requires a lot of financial resources and time. Summary of the invention

[0004] In view of the above problems, the present invention designs a flight trainee fatigue assessment system based on eye movement data, which is used to solve the problems existing in the above-mentioned prior art, expand the data of pilots in fatigue status, and evaluate the flight effect on flight trainees who produce fatigue status during long-term flight processes.

[0005] To achieve the above object, the technical solution of the present invention is:

[0006] A flight trainee fatigue assessment system based on eye movement data, comprising a data acquisition module, a data processing module, a flight fatigue state analysis module and a result output module;

[0007] The data acquisition module is used to build an eye movement data acquisition and storage platform, collect flight data of standard pilots under different fatigue levels and flight data of flight trainees to be tested, and generate a hotspot map of the pilots' eye movement data on the flight instrument area at different flight stages according to the time sequence of the collected eye movement data of all pilots;

[0008] The data processing module is used to perform preliminary processing on the heat map, and then use a deep convolutional generative adversarial network to simulate the heat map after processing the standard pilot data under different fatigue states, and generate a simulation database, expand the pilot's fatigue data based on the simulation database, and obtain the fatigue correlation and influence ratio of the fatigue degree and eye movement data through the expanded data analysis, and calculate the attention distribution area that affects the flight effect and the changes in the distribution of eye movement data under different fatigue degrees;

[0009] The flight fatigue state analysis module is used to assign different fatigue degree influence ratios to the eye movement data of the flight trainee at different stages, use the long short-term memory neural network to predict the later flight state of the assigned data, and use the prediction result as the flight result of the flight trainee under fatigue state;

[0010] The result output module is used to output the flight results of the flight trainee in a normal state and a simulated fatigue state.

[0011] Preferably, the eye movement data collection and storage platform collects the pilot's eye movement data through an eye tracker and a desktop flight procedure trainer, and sets the flight instrument area where the pilot's attention is distributed, wherein the flight instrument area includes the attitude indicator and external visual, altimeter, heading indicator, airspeed indicator, vertical speed indicator, and turn indicator.

[0012] Preferably, the flight phases are divided into four phases: take-off, front phase, rear phase, and landing, and the fatigue level of a standard pilot is divided into three types: no fatigue, moderate fatigue, and severe fatigue.

[0013] Preferably, in the data processing module, the heat map is subjected to grayscale processing and then binarization processing to obtain the distribution of attention areas of each pilot, and the eye movement ranges of the pilots in different attention distribution areas are counted, and then the processed binary heat map is input into a deep convolutional generative adversarial network to generate a simulation database.

[0014] Preferably, the steps of using a deep convolutional generative adversarial network to generate a simulation database are: standardizing input data; initializing network values, marking the heat map of standard pilots with no fatigue, moderate fatigue, and severe fatigue after being processed by the data processing module and inputting them into the initialization network; customizing the loop training network, judging whether the network training is completed through the loss value curves of the generator and the discriminator, and if the loss value curve meets expectations, outputting the image generated by the generative adversarial network as the simulation database, and verifying the network training effect by classifying the simulated data.

[0015] Preferably, the fatigue data of the standard pilot is expanded based on the simulation database, and the data sets of simulated and actual standard pilots in different fatigue states are used as statistical samples. The changes in the eye movement data of the standard pilots in different fatigue states are statistically analyzed, and the eye movement data under moderate fatigue and severe fatigue levels are compared with the eye movement data under no fatigue level. The influence ratio of the eye movement data and the fatigue level is analyzed and calculated, so as to obtain the changes in the attention distribution of each flight instrument area when the fatigue level affects the eye movement data.

[0016] Preferably, when calculating the correlation values ​​of various attention areas that affect the flight effect, the proportion and effective total duration of all attention distribution areas on the instrument in each flight stage of the standard pilot in a fatigue-free state are counted to obtain the attention distribution area that affects the flight effect, and this area is used as the area to be processed when the flight trainee to be tested is given a fatigue state.

[0017] Preferably, according to the proportion of influence of different fatigue levels on eye movement data, corresponding processing is performed on the flight trainee in the three stages of takeoff, front section and back section, and the attention distribution area that affects the flight effect in the binary heat map of the flight trainee processed by the data processing module is assigned a value. By utilizing the characteristics of the binary image, according to the proportion of fatigue level affecting eye movement data, if the influence proportion increases, the range of eye movement in the area marked as the trainee's eye movement record in the image is marginally expanded; if the influence proportion decreases, the range of eye movement in the area marked as the trainee's eye movement record in the image is marginally reduced.

[0018] Preferably, the eye movement data of the standard pilot in a non-fatigue state is input as a training set into the long short-term memory neural network for fitting training, and then the data of the flight trainee to be tested after fatigue assignment is input into the trained long short-term memory network fitting model for prediction, wherein each flight trainee to be tested presents a fatigue state or a progressive fatigue state in different flight stages, that is, the flight trainee to be tested has no fatigue in the take-off stage, moderate fatigue in the first part of the flight, and severe fatigue in the second part of the flight, or the flight trainee to be tested maintains a moderate fatigue state in both the first and second parts.

[0019] Preferably, the eye movement data of the standard pilot in a non-fatigue state is used as a standard to evaluate the flight result values ​​of the flight trainees to be tested in a normal state and a fatigue state, and a flight result graph of each flight trainee to be tested in a non-fatigue state and a fatigue state is generated.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: after preliminary collection of eye movement data of standard pilots in a fatigue state, fatigue visual data is learned and simulated using a deep convolutional generative adversarial network, a large amount of image data of pilot fatigue state is generated by the deep convolutional generative adversarial network, and the data acquisition cost is reduced; the influence of fatigue degree on the attention area during normal flight and the attention area that mainly affects the flight effect during flight are obtained by analyzing actual data and simulated data; then, data of flight test of flight trainees are simulated and assigned according to the influence of fatigue degree, thereby simulating the influence of different fatigue degrees on the eye movement data of flight trainees; then, a long short-term memory network is used to predict the flight results under different flight fatigue states and degrees during flight of the flight trainees, so as to evaluate the flight effect of the fatigue state of the flight trainees during the entire flight process during teaching, and achieve the role of early warning of teaching for flight trainees. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 This is a structural diagram of a flight trainee fatigue assessment system based on eye movement data proposed in a specific embodiment of the present invention;

[0023] Figure 2 This is an example diagram of the results of training a deep convolutional generative adversarial network in an embodiment of the present invention;

[0024] Figure 3 This is a diagram of flight prediction simulation results when the flight trainee 1 to be tested is in the same fatigue state in an embodiment of the present invention;

[0025] Figure 4 This is a diagram of flight prediction simulation results when the flight trainee 2 to be tested maintains the same fatigue state in an embodiment of the present invention;

[0026] Figure 5 This is a diagram of flight prediction simulation results when the flight trainee 1 to be tested is kept in a state of progressively increasing fatigue in an embodiment of the present invention;

[0027] Figure 6 This is a diagram of flight prediction simulation results when the flight trainee 2 to be tested is in a state of progressively increasing fatigue in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention. It should be noted that relational terms such as the terms "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0030] Moreover, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of more restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0031] This embodiment proposes a flight trainee fatigue assessment system based on eye movement data, which can simulate the flight results of flight trainees when they are fatigued at different flight stages during the teaching process, thereby increasing the attention paid by flight trainees to the fatigue state during flight and improving flight safety awareness. Among them, the image processing and learning capabilities of the deep convolution generative adversarial network are used to expand the flight eye movement hotspot image data after preliminary processing, thereby reducing the data acquisition cost. Then, according to the characteristics of the eye movement data of the flight trainees collected during teaching, the data of the set eye tracker attention distribution area and attention duration are statistically analyzed. In order to enhance the versatility of the method, the change in the proportion of the effective range of eye movement attention distribution in each interest area is used to statistically calculate the impact of fatigue, thereby replacing the commonly used but difficult to implement fatigue tests on flight trainees, such as psychomotor vigilance test, number of evaluation errors, reaction time test, etc. Finally, the flight effect of the flight trainees when they are given a fatigue state is predicted by the long short-term memory network, thereby increasing the attention paid to flight fatigue during the training of flight trainees.

[0032] Figure 1 It is a structural diagram of a flight trainee fatigue assessment system based on eye movement data according to an embodiment of the present application, which includes a data acquisition module, a data processing module, a flight fatigue status analysis module, and a result output module.

[0033] The implementation process of the data collection module is as follows: build a data collection platform, use a desktop primary flight procedure trainer loaded with Microsoft Flight Simulator 2020 software, a Tobii Glasses 3 Eyetracker and a flight simulation scoring software to collect the pilot's eye tracker data during the flight. A one-hour flight mission is set, and the flight phase is divided into four phases: takeoff, front section, back section, and landing. The operations that the pilot needs to perform at each phase of the mission are clarified. The pilot performs the flight mission on the simulation platform according to the flight mission requirements, and uses the eye movement acquisition device to obtain the pilot's eye movement data during the trial flight mission. The eye movement data is denoised to eliminate the influence of other factors such as the environment, nystagmus, and eye microsaccades. The gaze time in the set attention area is recorded, and various flight data of the simulation platform are integrated during the mission. The gaze area data is extracted to generate a heat map based on the eye movement data in each flight phase. After visualizing the eye tracker data in the form of a heat map, the eye movement fixation points of the corresponding interest area are recorded. Among them, the flight instrument areas where the pilot's attention is distributed are set as follows: attitude indicator and external visual, altimeter, heading indicator, airspeed indicator, vertical speed indicator, and turn indicator.

[0034] The drone pilots who perform flight missions are divided into: pilots with certain flight experience (standard pilots) and pilots who are still learning (flight trainees to be tested); they can master the simulation platform proficiently, and the subjects need to avoid large head turns during the flight mission. Among them, the standard pilot in the fatigue-free state is set to have accumulated flight time of 0 hours on the day; the standard pilot in the moderate fatigue state is set to have accumulated flight time of 4 hours on the day (that is, the flight mission has been repeated four times); the standard pilot in the severe fatigue state is set to have accumulated flight time of 7 hours on the day (that is, the flight mission has been repeated seven times).

[0035] The initial processing process of eye movement data is to generate a corresponding hotspot map for the pilot's eye movement data every minute, perform grayscale processing on the hotspot map of the flight instrument area where attention is distributed, and then perform binarization processing. The color range of the binary image is set to [0,1], and the black area in the binary image is marked as 1, and the white area is marked as 0. The binarized image data is convenient for the subsequent simulation generation of the deep convolutional generative adversarial network, and it is also convenient to count the distribution of each pilot's main attention area and the eye movement range of the pilot in each attention area through the area occupied by the white value of each interest area image. Then, the processed binary hotspot image is input into the deep convolutional generative adversarial network to generate simulation data.

[0036] One of the embodiments records three standard pilots in the states of no fatigue, moderate fatigue, and severe fatigue, and generates one image data for each flight of 1 minute. Invalid data will inevitably appear during the flight. Therefore, in order to balance the data of each flight stage, if invalid data appears during the flight, the data will be cleared, and valid image data will be cut in the remaining time period after the stage of excluding invalid data. Since the operation time of each flight stage is different, in order to ensure the balance of data in each stage, 10 valid image data are recorded in the four stages of takeoff, front stage, back stage, and landing. Then the label of the image of the takeoff stage is recorded as 1, the image of the front stage is recorded as 2, the image of the back stage is recorded as 3, and the image of the landing stage is recorded as 4. The above labeled data is input into the deep convolutional generative adversarial network for data expansion processing.

[0037] Before data expansion, the data is standardized, and then the parameters of the deep convolutional generative adversarial network are set. Specifically, the batch size is set to 4, the noise is set to 100, the learning rate of d_optimizer is set to 0.0002, the learning rate of g_optimizer is set to 0.0002, the iteration is 1500 generations, and the L1Loss loss function is selected as the evaluation value of the model training results. Figure 2This is the loss function result evaluation diagram after the deep convolution generative adversarial network training is completed. After the network training is completed, the loss values ​​of the generative network and the adversarial network are close to 0. Then the softmax function is directly used as the output layer to output the classification results of the network for the simulated data to verify the network training effect. The trained network model is used to expand the image data, and the real 120 image data are expanded into a simulated data set of 120 real images and 120 generated images, that is, the image data in each fatigue state is doubled. Because the eye movement data of pilots in fatigue state not only has the characteristics of shortened attention time, but also shows that the attention range is reduced due to long-term attention to a certain point due to mental fatigue. Therefore, the effective attention area of ​​the pilot in the flight instrument attention area can be characterized by the binary image. When the standard pilot stares at a certain effective area for a long time due to fatigue, his attention range becomes smaller due to the small fluctuation of the eye movement data. Therefore, the characteristics of the binary image are used to count the changes in the eye movement data range of the standard pilot under different degrees of fatigue. After taking the average of the attention area of ​​each attention area in the four flight stages, the eye movement data under moderate and severe fatigue levels are compared with the data without fatigue. By comparison, the influence ratio of eye movement data and fatigue degree is calculated based on the change of the effective range of attention, so as to obtain the change of the flight instrument area of ​​each attention distribution when the fatigue degree affects the eye movement data. From the data set generated in this embodiment, it is analyzed that the effective area of ​​the pilot's planned attention area will be changed in the defined moderate fatigue state as follows: the attitude instrument and the external visual are reduced by 6.17%, the altimeter is increased by 1.12%, the heading indicator is reduced by 0.03%, the airspeed indicator is reduced by 0.13%, the vertical speed indicator is reduced by 0.04%, and the turn indicator is increased by 0.03%; the effective area of ​​the pilot's planned attention area will be changed in the defined severe fatigue state as follows: the attitude instrument and the external visual are reduced by 11.45%, the altimeter is reduced by 0.07%, the heading indicator is increased by 0.11%, the airspeed indicator is reduced by 0.17%, the vertical speed indicator is reduced by 0.07%, and the turn indicator is reduced by 0.02%;

[0038] Then, the correlation values ​​of each attention area that affects the flight effect are calculated, and the proportion and effective total duration of all attention distribution areas on the instrument in each flight stage of the standard pilot in the fatigue-free state are counted, and the eye movement data that mainly affects the pilot's flight effect is analyzed, and this area is used as the area to be image processed when the flight trainee is given a fatigue state. In this embodiment, it is concluded through time analysis that in each flight process, the instrument areas that the pilot pays attention to are the attitude instrument and the external visual allocation and airspeed indicator areas. Therefore, the influence ratio of different fatigue levels on the attitude instrument and the external visual allocation and airspeed indicator areas is assigned to the eye movement data of the flight trainee. The specific steps are as follows: in the flight fatigue status analysis module, according to the proportion of eye movement data changes at different fatigue levels, the binary heat map of the take-off, front section and back section of the flight trainee to be tested is assigned after being processed by the data processing module; using the characteristics of the binary image, according to the proportion of fatigue level affecting the eye movement data, if the influence of the fatigue state increases the area ratio of the attention area, then the range of valid values ​​in the area marked as the trainee's eye movement record in the image is expanded; after the binary image is matrixed, the edge of the white range is expanded by changing the peripheral 1 of the white range of the corresponding attention area range in the matrix to 0; if the influence of the fatigue state reduces the area ratio of the attention area, then the range of valid values ​​in the area marked as the trainee's eye movement record in the image is reduced; after the binary image is matrixed, the edge of the white range is reduced by changing the peripheral 0 of the white range of the corresponding attention area range in the matrix to 1. In addition, in order to highlight the attention area that affects the pilot's actual flight performance, in the embodiment, only the gaze area of ​​the instrument area that mainly affects the pilot's flight performance is assigned a value, that is, the attention area of ​​the binary eye movement image data of the two areas of the flight student's attitude instrument and the external visual distribution and airspeed indicator is correspondingly expanded or reduced.

[0039] After the above-mentioned assignment is made to the eye movement data of the flight trainee to be tested, the data of the standard pilot in the non-fatigue state is used as a prediction training set and input into the long short-term memory neural network for fitting. Then, the data of the flight trainee to be tested after the fatigue assignment is input into the trained long short-term memory network fitting model for prediction. Each flight trainee to be tested may present one or more fatigue states in different flight stages, that is, the flight trainee to be tested may be free of fatigue during takeoff, moderately fatigued in the first part of the flight, severely fatigued in the second part of the flight, or may maintain a moderate fatigue state during takeoff, the first part, and the last part of the flight.

[0040] The specific prediction steps are to use the evaluation values ​​of ten image data of takeoff, front section, back section and landing of the flight trainee to be tested as the fitting standard of the trainee, and take the attitude instrument and external visual, altimeter, heading table, airspeed table, vertical speed table, turn meter, etc. as feature values, wherein the similarity evaluation of each region under the standard data of the standard pilot in the fatigue-free state in each flight period is used as the feature evaluation value, and each image data is evaluated for similarity with the corresponding image data of the standard pilot in the fatigue-free state in the same time series as the flight result value, the higher the similarity, the higher the evaluation, and the range of the result value is limited to [0,100]. Then replace the data of the front section and the back section with the eye movement data assigned with the moderate fatigue state, generate a new evaluation value for the replaced data, and use the trained long short-term memory network to predict the subsequent operation results of the flight trainee after the data is replaced, and then replace the data of the front section and the back section with the evaluation value of the eye movement data assigned with the severe fatigue state, perform the same operation, and predict the subsequent operation results of the flight trainee to be tested, refer to Figure 3-Figure 4 The evaluation results of the flight trainee under no fatigue, simulated moderate fatigue state, and simulated severe fatigue state are plotted and output together with the actual flight results, thereby realizing the simulation of the flight effect of the flight trainee under fatigue state. Figure 3-Figure 4 Two typical student data are used as examples. Student 1's eye movement data of the whole flight in a fatigue-free state maintains a high similarity with the standard eye movement data, while Student 2's eye movement data of the whole flight in a fatigue-free state maintains a large floating level with the standard eye movement data, but the final evaluation result is qualified for flight. Among them, the long short-term memory network model includes LSTM layers and Dense layers established in sequence. When training the network, it is necessary to initialize the parameters of the long short-term memory network model first, set the network batch size to 4, the number of network hidden layers to 100, epochs to 200, and dropout to 0.43, and take the characteristic evaluation data of the takeoff and the front section of the student to be tested in a fatigue-free state as input to obtain the output prediction value, compare the predicted value with the actual flight result value in the fatigue-free state, and modify the parameters of the long short-term memory network model according to the comparison result until the error of the comparison result is within the set range.

[0041] Similarly, using the above method, the eye movement data of the flight trainee to be tested is used in a method of gradually increasing fatigue state to simulate the flight effect of the flight trainee to be tested in a gradually increasing fatigue state during flight, that is, no fatigue state in the takeoff stage, moderate fatigue in the first stage, and severe fatigue in the second stage, and then generate the flight result values ​​of the trainees with gradually increasing fatigue degree and no fatigue state. In this embodiment, student 1 and student 2 are still taken as examples, and the final simulation results are referenced Figure 5-Figure 6 .

[0042] Through the above embodiments, it can be seen that the present invention can reduce the cost of obtaining eye movement data of pilots in a fatigue state, and realize the impact of fatigue state of flight trainees on the flight process when performing flight missions through prediction, thereby increasing the attention of flight trainees to fatigue during training, improving trainees' awareness of safe flight, and reducing the difficulty of judging the fatigue state of flight trainees through model prediction.

[0043] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

[0044] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A flight trainee fatigue assessment system based on eye movement data, characterized in that: It includes data acquisition module, data processing module, flight fatigue status analysis module and result output module; The data acquisition module is used to build an eye movement data acquisition and storage platform, collect flight data of standard pilots under different fatigue levels and flight data of flight trainees to be tested, and generate a hotspot map of the pilots' eye movement data on the flight instrument area at different flight stages according to the time sequence of the collected eye movement data of all pilots; The data processing module is used to perform preliminary processing on the heat map, and then use a deep convolutional generative adversarial network to simulate the heat map after processing the standard pilot data under different fatigue states, and generate a simulation database, expand the pilot's fatigue data based on the simulation database, and obtain the fatigue correlation and influence ratio of the fatigue degree and eye movement data through the expanded data analysis, and calculate the attention distribution area that affects the flight effect and the changes in the distribution of eye movement data under different fatigue degrees; The fatigue correlation and influence ratio between the fatigue degree and the eye movement data are obtained through the expanded data analysis, including: expanding the fatigue data of the standard pilot based on the simulation database, using the simulated and actual data sets of the standard pilots in different fatigue states as statistical samples, statistically analyzing the changes in the eye movement data of the standard pilots in different fatigue states, and comparing the eye movement data under moderate fatigue and severe fatigue with the eye movement data under no fatigue, analyzing and calculating the influence ratio between the eye movement data and the fatigue degree, thereby obtaining the changes in the attention distribution of each flight instrument area when the fatigue degree affects the eye movement data; When calculating the correlation values ​​of various attention areas that affect flight performance, the proportions and effective total durations of all attention distribution areas on the instrument at various flight stages in the standard pilot's fatigue-free state are counted to obtain the attention distribution area that affects flight performance, and this area is used as the area to be processed when the fatigue state is given to the flight trainee to be tested; The flight fatigue state analysis module is used to assign different fatigue degree influence ratios to the eye movement data of the flight trainee at different stages, use the long short-term memory neural network to predict the later flight state of the assigned data, and use the prediction result as the flight result of the flight trainee under fatigue state; The method for assigning the influence ratio of different fatigue levels to the eye movement data of the flight trainee at different stages includes: performing corresponding processing on the flight trainee at the take-off, front section and back section stages according to the influence ratio of different fatigue levels on the eye movement data, assigning a value to the attention distribution area that affects the flight effect in the binary heat map of the flight trainee after being processed by the data processing module, and utilizing the characteristics of the binary image to perform marginal expansion of the eye movement range in the area marked as the eye movement record of the trainee in the image if the influence ratio increases according to the influence ratio of the fatigue level, and marginal reduction of the eye movement range in the area marked as the eye movement record of the trainee in the image if the influence ratio decreases; The method of using the long short-term memory neural network to predict the later flight state of the assigned data includes: inputting the eye movement data of the standard pilot in a non-fatigue state as a training set into the long short-term memory neural network for fitting training, and then inputting the data of the flight trainee to be tested after the fatigue assignment into the trained long short-term memory network fitting model for prediction, wherein each flight trainee to be tested presents a fatigue state or a progressive fatigue state in different flight stages, that is, the flight trainee to be tested has no fatigue in the take-off stage, moderate fatigue in the first stage of the flight, and severe fatigue in the second stage of the flight, or the flight trainee to be tested maintains a moderate fatigue state in both the first stage and the second stage; The result output module is used to output the flight results of the flight trainee in a normal state and a simulated fatigue state.

2. The flight trainee fatigue assessment system based on eye movement data according to claim 1, characterized in that: The eye movement data collection and storage platform collects the pilot's eye movement data through an eye tracker and a desktop flight procedure trainer, and sets the flight instrument area where the pilot's attention is distributed, wherein the flight instrument area includes an attitude indicator and external visual, an altimeter, a heading indicator, an airspeed indicator, a vertical speed indicator, and a turn indicator.

3. The flight trainee fatigue assessment system based on eye movement data according to claim 2, characterized in that: The flight phases are divided into four stages: take-off, front stage, rear stage, and landing. The standard pilot fatigue level is divided into three types: no fatigue, moderate fatigue, and severe fatigue.

4. The flight trainee fatigue assessment system based on eye movement data according to claim 3, characterized in that: In the data processing module, the heat map is gray-scaled and then binarized to obtain the distribution of attention areas of each pilot, and the eye movement range of the pilots in different attention distribution areas is counted. The processed binary heat map is then input into the deep convolutional generative adversarial network to generate a simulation database.

5. The flight trainee fatigue assessment system based on eye movement data according to claim 4, characterized in that: The steps of using deep convolutional generative adversarial networks to generate a simulated database are as follows: standardize input data; initialize network values, and mark the heat maps of standard pilots’ no fatigue, moderate fatigue, and severe fatigue levels processed by the data processing module and input them into the initialization network; A custom loop training network is used to determine whether the network training is complete through the loss value curves of the generator and the discriminator. If the loss value curve meets expectations, the image output generated by the adversarial network is generated as the simulation database, and the network training effect is verified by classifying the simulated data.

6. The flight trainee fatigue assessment system based on eye movement data according to claim 1, characterized in that: The eye movement data of the standard pilot in a non-fatigue state is used as a standard to evaluate the flight result values ​​of the flight trainees to be tested in a normal state and a fatigue state, and a flight result graph of each flight trainee to be tested in a non-fatigue state and a fatigue state is generated.

Citation Information

Cited By

  • Rapid fatigue analysis system based on eye movement data

    CN122251010A