Pilot cognitive overload prevention and control method based on Bayesian combined classifier

Abstract

The invention provides a pilot cognitive overload prevention and control method based on a Bayesian combined classifier, which is based on pilot multi-modal physiological data (eye movement data and electroencephalogram data), and outputs a predicted cognitive overload condition and a prevention and control strategy through a combined classifier obtained by fusing a Bayesian estimation method with KNN, SVM, DT, RF and BP algorithms. The dynamic regulation and control capability enables the auxiliary system to provide personalized support and feedback according to task situations of different levels and changes, and helps a pilot to effectively process information and make a correct decision, so that the cognitive burden of the pilot is remarkably relieved, the operation efficiency and the flight safety are improved, and the flight safety is improved. And task requirements of complex air combat scenes can be met, and improvement of flight safety and optimization of pilot performance are facilitated, so that a more efficient man-machine cooperation mode is realized.

Description

Technical Field

[0001] The present invention relates to the field of human factors engineering, and in particular to research on prediction and response methods for pilot cognitive overload by an aircraft intelligent assistance system, and specifically to a pilot cognitive overload prevention and control method based on a Bayesian combination classifier. Background Art

[0002] The pilot's cognitive state refers to an internal state generated during the pilot's interaction with external things, including cognitive processes such as perception, thinking, memory, and decision-making. Cognitive overload refers to the psychological burden and pressure generated when the amount of information received by an individual exceeds his or her cognitive processing ability. This situation usually occurs when the task is too complex or the amount of information is too large, resulting in the individual being unable to effectively process and understand the received information.

[0003] With the rapid development of technology, the number of control tasks required of pilots in flight missions such as reconnaissance and weapon deployment has gradually increased. They may need to process hundreds of data points at the same time, including sensor data, map information, flight parameters, etc., and collect and analyze a large amount of information in a very short time, make decisions quickly, and perform correct operations to deal with complex aerial scenarios. Pilots need to quickly collect and analyze a large amount of information, make timely decisions, and perform correct operations to deal with complex aerial scenarios. This highly complex and dynamic scenario makes pilots face a huge information load and mental workload at all times, which is a huge challenge to their cognitive abilities. Therefore, the number of pilots experiencing cognitive overload, which in turn leads to an increase in safety risks.

[0004] In order to prevent and control the cognitive overload of pilots, a variety of methods and technologies are integrated into the onboard avionics system, especially through the aircraft's intelligent assistance system to help pilots reduce their cognitive burden. For example, the autopilot can take over the flight mission during most flight phases, reducing the pilot's operational burden; the flight management system can help pilots plan and manage flight routes, fuel consumption, etc., reducing the pilot's calculation and planning burden. Existing methods for preventing and controlling pilot cognitive overload mainly focus on reducing the pilot's cognitive burden through automation and auxiliary systems in onboard avionics systems. However, these methods often have some limitations and shortcomings:

[0005] (1) Over-reliance on automated systems: Although autopilots and flight management systems can reduce the operational burden on pilots, over-reliance on these systems may lead to a decline in pilot operational skills. When the system faces complex or emergency situations, pilots may not be able to intervene and respond in time, increasing flight safety risks.

[0006] (2) Single decision logic: The decision logic of existing systems is usually fixed or based on simple rules, which cannot dynamically adapt to different levels and changing mission situations. This static processing method may not be able to effectively cope with complex and dynamic air situations.

[0007] (3) Lack of personalization and predictability: Existing systems have limited ability to identify and predict the pilot's status and cognitive load. They often react based on static thresholds or simple input conditions and lack personalization and predictability. Summary of the invention

[0008] In order to solve the problems and shortcomings of the existing methods, the present invention provides a pilot cognitive overload prevention and control method based on a Bayesian combined classifier. The method is based on the pilot's multimodal physiological data (eye movement data, EEG data), and a combined classifier obtained by fusing KNN, SVM, DT, RF, and BP algorithms through the Bayesian estimation method to output the predicted cognitive overload situation and prevention and control strategy.

[0009] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0010] A method for preventing and controlling pilot cognitive overload based on a Bayesian combined classifier, characterized by comprising the following steps:

[0011] Step 1: Data collection and data label determination:

[0012] Step 1.1: First, prepare the multimodal physiological data features for training the combined classifier and determine the data labels, that is, the cognitive overload levels corresponding to different data groups;

[0013] Step 1.2: According to the five-level label quantification method based on subjective evaluation and objective performance, complete the label determination of the cognitive overload level corresponding to the pilot's physiological data characteristics for a certain task;

[0014] Step 2: The combined classifier obtained by fusing KNN, SVM, DT, RF, and BP algorithms through the Bayesian estimation method outputs the predicted cognitive overload situation and prevention and control strategies.

[0015] Furthermore, in step 1.1, the multimodal physiological data features include 16 eye movement and EEG features, and their specific definitions are shown in Table 1:

[0016] Table 1 Definition of eye movement and EEG features

[0017]

[0018]

[0019] Furthermore, in step 1.2, the five-level label quantification method based on subjective evaluation and objective performance includes the following steps:

[0020] Step 1.2.1: Subjective evaluation, which consists of two parts: pilot self-evaluation and expert evaluation, using the NASA-TLX scale and OBX scale designed for pilots respectively;

[0021] Step 1.2.2: Objective performance, which consists of two parts: primary task performance and secondary task performance, which are the performance scores of the flight control task and the cognitive test task respectively;

[0022] Step 1.2.3: Determine the weights of different indicators based on the hierarchical analysis method and establish a five-level label quantification model.

[0023] Furthermore, in step 1.2.1,

[0024] The NASA-TLX scale consists of 6 dimensions, including the mental demands, physical demands, time demands, effort, performance level, and frustration level of the pilot. Each dimension is divided into 21 levels. The pilot needs to mark the corresponding level of each dimension in the form of "×" according to the actual situation of the task. The score calculation formula of the NASA-TLX scale is as follows:

[0025]

[0026] In the formula,

[0027] E NASA-TLX is the cognitive state value obtained through the NASA-TLX scale; n1 is the scale dimension; α1 is the score corresponding to each level; L i is the level marked on the i-th dimension;

[0028] The OBX scale consists of five dimensions, including omission, redundancy, overrun, skewness, and misorder. Each dimension is divided into 21 levels. Experts need to give an evaluation of each dimension based on the pilot's operating behavior, and mark the corresponding level of each dimension in the form of "×". The score calculation formula of the OBX scale is as follows:

[0029]

[0030] In the formula,

[0031] E OBX is the cognitive state value obtained through the OBX scale; n2 is the scale dimension; α2 is the score corresponding to each level; M i is the level marked on the i-th dimension.

[0032] Furthermore, in step 1.2.2,

[0033] The calculation formula of the main task performance is as follows:

[0034]

[0035] In the formula,

[0036] A main is the cognitive state value obtained through the main task performance; n3 is the scale dimension; γ1 is the score corresponding to each dimension; R i is the percentage rate obtained by statistics on the i-th dimension;

[0037] When the task execution time does not exceed the set time, the performance evaluation value is 0%; when the task resource utilization rate does not exceed the set range, the performance evaluation value is 0%;

[0038] The calculation formula of sub-task performance is as follows:

[0039]

[0040] In the formula,

[0041] A Secondary is the cognitive state value obtained through the sub-task performance; n4 is the scale dimension; γ2 is the score corresponding to each dimension; T i is the percentage rate obtained by statistics on the i-th dimension.

[0042] Furthermore, in step 1.2.3, the construction of the model is divided into the following four steps:

[0043] a. Establish a hierarchical model:

[0044] The target layer was set as the comprehensive assessment score of cognitive status, and the criterion layers were set as NASA-TLX scale score, OBX scale score, flight control task score, and cognitive test task score;

[0045] b. Construct a judgment matrix:

[0046] First, invite flight experts to construct a judgment matrix A = (a ij ) n×n ,a ij >0, Then use the square root method to solve the approximate value of the eigenvector of the judgment matrix and obtain the maximum eigenroot of the judgment matrix:

[0047]

[0048] c.Consistency check:

[0049] The collected judgment matrices are checked for consistency, and only matrices with consistency within the allowed range are retained, and the rest are eliminated. The consistency ratio is defined as CR:

[0050]

[0051] in,

[0052] CI is an indicator for measuring the deviation of the judgment matrix from consistency;

[0053] RI is the average random consistency index;

[0054] When CR is less than 0.1, the normalized eigenvector is used as the weight vector after the consistency check; when CR is greater than or equal to 0.1, it is considered that the inconsistency of the judgment matrix is ​​not within the permissible range and the matrix is ​​discarded;

[0055] d. Five-level label quantization model:

[0056] The arithmetic average of all the retained m weight vectors is the final weight value of the four scores of NASA-TLX scale, OBX scale, flight control task, and cognitive test task, that is:

[0057]

[0058] Define β1 as the NASA-TLX scale weight value, β2 as the OBX scale weight value, β3 as the flight control task weight value, and β4 as the cognitive test task weight value. Then the comprehensive cognitive overload assessment score S is:

[0059] S=β1E NASA-TLX +β2E OBX +β3A main +β4A Secondary

[0060] The comprehensive score ranges from [0,100]. According to the definition of the value, the higher the score, the more severe the corresponding cognitive overload. On the contrary, the lower the score, the milder the corresponding cognitive overload.

[0061] According to application requirements, cognitive overload is divided into 5 levels, and the cognitive overload level can be expressed as follows:

[0062]

[0063] Furthermore, the step 2 comprises the following steps:

[0064] Step 2.1: Input the multimodal data with the determined cognitive overload level labels into the KNN, SVM, DT, RF, and BP algorithms for training to obtain five weak classifiers;

[0065] Step 2.2: Use the Bayesian estimation strategy to integrate multiple classifiers into a strong classifier to obtain a combined classifier;

[0066] Step 2.3: Use the combined classifier to output the prediction of cognitive overload level and the regulation of prevention and control strategies.

[0067] Furthermore, the output result of the combined classifier is the cognitive overload level with the maximum predicted probability, that is:

[0068] P(A|B1,B2,B3,B4,B5)=max[P(A i |B1,B2,B3,B4,B5)]

[0069] In the formula,

[0070]

[0071] P(B1,B2,B3,B4,B5|A i )=P(B1|A i )P(B2|A i )P(B3|A i )P(B4|A i )P(B5|A i );

[0072] P(B j |A i ) means that when the actual prediction target is A i Under the condition that j probability;

[0073] P(A i ) represents the ratio of the number of samples with the actual cognitive overload level of the i-th category to the total number of samples, and A i Satisfy ∑P(A i )=1;

[0074] B j is the output of the jth classifier, and the occurrence of the actual state is represented by events A1, A2, …, A5.

[0075] The beneficial effects of the present invention are as follows:

[0076] 1. The present invention realizes the dynamic prediction and evaluation of the pilot's cognitive overload level through the Bayesian combination classifier, and the automation level of the auxiliary system is adaptively and dynamically regulated based on real-time multimodal physiological parameter information. This dynamic regulation capability enables the auxiliary system to provide personalized support and feedback according to different levels and changing mission scenarios, helping pilots to effectively process information and make correct decisions, thereby significantly reducing the pilot's cognitive burden, improving operational efficiency and flight safety, and meeting the mission requirements of complex air combat scenarios, which has a significant promoting effect on improving human-machine collaboration capabilities and promoting the development of human-machine collaboration models.

[0077] 2. The present invention adopts the subjective and objective hierarchical analysis method to establish a comprehensive evaluation model of mission scenarios, which makes up for the deficiency of the single decision-making logic of the existing system. The model comprehensively considers the weights of multiple evaluation indicators and dynamically adapts to different flight mission scenarios, so that the system can handle complex and dynamic air situations more intelligently, thereby improving the pilot's ability to cope with complex and changeable battlefield environments, and providing a basis for improving human-machine collaboration capabilities. By combining the subjective evaluation method and the objective performance method, the present invention takes into account the subjective experience of cognitive state and the performance of mission performance. The weights of the evaluation indicators are determined by the hierarchical analysis method, which reduces the influence of subjective factors on the evaluation results and improves the objectivity and accuracy of the evaluation. Through the five-level cognitive overload level prediction, the system can provide more accurate mission scenario analysis and evaluation and more accurate decision support, effectively respond to various challenges in flight, and minimize the risk of pilot cognitive overload.

[0078] 3. This invention provides important support for the study of human-machine collaborative mode. The pilot cognitive overload prevention and control method based on Bayesian combination classifier helps to improve flight safety and optimize pilot performance, thereby achieving a more efficient human-machine collaborative mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is the NASA-TLX scale;

[0080] Figure 2 For the OBX scale. DETAILED DESCRIPTION

[0081] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention.

[0082] In order to solve the problems and shortcomings of the existing methods, the present invention provides a pilot cognitive overload prevention and control method based on a Bayesian combination classifier, which evaluates the pilot's cognitive overload level through multimodal physiological data to avoid the problem of pilots' over-reliance on the automation system; establishes a comprehensive evaluation model of task scenarios through the subjective and objective hierarchical analysis method to solve the shortcomings of a single decision-making logic; predicts the pilot's cognitive overload based on the Bayesian combination classifier and regulates the automation level of the auxiliary system, so that the auxiliary system can perform dynamic adaptive assistance according to different levels of task conditions, thereby achieving prevention and control of pilot cognitive overload.

[0083] The technical solution adopted by the present invention comprises the following steps:

[0084] Step 1: Data collection and data label determination

[0085] (1) Data Collection

[0086] First, prepare the multimodal physiological data features for training the combined classifier; there are a total of 16 eye movement and EEG features used in this embodiment, and their specific definitions are shown in Table 1.

[0087] Table 1 Definition of eye movement and EEG features

[0088]

[0089]

[0090] (2) Data label determination

[0091] Based on the multimodal physiological data features collected above, the data labels, i.e., the cognitive overload levels corresponding to different data groups, are determined through the following process.

[0092] In this embodiment, a five-level label quantification method based on subjective evaluation and objective performance is adopted. The subjective evaluation consists of two parts: pilot self-evaluation and expert evaluation, which are scored using the NASA-TLX (Task Load Index) scale and OBX (Operation behaviors Index) scale designed for pilots respectively; objective performance consists of two parts: primary task performance and secondary task performance, which are performance scores for flight control tasks and cognitive test tasks respectively. Finally, based on the hierarchical analysis method, the weights of different indicators are determined, and a five-level label quantification model is established to complete the label determination of the cognitive overload level corresponding to the physiological data characteristics of the pilot for a certain task.

[0093] ① Subjective evaluation: pilots’ self-evaluation

[0094] Have the pilot fill out the NASA-TLX scale ( Figure 1) to conduct a subjective evaluation of their cognitive state during the flight mission. The scale consists of 6 dimensions, including the mental demand, physical demand, time demand, pilot's effort, performance level and frustration level. Each dimension is divided into 21 levels. Pilots need to mark the corresponding level of each dimension in the form of "×" according to the actual situation of the task.

[0095] The NASA-TLX scale score calculation formula is as follows:

[0096]

[0097] In the formula, E NASA-TLX is the cognitive state value obtained by the NASA-TLX scale; n is the scale dimension, here n = 6; α is the score corresponding to each level, here α = 5; L i is the level marked on the i-th dimension, and its value is [0,20].

[0098] ② Subjective evaluation - expert evaluation

[0099] Experts with rich flight experience were asked to fill in the OBX scale based on the pilots' operational performance ( Figure 2 ) to conduct a subjective evaluation of their cognitive state during the flight mission. The scale consists of 5 dimensions, including omission, redundancy, overlimit, skewness, and misorder. Each dimension is divided into 21 levels. Experts need to give an evaluation of each dimension based on the pilot's operating behavior, and mark the corresponding level of each dimension in the form of "×". The score calculation formula of the OBX scale is as follows:

[0100]

[0101] In the formula, E OBX is the cognitive state value obtained through the OBX scale; n is the scale dimension, here n = 5; α is the score corresponding to each level, here α = 5; M i is the level marked on the i-th dimension, and its value is [0,20].

[0102] ③Objective performance-main task performance method

[0103] The main task refers to the flight control task. The evaluation of task performance is divided into four dimensions, as shown in Table 2.

[0104] Table 2 Main task performance evaluation indicators

[0105]

[0106] The calculation formula of the main task performance is as follows:

[0107]

[0108] In the formula, A main is the cognitive state value obtained through the main task performance; n is the scale dimension, here n = 4; β is the score corresponding to each dimension, here β = 100; R i is the percentage rate obtained by statistics on the i-th dimension, and its value is [0,1]. It is worth noting that when the task execution time does not exceed the set time, the performance evaluation value is 0%; similarly, when the task resource utilization rate does not exceed the set range, the performance evaluation value is 0%.

[0109] ④Objective performance-sub-task performance method

[0110] The secondary task refers to a cognitive test task, which is an additional task that the pilot needs to perform on the premise that the flight mission can be completed, and is intended to increase the difficulty of operation through the secondary task, thereby inducing different cognitive states of the pilot. The cognitive test tasks specified in the present invention include 4 items, as shown in Table 3.

[0111] Table 3 Task performance evaluation indicators

[0112]

[0113] The calculation formula of sub-task performance is as follows:

[0114]

[0115] In the formula, A Secondary is the cognitive state value obtained through the sub-task performance; n is the scale dimension, here n = 4; β is the score corresponding to each dimension, here β = 100; T i It is the percentage rate obtained by statistics on the i-th dimension, and its value is [0,1].

[0116] ⑤ Five-level label quantification model based on analytic hierarchy process weighting

[0117] Since the four evaluation values ​​have different weights for cognitive status, the hierarchical analysis method is used to determine the weights of the four scores of NASA-TLX scale, OBX scale, flight control task, and cognitive test task, so as to construct a five-level label quantification model of cognitive status. The model construction is divided into the following four steps:

[0118] a. Establish a hierarchical model

[0119] The target layer was set as the comprehensive assessment score of cognitive status, and the criterion layers were set as NASA-TLX scale score, OBX scale score, flight control task score, and cognitive test task score.

[0120] b. Construct judgment matrix

[0121] Invite flight experts to construct a judgment matrix A = (a ij ) n×n ,a ij >0, The judgment scale and its meaning are shown in Table 4 below:

[0122] Table 4 Judgment scale and its meaning

[0123]

[0124] The square root method is used to find the approximate value of the eigenvector of the judgment matrix. The steps are as follows:

[0125] Calculate the nth root of the product of each row of the judgment matrix A:

[0126]

[0127] M i Normalized, the formula is as follows:

[0128]

[0129] Calculate the maximum eigenvalue of the judgment matrix:

[0130]

[0131] c.Consistency check

[0132] The collected judgment matrices are checked for consistency, and only matrices with consistency within the allowed range are retained, and the rest are discarded. The steps for checking the consistency of the judgment matrix are as follows:

[0133] Define the consistency ratio CR:

[0134]

[0135] Where CI is the indicator of deviation of the judgment matrix from consistency, which is defined as:

[0136]

[0137] RI is the average random consistency index, and its value can be found in the following table:

[0138] Table 5 Values ​​of random consistency index RI

[0139]

[0140] In the present invention, when CR < 0.1, it is considered that the inconsistency of the judgment matrix is ​​within the permissible range, and its normalized eigenvector can be used as the weight vector through consistency check. When CR ≥ 0.1, it is considered that the inconsistency of the judgment matrix is ​​not within the permissible range, and the matrix is ​​discarded.

[0141] d. Five-level label quantization model

[0142] The arithmetic average of all the retained m weight vectors is the final weight value of the four scores of NASA-TLX scale, OBX scale, flight control task, and cognitive test task, as shown in the following formula:

[0143]

[0144] Assume that β1 is the weight value of NASA-TLX scale, β2 is the weight value of OBX scale, β3 is the weight value of flight control task, and β4 is the weight value of cognitive test task. Then the comprehensive cognitive overload assessment score S is:

[0145] S=β1E NASA-TLX +β2E OBX +β3A main +β4A Secondary (1-11)

[0146] The range of the comprehensive score is [0,100]. According to the definition of the value, the higher the score, the more severe the corresponding cognitive overload; on the contrary, the lower the score, the milder the corresponding cognitive overload. According to the application requirements, cognitive overload is divided into 5 levels, based on which the cognitive overload level (Level of CognitiveOverload) corresponding to the data feature can be determined according to formula (1-12).

[0147]

[0148] The detailed definitions of cognitive overload levels are shown in Table 6.

[0149] Table 6 Definition of cognitive overload level and auxiliary system regulation mechanism

[0150]

[0151] Step 2: Design and application of combined classifier based on Bayesian estimation method

[0152] The combined classifier obtained by fusing the K-nearest neighbors (KNN), support vector machine (SVM), decision tree (DT), random forest (RF), and back propagation neural network (BP) algorithms using the Bayesian estimation method outputs the predicted cognitive overload situation and prevention and control strategies.

[0153] The multimodal data with the determined cognitive overload level labels were input into the KNN, SVM, DT, RF, and BP algorithms for training, and five weak classifiers were obtained. The Bayesian estimation strategy was then used to integrate the multiple classifiers into a strong classifier to obtain a combined classifier. The combined classifier was used to output the prediction of cognitive overload level and the regulation of prevention and control strategies.

[0154] (1) Weak classifier parameter setting

[0155] The optimal parameters are determined by grid search combined with cross-validation. The parameter settings of KNN, SVM, DT, RF, and BP algorithms obtained by optimization in the present invention are shown in Table 7.

[0156] Table 7 Algorithm parameter settings

[0157]

[0158] (2) Multi-classifier Bayesian estimation method

[0159] ①Bayesian basic theory

[0160] Prior knowledge: P(A1), P(A2), …, P(A n ) represents events A1, A2, ..., A n Probability of occurrence.

[0161] A posteriori knowledge: The appearance of a test result B changes events A1, A2, ..., A n The probability of occurrence. After testing, events A1, A2, ..., A n The probability of occurrence is expressed as conditional probability: P(A1|B), P(A2|B),…, P(A n |B), and P(A i |B)≥0,∑P(A i |B)=1.

[0162] Conditional probability formula:

[0163]

[0164] The total probability formula is:

[0165]

[0166] In the formula, A i is a partition of the sample space, that is, A i are mutually exclusive events and ∑P(A i )=1.

[0167] For a set of mutually exclusive events A1, A2, ..., A n When event B occurs, A i The probability of occurrence is:

[0168]

[0169] ②Combined classifier real-time evaluation model

[0170] Based on Bayesian probability, SVM, KNN, DT, RF and BP are combined to obtain the real-time evaluation result of cognitive overload status output by the combined classifier, which improves the accuracy and robustness of classification.

[0171] The combined classifier of the present invention includes 5 weak classifiers in total, B j is the output of the jth classifier. The actual state is represented by events A1, A2, …, A5, and its prior probability P(A i ) represents the ratio of the number of samples with the actual cognitive overload level of the i-th category to the total number of samples, and A i satisfy

[0172] According to the confusion matrix of the jth classifier, the uncertainty P(B j |A i ). P(B j |A i ) means that when the actual prediction target is A i Under the condition that j The probability of B in the confusion matrix j The category sample is A i The decision fusion process of the five weak classifiers is as follows:

[0173] a. Obtain the predicted probability distribution P(B) of each classifier j |A i ),i=1,2,…,5:

[0174] P(B1,B2,B3,B4,B5|A i )=P(B1|A i )P(B2|Ai )P(B3|A i )P(B4|A i )P(B5|A i ) (1-16)

[0175] b. Calculate the posterior prediction probability of the combined classifier:

[0176]

[0177] c. Final decision: The output of the combined classifier is the cognitive overload level with the highest predicted probability.

[0178] P(A|B1,B2,B3,B4,B5)=max[P(A i |B1,B2,B3,B4,B5)] (1-18)

[0179] Embodiments of the present invention are described in detail below. The embodiments are exemplary and intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0180] This embodiment builds a flight control simulation platform for the context of human-machine collaboration, with fighter pilots performing typical combat missions as the research object, and the mission goal is to destroy important enemy military bases through tactical bombing. During the pilot's mission, his eye movement and EEG data are collected, and the real-time cognitive overload level is predicted based on the combined classifier model, and the feedback mechanism of the auxiliary system is regulated to achieve the purpose of cognitive overload prevention and control.

[0181] The specific steps include:

[0182] Step 1: Have pilots perform typical combat missions on a flight control simulation platform, while recording each pilot's eye movements and EEG characteristics.

[0183] The sample implementation process is as follows: according to the actual combat mission requirements, a typical combat mission of tactical bombing of enemy military bases is designed, and 5 subjects are recruited to simulate the driving and control of fighter jets through the flight control simulation platform and complete the designated tasks. After the mission is completed, the task performance of the subjects is counted, and the subjects are asked to fill in the NASA-TLX scale based on the task experience, and the third-party collaborators are asked to fill in the OBX scale based on the task performance of the subjects.

[0184] (1) Design of Typical Combat Missions

[0185] In order to evaluate the pilots' cognitive overload level, the static structures of their combat plans and action plans were fixed, and common classic combat tasks were selected. In order to focus on the changes in the cognitive state of our pilots, the uncontrollable random factors in the game were eliminated, and the force organization, weapon configuration and command relationship were further determined. The corresponding combat background was set up, and through relatively simple single combat experiments, a comprehensive combat experiment was formed in 6 different stages, and finally the design of the combat mission experiment was formed. The design plan is as follows:

[0186] The combat mission goal is to complete tactical bombing to destroy important enemy military bases. The specific combat process is that our aircraft takes off from the aircraft carrier and passes through an enemy defensive threat area. There are enemy ground defense radar facilities in the defensive area, which search and attack our aircraft; after passing through the evasion threat area, our aircraft arrives near the bombing target, carries out tactical bombing missions, and evacuates after completion. The stages of the typical combat scenario experiment for human-machine collaboration are divided into 6 stages, namely: rest stage, take-off stage, evasion stage, test stage, bombing stage, and evacuation stage. The following is a detailed design description of the entire task stage of the typical combat scenario experiment for human-machine collaboration:

[0187] ① Rest phase: At the beginning of the experiment, the subject is in the cockpit, with both hands on the throttle control lever and the flight control stick, and feet lightly placed on the tail control pedals, waiting for the take-off command. The rest phase lasts for 1 minute.

[0188] ② Take-off phase: The test subjects receive the take-off command and take off from an aircraft carrier in the coastal area. During this phase, they mainly control the throttle control lever and the flight operation joystick, and are required to complete the take-off of the aircraft safely and smoothly. They need to climb to a designated altitude of more than 2000m, and maintain a speed of about 350km / h during the climb. During the process, the test subjects are mainly responsible for the control of the aircraft flight, monitoring performance instruments and monitoring navigation instruments, and observing external scenes. The scheduled time for the take-off phase is 5 minutes.

[0189] ③ Avoidance phase: When the fighter enters the threat zone, due to the enemy's anti-aircraft gun attack, our onboard computer predicts a series of accurate advances to form a route plan. At this time, the pilot has two sets of avoidance maneuvers. One is to choose to accelerate in a straight line to avoid, and the other is to constantly adjust the route to avoid. The two plans correspond to different route plans, which can effectively avoid enemy fire. During this stage, while controlling the aircraft, the subjects need to observe external weather, terrain and other environmental information, evaluate the impact of the external environment, choose a suitable avoidance maneuver plan, and choose appropriate waypoints for avoidance flight. The set time for the avoidance phase is 5 minutes.

[0190] ④Testing phase: The testing phase is used to simulate the process of pilots conducting information communication and human-computer interaction when special situations occur in combat scenarios, and to reflect the changes in the pilots' cognitive overload levels during the mission. During this phase, the subjects need to complete cognitive test tasks.

[0191] ⑤ Bombing phase: When flying near the target point, the subject needs to control the aircraft, lower the flight altitude, and increase the aircraft to the maximum speed. When the attack zone prompt appears on the external view of the screen, the pilot needs to control the aircraft to dive at low altitude to approach the target point. When receiving the attack command, press the launch button on the joystick to complete the missile launch, and then control the aircraft to climb and leave. The set time for the avoidance phase is 30 seconds.

[0192] ⑥ Evacuation phase: After completing the bombing phase, evacuate the battlefield according to the scheduled route. The fuel consumption should not exceed 10,000 liters when completing the mission.

[0193] Step 2: Count the cognitive overload level assessment scores of the NASA-TLX scale, OBX scale, flight control tasks, and cognitive test tasks respectively.

[0194] ① The results of the NASA-TLX scale filled in by the subjects are shown in Table 8, and the NASA-TLX scale scores of each subject are calculated according to formula (1-1):

[0195] Table 8 NASA-TLX scale scores

[0196]

[0197] ② The results of the OBX scale filled in by the subjects are shown in Table 9, and the OBX scale scores of each subject are calculated according to formula (1-2):

[0198] Table 9 OBX scale scores

[0199]

[0200] ③ The flight control task performance of the subjects is shown in Table 10, and the main task performance score of each subject is calculated according to formula (1-3):

[0201] Table 10 Main task performance scores

[0202]

[0203] ④ The cognitive test task performance of the subjects is shown in Table 11, and the sub-task performance score of each subject is calculated according to formula (1-4):

[0204] Table 11 Task performance scores

[0205]

[0206] Step 3: Determine the weight values ​​of different indicators, calculate the comprehensive evaluation score of cognitive overload level, and determine the corresponding cognitive overload level of the subjects.

[0207] A total of 3 flight expert construction judgment matrices were collected, as follows:

[0208] Table 12 Judgment matrix constructed by expert 1

[0209]

[0210] Table 13 Judgment matrix constructed by expert 2

[0211]

[0212] Table 14 Judgment matrix constructed by expert three

[0213]

[0214] The three judgment matrices were tested for consistency respectively: the maximum eigenvalue of the judgment matrix constructed by expert one was 5.358, and the CI was 0.45. The corresponding RI value found in the RI table was 0.88, so CR = CI / RI = 0.511>0.1, so the judgment matrix failed the consistency test and was discarded; the maximum eigenvalue of the judgment matrix constructed by expert two was 4.032, and the CI was 0.01. The corresponding RI value found in the RI table was 0.88, so CR = CI / RI = 0.011<0.1, so the judgment matrix passed the consistency test, and its weight vector was: [0.10555, 0.0929, 0.49407, 0.30748]. The maximum eigenvalue constructed by expert three is 4.071, and the CI is 0.024. According to the RI table, the corresponding RI value is 0.88, so CR = CI / RI = 0.027 < 0.1. Therefore, the judgment matrix passes the consistency test, and its weight vector is: [0.39813, 0.35975, 0.12719, 0.11493].

[0215] The two retained weight vectors are arithmetic averaged to obtain the weighted average of the four scores of NASA-TLX scale, OBX scale, flight control task, and cognitive test task (retain four decimal places):

[0216] Table 15 Average weight values

[0217]

[0218] Substituting the weight value into formula (1-11), we can get the comprehensive evaluation score of each subject and the corresponding cognitive overload level label:

[0219] Table 16 Cognitive overload level labels of subjects

[0220]

[0221] Therefore, the data labels are: Subject 1 did not experience cognitive overload, Subject 2 experienced extremely mild cognitive overload, Subject 3 experienced mild cognitive overload, Subject 4 experienced moderate cognitive overload, and Subject 5 experienced extremely severe cognitive overload.

[0222] Step 4: Train the weak classifiers separately and calculate the Bayesian probability to determine the cognitive overload level prediction result of the combined classifier.

[0223] ① After processing and checking, a total of 100 sets of usable data samples are collected. We train weak classifiers accordingly and obtain the predicted probability distribution of each classifier. Set up a three-dimensional matrix prob for storage. Prob is a matrix of size [number of samples × number of categories × number of weak classifiers]. In this example, prob is [100×5×5].

[0224] ② Calculate the posterior prediction probability of the combined classifier, average the prediction probabilities on the third dimension (i.e., each classifier), and obtain a matrix combined_prob of size [number of samples × number of categories], which represents the average prediction probability of the combined classifier for each sample in the training set belonging to each category. In this example, combined_prob is [100×5].

[0225] ③ Make the final classification decision based on the posterior probability of the combined classifier. Take the maximum value for each row (each sample), return the maximum value and its index and store it in the column vector final_pred, where 1 to 5 are used to identify the final predicted cognitive overload level of the sample. 1 means no cognitive overload state, 2 means extremely mild cognitive overload state, 3 means mild cognitive overload state, 4 means moderate cognitive overload state, and 5 means extremely severe cognitive overload state.

[0226] ④Performance verification: 5-fold cross validation was used to test the performance of the combined classifier, and accuracy and confusion matrix were used for performance evaluation.

[0227] ⑤ Application and installation of combined classifiers: When the performance of the combined classifier meets the application requirements, multimodal EEG and eye movement physiological characteristics can be input into it in real time, and the overload prevention and control mechanism of the auxiliary system can be regulated according to the output cognitive overload level.

[0228] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A pilot cognitive overload prevention and control method based on Bayesian combination classifier, characterized in that: The following steps are involved: Step 1: Data collection and data label determination: Step 1.1: First, prepare the multimodal physiological data features for training the combined classifier and determine the data labels, that is, the cognitive overload levels corresponding to different data groups; Step 1.2: According to the five-level label quantification method based on subjective evaluation and objective performance, complete the label determination of the cognitive overload level corresponding to the pilot's physiological data characteristics for a certain task; Step 2: The combined classifier obtained by fusing KNN, SVM, DT, RF, and BP algorithms through the Bayesian estimation method outputs the predicted cognitive overload situation and prevention and control strategies.

2. A pilot cognitive overload prevention and control method based on Bayesian combination classifier as claimed in claim 1, characterized in that: In step 1.1, the multimodal physiological data features include 16 eye movement and EEG features, with specific definitions as shown in Table 1: Table 1 Definition of eye movement and EEG features 3. The pilot cognitive overload prevention and control method based on Bayesian combination classifier according to claim 1 is characterized in that: In step 1.2, the five-level label quantification method based on subjective evaluation and objective performance includes the following steps: Step 1.2.1: Subjective evaluation, which consists of two parts: pilot self-evaluation and expert evaluation, using the NASA-TLX scale and OBX scale designed for pilots respectively; Step 1.2.2: Objective performance, which consists of two parts: primary task performance and secondary task performance, which are the performance scores of the flight control task and the cognitive test task respectively; Step 1.2.3: Determine the weights of different indicators based on the hierarchical analysis method and establish a five-level label quantification model.

4. A pilot cognitive overload prevention and control method based on Bayesian combination classifier as claimed in claim 3, characterized in that: In step 1.2.1, The NASA-TLX scale consists of 6 dimensions, including the mental demands, physical demands, time demands, effort, performance level and frustration level of the pilot. Each dimension is divided into 21 levels. The pilot needs to mark the corresponding level of each dimension in the form of "×" according to the actual situation of the task. The score calculation formula of the NASA-TLX scale is as follows: In the formula, E NASA-TLX is the cognitive state value obtained through the NASA-TLX scale; n1 is the scale dimension; α1 is the score corresponding to each level; L i is the level marked on the i-th dimension; The OBX scale consists of five dimensions, including omission, redundancy, overrun, skewness, and misorder. Each dimension is divided into 21 levels. Experts need to give an evaluation of each dimension based on the pilot's operating behavior, and mark the corresponding level of each dimension in the form of "×". The score calculation formula of the OBX scale is as follows: In the formula, E OBX is the cognitive state value obtained through the OBX scale; n2 is the scale dimension; α2 is the score corresponding to each level; M i is the level marked on the i-th dimension.

5. The pilot cognitive overload prevention and control method based on Bayesian combination classifier according to claim 3 is characterized in that: In step 1.2.2, The calculation formula of the main task performance is as follows: In the formula, A main is the cognitive state value obtained through the main task performance; n3 is the scale dimension; γ1 is the score corresponding to each dimension; R i is the percentage rate obtained by statistics on the i-th dimension; When the task execution time does not exceed the set time, the performance evaluation value is 0%; when the task resource utilization rate does not exceed the set range, the performance evaluation value is 0%; The calculation formula for subtask performance is as follows: In the formula, A Secondary is the cognitive state value obtained through the sub-task performance; n4 is the scale dimension; γ2 is the score corresponding to each dimension; T i is the percentage rate obtained by statistics on the i-th dimension.

6. A pilot cognitive overload prevention and control method based on Bayesian combination classifier as claimed in claim 3, characterized in that: In step 1.2.3, the construction of the model is divided into the following four steps: a. Establish a hierarchical model: The target layer was set as the comprehensive assessment score of cognitive status, and the criterion layers were set as NASA-TLX scale score, OBX scale score, flight control task score, and cognitive test task score; b. Construct a judgment matrix: First, invite flight experts to construct a judgment matrix A = (a ij ) n×n ,a ij >0, Then use the square root method to solve the approximate value of the eigenvector of the judgment matrix and obtain the maximum eigenroot of the judgment matrix: c.Consistency check: The collected judgment matrices are checked for consistency, and only matrices with consistency within the allowed range are retained, and the rest are eliminated. The consistency ratio is defined as CR: in, CI is an indicator for measuring the deviation of the judgment matrix from consistency; RI is the average random consistency index; When CR is less than 0.1, the normalized eigenvector is used as the weight vector after the consistency check; when CR is greater than or equal to 0.1, it is considered that the inconsistency of the judgment matrix is ​​not within the permissible range and the matrix is ​​discarded; d. Five-level label quantization model: The arithmetic average of all the retained m weight vectors is the final weight value of the four scores of NASA-TLX scale, OBX scale, flight control task, and cognitive test task, that is: Define β1 as the NASA-TLX scale weight value, β2 as the OBX scale weight value, β3 as the flight control task weight value, and β4 as the cognitive test task weight value. Then the comprehensive cognitive overload assessment score S is: S=β1E NASA-TLX +β2E OBX +β3A main +β4A Secondary The comprehensive score ranges from [0,100]. According to the definition of the value, the higher the score, the more severe the corresponding cognitive overload. On the contrary, the lower the score, the milder the corresponding cognitive overload. According to application requirements, cognitive overload is divided into 5 levels, and the cognitive overload level can be expressed as follows:

7. The pilot cognitive overload prevention and control method based on Bayesian combination classifier according to claim 1 is characterized in that: The step 2 comprises the following steps: Step 2.1: Input the multimodal data with the determined cognitive overload level labels into the KNN, SVM, DT, RF, and BP algorithms for training to obtain five weak classifiers; Step 2.2: Use the Bayesian estimation strategy to integrate multiple classifiers into a strong classifier to obtain a combined classifier; Step 2.3: Use the combined classifier to output the prediction of cognitive overload level and the regulation of prevention and control strategies.

8. A method for preventing and controlling pilot cognitive overload based on a Bayesian combination classifier as claimed in claim 1 or 7, characterized in that: The output result of the combined classifier is the cognitive overload level with the largest predicted probability, namely: P(A|B1,B2,B3,B4,B5)=max[P(A i |B1,B2,B3,B4,B5)] In the formula, P(B1,B2,B3,B4,B5|A i )=P(B1|A i )P(B2|A i )P(B3|A i )P(B4|A i )P(B5|A i ); P(B j |A i ) means that when the actual prediction target is A i Under the condition that j probability; P(A i ) represents the ratio of the number of samples with the actual cognitive overload level of the i-th category to the total number of samples, and A i Satisfy ∑P(A i )=1; B j is the output of the jth classifier, and the occurrence of the actual state is represented by events A1, A2, …, A5.

Images