SYSTEM FOR DETERMINING THE OPERATIONAL STATE OF AN AIRCRAFT CREW DEPENDING ON AN ADAPTIVE TASK PLAN AND CORRESPONDING PROCEDURE

Description

[0001] The present invention relates to a system for determining the operational state of a crew of a manned or unmanned aircraft, according to an adaptive task plan.

[0002] EP2437033 describes a crew task tracking and task reassignment system.

[0003] The operational status of an aircraft is, in particular, the ability of a crew to perform the tasks of its mission based on its current state.

[0004] This system is designed, in particular, to assess the crew's operational status in real time, considering all the tasks they must perform with the aircraft during a mission. For example, the aircraft may be manned, with the crew present in the cockpit. Alternatively, the aircraft may be unmanned, with the crew in a ground control station for a drone.

[0005] The mission is, for example, a mission of a civil aircraft, such as a commercial or private flight between a geographical point of departure and a geographical point of arrival, in which, during the phases of the mission, numerous tasks must be carried out in accordance with applicable procedures.

[0006] In one variant, the mission is a mission of a military aircraft (manned aircraft or drone), including in particular reconnaissance or combat tasks, for example air-to-ground combat tasks to neutralize targets.

[0007] In such missions, the crew typically follows a task plan that includes a list of tasks to be performed, along with the time at which the task must be completed.

[0008] During a mission, the task plan is adaptive. This means that the list of tasks to be performed is automatically updated by the aircraft and validated by the crew, particularly in light of events occurring during the mission. The crew can also decide to modify the task plan themselves.

[0009] For example, the objectives of the mission plan or flight plan are likely to change, particularly in the event of a diversion for a civil aircraft, or when a new target needs to be dealt with for a military aircraft.

[0010] Similarly, the mission environment is likely to change, for example depending on the weather in the civil or military fields, or the emergence of additional threats in the military field.

[0011] In some cases, the crew's resources, that is, the crew's ability to perform its tasks, are not suited to the task plan to be carried out.

[0012] This can be explained, for example, by the fact that a large number of additional tasks need to be carried out due to a malfunction or emergency on the aircraft. The crew may also be tired or stressed, preventing them from performing their assigned tasks.

[0013] Finally, in some critical cases, the aircraft crew may find themselves overwhelmed by all the tasks to be performed, or conversely, focus too much on a particular task and neglect important tasks that should be accomplished.

[0014] To monitor the crew's condition, it is common practice to equip the aircraft cockpit with sensors that measure the crew's physiological state. However, these sensors only capture the instantaneous state of a crew member, without relating this instantaneous state to the stresses the crew faces in carrying out the mission plan.

[0015] One aim of the invention is to obtain a system for monitoring crew behavior during the execution of a task plan that reliably and dynamically determines the crew's ability to accomplish its mission.

[0016] For this purpose, the invention relates to a system according to claim 1.

[0017] The system according to the invention may comprise one or more of the features of claims 2 to 11, or one or more of the following features taken individually or in any technically feasible combination: The module for determining global cognitive states also determines global cognitive states from the interaction state measurements of the interaction state measurements module; the module for determining local cognitive states also determines local cognitive states from the interaction state measurements of the interaction state measurements module; the interaction states are chosen from at least one level of attention, one level of performance, one task management strategy; the crew resource determination module is designed to assign, via the resource model, for all tasks to be performed, the global cognitive state necessary to perform the tasks, advantageously based on the crew member profile given by the database and the context indicators defined by the context indicator determination module;The crew resource determination module is designed to assign, via the resource model, for each task in the task list, the local cognitive state necessary to perform said task advantageously, based on the crew member's profile.

[0018] The invention also relates to a method according to claim 12.

[0019] The method according to the invention may include one or more of the features of claims 13 and 14.

[0020] The invention will be better understood upon reading the following description, given solely by way of example, and made with reference to the attached drawings, in which: [ Fig 1 ] there figure 1 is a synoptic diagram illustrating a first system for determining the operational crew state according to the invention, integrated within an aircraft; [ Fig 2 ] there figure 2 is a functional diagram illustrating the interactions between the different modules of the system of the figure 1 ; Fig 3 ] there figure 3 is a flowchart illustrating a first example of implementing a process for determining the operational status of a crew via the system of the figure 1 ; Fig 4 ] there figure 4 is a flowchart illustrating a second example of implementing a method for determining the operational status of a crew via the system of the figure 1 ; Fig 5 ] there figure 5 is a flowchart illustrating a third example of implementing a method for determining the operational status of a crew via the system of the figure 1 ; And [ Fig 6 ] there figure 6 is a flowchart illustrating a fourth example of implementing a method for determining the operational status of a crew via the system of the figure 1 . [ Fig 7 ] there figure 7 is a flowchart illustrating a fifth example of implementing a method for determining the operational status of a crew via the system of the figure 1 .

[0021] A first system 10 for determining the operational status of the crew of an aircraft 12, based on a task plan to be carried out by the crew of aircraft 12, is schematically illustrated on the figure 1 .

[0022] The system 10 is intended to be connected to or integrated into a central avionics system 14 comprising a central avionics unit 16 and at least one display unit 18 placed in a control interface 19 of the aircraft 12.

[0023] The control interface 19 of aircraft 12 is for example located in aircraft 12 itself (in the cockpit), or in a remote control room of aircraft 12 (in a ground station).

[0024] The central avionics unit 16 is notably connected to aircraft equipment 12, intended to interact within the aircraft's functional systems.

[0025] The functional systems of aircraft 12 include, for example, aircraft state measurement systems 20, external communication systems 22, and aircraft control actuation systems 24. The functional systems of aircraft 12 also advantageously include weapon systems 26, in the case of a military aircraft.

[0026] Measurement systems 20 include, for example, components comprising sensors for measuring parameters external to the aircraft, such as temperature, pressure or speed, sensors for measuring parameters internal to the aircraft and its various functional systems, and positioning sensors, such as GPS sensors, inertial measurement units, and / or an altimeter.

[0027] External communication systems 22 include, for example, components including radio systems, VOR / LOC, ADS, DME, ILS, radar systems, and / or satellite communication systems such as "SATCOM".

[0028] The 24 control systems include components comprising actuators to actuate aircraft controls, such as flaps, rudders, pumps, or mechanical, electrical and / or hydraulic circuits, and software actuators to configure the aircraft's avionics states.

[0029] Weapon systems 26, when present, advantageously include target detection and weapon guidance components and weapons themselves, such as bombs or missiles.

[0030] The various systems 20 to 26 are connected to the central avionics unit 16, for example digitally, by at least one data bus running on an internal aircraft network 12.

[0031] The central avionics unit 16 includes at least one task management system 28 capable of establishing, in real time, a list of crew tasks to be performed, which depends on mission objectives updated by environmental measurements taken, for example, by measurement systems 20 or received by communication systems 22.

[0032] The tasks to be performed are, for example, navigation tasks, to guide aircraft 12 and move it according to a flight plan including at least geographical points of passage of aircraft 12 during the mission and times of passage at the points of passage.

[0033] Also, the tasks to be performed are, for example, communication tasks to be carried out according to a communication plan including geographical communication points and / or times spent at geographical communication points.

[0034] In addition, the tasks to be performed may be related to events external to the aircraft, such as meteorology, including geographical points of passage or avoidance of meteorological phenomena, for example, in areas of turbulence and / or storms.

[0035] In the context of a military aircraft, the tasks to be performed also advantageously include reconnaissance tasks, tasks in preparation for air or air-to-ground combat, and air or air-to-ground combat tasks such as image analysis or weapon firing carried out from the aircraft.

[0036] The task list is generally determined initially, before the flight, using a mission planning system, based on mission objectives. The tasks are updated as the mission progresses, for example, because new mission objectives are determined by the mission commander in response to malfunctions, emergencies, and / or external meteorological or tactical events.

[0037] The task management system 28 is then capable of modifying the task list to add, delete, and / or change the order or characteristics of the tasks to be performed by the crew, based on updated mission objectives and an updated mission context that takes into account the environment around aircraft 12. The task plan, including the task list, is thus adaptive. These modifications are made within the limits of the system's authority as defined by the crew and may be subject to crew approval.

[0038] The crew operational status determination system 10 is schematically illustrated on the figure 1 In this example, the system 10 includes a computer 30 comprising a processor 32 and a memory 34 that receives software modules to be executed by the processor 32 to perform functions. Alternatively, the computer 30 includes programmable logic components or dedicated integrated circuits, intended to perform the functions of the modules that will be described below.

[0039] With reference to the figure 1 , memory 34 contains a module 40 for determining crew tasks to be performed at the current time, capable of retrieving a list of tasks to be performed by the crew, according to updated mission objectives and / or the updated mission context, and of assigning the tasks to crew members, according to crew profile data contained in a database 42.

[0040] Memory 34 also contains a crew resource determination module 44, capable of calculating the resources required by each crew member to perform each task in the list of tasks assigned to him / her and of determining a theoretical reference operational state for each crew member, to perform the tasks assigned to him / her, on the basis of a resource model.

[0041] Memory 34 also contains a module 46 for determining at least one context indicator, capable of defining at least one context indicator on the basis of mission context data developed from sensors 48 of the measurement systems 20 of the aircraft 12, or from data transmitted to the aircraft 12 via the communication systems 20.

[0042] Memory 34 also contains a module 51 for determining interactions, designed to determine the expected interactions with aircraft systems, during the implementation of tasks from the task list by each crew member.

[0043] Memory 34 also contains a module 50 for measuring interaction states with aircraft systems, based on measurements taken by interaction measurement sensors 52 and expected interactions determined by module 51 and a module 55 for measuring physiological states of the crew connected to sensors 54 for measuring physiological data of the aircraft crew 12 and to a database 56 of basic physiological data of the crew.

[0044] Memory 34 finally contains a module 58 for determining local cognitive states, designed to evaluate at least one local cognitive state of the crew with respect to the execution of each task defined in the list of tasks to be performed, and a module 60 for determining global cognitive states of the crew related to the general physiological state of the crew with respect to all the tasks to be performed.

[0045] A local cognitive state of the crew is a state of a crew member's mental process associated with the completion of a particular task to be performed within the task list.

[0046] An overall cognitive state of the crew is a state of a crew member's mental process associated with the set of tasks to be performed by the crew member.

[0047] According to the invention, the memory 34 also contains a crew operational state determination module 62, suitable for comparing at least one actual operational state of a crew member, in particular, a global cognitive state and / or a local cognitive state determined by modules 58, 60, with a theoretical reference operational state, in particular with a theoretical global reference cognitive state and / or a theoretical local reference cognitive state, determined by the determination module 44 using the resource model.

[0048] Memory 34 also contains a task reconfiguration module 64 based on the operational state obtained by the state determination module 62, suitable for modifying the list of tasks to be performed by the crew member or the information provided to the crew.

[0049] The task determination module 40 is designed to import, at any given time, tasks from the task management system 28 to obtain an ordered list of tasks, including, for example, target identification, flight plan monitoring, firing tasks to be performed, etc.

[0050] Each task is associated with at least one crew member, whose profile is obtained from database 42. The crew profile includes, for example, the crew member's level of expertise in performing the tasks in the task list, as well as preferred modes of interaction.

[0051] Preferred interaction methods include, for example, alternative strategies for completing tasks, particularly variations in the order in which tasks are performed. For instance, certain task completion strategies are known to an expert user who has expertise in performing the task, allowing them, for example, to use shortcuts or optimize their interactions with the interfaces.

[0052] Resource determination module 44 is designed, based on task lists and the resources required to implement the task list, obtained from a resource model, to determine, for each crew member, at least one theoretical operational reference state of the crew member, including at least one theoretical global cognitive reference state and / or at least one theoretical local cognitive reference state associated with each task for the crew member.

[0053] Examples of global cognitive reference states are a theoretically acceptable level of mental workload for the crew member, a theoretically acceptable level of drowsiness for the crew member, a theoretically acceptable level of crew member engagement for the crew member, a theoretically acceptable level of hypoxia for the crew member (this accepted level being zero) and / or a theoretically acceptable level of stress for the crew member.

[0054] Mental workload level is a numerical indicator representing the demands placed on the crew member's mental capacities. Drowsiness level is a numerical indicator representing the degree to which the crew member is sleepy, particularly in a state intermediate between wakefulness and sleep. Engagement level is a numerical indicator reflecting the crew member's overall attention to task completion. Hypoxia level is a Boolean indicator representing the adequacy or inadequacy of the oxygen supply to the crew member's tissue requirements. Stress level is a numerical indicator representing the crew member's nervous tension.

[0055] These theoretical levels are predefined globally based on the crew profile and are intended to be compared to actual levels, determined from data measured on the crew, as described below. These levels are, for example, numerical levels measuring a value within a range of possible values, discrete levels measuring a value among a plurality of possible discrete values, or Boolean levels presenting either an acceptable value or an unacceptable value.

[0056] Examples of local cognitive reference states for task completion include a minimum level of attention, a level of perseverance in completing each task, and a maximum level of visual and / or auditory tunneling to perform each task on the task list.

[0057] The level of attention associated with a task is a numerical indicator that reflects a crew member's readiness to perform the task, specifically their ability to select and focus on specific information related to that task. The level of perseverance is a numerical indicator that, in a context where it is needed, reflects a lack of revision of a previously made decision in the execution of a task. The level of tunneling is a numerical indicator that reflects the allocation of the crew's attention to a specific sensory channel, to the detriment of others, resulting in an inability of the crew to consider certain types of stimuli (e.g., auditory and / or visual) related to the task at hand.

[0058] These levels are defined according to the crew member's crew profile and the list of tasks assigned to the crew member.

[0059] The theoretical reference cognitive states are determined from the resource model which advantageously links each task to be performed to a time and / or resource level to be implemented by the crew member to perform the task.

[0060] The determination module 44 is thus able to assign for all tasks the time required as well as the overall state required (for example: the mental workload required) to perform the tasks according to the profile of the crew member given by the database 42 and the context indicators defined by the context indicator determination module 46. Similarly, it is able to assign for each task in the task list the time required as well as the local state required (for example: the level of attention required) to perform said task according to the profile of the crew member.

[0061] The context indicator determination module 46 is designed to define a level of complexity and / or a level of danger, based on measurements obtained by environmental sensors 48, in particular on the number of adversaries present in the mission environment, the state of the weather, and / or the presence of dangerous areas.

[0062] The level of complexity and / or the level of danger is used by the Determination Module 44 to weight the resources needed to perform each task, particularly in terms of the time or mental effort required to complete the tasks. For example, if the level of complexity or the level of danger increases, the time and mental effort required to complete a task are likely to increase as well.

[0063] Module 51 for determining interactions is designed to determine the expected interactions with aircraft systems by each crew member, and the expected performance to carry out these interactions, based on the list of tasks assigned to the crew member obtained from the determination module 40, the crew member's crew profile, as determined in database 42 and the context indicators determined by the determination module 46.

[0064] Interactions with aircraft systems include, for example, screen areas or elements to observe, keys or screen regions to select, or commands to activate. Expected performance includes, for example, a minimum speed for chaining actions to perform one or more tasks, an error rate in task execution, or the overall task completion rate among all tasks to be performed.

[0065] The 52 interaction measurement sensors are for example capable of determining the position of the gaze on the display units 18 of the cockpit 19, the support and force of support on the controls, the position of a control cursor of the aircraft systems on a display unit 18 or on another interface.

[0066] Based on the measurements taken by the interaction measurement sensors 52, the interaction state measurement module 50 is designed to compare the interactions with aircraft systems performed by the crew member and the interactions with aircraft systems expected for the crew member obtained from module 51 on the basis of the ordered list of tasks, to determine whether the crew member is performing each of the expected interactions.

[0067] The interaction state measurement module 50 is also suitable for determining whether the crew member meets the expected performance in terms of speed of execution, error rate and completion rate.

[0068] Based on the performance and actions performed by the crew member, the interaction state measurement module 50 is designed to determine a level of attention of the crew member to perform each task, a level of performance of the crew member in the execution of each task and a level of effectiveness of the strategy used to manage the tasks according to the preferred interaction modes defined above.

[0069] Physiological measurement sensors 54 include, for example, sensors for measuring heart rate, pupil diameter, blood oxygenation, emitted brain waves, posture (in particular via pressure pads in the seat and / or via camera image analysis systems), perspiration, or frontal oxygenation.

[0070] Physiological measurement sensors 54 may be partially common with interaction measurement sensors 52.

[0071] Physiological database 56 contains, for each crew member, basic physiological state data for each crew member at rest.

[0072] These basic physiological state data include, for example, resting heart rate, resting pupil diameter, resting blood oxygenation level, resting brain wave rate, resting posture, resting sweat rate, and resting frontal oxygenation level.

[0073] Based on the measurements taken by the physiological measurement sensors 54, compared with the basic physiological state data present in the database 56, the physiological state measurement module 55 is suitable for determining in particular a level of mental fatigue of the crew member, and / or a level of concentration of the crew member.

[0074] The level of mental fatigue is determined, for example, from a measured percentage of eye closure, the number and duration of blinks, the level of postural relaxation, the level of repetitive gestures, and / or certain frequency spectral powers (e.g., alpha waves) predominant in the brain.

[0075] The concentration level is determined, for example, from a level of prefrontal oxygenation, e.g., a high level of prefrontal oxygenation, from heart rate variability, e.g., low heart rate variability, from a variation in ocular dispersion, e.g., a decrease in ocular dispersion, and from certain high frequency spectral powers of the brain (theta waves in the frontal area).

[0076] The local cognitive state determination module 58 is designed to use the data produced by the physiological state measurement module 55 and the interaction state measurement module 50 to determine local cognitive states, linked to the performance of each of the specific tasks that are planned in the ordered task list.

[0077] As mentioned above, local cognitive states include, for example, a level of attention in the performance of tasks, for example determined from the analysis of the distribution of the pilot's gaze on his human / machine interfaces.

[0078] Local cognitive states include, for example, a level of perseveration. The level of perseveration is determined, for example, from frequency measurements of the brain, by a rate of reaction to stimuli, in particular a low rate of reaction to stimuli, by a rate of fixation on the task, in particular a high rate of fixation on the task.

[0079] A high level of perseverance is, for example, achieved when a pilot continues at all costs to try to land even though external conditions would require a go-around.

[0080] Local cognitive states also preferably include a level of visual and / or auditory tunneling, which reflects excessive focus on a specific task, neglecting other tasks on the task list and / or visual and / or auditory stimuli external to the task. The level of tunneling is determined, for example, from eye measurements, the observation of a lack of interaction on certain tasks, and the observation of a lack of response to certain stimuli (represented, for example, by the amplitude of P300 evoked potentials in the brain, particularly a low amplitude).

[0081] The module for determining global cognitive states 60 is designed to use the data produced by the module for measuring physiological states 55 and the module for measuring interaction states 50 to determine global cognitive states of each crew member.

[0082] Global cognitive states include, for example, an overall level of mental workload. This overall level of mental workload is determined, for example, from prefrontal oxygenation, in particular high prefrontal oxygenation, heart rate variability, in particular a decrease in heart rate variability, performance on secondary tasks (delay, etc.), in particular a decrease in performance, and variation in pupil diameter, in particular an increase in pupil diameter.

[0083] Global cognitive states advantageously include a level of overall sleepiness. This level of overall sleepiness is determined, for example, from posture, particularly a relaxed posture, the number of blinks, particularly an increase in the number of blinks, and a determination of the absence of interactions.

[0084] Global cognitive states advantageously include a level of engagement, which reflects the rate of tasks completed relative to tasks to be completed. The level of engagement is determined, for example, from a ratio of brain frequency waves (beta / alpha+delta), also called the "engagement index," which is notably higher than at rest, and / or from an analysis of eye data, for example, to identify a ratio of "long fixations + short saccades / long saccades + short fixations," particularly when this ratio is low.

[0085] Global cognitive states advantageously include a hypoxia level, representing the mismatch between tissue oxygen requirements and oxygen supply for the crew member. This hypoxia level is a Boolean level. It is determined, for example, by a heart and respiratory rate exceeding predetermined thresholds indicating abnormality, a drop in blood oxygen saturation below a certain threshold, incoherent speech, and / or eye blinking above a certain threshold.

[0086] Global cognitive states also preferably include a level of stress. The level of stress is determined, for example, from microsweating, especially heavy microsweating, muscle tension, especially heavy muscle tension, heart rate variability, especially high heart rate variability, saccade / fixation ratio of gaze, especially a high saccade / fixation ratio, voice frequency, especially a voice becoming higher pitched, and / or an upright posture in the seat.

[0087] The crew operational state determination module 62 is designed to analyze global cognitive states and local cognitive states and to compare them respectively with the global reference cognitive states and the local reference cognitive states calculated by the resource determination module 44 in order to determine a crew member's fitness indicator, for example boolean, suitable for moving between a state of fitness of the crew member to perform the tasks in the list of tasks assigned to him and a state of unfitness of the crew member to perform the tasks in the list of tasks assigned to him.

[0088] For example, the crew operational state determination module 62 is suitable for evaluating whether the actual mental workload level determined by the global cognitive state determination module 60 is greater than the maximum theoretical mental workload level calculated by the resource determination module 44. In this case, it is suitable for changing the crew member's fitness indicator from a fit state to an unfit state.

[0089] Similarly, the crew operational state determination module 62 is advantageously suited to assess whether the effective stress level determined by the global cognitive state determination module 60 is greater than the maximum stress level calculated by the resource determination module 44. In this case, it is suitable for changing the crew member's fitness indicator from a fit state to an unfit state.

[0090] The crew operational state determination module 62 is advantageously suited to assess whether the actual level of drowsiness determined by the global cognitive state determination module 60 is greater than the maximum level of drowsiness calculated by the resource determination module 44. In this case, it is suitable for changing the crew member's fitness indicator from a state of fitness to a state of unfitness.

[0091] The crew operational state determination module 62 is advantageously suited to assess whether the effective level of engagement determined by the global cognitive state determination module 60 is greater than the level of engagement calculated by the resource determination module 44. In this case, it is suitable for changing the crew member's fitness indicator from a state of fitness to a state of unfitness.

[0092] The crew operational status determination module 62 is advantageously suited to assessing whether the actual hypoxia level determined by the global cognitive state determination module 60 exceeds the permitted hypoxia level, in this case zero. If so, it is suitable for changing the crew member's fitness indicator from a fit to an unfit state.

[0093] In addition, the crew operational state determination module 62 is advantageously suited to assess whether the effective tunneling level and / or the perseverance level determined by the local cognitive state determination module 58 exceed respectively the theoretical maximum tunneling level and / or the theoretical maximum perseverance level determined by the resource determination module 44, and whether the attention level determined by the local cognitive state determination module 58 is less than the attention level determined by the resource determination module 44.

[0094] In each of these cases, it is specific to changing the crew member's fitness indicator from a state of fitness to a state of unfitness.

[0095] The operational state determination module 62 is specific, when a state of unfitness is determined, to define a type of task reconfiguration or crew information modification to be carried out by the task configuration module 64.

[0096] The task configuration module 64 is advantageously suited to modifying the task list to perform a task in place of the crew, to delete a task, to postpone a task to be performed.

[0097] It is also capable of generating an alert and / or an alarm in the event that the operational status determination module 62 determines a state of unfitness of the crew, or of temporarily removing information displayed to the crew on the display units 18, and / or of displaying new information replacing the existing information.

[0098] Examples of implementing a method for determining the operational state of a crew based on a task plan using the determination system 10 according to the invention will now be described, with regard to the figures 3 à 7 .

[0099] In the example illustrated by the figure 3 In a civil aircraft context, aircraft 12 operates a cruise phase, after the takeoff of aircraft 12.

[0100] The task management system 28 defines the tasks to be performed, which are essentially tasks of monitoring the aircraft's trajectory, in particular the passage through waypoints, the aircraft's fuel consumption 12 and communication tasks, when passing through various airspaces.

[0101] The crew member in charge of monitoring is likely to become drowsy.

[0102] As indicated above, in step 100 of the process according to the invention, the task determination module 40 establishes at each moment the list of tasks to be performed, and the distribution of tasks among the crew members intended to perform them using the crew profile database 42.

[0103] Then, in step 102, the resource determination module 44 defines a maximum level of sleepiness acceptable for performing these tasks, based for example on a maximum level of sleepiness associated with each individual task.

[0104] At step 104, based on measurements obtained from physiological sensors 54, the physiological state measurement module 55 determines an increase in the frequency and duration of blinks, an increase in the percentage of eyelid closure, and a relaxed posture.

[0105] Based on these parameters, the module for determining global cognitive states 60 calculates an effective level of sleepiness.

[0106] At step 106, the operational status determination module 62 determines that the actual level of drowsiness exceeds the maximum permissible level of drowsiness for all ongoing tasks. It therefore changes the crew member's fitness indicator from "fit to perform assigned tasks" to "unfit to perform assigned tasks."

[0107] The operational state determination module 62 also determines that the reconfiguration type is to trigger a wake-up alarm and temporarily automate the piloting of the aircraft.

[0108] At step 108, the reconfiguration module 64 therefore sends instructions to the aircraft system to generate an alarm in the cockpit 19.

[0109] In a second example, illustrated by the figure 4 During an approach phase to a landing strip with a strong crosswind, the pilot is focused on a piloting task.

[0110] If an important alarm is triggered, it is likely not to be treated as a priority.

[0111] In this regard, the task determination module 40 determines a list of tasks relating to piloting, and a list of tasks relating to alarm handling, which may be a red CAS alarm requiring priority action from the pilot from the task list.

[0112] At step 110, the interaction modelling module 51 determines that the crew member should process the tasks associated with the alarm, by performing a number of interactions in the cockpit 19 including on the display units 18 and / or using the controls.

[0113] At step 112, the interaction state measurement module 50 determines, from the cursor position, the gaze position, that the crew member is not performing any of the interactions related to the alarm.

[0114] In addition, the interaction state measurement module 50 establishes that the crew member does not react to the auditory and visual stimuli resulting from the alarm that are outside his field of vision.

[0115] On this basis, it determines a level of attention for the crew member to perform each task, and a level of performance for the crew member in performing each task.

[0116] Similarly, in step 114, the physiological state measurement module 55 determines a pupil diameter with little visual dispersion, low heart rate variability, e.g. less than 20% relative to resting variability and high heart rate relative to resting heart rate, e.g. at least greater than 20% of resting heart rate, from the basic physiological data databases 56.

[0117] The physiological state measurement module 55 further determines an absence of amplitude of evoked potentials P300 following a sound stimulus (alarm).

[0118] Based on the information obtained by modules 50 and 55, module 60, which determines global cognitive states, determines a level of mental workload and a level of stress. Module 58, which determines local cognitive states, determines a level of tunnel vision.

[0119] At step 116, the operational state determination module 62 determines that the actual mental workload level exceeds the maximum mental workload level determined by the determination module 44, and that the tunneling level exceeds the maximum tunneling level determined by the determination module 44. It defines a state of unfitness for the crew member to perform the required tasks, particularly to handle mission alarms. The crew member is thus in a state of attentional deafness.

[0120] The operational state determination module 62 further determines that the action to be taken would be to temporarily suppress the display of the information on which the pilot is focusing.

[0121] At step 118, the reconfiguration module 64 therefore directs the display units 18 in the cockpit 19 to suppress at least one piece of information relating to crosswind piloting, in order to allow the pilot to identify that another urgent alarm is present and must be dealt with.

[0122] In a third example, shown on the figure 5 , system 10 detects a state of panic present in a crew member.

[0123] This can occur, for example, when multiple alarms are displayed.

[0124] As before, in step 120, the task determination module 40 determines that a plurality of tasks must be performed simultaneously and defines for each task a level of mental workload and a level of stress associated with the task.

[0125] Module 51, the interaction modeling module, determines the expected interactions with the cockpit interface and the expected performance to perform the tracking tasks.

[0126] In step 122, based on measurements from interaction measurement sensors 52, the interaction state measurement module 50 compares the expected interactions for each crew member with the aircraft systems obtained from module 51 based on the ordered task list. It identifies a dispersion in the interactions, and the fact that no task is completed.

[0127] It therefore calculates a level of attention and a level of engagement. These levels are low.

[0128] Furthermore, at step 124, based on physiological sensors 54, the physiological state measurement module 55 determines the presence of numerous saccades in eye movement, a very high heart rate, and sudden and heavy sweating.

[0129] Based on this, the module for determining global cognitive states 60 calculates a stress level. This stress level is high.

[0130] At step 126, the operational state determination module 62 then identifies a high stress level, higher than the maximum stress level defined by the determination module 44. This stress level is characteristic of a state of panic which renders the crew member unfit to perform safety-critical tasks.

[0131] It identifies a level of attention and a level of engagement lower respectively than the minimum level of attention and the minimum level of engagement determined by the determination module 44.

[0132] It therefore defines a state of unsuitability of the crew member to perform the required tasks.

[0133] The operational state determination module 62 further determines that a synthesis of the information and the procedure to be followed must be carried out by the reconfiguration module 64.

[0134] At step 128, the reconfiguration module controls the display units 18 of the avionics central unit 16 to provide an oral and graphic summary of the information and the procedure to follow.

[0135] Another example will now be described with reference to the figure 6 , in the context of a military aircraft during high-altitude aerial combat.

[0136] As before, in step 130, the task determination module 40 determines the list of tasks to be performed during this phase of air combat. The interaction determination module 51 determines the expected interactions with aircraft systems by the crew member, and the expected performance to carry out these interactions, based on the list of tasks assigned to the crew member obtained from the determination module 40, the crew member's crew profile as determined in database 42, and the context indicators determined by the determination module 46.

[0137] At step 132, based on the data from interaction sensors 52, the interaction state measurement module determines that the reaction time increases significantly, and that the responses do not correspond to what is expected at the interaction level.

[0138] The module for determining global cognitive states 60 thus determines a level of engagement, and the module for determining local cognitive states 58 determines a level of attention. These levels are low.

[0139] In parallel, at step 134, based on data from physiological sensors 54, the physiological state measurement module 55 determines a decrease in blood oxygen saturation, and an abnormal posture characterized by an absence of movement.

[0140] The module for determining global cognitive states 60 infers a level of hypoxia. This level of hypoxia is unacceptable.

[0141] At step 136, the operational status determination module 62 identifies that the hypoxia level is higher than a maximum hypoxia level determined by the determination module 44, in this case a level of zero. It therefore defines a state of unfitness for the crew member to perform the required tasks.

[0142] It also determines the actions to be taken for reconfiguration module 64, which consist of letting the central avionics unit 16 resume piloting, and performing an emergency descent.

[0143] At step 138, the reconfiguration module 64 sends orders to the central avionics unit 16 to perform this takeover and emergency descent.

[0144] In another example, shown on the figure 7 , still in the context of a military aircraft, during an air-to-ground combat mission, the pilot is forced to perform numerous tasks in a limited time.

[0145] In step 140, the task determination module 40 determines the list of tasks to be performed. The interaction determination module 51 determines the expected interactions with aircraft systems by the crew member, and the expected performance to perform these interactions, based on the list of tasks assigned to the crew member obtained from the determination module 40, the crew member's crew profile, as determined in the database 42, and the context indicators determined by the determination module 46.

[0146] At step 142, based on the data from the interaction sensors 52, the interaction state measurement module 50 determines repetitive errors on the interactions to be performed and a decrease in performance, particularly on tasks that are not a priority or not included in the initial task list.

[0147] The module for determining global cognitive states 60 thus determines a level of engagement, and the module for determining local cognitive states 58 determines a level of attention. These levels are very high.

[0148] In parallel, at step 144, based on data from physiological sensors 54, the physiological state measurement module 55 determines high prefrontal oxygenation, exceeding a given threshold, the dominance of theta and beta waves at the frontal level, a large pupil diameter, and very low heart rate variability. It deduces from this a high concentration level.

[0149] The module for determining global cognitive states 60 calculates a level of mental workload on this basis. This level of mental workload is high.

[0150] At step 146, the operational state determination module 62 then determines a level of mental workload higher than the maximum level of mental workload determined by the determination module 44. It therefore defines a state of unfitness of the crew member to perform all the tasks required.

[0151] The operational state determination module 62 then determines a type of reconfiguration in which the avionics central unit 16 automatically takes over certain lower priority tasks.

[0152] At step 148, the reconfiguration module 64 therefore sends instructions to the avionics central unit 16 so that it can take charge of the tasks thus defined.

[0153] The determination system 10 according to the invention is therefore capable of developing an operational state of the crew, based on a resource model built on the list of crew tasks, by separately measuring the physiological states and interaction states of the crew, even if these measurements come a priori from data from the same sensors and by separately determining local cognitive states, relating to the execution of a task and global cognitive states, relating to all tasks.

[0154] The determination system 10 updates the operational state as changes occur in the task list, for example when a new task appears, or based on changes in mission objectives, or the context observed in the environment.

[0155] This makes the determination system 10 very interactive, since it dynamically takes into account the mission context and the crew's resources.

[0156] Thanks to its generic functional architecture, the 10 determination system is also easily configurable to adapt to different types of sensors, missions, and platforms. It takes into account the specific characteristics of the crew members.

[0157] Advantageously, the determination system 10 also establishes reconfiguration actions that help to resolve the identified problems.

[0158] Thus, the determination system 10 easily interprets the crew's condition, which improves crew performance, reduces their stress and mental workload.

[0159] As stated above, the local cognitive state determination module 58 is designed to use the data produced by the physiological state measurement module 55 and the interaction state measurement module 50 to determine local cognitive states, linked to the performance of each of the specific tasks that are planned in the ordered task list.

[0160] Similarly, the global cognitive state determination module 60 is designed to use the data produced by the physiological state measurement module 55 and the interaction state measurement module 50 to determine global cognitive states of each crew member.

[0161] Thus, the same sensor 54 for physiological measurements or the same sensor for measuring interactions 52 can be used to determine at least one local cognitive state referring to a task and at least one global cognitive state.

[0162] For example, when sensor 52, 54 is an eye tracker, observing the percentage of eye opening and blink rate parameters can advantageously identify the crew member's level of drowsiness, which is a general physiological state. Furthermore, a significantly reduced percentage of eye opening can also indicate a state of drowsiness.

[0163] The area of ​​gaze fixation, combined with the level of saccadic eye movements, allows, for example, the identification of a level of attention, advantageously defined by the rate of attentional fixation on a given task, knowing that a task can be associated with a set of attentional areas in the cockpit. This characterizes a local physiological state.

[0164] In the case of a 52, 54 sensor allowing the establishment of an electroencephalogram, the observation of brain waves establishes global physiological states (drowsiness, mental workload).

[0165] Observing responses (measured brain electrical potential) to auditory stimuli within the brain (N100, P300) establishes local physiological states (e.g., a level of attention to an alarm or information).

[0166] For the determination of effective cognitive states, the determination modules 58, 60 are suitable for implementing machine learning algorithms or algorithms without learning.

[0167] Machine learning algorithms can be neural networks and / or decision trees. Algorithms without machine learning include, for example, expert rules.

[0168] The training of algorithms is advantageously carried out as follows.

[0169] The measurement data from sensors 52, 54 are recorded over experimental phases carried out by a number of crew members allowing supervised learning.

[0170] The training involves several steps including labeling measurement data to establish local or global cognitive states.

[0171] This labeling can be done manually by external observers who attended the experimentation phases, and validated by the crew members concerned, or manually, a posteriori, by reviewing a video recording of the experimentation phase with the crew member concerned.

[0172] Alternatively, labeling can be done automatically, based on the results of subjective tests given to crew members during the experimental phases (for example, NASA-TLX test for mental workload) or automatically, based on numerical results of crew member performance during data recording (score, reaction time...).

[0173] Alternatively, the labeling is achieved through a mixture of all these techniques.

[0174] The labeled data are then anonymized, normalized by each crew member's baseline to eliminate inter-individual variability, advantageously mixed and separated into several sets allowing for the training, development and testing of algorithms.

[0175] Next, machine learning algorithms are trained from this data or business rules for non-learning algorithms are established.

[0176] In one example, it is possible to first consider an existing fleet of aircraft and take measurements from anonymized sensors on all pilots in the fleet, then correlate the ground measurements after the flights with the observed state (following pilot questionnaires, and / or expert labeling), and then integrate these results into the determination system 10. This can be done iteratively.

[0177] Having done this, during a flight, the determination module 60 is able to establish at each instant the effective global cognitive states of the crew member based on the physiological and interaction measurements of the crew member and the determination module 58 is able to establish at each instant the effective local cognitive states relating to the execution of tasks by the crew member based on the physiological and interaction measurements of the crew member, by implementing the aforementioned algorithms.

[0178] Similarly, as explained above, module 44, crew resource determination, is designed to determine, for the crew member or for each crew member, at least one theoretical reference cognitive state of the crew member based on the list of tasks assigned to the crew member, using a resource model.

[0179] These reference cognitive states are advantageously global reference cognitive states (e.g., level of mental workload, level of sleepiness) or local reference cognitive states (e.g., prioritization of attention on task A over task B).

[0180] Each global or local reference cognitive state determined using the resource model by module 44 corresponds to a global or local effective cognitive state determined by a determination module 58, 60.

[0181] Thus, during the flight, the determination module 62 compares each global or local effective cognitive state determined by a determination module 58, 60 to a corresponding global or local reference cognitive state, determined using the resource model by module 44, in order to obtain the aptitude indicator, as described above.

[0182] The resource model is designed to determine at least the overall reference cognitive state (e.g., the necessary mental workload) based on the current task(s) in the task list, based on the crew member profile given by the database 42, and possibly based on context indicators defined by the context indicator determination module 46.

[0183] Similarly, the resource model is designed to determine at least one local reference cognitive state (e.g., the level of attention required) as a function of at least one task from the task list, based on the crew member profile and possibly based on a defined prioritization of tasks.

[0184] The resource model advantageously implements machine learning algorithms and non-learning algorithms.

[0185] Machine learning algorithms can be neural networks and / or decision trees. Algorithms without machine learning include, for example, expert rules.

[0186] Training algorithms is advantageously carried out on datasets created or by expertise.

[0187] The training of algorithms is advantageously carried out as described above, by implementing phases of experimentation.

[0188] Reference cognitive states are, for example, predefined by expertise. Alternatively, reference cognitive states are labeled based on missions that have been carried out correctly, for which the operator reacted appropriately to the tasks performed (for example, with subjective feedback from the operator stating that they did not experience any problems with the tasks performed).

[0189] Once the resource model has been trained or adjusted, it can be used by providing it with input a list of tasks to be performed, a crew member profile given by the database 42, and possibly context indicators defined by the context indicator determination module 46.

[0190] The resource model then provides at least one reference cognitive state as output, including a global reference cognitive state corresponding to all the input tasks (e.g., level of mental workload, level of sleepiness) and / or a local reference cognitive state corresponding to at least one task from the list of tasks provided as input (e.g., prioritization of attention on task A over task B).

[0191] The crew operational state determination module 62 is then specific to determining the aptitude indicator of each crew member to perform the task(s) in the list of tasks assigned to the crew member, from the comparison between each actual cognitive state of the crew member determined on the basis of physiological measurements on the crew member and measurements of interaction of the crew member with the aircraft 12, and the corresponding theoretical reference cognitive state, determined by the resource model.

[0192] In a first illustrative example, during a transit for a long-haul flight, the resource model indicates that a high level of mental workload (above a predefined threshold) is not normal, while a level of drowsiness above a predefined threshold is acceptable.

[0193] Based on physiological and interaction measurements, the 60-step determination module establishes that the pilot has a normal workload and is drowsy. The 62-step determination module then establishes that the crew member's fitness indicator is in the state where the crew member is fit to perform the tasks on the task list.

[0194] Conversely, during a landing with degraded weather, the resource model indicates that a high level of mental workload (above a predetermined threshold) is acceptable, while a level of drowsiness above a predefined threshold is not acceptable.

[0195] Based on physiological and interaction measurements, the determination module 60 establishes that the pilot has a normal workload and is drowsy. The determination module 62 then establishes that the crew member's fitness indicator is in a state of unfitness for the crew member to perform the tasks on the task list.

[0196] In another illustrative example, the resource model indicates that the pilot's mental workload level (which corresponds to the global reference state) must avoid exceeding a predefined maximum reference level, and that the pilot's attention (which corresponds to a local reference state) must be focused on task A as a priority.

[0197] Based on physiological and interaction measurements, the determination module 60 establishes the pilot's mental workload level, and the determination module 62 determines that this effective mental workload level is well below the predefined maximum reference level value established by the resource model.

[0198] However, still based on physiological and interaction measurements, the determination module 60 establishes that the pilot's attention is focused on task B, which is less of a priority than task A, according to the resource model.

[0199] The determination module 62 then establishes that the crew member's fitness indicator is in the state of the crew member being unfit to perform the tasks in the task list.

Claims

1. A system (10) for determining an operational state of an aircrew of a manned or unmanned aircraft (12) according to an adaptive task plan, the system (10) comprising: - a module (40) for determining crew tasks to be carried out, capable of defining, at each moment during a mission of the aircraft (12), a list of tasks to be carried out by at least one crew member as a function of updated mission objectives, and of an instantaneous mission context determined from measurements of the mission environment; characterized by : - a crew resource determination module (44), adapted to determine for the or each crew member at least one theoretical baseline crew member cognitive state according to the task list assigned to the crew member, using a resource model; - a crew operational state determination module (62), adapted to determine an indicator of the fitness of the or each crew member to perform the task(s) of the task list assigned to the crew member, from an actual crew member cognitive state determined on the basis of physiological measurements on the crew member and measurements of the crew member's interaction with the aircraft (12), and from the theoretical baseline crew member cognitive state determined by the crew resource determination module (44). - a module (60) for determining overall cognitive states of the crew member as a function of the physiological and interaction measurements of the crew member and - a module (58) for determining local cognitive states of the crew, adapted to determine local cognitive states related to the execution of tasks by the crew member as a function of the crew member's physiological measurements and aircraft (12) interaction measurements, the crew operational state determination module (62) being adapted to calculate the actual crew member cognitive state as a function of the local cognitive states and the overall cognitive states determined by the overall cognitive state determination module (60) and the local cognitive state determination module (58), respectively, and to compare each local or overall crew member cognitive state with the theoretical baseline local or overall crew member cognitive state determined by the crew resource determination module (44) based on the resource model.

2. The system (10) according to claim 1, wherein the operational state determination module (62) is adapted to transition the fitness indicator between a state of fitness of the crew member to perform the task(s) of the task list assigned to the crew member and a state of unfitness of the crew member to perform the task(s) of the task list assigned to the crew member, based on a comparison between the actual crew member cognitive state and the theoretical crew member cognitive state.

3. The system (10) according to claim 2, comprising a mission task reconfiguration module (64), suitable for modifying the task list to be performed by the crew member when the fitness indicator changes to the state of unfitness of the crew member to perform the task(s) of the task list assigned to the crew member.

4. The system (10) according to claim 3, wherein the task reconfiguration module (64) is suitable for deleting or / and postponing a task to be performed, performing a task instead of the crew member, deleting information given to the crew member, and / or replacing the information given to the crew member with other information.

5. The system (10) according to any of the preceding claims, wherein the overall cognitive states are selected from a level of mental load, a level of engagement, a level of sleepiness, a level of hypoxia, and / or a level of stress of the crew member.

6. The system (10) according to any of the preceding claims, wherein the local cognitive states are selected from a level of attention, a level of perseveration, a level of visual tunnelling, or a level of auditory tunnelling in the performance of a task.

7. The system (10) according to any of the preceding claims, comprising a module (55) for measuring physiological states based on sensors for measuring physiological data (54), in particular based on sensors for measuring heart rate, pupil diameter, number of blinks, blood oxygenation, brain waves, evoked potentials, and based on a database (56) of basic physiological states of the crew, the module for determining overall cognitive states determining the overall cognitive states (60) from the physiological state measurements of the physiological state measurement module (55).

8. The system (10) according to any of the preceding claims, comprising a module (50) for measuring crew interaction states, based on sensors (52) for measuring crew interaction states, in particular gaze position, the use of physical commands, and the use of touch, and based on a model of expected interactions based on the list of tasks determined by the module (40) for determining current crew tasks, the local cognitive state determination module (58) determining local cognitive states from the interaction state measurements of the module for measuring interaction states (50).

9. The system (10) according to claim 8, wherein the interaction states are selected from at least one level of attention, one level of performance, and one task completion strategy.

10. The system (10) according to any of the preceding claims, comprising a context indication module (46), which based on mission environment measurements is adapted to define at least one context indicator, the crew resource determination module (44) being adapted to determine the theoretical baseline crew member cognitive state based on the or each context indicator.

11. The system (10) according to any of the preceding claims, wherein the current crew task determination module (40) is adapted to assign the list of tasks to the crew member according to a database (42) of crew profiles determining the characteristics of each crew member and advantageously the task handling capabilities of the aircraft, and according to the mission objectives.

12. A method of determining an operational state of an aircrew of a manned or unmanned aircraft (12), based on an adaptive task plan comprising the following steps: - providing a determination system (10) according to any one of the preceding claims; - defining, by the module (40) for determining current crew tasks, at each moment during a mission of the aircraft (12), of a list of tasks to be carried out by at least one crew member as a function of updated mission objectives, and of an instantaneous mission context determined from environmental measurements of the mission; - determination, by the crew resource determination module (44), for the or each crew member, of at least one theoretical baseline crew member cognitive state based on the task list assigned to the crew member, using a resource model; - determination, by the crew operational state determination module (62), adapted to determine an indicator of the fitness of the or each crew member to perform the task(s) of the task list assigned to the crew member, from an actual crew member cognitive state determined on the basis of physiological measurements on the crew member and measurements of the crew member's interaction with the aircraft (12), and from the theoretical baseline crew member cognitive state determined by the crew resource determination module (44); - determination of the crew member's overall cognitive states as a function of the crew member's physiological and interaction measurements, by the module (60) for determining the crew member's overall cognitive states, and - determination of local cognitive states of the crew, relative to the execution of tasks by the crew member as a function of the crew member's physiological and interaction measurements with the aircraft (12) by the module (58) for determining local cognitive states of the crew; - calculation by the crew operational state determination module (62) of the crew member's actual cognitive state as a function of the local cognitive states and global cognitive states determined by the global cognitive state determination module (60) and local cognitive state determination module (58) respectively, and comparison of each local or global cognitive state of the crew member with the crew member's theoretical reference local or global cognitive state determined by the crew resource determination module (44) on the basis of the resource model.

13. The method according to claim 12, comprising the transition of the fitness indicator between a state of fitness of the crew member to perform the task(s) on the crew member's assigned task list and a state of unfitness of the crew member to perform the task(s) on the crew member's assigned task list based on a comparison between the actual crew member cognitive state and the theoretical crew member cognitive state.

14. The method according to claim 13, comprising a modification, by a mission task reconfiguration module (64), of the task list to be performed by the crew member when the fitness indicator changes to the state of unfitness of the crew member to perform the task(s) on the task list.