System for determining the operational status of an aircraft crew based on an adaptive task plan and associated method
The system dynamically evaluates crew capability by integrating task and resource management with real-time physiological and interaction data, addressing the challenge of crew overload or neglect in adapting to mission tasks, enhancing performance and reducing stress.
Patent Information
- Application Number
- FR2020013237
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-15
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-12-15
AI Technical Summary
Existing systems fail to dynamically assess an aircraft crew's ability to perform mission tasks due to reliance on instantaneous physiological measurements without considering the crew's adaptive task plan and evolving mission context, leading to potential overloading or neglect of critical tasks.
A system that includes modules for determining crew tasks, resource management, operational state assessment, and task reconfiguration based on real-time mission objectives, crew profiles, and environmental conditions, using sensors for physiological and interaction data to compare actual and theoretical cognitive states.
The system provides reliable and dynamic evaluation of crew capability, enabling adaptive task reconfiguration to ensure effective mission execution by addressing crew overload or neglect, thereby improving performance and reducing stress.
Smart Images

Figure 00000026_0000 
Figure 00000027_0000 
Figure 00000028_0000
Abstract
Description
Title of the invention: System for determining an operational state of an aircraft crew based on an adaptive task plan and associated method
[0001] The present invention relates to a system for determining an operational state of a manned or unmanned aircraft crew, based on an adaptive task plan.
[0002] The operational state of an aircraft is in particular the capacity of a crew to carry out the tasks of its mission according to its current state.
[0003] This system is intended in particular to evaluate in real time the operational state of the crew, with regard to all the tasks that it must carry out with the aircraft in the context of a mission. The aircraft is for example manned and the crew is present in the cockpit of the aircraft. Alternatively, the aircraft is unmanned, the crew is then in a drone control ground station.
[0004] The mission is for example a mission of a civil aircraft, such as a commercial or private flight between a geographical point of departure and a geographical point of arrival, in which, during the phases of the mission, numerous tasks must be carried out in accordance with the applicable procedures.
[0005] In a variant, the mission is a mission of a military aircraft (manned aircraft or drone), including in particular reconnaissance or combat tasks, for example air-to-ground combat tasks to neutralize targets.
[0006] In such missions, the crew generally follows a task plan comprising a list of tasks to be performed, associated with the time by which the task must be performed.
[0007] During a mission, the task plan is adaptive. This means that the list of tasks to be performed is updated automatically by the aircraft and is validated by the crew, in particular based on events occurring during the mission. The crew can also decide to modify the task plan itself.
[0008] For example, the objectives of the mission plan or the flight plan are likely to change, in particular during a diversion for a civil aircraft, or when a new target must be processed for a military aircraft.
[0009] Similarly, the mission environment is likely to change, for example depending on the weather in the civil or military domains, or the appearance of additional threats in the military domain.
[0010] In certain cases, the crew's resources, i.e. the crew's ability to carry out its tasks, are not adapted to the task plan to be carried out.
[0011] This can be explained, for example, by the fact that a large number of additional tasks have to be carried out, due to a breakdown or an emergency occurring on the aircraft. The crew may also be tired or stressed, which does not allow them to carry out the tasks assigned to them.
[0012] Finally, in certain critical cases, the aircraft crew may find themselves overwhelmed by all the tasks to be performed, or on the contrary concentrate too much on a particular task, and neglect important tasks which should be accomplished.
[0013] To check the state of the crew, it is known to equip the cockpit of the aircraft with sensors measuring the physiological state of the crew. However, these sensors are only interested in the instantaneous state of a crew member, without linking this instantaneous state with the constraints that the crew undergoes to carry out the mission plan.
[0014] An aim of the invention is to obtain a system for monitoring the behavior of the crew during the execution of a task plan which reliably and dynamically determines the ability of the crew to carry out its mission.
[0015] To this end, the invention relates to a system of the aforementioned type, comprising:
[0016] - a module for determining crew tasks to be carried out, capable of defining, at each moment during an aircraft mission, a list of tasks to be carried out by at least one crew member based on updated mission objectives, and an instantaneous mission context determined from mission environment measurements;
[0017] - a crew resource determination module, suitable for determining for the or for each crew member, at least one theoretical reference cognitive state of the crew member based on the task list assigned to the crew member, using a resource model;
[0018] - a crew operational status determination module, suitable for determining a indicator of the ability of the or each crew member to perform the task(s) on the task list assigned to the crew member, based on an actual cognitive state of the crew member determined on the basis of physiological measurements on the crew member and measurements of interaction of the crew member with the aircraft, and from the theoretical reference cognitive state of the crew member determined by the crew resource determination module.
[0019] The system according to the invention may comprise one or more of the following characteristics, taken in isolation or in any technically possible combination:
[0020] - the operational state determination module is capable of passing the indicator of ability between a state of ability of the crew member to perform the task(s) on the task list assigned to the crew member and a state of inability of the crew member to perform the task(s) on the task list assigned to the crew member, based on a comparison between the actual cognitive state of the crew member crew and the theoretical cognitive state of the crew member;
[0021] - it includes a mission task reconfiguration module, suitable for modifying the list of tasks to be performed by the crew member when the suitability indicator changes to the state of the crew member being unable to perform the task(s) on the task list assigned to the crew member.
[0022] - the task reconfiguration module is capable of deleting and / or postponing a task to be performed, to perform a task in place of the crew member, to delete information given to the crew member, and / or to replace information given to the crew member with other information.
[0023] -it comprises a module for determining global cognitive states of the crew member as a function of the physiological and interaction measurements of the crew member and a module for determining local cognitive states of the crew, capable of determining local cognitive states relating to the execution of tasks by the crew member as a function of the physiological and interaction measurements of the crew member with the aircraft,the crew operational state determination module being capable of calculating the effective cognitive state of the crew member as a function of the local cognitive states and the global cognitive states determined respectively by the global cognitive state determination module and by the local cognitive state determination module and of comparing each local or global cognitive state of the crew member with the theoretical local or global reference cognitive state of the crew member determined by the crew resource determination module on the basis of the resource model;
[0024] - the global cognitive states are chosen from a level of mental load, a engagement level, drowsiness level, hypoxia level, and / or crew member stress level;
[0025] - the local cognitive states are chosen from an attention level, a performance level severity, a level of visual tunneling, a level of auditory tunneling in the performance of a task;
[0026] - it includes a module for measuring physiological states from sensors of physiological data measurements, in particular from sensors measuring heart rate, pupil diameter, number of blinks, blood oxygenation, brain waves, evoked potentials, and from a database of basic physiological states of the crew, the global cognitive state determination module determining the global cognitive states from the physiological state measurements of the physiological state measurement module;
[0027] -it comprises a module for measuring states of interactions with the crew, from sensors for measuring interactions with the crew, in particular the position of the gaze, the use of physical controls, the use of touch, and from a model expected interactions based on the task list determined by the current crew task determination module, the local cognitive state determination module determining the local cognitive states from the interaction state measurements of the interaction state measurement module;
[0028] - the interaction states are chosen from at least one level of attention, one performance level, a task management strategy;
[0029] - it includes a context indication module, based on measurements mission environment, capable of defining at least one context indicator, the crew resource determination module being capable of determining the theoretical cognitive reference state of the crew member based on the or each context indicator;
[0030] - the current crew task determination module is capable of assigning the list of tasks to the crew member based on a database of crew profiles determining the characteristics of each crew member and advantageously the capabilities of the aircraft to take charge of the tasks, and based on the objectives of the mission.
[0031] The invention also relates to a method for determining an operational state of a crew of a manned or unmanned aircraft, as a function of an adaptive task plan comprising the following steps:
[0032] - provision of a determination system as defined above;
[0033] - definition, by the current crew task determination module, at each instant during an aircraft mission, a list of tasks to be performed by at least one crew member based on updated mission objectives, and an instantaneous mission context determined from mission environment measurements;
[0034] - determination, by the crew resource determination module, for the or for each crew member, at least one theoretical cognitive reference state of the crew member based on the task list assigned to the crew member, using a resource model;
[0035] - determination by the crew operational state determination module, of a indicator of the ability of the or each crew member to perform the task(s) on the task list assigned to the crew member based on an actual cognitive state of the crew member determined on the basis of physiological measurements on the crew member and measurements of interaction of the crew member with the aircraft, and from the theoretical reference cognitive state of the crew member determined by the crew resource determination module.
[0036] The method according to the invention may comprise one or more of the following characteristics, taken in isolation in any technically possible combination:
[0037] - the passage of the fitness indicator between a fitness state of the crew member to perform the task(s) on the task list assigned to the crew member and a state of inability of the crew member to perform the task(s) on the task list assigned to the crew member based on a comparison between the actual cognitive state of the crew member and the theoretical cognitive state of the crew member;
[0038] - it includes a modification, by a task reconfiguration module of mission, from the list of tasks to be carried out by the crew member when the suitability indicator changes to the state of the crew member being unable to carry out the task(s) on the task list.
[0039] The invention will be better understood on reading the following description, given solely by way of example, and made with reference to the appended drawings, in which:
[0040] - [fig.l] [fig.l] is a block diagram illustrating a first system of determination mination of the operational state of crew according to the invention, integrated within an aircraft;
[0041] - [fig.2] [fig.2] is a functional diagram illustrating the interactions between the different modules of the system of [fig.l];
[0042] - [fig.3] [fig.3] is a flowchart illustrating a first example of implementation of a method for determining the operational status of a crew by means of the system of [fig.l];
[0043] - [fig.4] [fig.4] is a flowchart illustrating a second example of implementation implementation of a method for determining the operational status of a crew via the system of [fig.l];
[0044] - [fig.5] [fig.5] is a flowchart illustrating a third example of implementation implementation of a method for determining the operational status of a crew via the system of [fig.l]; and
[0045] - [fig.6] [fig.6] is a flowchart illustrating a fourth example of implementation implementation of a method for determining the operational status of a crew via the system of [fig.l].
[0046] - [fig.7] [fig.7] is a flowchart illustrating a fifth example of implementation implementation of a method for determining the operational status of a crew via the system of [fig.l].
[0047] A first system 10 for determining the operational state of the crew of an aircraft 12, based on a plan of tasks to be carried out by the crew of the aircraft 12, is illustrated schematically in [fig.l].
[0048] The system 10 is intended to be connected to or integrated into a central avionics system 14 comprising a central avionics unit 16 and at least one display unit 18 placed in a control interface 19 of the aircraft 12.
[0049] The control interface 19 of the aircraft 12 is for example located in the aircraft 12 itself (in the cockpit), or in a remote control room of the aircraft 12. (in a ground station).
[0050] The central avionics unit 16 is in particular connected to equipment of the aircraft 12, intended to interact within functional systems of the aircraft.
[0051] The functional systems of the aircraft 12 include, for example, systems 20 for measuring the state of the aircraft, systems 22 for external communication, and systems 24 for actuating the controls of the aircraft. The functional systems of the aircraft 12 further advantageously comprise weapons systems 26, in the case of a military aircraft.
[0052] The measurement systems 20 comprise, for example, components comprising sensors for measuring parameters external to the aircraft, such as temperature, pressure or speed, sensors for measuring parameters internal to the aircraft and its various functional systems and positioning sensors, such as GPS sensors, inertial units, and / or an altimeter.
[0053] The external communication systems 22 include, for example, components comprising radio systems, VOR / LOC, ADS, DME, ILS, radar systems, and / or satellite communication systems such as “SATCOM”.
[0054] The control systems 24 include components comprising actuators capable of actuating controls of the aircraft, such as flaps, control surfaces, pumps, or even mechanical, electrical and / or hydraulic circuits, and software actuators capable of configuring the avionic states of the aircraft.
[0055] The weapons systems 26, when present, advantageously comprise target tracking and weapon guidance components and weapons as such, such as bombs or missiles.
[0056] The different systems 20 to 26 are connected to the central avionics unit 16, for example digitally, by at least one data bus circulating on a network internal to the aircraft 12.
[0057] The avionics central unit 16 comprises at least one task management system 28 capable of establishing, in real time, a list of crew tasks to be carried out, which depends on mission objectives updated by the environmental measurements carried out for example by the measurement systems 20 or received by the communication systems 22.
[0058] The tasks to be performed are, for example, navigation tasks, to guide the aircraft 12 and move it according to a flight plan comprising at least geographical points of passage of the aircraft 12 during the mission and times of passage at the points of passage.
[0059] Also, the tasks to be carried out are, for example, communication tasks to be carried out according to a communication plan comprising geographical communication points and / or times spent at geographical communication points. communication.
[0060] Furthermore, the tasks to be performed may be tasks linked to events external to the aircraft, such as meteorology, including geographical points of passage or avoidance of meteorological phenomena, for example, in areas of turbulence and / or storms.
[0061] In the context of a military aircraft, the tasks to be performed further advantageously include reconnaissance tasks, tasks of preparation for air or air-ground combat, and air or air-ground combat tasks such as including, for example, image analysis or weapon firing carried out from the aircraft.
[0062] The task list is generally determined initially, before the flight from a mission preparation system, from mission objectives. The tasks are updated as a function of the mission execution, for example because new mission objectives are determined by the mission commander as a function of breakdowns and / or emergencies and / or external weather or tactical events.
[0063] The task management system 28 is then able to modify the list of tasks, to add, delete, and / or change the order or characteristics of the tasks to be performed by the crew, based on updated mission objectives and a mission context updated based on the environment around the aircraft 12. The task plan comprising the list of tasks is thus adaptive. These modifications are carried out within the delegation limit of the system provided by the crew and may be subject to the agreement of the crew.
[0064] The crew operational status determination system 10 is illustrated schematically in [fig.l]. In this example, the system 10 comprises a computer 30 comprising a processor 32 and a memory 34 receiving software modules capable of being executed by the processor 32 to perform functions. Alternatively, the computer 30 comprises programmable logic components or dedicated integrated circuits, intended to perform the functions of the modules which will be described below.
[0065] With reference to [fig. 1], the memory 34 contains a module 40 for determining crew tasks to be performed at the current time, capable of retrieving a list of tasks to be performed by the crew, as a function of updated mission objectives and / or the updated mission context, and of assigning the tasks to the crew members, as a function of crew profile data contained in a database 42.
[0066] The memory 34 further contains a module 44 for determining crew resources, capable of calculating the resources required by each crew member to carry out each task on the list of tasks assigned to them and of determining a theoretical operational reference state for each crew member, to carry out the tasks assigned to them, on the basis of a resource model.
[0067] The memory 34 also contains a module 46 for determining at least one context indicator, capable of defining at least one context indicator on the basis of mission context data developed from sensors 48 of the measurement systems 20 of the aircraft 12, or from data transmitted to the aircraft 12 via the communication systems 20.
[0068] The memory 34 also contains a module 51 for determining interactions, suitable for determining the expected interactions with the aircraft systems, during the implementation of the tasks on the task list by each crew member.
[0069] The memory 34 further contains a module 50 for measuring states of interactions with the aircraft systems, on the basis of measurements carried out by interaction measurement sensors 52 and expected interactions determined by the module 51 and a module 55 for measuring physiological states of the crew connected to sensors 54 for measuring physiological data of the crew of the aircraft 12 and to a database 56 of basic physiological data of the crew.
[0070] The memory 34 finally contains a module 58 for determining local cognitive states, suitable for evaluating at least one local cognitive state of the crew with respect to the execution of each task defined in the list of tasks to be carried out, and a module 60 for determining global cognitive states of the crew linked to the general physiological state of the crew with respect to all of the tasks to be carried out.
[0071] A local cognitive state of the crew is a state of a mental process of a crew member associated with the performance of a particular task to be performed within the task list.
[0072] A global cognitive state of the crew is a state of a mental process of a crew member associated with the set of tasks to be performed by the crew member.
[0073] According to the invention, the memory 34 also contains a module 62 for determining the operational state of the crew, capable of comparing at least one actual operational state of a crew member, in particular, a global cognitive state and / or a local cognitive state determined by the modules 58, 60, with a theoretical reference operational state, in particular with a theoretical global cognitive state of reference and / or a theoretical local cognitive state of reference, determined by the determination module 44 using the resource model.
[0074] The memory 34 further contains a module 64 for reconfiguring tasks based on the operational state obtained by the state determination module 62, capable of modifying the list of tasks to be carried out by the crew member or the information provided to the crew.
[0075] The task determination module 40 is capable of importing, at any time, the tasks coming from the task management system 28 to obtain an ordered list of tasks, including for example the identification of targets, the monitoring of the flight plan, the shooting tasks to be carried out, etc.
[0076] Each task is associated with at least one crew member, whose profile is obtained from the database 42. The crew profile includes, for example, a level of expertise of the crew member in performing the tasks on the task list, as well as preferred modes of interaction.
[0077] The preferred interaction modes are, for example, possible alternative strategies in carrying out tasks, in particular variants of task sequence orders. For example, certain strategies for carrying out a task are known to an expert user having expertise in carrying out the task, allowing him, for example, to use shortcuts or to optimize his interactions with the interfaces.
[0078] The resource determination module 44 is capable, on the basis of the task lists, and on the basis of the resources necessary to implement the task list, obtained from a resource model, of determining, for each crew member, at least one theoretical reference operational state of the crew member, in particular at least one theoretical reference global cognitive state and / or at least one theoretical reference local cognitive state associated with each task for the crew member.
[0079] Examples of global cognitive reference states are a theoretical level of mental load accepted for the crew member, a theoretical level of sleepiness accepted for the crew member, a theoretical level of crew member engagement accepted for the crew member, a theoretical level of hypoxia accepted for the crew member (this accepted level being a zero level) and / or a theoretical level of stress accepted for the crew member.
[0080] The mental load level is a numerical indicator representative of the demand for use of the crew member's mental capacities. The sleepiness level is a numerical indicator representative of the degree of sleepiness of the crew member, in particular in an intermediate state between wakefulness and sleep. The engagement level is a numerical indicator reflecting the overall attention of the crew member for carrying out tasks. The hypoxia level is a Boolean indicator representative of the adequacy or inadequacy between the tissue oxygen requirements and the oxygen supply for the crew member. The stress level is a numerical indicator representative of the nervous tension of the crew member.
[0081] These theoretical levels are predefined globally according to the crew profile and are intended to be compared to real levels, determined from data measured on the crew, as described below. These levels are for example digital levels measuring a value in an interval of possible values, discrete levels measuring one of a plurality of possible discrete values, or Boolean levels exhibiting either an acceptable value or an unacceptable value.
[0082] Examples of local cognitive reference states for performing a task are a minimum level of attention, a level of perseverance in performing each task, and a maximum level of visual and / or auditory tunneling for performing each task on the task list.
[0083] The level of attention associated with a task is a numerical indicator that reflects the crew member's availability to perform the task, in particular the ability of a crew to select and focus on specific information of a task. The level of perseveration is a numerical indicator that reflects, in a context that requires it, a lack of revision of a decision previously taken in the performance of a task. The level of tunneling is a numerical indicator reflecting the allocation of the crew's attention to a specific sensory channel, to the detriment of others, which results in an inability of the crew to take into account certain types of stimuli (for example, sound and / or visual) related to the task to be performed.
[0084] These levels are defined based on the crew member's crew profile and the task list assigned to the crew member.
[0085] The theoretical cognitive reference states are determined from the resource model which advantageously links each task to be carried out to a time and / or a resource level to be implemented by the crew member to carry out the task.
[0086] The determination module 44 is thus able to assign for all the tasks the necessary time as well as the necessary global state (for example: the necessary mental load) to carry out the tasks according to the profile of the crew member given by the database 42 and the context indicators defined by the context indicator determination module 46. Similarly, it is able to assign for each task in the list of tasks the necessary time as well as the necessary local state (for example: the necessary level of attention) to carry out said task according to the profile of the crew member.
[0087] The context indicator determination module 46 is capable of defining a level of complexity and / or a level of danger, as a function of measurements obtained by the environmental sensors 48, in particular on the number of adversaries present in the mission environment, the state of the meteorology, and / or the presence of dangerous zones.
[0088] The level of complexity and / or the level of danger is used by the determination module 44 to weight the resources necessary to execute each task, in particular in terms of time required or mental load required to perform tasks. For example, if the level of complexity or the level of danger increases, the time and mental load required to perform a task are likely to increase.
[0089] The interaction determination module 51 is capable of determining the expected interactions with the aircraft systems by each crew member, and the expected performance to carry out these interactions, on the basis of the list of tasks assigned to the crew member obtained from the determination module 40, the crew profile of the crew member, as determined in the database 42 and the context indicators determined by the determination module 46.
[0090] The interactions with the aircraft systems are, for example, screen areas or elements to be observed, keys or screen regions to be selected, or even commands to be actuated. The expected performance is, for example, a minimum speed of sequence of actions to be performed to implement one or more tasks, an error rate in the implementation of the tasks or even a total task completion rate among all the tasks to be performed.
[0091] The interaction measurement sensors 52 are for example capable of determining the position of the gaze on the display units 18 of the cockpit 19, the pressure and the pressure force on the controls, the position of a control cursor of the aircraft systems on a display unit 18 or on another interface.
[0092] Based on the measurements made by the interaction measurement sensors 52, the interaction state measurement module 50 is capable of comparing the interactions with the aircraft systems made by the crew member and the interactions with the aircraft systems expected for the crew member obtained from the module 51 based on the ordered list of tasks, to determine whether the crew member is performing each of the expected interactions.
[0093] The interaction state measurement module 50 is furthermore capable of determining whether the crew member meets the expected performance in terms of speed of execution, error rate and completion rate.
[0094] Based on the performance and actions performed by the crew member, the interaction state measurement module 50 is capable of determining a level of attention of the crew member to perform each task, a level of performance of the crew member in the execution of each task and a level of effectiveness of the strategy used to manage the tasks according to the preferred interaction modes defined above.
[0095] The physiological measurement sensors 54 comprise, for example, sensors for measuring heart rate, pupil diameter, blood oxygenation, emitted brain waves, posture (in particular via pressure pads in the seat and / or via image analysis systems using cameras), sweating, or frontal oxygenation.
[0096] The physiological measurement sensors 54 may be partially common with the interaction measurement sensors 52.
[0097] The physiological database 56 comprises, for each crew member, basic physiological state data of each crew member at rest.
[0098] These basic physiological state data are, for example, a resting heart rate, a resting pupil diameter, a resting blood oxygenation rate, a resting brain wave rate, a resting posture, a resting sweat rate, a resting frontal oxygenation rate.
[0099] On the basis of the measurements carried out by the physiological measurement sensors 54, compared to the basic physiological state data present in the database 56, the physiological state measurement module 55 is capable of determining in particular a level of mental fatigue of the crew member, and / or a level of concentration of the crew member.
[0100] The level of mental fatigue is for example determined from a measured percentage of eye closure, a number and duration of blinks, a level of relaxation of posture, a level of repetition of gestures, and / or certain frequency spectral powers (for example alpha waves) predominant in the brain.
[0101] The concentration level is for example determined from a prefrontal oxygenation level, for example a high prefrontal oxygenation level, heart rate variability, for example a low heart rate variability, ocular dispersion variation, for example a decrease in ocular dispersion, and certain high brain frequency spectral powers (theta waves in the frontal area)
[0102] The local cognitive state determination module 58 is capable of using the data produced by the physiological state measurement module 55 and the interaction state measurement module 50 to determine local cognitive states, linked to the performance of each of the specific tasks which are provided in the ordered list of tasks.
[0103] As indicated above, local cognitive states include, for example, a level of attention in carrying out tasks, for example determined from the analysis of the distribution of the pilot's gaze on his man / machine interfaces.
[0104] Local cognitive states include, for example, a level of perseveration. The level of perseveration is determined, for example, from frequency measurements of the brain, by a rate of reaction to stimuli, in particular a low rate of reaction to stimuli, by a rate of fixation on the task, in particular a high rate of fixation on the task.
[0105] A high level of perseverance is for example obtained when a pilot continues at all costs to try to land while external conditions would require a go-around.
[0106] The local cognitive states also preferably include a level of visual and / or auditory tunneling, which reflects excessive attention to a specific task, neglecting other tasks on the task list and / or visual and / or auditory stimuli external to the task. The level of tunneling is determined, for example, from ocular measurements, determination of absence of interaction on certain tasks, and determination of absence of response to certain stimuli (translated, for example, by the amplitude of the P300 evoked potentials in the brain, in particular by a low amplitude).
[0107] The global cognitive state determination module 60 is capable of using the data produced by the physiological state measurement module 55 and the interaction state measurement module 50 to determine global cognitive states of each crew member.
[0108] Global cognitive states include, for example, a level of global mental load. This level of global mental load is determined, for example, from prefrontal oxygenation, in particular high prefrontal oxygenation, heart rate variability, in particular a decrease in heart rate variability, performance on secondary tasks (delay, etc.), in particular a decrease in performance, variation in pupil diameter, in particular an increase in pupil diameter.
[0109] The global cognitive states advantageously comprise a global level of sleepiness. This global level of sleepiness is determined for example from the posture, in particular a relaxed posture, the number of blinks of the eyes, in particular an increase in the number of blinks of the eyes, and a determination of absence of interactions.
[0110] The global cognitive states advantageously comprise a level of engagement, which reflects the rate of tasks performed in relation to the tasks to be performed. The level of engagement is determined for example from a ratio of the brain's frequency waves (beta / alpha+delta), also called "engagement index", in particular higher than at rest, and / or from an analysis of ocular data for example to identify a ratio of "long fixations + short saccades / long saccades + short fixations", in particular when this ratio is low.
[0111] The global cognitive states advantageously include a level of hypoxia, representative of the mismatch between tissue oxygen requirements and oxygen supplies for the crew member. This level of hypoxia is a Boolean level. It is determined for example from a heart rate and respiratory rate above predetermined thresholds reflecting an abnormality, a drop in blood oxygen saturation below a threshold, incoherent sentences, and / or blinking of the eyes above a determined threshold.
[0112] The overall cognitive states also preferably include a stress level. The stress level is determined, for example, from microsweating, in particular heavy microsweating, muscle tension, in particular heavy muscle tension, heart rate variability, in particular heavy heart rate variability, gaze jerk / fixation ratio, in particular a high gaze jerk / fixation ratio, voice frequency, in particular a voice that becomes higher pitched, and / or an upright posture in the seat.
[0113] The crew operational state determination module 62 is capable of analyzing the global cognitive states and the local cognitive states and comparing them respectively to the reference global cognitive states and to the reference local cognitive states calculated by the resource determination module 44 to determine an indicator of the crew member's ability, for example a Boolean indicator, capable of switching between a state of the crew member's ability to perform the tasks on the list of tasks assigned to them and a state of the crew member's inability to perform tasks on the list of tasks assigned to them.
[0114] For example, the crew operational state determination module 62 is capable of evaluating whether the effective mental load level determined by the global cognitive state determination module 60 is greater than the maximum theoretical mental load level calculated by the resource determination module 44. In this case, it is capable of changing the crew member's aptitude indicator from a aptitude state to an unaptitude state.
[0115] Similarly, the crew operational state determination module 62 is advantageously capable of evaluating whether the effective stress level determined by the global cognitive state determination module 60 is greater than the maximum stress level calculated by the resource determination module 44. In this case, it is capable of changing the crew member's aptitude indicator from a state of aptitude to a state of inaptitude.
[0116] The crew operational state determination module 62 is advantageously capable of evaluating whether the effective level of drowsiness determined by the global cognitive state determination module 60 is greater than the maximum level of drowsiness calculated by the resource determination module 44. In this case, it is capable of changing the crew member's fitness indicator from a fit state to a unfit state.
[0117] The crew operational state determination module 62 is advantageously capable of evaluating whether the effective engagement level determined by the crew operational state determination module mination of global cognitive states 60 is higher than the level of engagement calculated by the resource determination module 44. In this case, it is suitable for changing the crew member's aptitude indicator from a state of aptitude to a state of inaptitude.
[0118] The crew operational state determination module 62 is advantageously capable of evaluating whether the effective hypoxia level determined by the global cognitive state determination module 60 is greater than the authorized hypoxia level, here a zero level. In this case, it is capable of changing the crew member's fitness indicator from a fit state to an unfit state.
[0119] Furthermore, the crew operational state determination module 62 is advantageously capable of evaluating whether the effective tunneling level and / or the perseverance level determined by the local cognitive state determination module 58 exceed respectively the theoretical maximum tunneling level and / or the theoretical maximum perseverance level determined by the resource determination module 44, and whether the attention level determined by the local cognitive state determination module 58 is lower than the attention level determined by the resource determination module 44.
[0120] In each of these cases, it is suitable to change the crew member's fitness indicator from a fit state to an unfit state.
[0121] The operational state determination module 62 is capable, when an unfitness state is determined, of defining a type of task reconfiguration or crew information modification to be carried out by the task configuration module 64.
[0122] The task configuration module 64 is advantageously capable of modifying the list of tasks to carry out a task in place of the crew, to delete a task, or to postpone a task to be carried out in time.
[0123] It is also capable of generating an alert and / or an alarm in the case where the operational status determination module 62 determines a state of unfitness of the crew, or even of temporarily deleting information displayed to the crew on the display units 18, and / or of displaying new information replacing the existing information.
[0124] Examples of implementation of a method for determining the operational state of a crew according to a task plan using the determination system 10 according to the invention will now be described, with reference to FIGS. 3 to 7.
[0125] In the example illustrated by [fig.3], in a civil aircraft context, the aircraft 12 operates a cruise phase, after the takeoff of the aircraft 12.
[0126] The task management system 28 defines the tasks to be performed, which are essentially tasks of tracking the trajectory of the aircraft, in particular the passage by waypoints, aircraft fuel consumption 12 and communication tasks, when passing through various airspaces.
[0127] The crew member in charge of monitoring is likely to enter a state of drowsiness.
[0128] As indicated above, in step 100 of the method according to the invention, the task determination module 40 establishes at each instant the list of tasks to be carried out, and the distribution of tasks between the crew members intended to carry them out using the crew profile database 42.
[0129] Then, in step 102, the resource determination module 44 defines a maximum level of sleepiness accepted for performing these tasks, based for example on a maximum level of sleepiness associated with each individual task.
[0130] In step 104, based on the measurements obtained from the physiological sensors 54, the physiological state measurement module 55 determines an increase in the frequency and duration of eye blinks, an increase in the percentage of eyelid closure, and a relaxed posture.
[0131] Based on these parameters, the global cognitive state determination module 60 calculates an effective drowsiness level.
[0132] In step 106, the operational state determination module 62 determines that the actual drowsiness level is greater than the maximum permissible drowsiness level for all current tasks. It therefore changes the crew member's fitness indicator from the crew member's fitness state to perform the tasks on the task list assigned to them to the crew member's fitness state to perform tasks on the task list assigned to them.
[0133] The operational state determination module 62 also determines that the type of reconfiguration is to trigger a wake-up alarm and temporarily automate the piloting of the aircraft.
[0134] In step 108, the reconfiguration module 64 therefore sends instructions to the aircraft system to generate an alarm in the cockpit 19.
[0135] In a second example, illustrated by [fig.4], during an approach phase to a landing runway with a strong crosswind, the pilot is concentrated on a piloting task.
[0136] If an important alarm is triggered, it may not be treated as a priority.
[0137] In this regard, the task determination module 40 determines a list of tasks relating to piloting, and a list of tasks relating to the processing of the alarm, which may be a red CAS alarm requiring priority action from the pilot from the list of tasks.
[0138] In step 110, the interaction modeling module 51 determines that the crew member should handle the tasks associated with the alarm, performing a number of interactions in the cockpit 19 including on the display units 18 and / or using the controls.
[0139] In step 112, the interaction state measurement module 50 determines, from the position of the cursor and the position of the gaze, that the crew member is not carrying out any of the interactions linked to the alarm.
[0140] Furthermore, the interaction state measurement module 50 establishes that the crew member does not react to the sound and visual stimuli resulting from the alarm which are outside his field of vision.
[0141] On this basis, it determines a level of attention of the crew member to carry out each task, and a level of performance of the crew member in carrying out each task.
[0142] Similarly, in step 114, the physiological state measurement module 55 determines a pupil diameter with little visual dispersion, a low heart rate variability, for example less than 20% compared to the variability at rest and a high heart rate compared to the heart rate at rest, for example at least greater than 20% of the heart rate at rest, from the databases 56 of basic physiological data.
[0143] The physiological state measurement module 55 further determines an absence of amplitude of the P300 evoked potentials following a sound stimulus (alarm).
[0144] Based on the information obtained by the modules 50, 55, the global cognitive state determination module 60 determines a level of mental load, and a level of stress. The local cognitive state determination module 58 determines a level of tunneling.
[0145] In step 116, the operational state determination module 62 then determines that the effective mental load level is greater than the maximum mental load level determined by the determination module 44 and that the tunneling level exceeds the maximum tunneling level determined by the determination module 44. It defines a state of inability of the crew member to carry out the requested tasks, in particular to process the mission alarms. The crew member is here in a state of attentional deafness.
[0146] The operational state determination module 62 further determines that the action to be taken would be to temporarily suppress the display of the information on which the pilot is focusing.
[0147] In step 118, the reconfiguration module 64 therefore controls the display units 18 in the cockpit 19 to delete at least one item of information relating to piloting in crosswinds, in order to allow the pilot to identify that another urgent alarm is present and must be processed.
[0148] In a third example, shown in [fig.5], the system 10 detects a state of panic present in a crew member.
[0149] This can occur for example when multiple alarms are displayed.
[0150] As previously, in step 120, the module 40 for determining tasks to be performed determines that a plurality of tasks must be performed simultaneously and defines for each task a level of mental load and a level of stress associated with the task.
[0151] The interaction modeling module 51 determines the expected interactions with the cockpit interface and the expected performance to perform the tracking tasks.
[0152] In step 122, based on the measurements from the interaction measurement sensors 52, the interaction state measurement module 50 compares the expected interactions for each crew member with the aircraft systems obtained from the module 51 based on the ordered list of tasks. It identifies a dispersion in the interactions, and the fact that no task is completed.
[0153] It therefore calculates a level of attention and a level of engagement. These levels are low.
[0154] Furthermore, in step 124, based on the physiological sensors 54, the physiological state measurement module 55 determines the presence of numerous saccades in the movement of the eyes, a very high heart rate, and sudden and heavy sweating.
[0155] On this basis, the global cognitive state determination module 60 calculates a stress level. This stress level is high.
[0156] In step 126, the operational state determination module 62 then identifies a high stress level, higher than the maximum stress level defined by the determination module 44. This stress level is characteristic of a state of panic which makes the crew member unfit to carry out the safety-critical tasks.
[0157] It identifies a level of attention and a level of engagement lower respectively than the minimum level of attention and the minimum level of engagement determined by the determination module 44.
[0158] It therefore defines a state of inability of the crew member to carry out the requested tasks.
[0159] The operational state determination module 62 further determines that a synthesis of the information and the procedure to be followed must be carried out by the reconfiguration module 64.
[0160] In step 128, the reconfiguration module controls the display units 18 of the central avionics unit 16 to provide an oral and graphic summary of the information and the procedure to follow.
[0161] Another example will now be described with reference to [fig.6], in the context of a military aircraft during a high altitude air combat.
[0162] As previously, in step 130, the task determination module 40 determines the list of tasks to be performed during this air combat phase. The interaction determination module 51 determines the expected interactions with the aircraft systems by the crew member, and the expected performance to perform these interactions, on the basis of the list of tasks assigned to the crew member obtained from the determination module 40, the crew profile of the crew member, as determined in the database 42 and the context indicators determined by the determination module 46.
[0163] In step 132, based on the data from the interaction sensors 52, the interaction state measurement module determines that the reaction time increases significantly, and that the responses do not correspond to what is expected at the interaction level.
[0164] The global cognitive states determination module 60 thus determines a level of engagement, and the local cognitive states determination module 58 determines a level of attention. These levels are low.
[0165] In parallel, in step 134, on the basis of the data from the physiological sensors 54, the physiological state measurement module 55 determines a decrease in blood oxygen saturation, and an abnormal posture characterized by an absence of movement.
[0166] The global cognitive state determination module 60 deduces a level of hypoxia. This level of hypoxia is unacceptable.
[0167] In step 136, the operational state determination module 62 identifies that the hypoxia level is greater than a maximum hypoxia level determined by the determination module 44, here a zero level. It therefore defines a state of inability of the crew member to carry out the requested tasks.
[0168] It also determines the actions to be carried out for the reconfiguration module 64, which consist of letting the avionics central unit 16 resume piloting, and carrying out an emergency descent.
[0169] In step 138, the reconfiguration module 64 sends orders to the avionics central unit 16 to carry out this takeover and the emergency descent.
[0170] In another example, shown in [fig.7], still in the context of a military aircraft, during an air-ground combat mission, the pilot is forced to carry out numerous tasks in a limited time.
[0171] In step 140, the task determination module 40 determines the list of tasks to be performed. The interaction determination module 51 determines the expected interactions with the aircraft systems by the crew member, and the performance expected to perform these interactions, based on the list of tasks assigned to the crew member obtained from the determination module 40, the crew profile of the crew member, as determined in the database 42 and the context indicators determined by the determination module 46.
[0172] In step 142, based on the data from the interaction sensors 52, the interaction state measurement module 50 determines repetitive errors on the interactions to be carried out and a reduction in performance, in particular on tasks which are not a priority or not planned in the list of initial tasks.
[0173] The global cognitive states determination module 60 thus determines a level of engagement, and the local cognitive states determination module 58 determines a level of attention. These levels are very high.
[0174] In parallel, in step 144, on the basis of the data from the physiological sensors 54, the physiological state measurement module 55 determines a high prefrontal oxygenation, above a given threshold, the dominance of theta and beta waves at the frontal level, a large pupil diameter, a very low heart rate variability. It deduces a high concentration level,
[0175] The global cognitive state determination module 60 calculates a level of mental load on this basis. This level of mental load is high.
[0176] In step 146, the operational state determination module 62 then determines a level of mental load greater than the maximum level of mental load determined by the determination module 44. It therefore defines a state of inability of the crew member to carry out all of the requested tasks.
[0177] The operational state determination module 62 then determines a type of reconfiguration consisting of the avionics central unit 16 automatically taking over certain lower priority tasks.
[0178] In step 148, the reconfiguration module 64 therefore sends instructions to the avionics central unit 16 so that it takes charge of the tasks thus defined.
[0179] The determination system 10 according to the invention is therefore capable of developing an operational state of the crew, as a function of a resource model built on the list of crew tasks, by measuring separately the physiological states and the interaction states of the crew, even if these measurements come a priori from data from the same sensors and by separately determining local cognitive states, relating to the execution of a task and global cognitive states, relating to all the tasks.
[0180] The determination system 10 updates the operational state as the task list changes, for example in the case where a new task appears, or as a function of changes in the mission objectives, or the context observed in the environment.
[0181] This makes the determination system 10 very interactive, since it dynamically takes into account the context of the mission, and the resources of the crew.
[0182] Thanks to its generic functional architecture, the determination system 10 is also easily configurable to adapt to different types of sensors, missions and platforms. It takes into account the specific characteristics of the crew members.
[0183] Advantageously, the determination system 10 further establishes reconfiguration actions helping to resolve the identified problems.
[0184] Thus, the determination system 10 easily interprets the state of the crew, which improves the performance of the crew, reduces its stress and its mental load.
Claims
Claims
1. System (10) for determining an operational state of a crew of a manned or unmanned aircraft (12), based on an adaptive task plan, the system (10) comprising: - a module (40) for determining crew tasks to be performed, capable of defining, at each instant during a mission of the aircraft (12), a list of tasks to be performed by at least one crew member based on updated mission objectives, and an instantaneous mission context determined from mission environment measurements; - a module (44) for determining crew resources, capable of determining for the or each crew member, at least one theoretical cognitive reference state of the crew member based on the task list assigned to the crew member, using a resource model;- a module (62) for determining the operational state of the crew, capable of determining an indicator of the ability of the or each crew member to carry out the task(s) on the task list assigned to the crew member, from an effective cognitive state of the crew member determined on the basis of physiological measurements on the crew member and measurements of interaction of the crew member with the aircraft (12), and from the theoretical reference cognitive state of the crew member determined by the crew resource determination module (44).;
2. The system (10) of claim 1, wherein the operational state determination module (62) is adapted to switch the ability indicator between a state of ability of the crew member to perform the task(s) on the task list assigned to the crew member and a state of inability of the crew member to perform the task(s) on the task list assigned to the crew member, based on a comparison between the actual cognitive state of the crew member and the theoretical cognitive state of the crew member.
3. System (10) according to claim 2, comprising a mission task reconfiguration module (64), capable of modifying the list of tasks to be carried out by the crew member when the ability indicator changes to the state of inability of the crew member to carry out the task(s) on the list of tasks assigned to the crew member.
4. The system (10) of claim 3, wherein the reconfi- task planning (64) is capable of deleting and / or postponing a task to be performed, of performing a task in place of the crew member, of deleting information given to the crew member, and / or of replacing the information given to the crew member with other information.
5. System (10) any one of the preceding claims, comprising a module (60) for determining global cognitive states of the crew member as a function of the physiological and interaction measurements of the crew member and a module (58) for determining local cognitive states of the crew, capable of determining local cognitive states relating to the execution of tasks by the crew member as a function of the physiological and interaction measurements of the crew member with the aircraft (12),the crew operational state determination module (62) being capable of calculating the effective cognitive state of the crew member as a function of the local cognitive states and the global cognitive states determined respectively by the global cognitive state determination module (60) and by the local cognitive state determination module (58) and of comparing each local or global cognitive state of the crew member with the theoretical local or global reference cognitive state of the crew member determined by the crew resource determination module (44) on the basis of the resource model.,
6. The system (10) of claim 5, wherein the global cognitive states are selected from a mental load level, an engagement level, a sleepiness level, a hypoxia level, and / or a crew member stress level.
7. System (10) according to any one of claims 5 or 6, wherein the local cognitive states are chosen from an attention level, a perseveration level, a visual tunneling level, an auditory tunneling level in the performance of a task.
8. System (10) according to any one of claims 5 to 7, comprising a module (55) for measuring physiological states from physiological data measurement sensors (54), in particular from sensors for measuring heart rate, pupil diameter, the number of blinks, blood oxygenation, brain waves, evoked potentials, and from a database (56) of basic physiological states of the crew, the module for determining global cognitive states (60) determining the cognitive states global from the physiological state measurements of the physiological state measurements module (55).
9. System (10) according to any one of the preceding claims, comprising a module (50) for measuring states of interactions with the crew, from sensors (52) for measuring interactions with the crew, in particular the position of the gaze, the use of physical controls, the use of touch, and from a model of expected interactions on the basis of the list of tasks determined by the module for determining current crew tasks (40), the module for determining local cognitive states (58) determining the local cognitive states from the measurements of states of interactions of the module for measuring states of interactions (50).
10. System (10) according to claim 9, in which the interaction states are chosen from at least one level of attention, one level of performance, one strategy for conducting tasks.
11. System (10) according to any one of the preceding claims comprising a context indication module (46), based on mission environment measurements, capable of defining at least one context indicator, the crew resource determination module (44) being capable of determining the theoretical cognitive reference state of the crew member based on the or each context indicator.
12. System (10) according to any preceding claim, in which the current crew task determination module (40) is capable of assigning the list of tasks to the crew member according to a database (42) of crew profiles determining the characteristics of each crew member and advantageously the capabilities of the aircraft to take charge of the tasks, and according to the objectives of the mission.
13. Method for determining an operational state of a crew of a manned or unmanned aircraft (12), based on an adaptive task plan comprising the following steps: - providing a determination system (10) according to any one of the preceding claims; - defining, by the current crew task determination module (40), at each instant during a mission of the aircraft (12), a list of tasks to be carried out by at least one crew member based on updated mission objectives, and an instantaneous mission context determined from mission environment measurements; - determination, by the crew resource determination module (44), for the or each crew member, of at least one theoretical cognitive reference state of the crew member based on the task list assigned to the crew member, using a resource model; - determination by the crew operational state determination module (62), of an indicator of the ability of the or each crew member to carry out the task(s) on the task list assigned to the crew member from an effective cognitive state of the crew member determined on the basis of physiological measurements on the crew member and measurements of interaction of the crew member with the aircraft (12), and from the theoretical reference cognitive state of the crew member determined by the crew resource determination module (44).
14. The method of claim 13, comprising switching the ability indicator between a state of ability of the crew member to perform the task(s) on the task list assigned to the crew member and a state of inability of the crew member to perform the task(s) on the task list assigned to the crew member based on a comparison between the actual cognitive state of the crew member and the theoretical cognitive state of the crew member.
15. A method according to claim 14, comprising a modification, by a mission task reconfiguration module (64), of the list of tasks to be carried out by the crew member when the suitability indicator changes to the state of inability of the crew member to carry out the task(s) on the task list.
