Method and device for evaluating workload of pilot in high dynamic scene

Abstract

The invention provides a method and a device for evaluating the workload of a pilot in a high-dynamic scene, and the method comprises the steps: inputting the audio and video information of a cockpit according to the real-time operation of a computer; the method comprises the following steps: decomposing uncertain actions made by a pilot when executing a task in a high-dynamic scene in a time window with a preset length into basic action elements with fine granularity and certainty; and evaluating the workload of the pilot according to the types of the basic action elements appearing in the time window with the preset length, the duration of each basic action element and the total resource consumption value of each basic action element. The method is real-time, accurate, quantitative and non-invasive, and a certain reference is provided for efficiency improvement and task arrangement optimization of new-generation aviation equipment. The method is suitable for flight training, task optimization and man-machine system design.

Description

Technical Field

[0001] This invention belongs to the field of aviation human factors engineering and workload assessment technology, specifically relating to a method and apparatus for assessing pilot workload in high dynamic scenarios. Background Technology

[0002] To prevent pilots from being under high workload for extended periods, which could lead to cumulative fatigue and pose a threat to flight safety, existing research has assessed pilot workload using various methods and information sources.

[0003] In patents CN117883085A, CN117454230A, CN116313079A, CN115177254A, and CN111407292A, the inventors collected physiological data such as EEG, eye movement, and ECG from pilots during flight missions and used data processing models to measure their workload. While using physiological data to measure pilot workload typically offers high accuracy, reliability, and real-time performance, the physiological data acquisition equipment often interferes with and affects the pilot's normal operations. Furthermore, high-dynamic flight scenarios are often accompanied by significant maneuvering overloads, which can greatly interfere with the acquisition of physiological data.

[0004] In patents CN118396449A, CN115115270A, CN117390437A, and CN116313079A, the inventors modeled, scored, or evaluated the scenarios or tasks pilots might face or perform during flight, constructing a pilot workload assessment based on task complexity and task completion performance. This method is suitable for pilot workload assessment in scenarios with standard flight manuals and fixed task sequences, such as pilot workload assessment for civil aircraft and military transport aircraft. In highly dynamic scenarios, the environment changes rapidly, and pilots need to react instantly to the constantly changing situation. Therefore, there is no fixed task sequence, and workload assessment cannot be based on task scoring.

[0005] The aforementioned invention is suitable for pilot workload assessment in scenarios where the flight process is relatively stable and pilots have standard operating manuals and standard mission sequences. In highly dynamic scenarios, pilots need to react instantly to constantly changing environments, and their actions are highly uncertain, making the pilot workload assessment method constructed in the aforementioned patents unsuitable. In summary, existing methods cannot achieve non-intrusive, dynamically adaptable, and high-precision workload assessment in highly dynamic scenarios. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the problem that existing pilot workload assessment technology cannot be used in scenarios where the pilot's tasks and actions are highly uncertain, and to provide a method and apparatus for pilot workload assessment in highly dynamic scenarios.

[0007] The first aspect of this invention provides a method for assessing pilot workload in highly dynamic scenarios, comprising: Based on the real-time operation input of the computer and the audio and video information of the cockpit, the uncertain actions made by the pilot when performing a mission in a high-dynamic scenario within a preset time window are decomposed into fine-grained, deterministic basic action elements. The pilot's workload is assessed based on the types of basic action elements appearing within a preset time window, the duration of each basic action element, and the total resource consumption of each basic action element.

[0008] Optionally, conduct a pilot workload assessment, including: Obtain the pilot's workload value caused by each basic action element that appears within the time window; The pilot workload values ​​caused by each basic motion element are scaled and varied. The cumulative values ​​after scaling changes form the pilot's workload assessment results.

[0009] Optionally, obtain the pilot workload values ​​caused by each basic motion element occurring within the time window, including: Using formula Get the pilot's workload value caused by each basic action element that appears within the time window; in, The first one to appear within the time window The workload value of each basic motion element for the pilot The length of the time window used for workload assessment; The first occurrence within this time window The duration of each element; For the first The total resource consumption of each basic action element; i is a positive integer from 1 to N, where N is the total number of basic action elements within the time window.

[0010] Optionally, the pilot workload values ​​caused by each basic motion element can be scaled, including: Using formula The scale of the pilot's workload caused by each basic motion element is varied. Where Q represents the total amount of resources.

[0011] Optional basic motion elements include: listening, turning, operating instruments, pressing, pulling, fine-tuning levers, pushing / pressing levers, left and right levers, pushing / pulling throttle, pedals, turning head, and twisting head.

[0012] A second aspect of the present invention provides an apparatus for assessing pilot workload in highly dynamic scenarios, comprising: The decomposition module is used to decompose the uncertain actions performed by the pilot in a high-dynamic scenario within a preset time window into fine-grained, deterministic basic action elements based on the real-time operation input of the computer and the audio and video information of the cockpit. The assessment module is used to assess the pilot's workload based on the types of basic action elements appearing in a preset time window, the duration of each basic action element, and the total resource consumption value of each basic action element.

[0013] Optionally, the evaluation module is specifically used for: The workload acquisition unit is used to acquire the pilot's workload value caused by each basic action element that appears within the time window. The scaling unit is used to scale the pilot workload values ​​caused by each basic motion element. The accumulation unit is used to accumulate the values ​​after scaling changes to form the pilot's workload evaluation results.

[0014] Optionally, the load value acquisition unit is specifically used for: Using formula Get the pilot's workload value caused by each basic action element that appears within the time window; in, The first one to appear within the time window The workload value of each basic motion element for the pilot The length of the time window used for workload assessment; The first occurrence within this time window The duration of each element; For the first The total resource consumption of each basic action element; i is a positive integer from 1 to N, where N is the total number of basic action elements within the time window.

[0015] Optionally, the scaling unit is specifically used for: Using formula The scale of the pilot's workload caused by each basic motion element is varied. Where Q represents the total amount of resources.

[0016] Optional basic motion elements include: listening, turning, operating instruments, pressing, pulling, fine-tuning levers, pushing / pressing levers, left and right levers, pushing / pulling throttle, pedals, turning head, and twisting head.

[0017] This invention provides a method and apparatus for assessing pilot workload in highly dynamic scenarios. By decomposing pilot actions into kinematic elements, it overcomes the problem that traditional workload assessment methods cannot be used in highly dynamic scenarios with uncertainty. Therefore, this workload assessment method is suitable for assessing pilot workload in highly dynamic scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 It is a process for generating a list of pilot dynamic elements and scoring the overall resource consumption value of each dynamic element in high dynamic scenarios; Figure 2 It is a pilot workload assessment process in high-dynamic scenarios. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The features and illustrative embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific setups and methods set forth below, but covers any improvements, substitutions, and modifications to structures, methods, and devices without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description to avoid unnecessarily obscuring the invention.

[0022] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing and simplifying the invention, and should not be construed as limiting the invention. Furthermore, the use of ordinal numbers (e.g., "first and second," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.

[0023] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly, encompassing both direct connection and indirect connection via an intermediate medium. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0024] It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other, and the various embodiments can be referenced and cited in each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0026] This invention provides a method and apparatus for assessing pilot workload in highly dynamic scenarios. The assessment method includes: decomposing the uncertain actions performed by the pilot in a highly dynamic scenario within a fixed time window into fine-grained, deterministic basic action elements (kinetic elements) based on real-time computer operation input and cockpit audio and video information; and using the types of kinematic elements appearing in the time window, the duration of each kinematic element, and the total resource consumption of each kinematic element as input to assess the pilot workload.

[0027] The list of motion elements used for pilot motion decomposition needs to be compiled in advance. Specifically, by observing and analyzing the audio and video of pilots conducting flight training in typical high-dynamic scenarios on a ground simulator, the uncertain movements of the pilots are decomposed into fine-grained, deterministic basic motion elements (motion elements). After obtaining a complete list of motion elements approved by the pilots and experts, the pilots and experts are invited to score the overall resource consumption value of each motion element.

[0028] Existing workload assessment technologies largely rely on physiological sensors (such as EEG and ECG) or fixed task sequence modeling, which suffers from problems such as high task invasiveness and inability to adapt to uncertainties in highly dynamic scenarios. This invention abandons physiological data acquisition and instead relies on real-time computer input and cockpit audio-visual information to acquire pilot motion data through non-contact means, completely eliminating interference from invasive devices on pilot operations. Simultaneously, this invention innovatively decomposes the pilot's complex and uncertain motion sequences into fine-grained basic motion elements, effectively solving the assessment failure problem caused by the non-fixed task sequences in high-dynamic scenarios of traditional methods.

[0029] The list of motion elements and scoring system summarized in this invention have been verified by experts to cover typical high-dynamic scene actions and support dynamic updates and expansions. When adding motion elements, only the scoring needs to be supplemented to seamlessly integrate them into the existing evaluation framework without reconstructing the model, demonstrating good versatility and adaptability. This feature makes it not only suitable for flight training but also extendable to other high-dynamic human-computer interaction fields.

[0030] Figure 1 The flowchart illustrates the generation of the dynamic element list and the scoring of the overall resource occupancy value for each dynamic element in the pilot workload assessment method under high dynamic scenarios according to the present invention. Figure 1 As shown, audio and video acquisition equipment is deployed in the cockpit of a ground simulator to collect video and cockpit audio of pilots performing typical high-dynamic mission training. Through observation and analysis of the audio and video, all fine-grained basic motion elements (kinetic elements) are extracted, such as "adjusting the throttle by hand," "operating buttons on the instrument panel by hand," and "adjusting the rudder by foot." The extracted kinematic elements are initially verified to ensure that each kinematic element meets the standards of clear boundaries, independent identifiability, and reusability across missions, thus forming a preliminary kinematic element list. This list is then jointly reviewed by aviation human factors engineering experts and senior pilots to ensure that it covers as many possible actions a pilot might perform in a high-dynamic scenario, forming a complete "High-Dynamic Scenario Pilot Kinematic Element List." Subsequently, aviation human factors engineering experts and senior pilots are invited to rate the overall resource consumption value of each kinematic element from 0 to 7. The average of the scores from different experts and pilots is used to obtain a score table of the overall resource occupancy value of each kinematic element for assessing pilot workload in high-dynamic scenarios, as shown in Table 1.

[0031] Table 1. List of pilot dynamic elements and their total resource consumption in high-dynamic scenarios.

[0032] Figure 2This document demonstrates the operational flow of a pilot workload assessment method for high-dynamic scenarios. During real flight or simulated training, real-time input from the onboard computer (such as joystick displacement, throttle angle, and button trigger states) and the cockpit's audio-visual acquisition equipment collect pilot motion data in real time. Based on a pre-summarized "high-dynamic scenario pilot motion element list," the uncertain motion data within a fixed-length time window is decomposed and transformed into a deterministic motion element sequence, forming a pilot workload assessment window. The default time window length is 5 seconds, and two time windows do not overlap. For the first motion element appearing within a time window... The workload of individual factors on pilots The calculation formula is as follows:

[0033] in, The length of the time window used for workload assessment; the default value is T = 5 seconds. The first occurrence within this time window The duration of each element; For the first The total resource consumption of each element is shown in Table 1.

[0034] Aviation human factors engineering experts have determined that within a 5-second time window, pilots typically perform no more than four actions. Based on this conclusion, the following basic assumption is made: if a pilot simultaneously performs four actions within a certain period, and the overall resource utilization of these four actions is 7, then the pilot's workload is 100%. Based on this basic assumption, for each... Perform the following scaling transformation:

[0035] Finally, the formula for calculating the pilot workload within this time window is as follows:

[0036] Example: Table 2 shows a pilot workload assessment window generated after decomposing and converting pilot actions into dynamic elements during a high-dynamic flight training session. In this table, the row direction represents time, and the column direction represents the type of dynamic element. If the subject performed a certain dynamic element at a certain moment, 1 is filled in the table corresponding to that dynamic element time; otherwise, 0 is filled in.

[0037] Table 1. Dynamic Elements-Timeline Table (Excerpt)

[0038] Table 2 shows that within this time window, the pilot performed a total of four actions: a 2-second earpiece cue and warning tone, a 2-second manual pull / push of the control stick, a 3-second manual left / right operation of the control stick, and a 5-second head-turn search. The pilot workload values ​​caused by these four actions are calculated as follows:

[0039]

[0040]

[0041]

[0042] The scale variations of the pilot workload values ​​caused by these four dynamic elements are as follows:

[0043]

[0044]

[0045]

[0046] Ultimately, the pilot workload during this time window is as follows:

[0047] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A method for assessing pilot workload in high-dynamic scenarios, characterized in that, include: Based on the real-time operation input of the computer and the audio and video information of the cockpit, the uncertain actions made by the pilot when performing a mission in a high-dynamic scenario within a preset time window are decomposed into fine-grained, deterministic basic action elements. The pilot's workload is assessed based on the types of basic action elements appearing within a preset time window, the duration of each basic action element, and the total resource consumption of each basic action element.

2. The method for assessing pilot workload in high-dynamic scenarios according to claim 1, characterized in that, Conduct pilot workload assessments, including: Obtain the pilot's workload value caused by each basic action element that appears within the time window; The pilot workload values ​​caused by each basic motion element are scaled and varied. The cumulative values ​​after scaling changes form the pilot's workload assessment results.

3. The method for assessing pilot workload in high-dynamic scenarios according to claim 2, characterized in that, Obtain the pilot workload values ​​caused by each basic action element occurring within the time window, including: Using formula Get the pilot's workload value caused by each basic action element that appears within the time window; in, The first one to appear within the time window The workload value of each basic motion element for the pilot The length of the time window used for workload assessment; The first occurrence within this time window The duration of each element; For the first The total resource consumption of each basic action element; i is a positive integer from 1 to N, where N is the total number of basic action elements within the time window.

4. The method for assessing pilot workload in high-dynamic scenarios according to claim 3, characterized in that, The pilot workload values ​​caused by each basic motion element are scaled, including: Using formula The scale of the pilot's workload caused by each basic motion element is varied. Where Q represents the total amount of resources.

5. The method for assessing pilot workload in high-dynamic scenarios according to claim 1, characterized in that, Basic movement elements include: listening, turning, operating instruments, pressing, pulling, fine-tuning levers, pushing / pressing levers, moving levers left and right, pushing / pulling the throttle, pedaling, turning the head, and twisting the head.

6. A device for assessing pilot workload in high-dynamic scenarios, characterized in that, include: The decomposition module is used to decompose the uncertain actions performed by the pilot in a high-dynamic scenario within a preset time window into fine-grained, deterministic basic action elements based on the real-time operation input of the computer and the audio and video information of the cockpit. The evaluation module is used to evaluate the pilot's workload based on the types of basic action elements appearing in a preset time window, the duration of each basic action element, and the total resource consumption value of each basic action element.

7. The apparatus for assessing pilot workload in high-dynamic scenarios according to claim 6, characterized in that, The evaluation module is specifically used for: The workload acquisition unit is used to acquire the pilot's workload value caused by each basic action element that appears within the time window. The scaling unit is used to scale the pilot workload values ​​caused by each basic motion element. The accumulation unit is used to accumulate the values ​​after scaling changes to form the pilot's workload evaluation results.

8. The apparatus for assessing pilot workload in high-dynamic scenarios according to claim 7, characterized in that, The load value acquisition unit is specifically used for: Using formula Get the pilot's workload value caused by each basic action element that appears within the time window; in, The first one to appear within the time window The workload value of each basic motion element for the pilot The length of the time window used for workload assessment; The first occurrence within this time window The duration of each element; For the first The total resource consumption of each basic action element; i is a positive integer from 1 to N, where N is the total number of basic action elements within the time window.

9. The apparatus for assessing pilot workload in high-dynamic scenarios according to claim 8, characterized in that, The scale-changing unit is specifically used for: Using formula The scale of the pilot's workload caused by each basic motion element is varied. Where Q represents the total amount of resources.

10. The apparatus for assessing pilot workload in high-dynamic scenarios according to claim 6, characterized in that, Basic movement elements include: listening, turning, operating instruments, pressing, pulling, fine-tuning levers, pushing / pressing levers, moving levers left and right, pushing / pulling the throttle, pedaling, turning the head, and twisting the head.

Images