Utilizing behavioral elicitation events for evaluating training performance against competency framework

Abstract

To provide a method of automatically evaluating the training performance of a training subject.SOLUTION: The method comprises: receiving a simulation state update from a computing system of a training environment; querying a behavioral elicitation event (BEE) database to determine a set of active BEEs based upon the simulation state update; receiving a plurality of simulation action notifications from the computing system; for each simulation action notification, determining whether an action represented by the simulation action notification matches an elicited action for an active BEE; storing information regarding the elicited action as a response to the active BEE; determining a set of completed BEEs comprising any active BEEs that have all corresponding elicited actions met; and, for each completed BEE, storing information regarding the completed BEE.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE

[0001] Embodiments of the present disclosure relate to the automated assessment of training subjects against a competency framework. [Background technology]

[0002]

[0002] Some industries assess professionals against a competency framework specified during professional training. The competency framework may include soft skills exhibited by the trainee in addition to hard skills performed during training within the training environment. Some exemplary soft skills include communication, leadership, teamwork, situational awareness, information management, hard skill task management, and hard skill procedural knowledge. For example, commercial pilot training uses competency-based training and assessment (CBTA) to assess trainee pilots on core pilot competencies defined by the International Civil Aviation Organization (ICAO). ICAO currently assesses trainee pilots based on nine pilot competencies (i.e., knowledge application, procedure application and rule compliance, communication, airplane flight path management for automation, airplane flight path management for manual pilots, leadership and teamwork, problem solving and decision making, situational awareness and information management, and workload management). The current method of soft skills assessment is by instructor observation during training. Such observational assessments can sometimes make it difficult for instructors to provide objective assessments and / or feedback to trainees. Summary of the Invention

[0003]

[0003] One disclosed embodiment provides a method for evaluating a training subject's training performance against a competency framework. Training takes place in a training environment simulating a physical environment. The method includes receiving simulation state updates from a computing system of the training environment, and querying a behavioral elicitation event (BEE) database to determine a set of active BEEs from a plurality of BEEs stored in a BEE database based on the simulation state updates. Each BEE of the plurality of BEEs includes one or more elicited actions and one or more observable behaviors. The method further includes receiving a plurality of simulation behavior notifications from the computing system of the training environment. The method further includes determining, for each simulation behavior notification, whether the action represented by the simulation behavior notification matches an elicited action for at least one active BEE in the set of active BEEs, and storing information regarding the elicited action as a response to at least one active BEE in the set of active BEEs. The method further includes determining a set of completed BEEs. The set of completed BEEs includes each active BEE for which all corresponding elicited behaviors have been satisfied. For each completed BEE in the set of completed BEEs, the method stores information regarding training performance, including one or more observable behaviors of the completed BEE.

[0004] Another embodiment provides a computing system including a logic system operatively coupled to a training environment simulating a physical environment and a computer readable storage system. The computer readable storage system includes a BEE database including a plurality of behavioral elicitation events (BEEs) and instructions executable by the logic system. Each BEE of the plurality of BEEs includes one or more elicited actions and one or more observable behaviors. The instructions are executable to receive simulation state updates from the computing system of the training environment, query the BEE database to determine a set of active BEEs based on the simulation state updates, and receive a plurality of simulation behavior notifications from the plurality of BEEs. The instructions are further executable to determine, for each simulation behavior notification, whether the action represented by the simulation behavior notification matches an elicited action for at least one active BEE in the set of active BEEs, and store information regarding the elicited action as a response to at least one active BEE in the set of active BEEs. The instructions are further executable to determine a set of completed BEEs. The set of completed BEEs includes each active BEE for which all corresponding invoked behaviors have been satisfied. The instructions are further executable to, for each completed BEE in the set of completed BEEs, store information about the completed BEE.

[0005]

[0005] Another embodiment provides a method for evaluating a training subject's training performance against a competency framework. Training takes place in a training environment simulating a physical environment. The method includes receiving a first simulation state update from a computing system of the training environment, and querying a behavioral elicitation event (BEE) database to determine a first set of active BEEs from a plurality of BEEs stored in a BEE database based on the first simulation state update. Each BEE of the plurality of BEEs includes one or more elicited actions and one or more observable behaviors. The method further includes receiving a plurality of simulation behavior notifications from the computing system of the training environment. The method further includes determining, for each simulation behavior notification, whether the action represented by the simulation behavior notification matches an elicited action for at least one active BEE in the first set of active BEEs, and storing information regarding the elicited action and the corresponding observable behavior in response to at least one active BEE in the first set of active BEEs. The method further includes receiving a second simulation state update from a computing system of the training environment, querying a BEE database to determine a second set of active BEEs based on the second simulation state update, and comparing the first set of active BEEs to the second set of active BEEs. For each active BEE included in the first set of active BEEs and not included in the second set of active BEEs, the method stores the active BEE as a non-completed BEE and also stores information about the non-completed BEE. [Brief description of the drawings]

[0006] [Figure 1]

[0006] A block diagram of an example computing system configured to utilize a BEE database to evaluate training performance is shown. [Diagram 2]

[0007] 2 illustrates a block diagram of an exemplary BEE database used by the computing system of FIG. 1. [Diagram 3]

[0008] FIG. 1 illustrates a block diagram of an exemplary engine ice protection BEE. [Figure 4A]

[0009] 4A and 4B illustrate generally an exemplary algorithm that uses the BEE database of FIG. 1 to assess the training performance of a test subject. [Figure 4B] 4A and 4B illustrate generally an exemplary algorithm that uses the BEE database of FIG. 1 to assess the training performance of a test subject. [Diagram 5]

[0010] 4 illustrates a table of example simulation state updates corresponding to successful completion of the engine anti-icing BEE of FIG. [Figure 6]

[0011] 6 and 7 show schematic diagrams of example sub-processes of the algorithm of FIGS. 4A and 4B. [Figure 7] 6 and 7 show schematic diagrams of example sub-processes of the algorithm of FIGS. 4A and 4B. [Figure 8]

[0012] 1 shows a flowchart illustrating an exemplary method for evaluating the training performance of a training subject against a competency framework. [Figure 9]

[0013] FIG. 1 illustrates a block diagram of an exemplary computing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007]

[0014] As previously discussed, pilot training uses competency-based training and assessment (CBTA) to assess trainees (pilots) against the ICAO Competency Framework. Current methods of using CBTA in training solutions have instructors assess trainees on the ICAO Competency Framework by observing the trainees during training runs, such as during training in a flight simulator. In addition, the instructor must also provide supporting evidence showing how the trainees succeeded or failed to perform observable behaviors corresponding to each competency in the ICAO Competency Framework. Such current assessment methods present several challenges. First, the ICAO Competency Framework mostly includes soft skills. Soft skills are difficult for instructors to objectively assess and provide feedback to trainees upon completion of the ICAO Competency Framework. For example, observational assessments may be difficult to provide traceable information between actions performed by the trainee and the ICAO Competency Framework. Due to the lack of traceability, it may be difficult to provide recommendations to assist the trainee in improving their training performance in the next training run over their current training performance. Another problem with observation is that similar training performance may be evaluated differently among different instructors. Furthermore, instructors need to be present during the training run for observation. This may add costs to pilot training to cover the instructor's salary and training.

[0008]

[0015] Accordingly, embodiments are disclosed that relate to utilizing computer-implemented behavioral call events (BEEs) to automate the evaluation of training subject's training performance against a competency framework. A BEE is an event that invokes a specific behavior from the training subject to demonstrate soft skill competencies specified in the competency framework. Training is performed by the training subject in a training environment that simulates a physical environment. Exemplary training environments include a fixed training device (FTD) flight simulator, a full motion simulator (FMS) flight simulator, and a virtual training environment. As described in more detail below, embodiments of the present disclosure utilize a BEE database that includes a plurality of BEEs to evaluate training performance. Each BEE of the plurality of BEEs includes one or more triggers, one or more invoked actions, and one or more observable behaviors, as described in more detail in connection with FIG. 2. The disclosed embodiments receive simulation state updates from the training environment and determine a set of active BEEs based on the simulation state updates by querying the BEE database. The simulation state updates indicate changes in the simulation state of the training environment.

[0009]

[0016] Next, the disclosed embodiments receive simulation behavior notifications from the training environment. The simulation behavior notifications indicate corresponding user inputs received by the training environment. When the behavior represented by the simulation behavior notifications matches an invoked behavior for an active BEE in the set of active BEEs, the disclosed embodiments store information about the invoked behavior and the corresponding observable behavior as a response to the active BEE. Further, when all corresponding invoked behaviors of the active BEE have been satisfied, the disclosed embodiments determine that the active BEE is a completed BEE.

[0010]

[0017] The disclosed embodiments further store information about any completed and / or incomplete BEEs. The information can be used as evidence to support how the training subject performed the various competencies in the competency framework during the training run. The information can also be used for instructional purposes and to create training scenarios tailored to individual training subjects. The use of the BEE database allows scenario-independent behavior detection for evaluation. This allows for faster creation of training scenarios than current methods where each scenario is created independently of each other.

[0011]

[0018] FIG. 1 illustrates a block diagram of an exemplary computing system 100 in communication with a training environment. The training environment, in this example, takes the form of an aircraft training environment 102, where the aircraft training environment 102 is configured to simulate an aircraft environment used in training pilots. In some examples, the aircraft training environment 102 may be configured as an FTD, an FMS, or a virtual training environment. A training subject 104 performs training in the aircraft training environment 102 using a training interface 106 to interact with a training computing system 108. The training computing system 108 is configured to execute computer-simulated training scenarios. In some examples, the training interface 106 and the aircraft training environment 102 may be configured to simulate a cockpit in the aircraft training environment 102 and receive user input from the training subject 104. Examples of user input include voice input, button presses or hardware selections into the training interface 106, video input, and other suitable input from the training subject 104.

[0012]

[0019] In some embodiments, the training computing system 108 may be further configured to utilize machine learning algorithms to analyze various user inputs, such as, for example, using speech-to-text recognition to determine the content of a voice input or using a classifier to identify a specified gesture in a video input. Additionally, machine learning algorithms may be used in conjunction with physiological measurements to help determine whether the training subject 104 is paying attention to a desired portion of a cockpit in the aircraft training environment 102. Such a configuration may help generate a perceptual model that may be used to detect evoked behaviors corresponding to monitoring various competencies in a competency framework, such as, for example, cognitive workload management competencies. The training subject 104 may further input user inputs in response to a computer-simulated training scenario, such as, for example, changes in one or more of a simulated flight phase, a simulated weather, or a simulated aircraft state. In other embodiments, the aircraft training environment 102 may have a different configuration. In still further embodiments, a training environment other than an aircraft training environment may be used.

[0013]

[0020] The computing system 100 is configured to evaluate the training performance of the training subject 104 against a competency framework, such as, for example, ICAO pilot competencies. The computing system 100 comprises a processor 110 operatively coupled to the aircraft training environment 102 and a storage system 112. The storage system 112 comprises a BEE database 114 including a plurality of BEEs, as described in connection with FIG. 2. The storage system 112 further comprises a BEE monitor 116 configured to be executable by the processor 110. The storage system 112 further comprises a learning record store (LRS) 118 configured to store information related to the evaluation of the performance of the training subject 104. In some embodiments, the LRS 118 may also be configured to store information from the aircraft training environment 102. In other embodiments, information from the training environment and / or information related to the performance evaluation may be stored in a different location or across multiple locations. Further aspects of the processor 110 and the storage system 112 are described in connection with FIG. 9. In other embodiments, the computing system 100 may be configured to receive Flight Operational Quality Assurance (FOQA) data recorded during live flight and detect pilot behavior against ICAO competency frameworks. Such configuration may help determine training recommendations for an airline fleet and / or determine the effectiveness of an airline fleet training program. Although described herein with reference to the aircraft training environment 102, the BEE database may be used to evaluate other training performance against any suitable competency framework in other embodiments.

[0014]

[0021] Each BEE stored in the BEE database indicates an event that elicits a particular action from the training subject. FIG. 2 shows a block diagram of the BEE database 114 in more detail. The BEE database 114 includes a first BEE 200. The first BEE 200 includes one or more triggers 202 that define conditions when the first BEE 200 is an active BEE. More specifically, each trigger may be an active trigger when a corresponding condition is met in a simulated state of the training environment. In the pilot training examples, the trigger may include conditions of a specified simulated air temperature, a specified flight phase of the simulated aircraft, and a specified fuel level, altitude, speed, and / or other status of the simulated aircraft. Further, the flight phase examples include stationary, cruise, takeoff, landing, taxiing, climbing, descending, and approach. The trigger 202 may include an optional timing condition 204 that indicates a specified time for responding to the trigger 202. The timing condition 204 may also indicate whether the training subject 104 is expected to respond before a specified time or to wait until after a specified time. As one particular example, a fuel leak BEE has a trigger with a timing condition to wait until a desired time before taking action on an increase in fuel flow rate. The first BEE 200 further includes one or more evoked actions 206 that define expected user inputs in response to when the first BEE 200 is an active BEE. Similar to the trigger 202, the evoked action 206 may include a timing condition 208. In some examples, the trigger 202 and / or the evoked action 206 may include logical statements (not shown for clarity), such as, for example, Boolean logic (e.g., AND, OR, etc.), sequence statements, and conditional statements. The first BEE 200 further includes one or more observable behaviors 210 that indicate competencies demonstrated by the training subject 104 by performing the one or more evoked actions 206. Further, each observable behavior 210 may be aligned to a specified competency in a competency framework, such as a specified pilot competency in ICAO competencies.One particular embodiment of a BEE is provided in FIG.

[0015]

[0022] Similarly, the BEE database 114 includes a second BEE 212 having one or more triggers, one or more evoked actions, and one or more observable behaviors. In this manner, the BEE database may be useful for aligning evoked actions with observable behaviors for any suitable competency framework. The BEE database may thereby be useful for detecting evidence of observable behaviors in another simulated training environment. Although FIG. 2 shows two BEEs for clarity, the BEE database 114 may include any suitable number of BEEs, such as a single BEE or three or more BEEs, in other embodiments. The BEE database 114 is exemplary. In other embodiments, the BEE database 114 may include other configurations.

[0016]

[0023] FIG. 3 illustrates a block diagram of an exemplary engine anti-icing BEE 300. The engine anti-icing BEE 300 may be used to evaluate a pilot's training performance within a training environment, as described with reference to FIGS. 4A and 4B. In one embodiment of FIG. 3, the engine anti-icing BEE 300 corresponds to an engine anti-icing scenario during a simulated stationary flight phase within the aircraft training environment 102. Accordingly, the engine anti-icing BEE 300 includes a stationary flight phase flight phase indicator 302 that indicates that the engine anti-icing BEE 300 may be active during the simulated stationary flight phase. The engine anti-icing BEE 300 further includes a first trigger 304 configured to be an active trigger when a simulated air temperature of the training environment is less than 10° C. Similarly, the second trigger 306 is configured to be an active trigger when the training environment has a simulated visible air humidity.

[0017]

[0024] The engine anti-icing BEE 300 further includes a first evoked action (EA) 308 and a second evoked action 310, where the first evoked action 308 defines a user input to the aircraft training environment 102 indicating that the pilot has selected an engine anti-icing procedure on an electronic flag bag of the aircraft training environment 102. The second evoked action 310 defines a user input indicating that the pilot has verbally communicated to the co-pilot that an engine anti-icing procedure has been selected.

[0018]

[0025] The engine anti-icing BEE 300 further includes a first set of observable behaviors (OB) 312 that define competencies related to the pilot's demonstrated behavior of performing the first evoked behavior 308. Here, the first set of observable behaviors 312 indicate that the training subject demonstrated knowledge of procedures, knowing where to find necessary information, situational awareness, communication, problem solving, following standard operating procedures, and proper task planning, prioritization, and scheduling. Similarly, the engine anti-icing BEE 300 also includes a second set of observable behaviors 314 that are related to responses to the second evoked behavior 310. The second set of observable behaviors 314 indicate that the training subject demonstrated conveying messages clearly, accurately, and concisely. The first and second sets of observable behaviors 312, 314 correspond to pilot competencies defined in ICAO competencies. FIG. 3 is exemplary. In other embodiments, the engine anti-icing BEE 300 may have another configuration.

[0019]

[0026] One or more computing systems may be configured to execute an algorithm that uses the BEE database to evaluate the training performance of a training subject in a training environment, such as the aircraft training environment 102. FIGS. 4A and 4B show a general flow chart of an example algorithm 400 that uses the BEE database 114, where the computing system 402 and the computing system 100 execute the algorithm 400. In some embodiments, the computing system 402 may be part of the training computing system 108. The computing system 402 comprises a learning record store (LRS) 404 configured to store information regarding user input from the training subject, such as, for example, interactions with other individuals participating in the training environment, and information regarding actions performed by the training subject. The LRS 404 is further configured to store information regarding simulated conditions of the training environment, such as, for example, simulated weather. The computing system 402 further comprises an experience application programming interface (xAPI) monitor 406 and an xAPI logger 408. Briefly, the xAPI monitor 406 is configured to monitor the LRS 404 for any new statements received from the training environment, and the xAPI logger 408 is configured to send information from the BEE monitor 116 to the LRS 404 for storage. In other embodiments, another configuration of one or more computing systems comprising a BEE database may execute the algorithm 400.

[0020]

[0027] 4A, the xAPI monitor 406 queries the LRS 404 session for update statements including simulation state updates or simulation behavior notifications at 410. In response, the LRS 404 returns the update statements to the xAPI monitor 406 at 412. The xAPI monitor 406 sends the update statements to the BEE monitor 116 at 414. Alternatively, in one embodiment of FIG. 4B, the querying for update statements at 410 and the returning of the update statements at 412 are omitted.

[0021]

[0028] After the BEE monitor 116 receives the update statement, when the update statement includes a simulation behavior notification, the algorithm 400 moves to a corresponding sub-process as described with reference to FIG. 7, as shown at 416. Alternatively, when the update statement includes a simulation behavior notification, the algorithm 400 moves to a corresponding sub-process as described with reference to FIG. 6, as shown at 418. The embodiment of FIG. 4A or FIG. 4B may be selected based on the configuration of the training environment. For example, an embodiment of FIG. 4A may be selected for use with an FTD or FMS. An embodiment of FIG. 4B may be selected for use with a virtual training environment. FIGS. 4A and 4B are exemplary. In other embodiments, the algorithm 400 may use another overview flowchart.

[0022]

[0029] For illustrative purposes, the sub-processes of the algorithm 400 are described with reference to the engine anti-icing BEE 300. Accordingly, FIG. 5 illustrates a table of example update statements 500 corresponding to the successful performance of the training subject 104 in response to the engine anti-icing BEE 300. Here, a first simulation state update 502 includes a simulated weather change reflecting a simulated air temperature of 0° C. Further, a second simulation state update 504 includes a simulated weather change reflecting the presence of a simulated visible air humidity. Next, a first simulation behavior notification 506 indicates a corresponding user input to select an engine anti-icing procedure in the electronic flight bag of the aircraft training environment 102. Further, a second simulation behavior notification 508 indicates a corresponding user input to communicate that an engine anti-icing procedure has been selected. Such user input may include, for example, a voice input. In other embodiments, the algorithm 400 may use another BEE and / or other update statements.

[0023]

[0030] As mentioned above, the algorithm 400 is described using one embodiment of the engine anti-icing BEE 300 during a simulated stationary flight phase. In one current embodiment, the algorithm 400 will execute two or more loops through various sub-processes. During each loop, the algorithm 400 executes various sub-processes and / or skips various sub-processes. The first loop begins when the xAPI monitor 406 sends a first simulation state update 502 to the BEE monitor 116 at 414, and the algorithm 400 moves to FIG. 6, as described above. Referring to FIG. 6, the BEE monitor 116 queries the BEE database 114 at 600 for any triggers associated with the first simulation state update 502 (simulated air temperature is 0° C.). In response, the BEE database 114 returns at 602 the first trigger 304 of the engine anti-icing BEE 300 (air temperature is less than 10° C.), where the first trigger 304 is an active trigger. Thus, the BEE monitor 116 stores the first trigger 304 in the set of active triggers at 604. In this example, based on the first simulation state update 502, there are no active triggers that become inactive triggers, so the BEE monitor 116 skips 606. In other examples, the BEE monitor 116 may remove any inactive triggers from the set of active triggers at 606. Inactive triggers are triggers that have conditions that are not met by the current state of the training environment. As a particular example, a change in the simulated flight phase may cause one or more triggers to become inactive. In this manner, the set of active triggers may be useful to allow for quick querying of any active BEEs.

[0024]

[0031] Next, the BEE monitor 116 queries the BEE database 114 for any active BEEs based on the set of active triggers at 608. As previously described, an active BEE has all corresponding triggers as active triggers. Here, the engine anti-icing BEE 300 has one active trigger (first trigger 304) and one inactive trigger (second trigger 306). As a result, the BEE database 114 does not send any active BEEs to the BEE monitor 116 in response to the query at 610. Furthermore, since there are no BEEs in the preceding set of active BEEs and there are no BEEs in the set of active BEEs as a result of the first trigger 304 being in the set of active triggers, the monitor 116 skips 612 and returns to 420.

[0025]

[0032] In other embodiments, when a BEE is in a prior set of active BEEs and is not in the set of active BEEs, the BEE monitor 116 determines a set of uncompleted BEEs at 612, including one or more BEEs that are in a prior set of active BEEs and are not in the set of active BEEs. For each uncompleted BEE in the set of uncompleted BEEs, the algorithm 400 may store information about the uncompleted BEE. In particular, the BEE monitor 116 queries the BEE database 114 for each corresponding trigger with optional timing conditions at 614 and / or for each corresponding evoked action at 616, for the observable behavior of the uncompleted BEE. In response, the BEE database 114 returns the corresponding observable behavior of the uncompleted BEE at 618 and 620, respectively. The BEE monitor 116 then sends an evaluation of the observable behavior for each uncompleted BEE to the xAPI logger 408 based on the corresponding observable behavior of the uncompleted BEE at 622. The evaluation of the observable behavior may indicate that a subset of the corresponding observable behaviors were exhibited and / or partially exhibited in response to the uncompleted BEE. As a specific example, the evaluation may indicate that two invoked actions were completed, but there was an opportunity for five invoked actions to be completed. Subsequently, at 624, the xAPI logger 408 then sends information regarding the observable behaviors that were exhibited, partially exhibited, and / or could not be exhibited to the LRS 404 for storage. In this manner, the LRS 404 may store information regarding the invoked actions that were completed, as well as information regarding the invoked actions that were not performed in response to the uncompleted BEE.

[0026]

[0033] Returning to one embodiment of the engine de-icing BEE 300, the algorithm 400 continues in the second loop when the xAPI monitor 406 sends a second simulation state update 504 (visible humidity in the air) at 414. The BEE monitor 116 skips 416 and moves to the sub-process associated with the simulation state update at 418. In FIG. 6, the BEE monitor 116 again queries the BEE database 114 at 600 for any triggers associated with the second simulation state update 504. Here, the BEE monitor 116 receives the second trigger 306 (simulated visible air humidity) at 602 and stores the second trigger 306 in a set of active triggers at 604. The BEE monitor 116 further queries the BEE database 114 at 608 for any active BEEs based on the set of active triggers. Now, the first trigger 304 (weather less than 10° C.) and the second trigger 306 (visible air humidity) of the engine anti-icing BEE 300 are active. Thus, the BEE database 114 stores the engine anti-icing BEE 300 in the set of active BEEs. As with the first loop, as a result of the second simulation state update 504, no BEEs become incomplete BEEs, and therefore the algorithm 400 returns to 420.

[0027]

[0034] Next in the third algorithm 400, the xAPI monitor 406 sends a first simulation behavior notification 506 (user input selecting an engine anti-icing procedure) at 414. Here, the behavior represented by the first simulation behavior notification 506 matches the first evoked behavior 308 of the engine anti-icing BEE 300, so the algorithm 400 moves to 7 at 416. Referring now to FIG. 7, the first observable behavior 308 does not have a corresponding timing condition. Similarly, there are no engine anti-icing BEEs 300 that have optional timing conditions. Thus, the algorithm 400 skips 700 and 702.

[0028]

[0035] In other embodiments, the observable behavior may have an optional timing condition, as shown at 704, and / or the corresponding trigger may have an optional timing condition, as shown at 706. In such embodiments, the BEE monitor 116 checks at 700 whether the simulation behavior notification is within the timing condition of the invoked behavior. As a specific example, a fuel leak BEE has a fuel flow trigger, an increase fuel flow indication, with a timing condition. Furthermore, the timing condition indicates that a specified duration will elapse before taking the corresponding invoked behavior in response to the fuel flow trigger. The BEE monitor 116 further checks at 702 whether the timing of the trigger corresponds to the invoked behavior. Information regarding the timing of the simulation behavior notification and / or the timing of the trigger may be included in the stored BEE evaluation message.

[0029]

[0036] Returning to one embodiment of the engine de-icing BEE 300, the BEE monitor 116 queries the BEE database 114 for one or more observable behaviors (OB) corresponding to the first evoked behavior 308 at 708. The BEE monitor 116 receives the first set of observable behaviors 312 at 710. Additionally, the BEE monitor 116 stores information regarding the first evoked behavior 308 in response to the engine de-icing BEE 300 by sending a BEE evaluation to the xAPI logger 408 at 712, which in response sends a response message to the LRS 404 for storage at 714. The information may include the first set of observable behaviors 312 as, for example, behavior exhibited by the training subject performing the first evoked behavior 308. In other embodiments, other information regarding the evoked behavior in response to an active BEE may be stored in another suitable manner.

[0030]

[0037] Continuing with the current example, the engine anti-icing BEE 300 has the first invoked action 308 satisfied and the second invoked action 310 not satisfied. Thus, at least one corresponding invoked action of the engine anti-icing BEE 300 is not satisfied and therefore the engine anti-icing BEE 300 is still an active BEE. Thus, the algorithm 400 skips 716 and moves to querying for active triggers at 600 in FIG. 6. In this example, the first simulation behavior notification 506 (user input selecting the engine anti-icing procedure) does not match the conditions for another trigger and therefore the BEE database 114 does not return any new active triggers. Furthermore, there is no new BEE to become an active BEE and therefore the algorithm 400 returns to 420.

[0031]

[0038] Similarly, the algorithm 400 begins a fourth loop and moves to 416 when the BEE monitor 116 receives a second simulation behavior notification 508 at 414. In Figure 7, the BEE monitor 116 queries for observable behaviors (OB) corresponding to the second evoked behavior 310 at 708 and receives a second set of observable behaviors 314 at 710. Additionally, the BEE monitor 116 stores information regarding the second evoked behavior 310 as a response to the engine de-icing BEE 300 using the xAPI logger 408, as described above with reference to the first evoked behavior 308.

[0032]

[0039] Now, in one current embodiment, the engine anti-icing BEE 300 has all corresponding invoked actions (e.g., both the first and second invoked actions 308, 310) fulfilled. Thus, the BEE monitor 116 determines at 716 that the engine anti-icing BEE 300 is a completed BEE. As previously discussed, the engine anti-icing BEE 300 does not have a trigger with a timing condition, so the BEE monitor 116 skips 718. The BEE monitor 116 then queries at 720 for the observable behavior exhibited by simply executing each corresponding invoked action of the completed BEE. In one current embodiment of the engine anti-icing BEE 300, the BEE database 114 returns at 722 a set of observable behaviors 312 for the first invoked action 308 and a second set of observable behaviors 314 for the second invoked action 310. In response, the BEE monitor 116 sends an observable behavior evaluation message to the xAPI logger 408 for each observable behavior in the first and second sets of observable behaviors 312, 314 at 724. At 726, the xAPI logger 408 sends an observable behavior indication message to the LRS 404 for storage. In this manner, the observable behaviors exhibited by the pilot in the context of the engine anti-icing BEE 300 are stored. In other embodiments, the BEE may have a trigger with a timing condition, as shown at 718. In such embodiments, the BEE monitor 116 queries the exhibited observable behavior for the timing of the corresponding invoked action relative to the timing of the BEE trigger at 728. At 730, the BEE monitor 116 receives the observable behavior and further includes the timing information in the evaluation of the observable behavior sent at 724.

[0033]

[0040] As with the third loop, the algorithm 400 then moves to querying for a trigger at 600 in FIG. 6. Since no new trigger was activated as a result of the training subject performing the second evoked behavior 310 or the completion of the engine anti-icing BEE 300, the algorithm 400 skips back to 420 to await a new simulation state update. In this manner, the algorithm 400 utilizing the BEE database 114 may evaluate a pilot's training performance against ICAO competencies. FIGS. 6 and 7 are exemplary. Although the algorithm 400 and corresponding sub-processes are described with reference to evaluating a pilot against the engine anti-icing BEE 300, the algorithm 400 may be used to evaluate the training performance of another training subject against another competency framework in other embodiments. Similarly, in further embodiments, another algorithm utilizing the BEE database 114 is used to evaluate any training subject against any suitable competency framework implemented within any suitable training environment that simulates a physical environment.

[0034]

[0041] 8 illustrates a flow chart depicting an example method 800 for evaluating a training subject's training performance against a competency framework. The training performance is performed in a training environment that simulates a physical environment, such as, for example, aircraft training environment 102. In one such example, the competency framework may include, for example, the ICAO competency framework. In other examples, a different training environment and / or a different competency framework may be used. Method 800 may be performed by any suitable computing system that includes a BEE database, such as, for example, computing system 100.

[0035]

[0042] Method 800 includes, at 802, receiving a simulation state update from a computing system of a training environment, such as, for example, training computing system 108. In some embodiments simulating an aircraft environment, the simulation state update may include a change in one or more of a simulated flight phase, a simulated weather, or a simulated aircraft state. In other embodiments, the simulation state update may include another change in a simulated state of the physical environment (e.g., a simulated weather, a simulated location, a simulated alarm, etc.).

[0036]

[0043] Returning, the method 800 queries a BEE database to determine a set of active BEEs from a plurality of BEEs stored in the BEE database based on the simulation state updates, at 804. For example, the BEE database 114 may be used. As described above, each BEE of the plurality of BEEs includes one or more triggers, one or more evoked actions, and one or more observable behaviors. Querying the BEE database to determine a set of active BEEs includes, at 806, querying the BEE database for one or more triggers having a condition that matches the simulation state updates. Determining the set of active BEEs also includes, at 808, storing the one or more triggers as a set of active triggers. Furthermore, at 810, determining the set of active BEEs includes determining one or more inactive triggers in the set of active triggers by comparing a corresponding condition for each active trigger in the set of active triggers with the simulation state updates, and removing the one or more inactive triggers from the set of active triggers. Determining the set of active BEEs further includes querying a BEE database to determine whether any BEEs are active BEEs based on the set of active triggers, at 811. In other embodiments, the set of active BEEs may be determined in another suitable manner.

[0037]

[0044] The method 800 further includes receiving, at 812, a plurality of simulation behavior notifications from a computing system of the training environment. The plurality of simulation behavior notifications are indicative of corresponding plurality of user inputs received by the training environment. Examples of user inputs include detected voice, selected input on a user interface of the training environment, detected eye movement, and / or detected gesture. In other embodiments, the method 800 may again query a BEE database to determine the set of active BEEs, as described with reference to 804. The method 800 further includes, at 814, for each simulation behavior notification, determining whether the behavior represented by the simulation behavior notification matches an invoked behavior for at least one active BEE in the set of active BEEs. When the behavior represented by the simulation behavior notification matches the invoked behavior, the method 800 includes, at 816, determining whether the invoked behavior has a timing condition and storing information regarding the timing of the simulation behavior notification. The timing condition may indicate that a desired timing of the simulation behavior notification is after a desired duration or at a desired duration. Alternatively, the timing condition may indicate that a desired timing of the simulated behavior notification is within a desired duration. Method 800 further includes, at 818, comparing the timing of the simulated behavior notification to a timing of a corresponding trigger of the at least one active BEE and storing information regarding the timing comparison. In other embodiments, method 800 may omit 816 and / or 818.

[0038]

[0045] Method 800 includes storing information about evoked actions and corresponding observable behavior in response to at least one active BEE in the set of active BEEs at 820. Method 800 compares a prior set of active BEEs to the set of active BEEs at 822, and for each active BEE included in the prior set of active BEEs and not included in the set of active BEEs, stores the active BEE as a non-completed BEE. Method 800 stores information about training performance at 824, including one or more non-evoked actions in response to the non-completed BEE. In other embodiments, 822 and 824 may be omitted.

[0039]

[0046] The method 800 includes, at 826, determining a set of completed BEEs. The set of completed BEEs includes each active BEE for which all corresponding triggered behaviors have been satisfied. At 828, the method 800 stores, for each completed BEE in the set of completed BEEs, information regarding training performance. The information includes one or more observable behaviors of the completed BEE. In some embodiments, the information may further include any suitable information regarding the completed BEEs, such as, for example, timing information.

[0040]

[0047] Using the BEE database for the evaluation of training performance as described herein may provide various advantages. The evaluation may provide data about training performance that helps instructors objectively evaluate soft skill competencies and increase the consistency of inter-rater reliability over instructor observational evaluation. The evaluation may also help provide detailed evidence to be used when asserting the proficiency of a training recipient performing training. Furthermore, the evaluation may help provide traceability between the evoked behaviors and the specified competencies within the competency framework. Such traceability helps provide feedback to the training recipient regarding recommendations for increasing the level of competencies during the next training run over the current training run. Furthermore, the evaluation may help enable tailoring of additional training scenarios for the training recipient based on the current training performance and / or prior training performance.

[0041]

[0048] Additionally or alternatively, using the BEE for evaluation across multiple training performances can provide data that allows for analysis across multiple training audiences to determine recommendations for updating training scenarios used in the training environment. Additionally, this data can also be useful for evaluating the effectiveness of the training environment, particularly a virtual training environment. Similarly, such evaluation can be useful for analyzing operational data to determine the effectiveness of training scenarios and / or to determine recommended training scenarios across multiple training audiences.

[0042]

[0049] In some embodiments, the methods and processes described herein may be coupled to a computing system of one or more computing devices. In particular, such methods and processes may be implemented in hardware, as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products, as described above.

[0043]

[0050] 9 illustrates a simplified representation of an exemplary computing system 900 configured to provide any of the computing functions described herein. The computing system 900 may take the form of one or more of a personal computer, a server computer, and an aircraft integrated computer, as examples. The computing system 100, the training computing system 108, and the computing system 402 are examples of computing systems 900.

[0044]

[0051] Computing system 900 includes a logic subsystem 902 and a storage subsystem 904. Computing system 900 may optionally include a display subsystem 906, an input subsystem 908, a communication subsystem 910, and / or other subsystems not shown in FIG.

[0045]

[0052] Logic subsystem 902 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions, which may be part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform an operation, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.

[0046]

[0053] The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processors of the logic subsystem may be single-core or multi-core, and the instructions executed by the processors may be configured for sequential, parallel, and / or distributed processing. Individual components of the logic subsystem may optionally be distributed across two or more separate devices. These devices may be remotely located and / or configured for coordinated processing. Aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured as a cloud computing arrangement.

[0047]

[0054] The storage subsystem 904 includes one or more physical devices configured to temporarily and / or permanently hold computer information such as data and instructions executable by the logic subsystem. When the storage subsystem includes two or more devices, these devices may be co-located and / or remotely located. The storage subsystem 904 may include volatile devices, non-volatile devices, dynamic devices, static devices, read / write devices, read-only devices, random access devices, sequential access devices, location addressable devices, file addressable devices, and / or content addressable devices. The storage subsystem 904 may include removable and / or embedded devices. As the logic subsystem executes instructions, the state of the storage subsystem 904 may be transformed, for example, to hold different data.

[0048]

[0055] The storage subsystem 904 may include removable and / or internal devices. The storage subsystem 904 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory, among others. The storage subsystem 904 may include volatile devices, non-volatile devices, dynamic devices, static devices, read / write devices, read-only devices, random access devices, sequential access devices, location addressable devices, file addressable devices, and / or content addressable devices.

[0049]

[0056] Aspects of logic subsystem 902 and storage subsystem 904 may be integrated into one or more hardware logic components, which may include, for example, program specific and application specific integrated circuits (PASICs / ASICs), program specific and application specific standard products (PSSPs / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0050]

[0057] The logic subsystem and the storage subsystem may cooperate to instantiate one or more logic machines. As used herein, the term "machine" collectively refers to a combination of hardware, firmware, software, instructions, and / or any other components that cooperate to provide a computer function. In other words, a "machine" is never an abstract idea, but always has a concrete form. A machine may be instantiated by a single computing device, or a logic machine may include two or more subcomponents instantiated by two or more different computing devices. In some implementations, a machine includes a local component (e.g., a software application executed by a computer processor) that cooperates with a remote component (e.g., a cloud computing service provided by a network of server computers). The software and / or other instructions that give a particular machine its functionality may optionally be stored as one or more unexecuted modules on one or more suitable storage devices.

[0051]

[0058] When included, the display subsystem 906 can be used to present a visual representation of the data maintained by the storage subsystem 904. This visual representation can take the form of a graphical user interface (GUI). As the methods and processes described herein change the data maintained by the storage subsystem and thus transform the state of the storage subsystem, the state of the display subsystem 906 can similarly be transformed to visually represent the changes in the underlying data. The display subsystem 906 can include one or more display devices utilizing virtually any type of technology. Such display devices can be combined with the logic subsystem and the storage subsystem in a shared enclosure. Alternatively, such display devices can be peripheral display devices.

[0052]

[0059] When included, the input subsystem 908 may include or interact with one or more input devices, such as a keyboard and / or touch screen. In some embodiments, the input subsystem may include or interact with selected natural user input (NUI) components. Such components may be integrated or peripheral, and input act delivery and / or processing may be handled on-board or off-board. Examples of NUI components include microphones for speech and / or voice data recognition, and infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition.

[0053]

[0060] When included, the communications subsystem 910 may be configured to communicatively couple the computing system 900 with one or more other computing devices. The communications subsystem 910 may include wired and / or wireless communication devices compatible with one or more various communications protocols. As non-limiting examples, the communications subsystem may be configured to communicate over a wireless telephone network, or a wired or wireless local or wide area network. In some examples, the communications subsystem may enable the computing system 900 to send and / or receive messages to and from other devices over a network, such as the Internet.

[0054]

[0061] Furthermore, the present disclosure includes multiple configurations according to the following clauses.

[0055]

[0062] Article 1. A method for evaluating a training performance of a training subject against a competency framework, the training taking place in a training environment simulating a physical environment, the method comprising: receiving simulation state updates from a computing system of the training environment; querying a behavioral elicitation event (BEE) database to determine a set of active BEEs based on the simulation state updates from a plurality of BEEs stored in a BEE database, each BEE of the plurality of BEEs including one or more elicited actions and one or more observable behaviors; receiving a plurality of simulation behavior notifications from the computing system of the training environment; for each simulation behavior notification, querying the BEE database to determine a set of active BEEs based on the simulation state updates. determining whether the behavior represented by the simulation behavior notification matches a evoked behavior for at least one active BEE in the set of active BEEs, and storing information regarding the evoked behavior in response to the at least one active BEE in the set of active BEEs; determining a set of completed BEEs, the set of completed BEEs including each active BEE for which all corresponding evoked behaviors have been satisfied; and for each completed BEE in the set of completed BEEs, storing information regarding the training performance, the information including the one or more observable behaviors of the completed BEE.

[0056]

[0063] Article 2. The method of claim 1, wherein each BEE of the plurality of BEEs further includes one or more triggers, and querying the BEE database to determine the set of active BEEs includes querying the BEE database for one or more triggers having a condition that matches the update of the simulation state, storing the one or more triggers as a set of active triggers, and querying the BEE database to determine whether any BEE is an active BEE based on the set of active triggers.

[0057]

[0064] Article 3. The method of claim 2, wherein querying the BEE database to determine the set of active BEEs further includes determining one or more inactive triggers in the set of active triggers by comparing a corresponding condition for each active trigger in the set of active triggers with the simulation state updates, and removing the one or more inactive triggers from the set of active triggers.

[0058]

[0065] Article 4. 2. The method of claim 1, further comprising: comparing a prior set of active BEEs to the set of active BEEs; and for each active BEE that is included in the prior set of active BEEs and is not included in the set of active BEEs, storing the active BEE as an uncompleted BEE; and storing information about the training performance, the information including one or more untaken evoked actions in response to the uncompleted BEE.

[0059]

[0066] Article 5. The method of claim 1, further comprising: determining whether the evoked behavior has a timing condition when the behavior represented by the simulation behavior notification matches the evoked behavior for at least one active BEE in the set of active BEEs; and storing information regarding the timing of the simulation behavior notification.

[0060]

[0067] Article 6. 6. The method of clause 5, further comprising: comparing the timing of the simulation behavior notification with a timing of a corresponding trigger of the at least one active BEE; and storing information relating to the comparison of the timing.

[0061]

[0068] Article 7. 2. The method of claim 1, wherein the simulation state updates include changes to one or more of a simulated flight phase, a simulated weather, or a simulated aircraft state.

[0062]

[0069] Article 8. 2. The method of claim 1, wherein the plurality of simulation behavior notifications indicate a corresponding plurality of user inputs received by the training environment.

[0063]

[0070] Article 9. 1. A computing system comprising: a logic system operatively coupled to a training environment simulating a physical environment; and a computer readable storage system, the computer readable storage system including a behavioral elicitation event (BEE) database including a plurality of BEEs, each BEE of the plurality of BEEs including one or more elicited actions and one or more observable behaviors, the instructions including: receiving simulation state updates from a computing system of the training environment; querying the BEE database from the plurality of BEEs to determine a set of active BEEs based on the simulation state updates; receiving a plurality of simulation behavior notifications from each BEE; A computing system executable to: for a simulation behavior notification, determine whether the behavior represented by the simulation behavior notification matches an evoked behavior for at least one active BEE in the set of active BEEs, and store information regarding the evoked behavior in response to the at least one active BEE in the set of active BEEs; determine a set of completed BEEs, the set of completed BEEs including each active BEE for which all corresponding evoked behaviors have been satisfied; and, for each completed BEE in the set of completed BEEs, store information regarding the completed BEE.

[0064]

[0071] Article 10. The computing system of clause 9, wherein each BEE of the plurality of BEEs further includes one or more triggers, and the instructions executable to query the BEE database to determine the set of active BEEs include instructions executable to query the BEE database for one or more triggers having a condition that matches an update to the simulation state, store the one or more triggers as a set of active triggers, and query the BEE database to determine whether any BEE is an active BEE based on the set of active triggers.

[0065]

[0072] Article 11. The computing system of claim 10, wherein the instructions executable to query the BEE database to determine the set of active BEEs further include instructions executable to determine one or more inactive triggers in the set of active triggers by comparing a corresponding condition for each active trigger in the set of active triggers with the simulation state updates, and removing the one or more inactive triggers from the set of active triggers.

[0066]

[0073] Article 12. 10. The computing system of claim 9, wherein the instructions are further executable to: compare a previous set of active BEEs to the set of active BEEs, and for each active BEE that is included in the previous set of active BEEs and is not included in the set of active BEEs, store the active BEE as a non-completed BEE and also store information about the non-completed BEE.

[0067]

[0074] Article 13. The computing system of clause 9, wherein the instructions are further executable to: determine whether the invoked behavior has a timing condition when the behavior represented by the simulated behavior notification matches the invoked behavior for the at least one active BEE in the set of active BEEs; and store information regarding the timing of the simulated behavior notification.

[0068]

[0075] Article 14. 10. The computing system of claim 9, wherein the simulation state updates include changes to one or more of a simulated flight phase, a simulated weather, or a simulated aircraft state.

[0069]

[0076] Article 15. 10. The computing system of claim 9, wherein the plurality of simulation behavior notifications are indicative of a corresponding plurality of user inputs received by the training environment.

[0070]

[0077] Article 16. 13. A method for evaluating a training performance of a training subject against a competency framework, the training taking place in a training environment simulating a physical environment, the method comprising: receiving a first simulation state update from a computing system of the training environment; querying a behavioral elicitation event (BEE) database to determine a first set of active BEEs based on the first simulation state update from a plurality of BEEs stored in a BEE database, each BEE of the plurality of BEEs including one or more elicited actions and one or more observable behaviors; receiving a plurality of simulation behavior notifications from the computing system of the training environment; for each simulation behavior notification, determining whether the behavior represented by the simulation behavior notification is consistent with at least one of the first set of active BEEs; determining whether an evoked action matches an active BEE and stores information regarding the evoked action and the corresponding observable behavior in response to the at least one active BEE in the first set of active BEEs; receiving a second simulation state update from the computing system of the training environment; querying the BEE database to determine a second set of active BEEs based on the second simulation state update; comparing the first set of active BEEs with the second set of active BEEs; and, for each active BEE included in the first set of active BEEs and not included in the second set of active BEEs, storing the active BEE as a non-completed BEE and also storing information regarding the non-completed BEE.

[0071]

[0078] Article 17. The method of claim 16, wherein each BEE of the plurality of BEEs further includes one or more triggers, and wherein querying the BEE database to determine the first set of active BEEs or the second set of active BEEs includes querying the BEE database for one or more triggers having a condition that matches the simulation state update, storing the one or more triggers as a set of active triggers, and querying the BEE database to determine whether any BEE is an active BEE based on the set of active triggers.

[0072]

[0079] Article 18. 18. The method of claim 17, wherein querying the BEE database to determine the first set of active BEEs or the second set of active BEEs further comprises determining one or more inactive triggers in the set of active triggers by comparing a corresponding condition for each active trigger in the set of active triggers with the simulation state updates, and removing the one or more inactive triggers from the set of active triggers.

[0073]

[0080] Article 19. The method of claim 16, further comprising: determining whether the invoked behavior has a timing condition when the behavior represented by the simulation behavior notification matches the invoked behavior for the at least one active BEE in the set of active BEEs; and storing information regarding the timing of the simulation behavior notification.

[0074]

[0081] Article 20. The method of claim 16, further comprising: determining a set of completed BEEs, the set of completed BEEs including each active BEE for which all corresponding triggered behaviors have been satisfied; and for each completed BEE in the set of completed BEEs, storing information regarding the training performance, the information including the one or more observable behaviors of the completed BEE.

[0075]

[0082] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples should not be considered in a limiting sense, as numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various operations illustrated and / or described may be performed in the order illustrated and / or described, in other orders, concurrently, or omitted. Similarly, the order of the processes described above may be altered.

[0076]

[0083] The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various process, system, and configurations disclosed herein, as well as other features, functions, operations, and / or properties, and any and all equivalents thereof. [Explanation of symbols]

[0077] 100, 900, 402: Computing Systems 102: Aircraft training environment 104: Training subjects 106: Training Interface 108: Training Computing Systems 110: Processor 112: Storage system 114:BEE Database 116:BEE monitor 118, 404: LRS 200, 212: BEE 202, 304, 306: Trigger 204, 208: Timing conditions 206, 308, 310: Evoked Action 210, 312, 314: Observable behavior 300: Engine anti-icing BEE 302: Flight Phase Indicator 400: Algorithm 406: Application Programming Interface (xAPI) Monitor 408:xAPI Logger 410, 412, 414: Step 500: Update statement 502, 504: Update simulation state 506, 508: Simulation action notification 600, 602, 604, 606, 608, 610, 614, 616, 618, 620, 622, 624: Step 700, 702, 708, 710, 712, 714, 720, 722, 724, 726, 728, 730: Step 800: Method 802, 804, 806, 808, 810, 811, 812, 814, 816, 818, 820, 822, 824, 826, 828: Step 902: Logic Subsystem 904: Storage Subsystem 906: Display Subsystem 908: Input Subsystem 910: Communications Subsystem

Claims

1. A method (800) for evaluating training performance of a training subject (104) against a competency framework, the training taking place in a training environment that simulates a physical environment, the method (800) comprising: receiving (802) simulation state updates (502, 504) from computing systems (100, 900, 402) of the training environment; Querying (804) a behavioral call-to-action (BEE) database (114) to determine a set of active BEEs based on the simulation state updates (502, 504) from a plurality of BEEs (200, 212) stored in the BEE database (114), wherein each BEE (200, 212) of the plurality of BEEs (200, 212) includes one or more evoked actions (206, 308, 310) and one or more observable behaviors (210, 312, 314); receiving (812) a plurality of simulation action notifications (506, 508) from the computing systems (100, 900, 402) of the training environment; for each simulation behavior notification (506, 508), determining (814) whether the behavior represented by the simulation behavior notification (506, 508) matches an evoked behavior (206, 308, 310) for at least one active BEE in the set of active BEEs, and storing (820) information regarding the evoked behavior (206, 308, 310) as a response to the at least one active BEE in the set of active BEEs; determining (826) a set of completed BEEs, the set of completed BEEs including each active BEE for which all corresponding evoked behaviors (206, 308, 310) have been satisfied; and A method (800) comprising, for each completed BEE in the set of completed BEEs, storing (828) information regarding the training performance, the information including the one or more observable behaviors (210, 312, 314) of the completed BEE.

2. Each BEE (200, 212) of the plurality of BEEs (200, 212) further includes one or more triggers (202, 304, 306), and querying the BEE database (114) to determine the set of active BEEs further comprises: Querying (806) the BEE database (114) for one or more triggers (202, 304, 306) having conditions that match the simulation state update (502, 504); storing (808) the one or more triggers (202, 304, 306) as a set of active triggers (202, 304, 306); and 2. The method (800) of claim 1, comprising: querying (811) the BEE database (114) to determine whether any BEE (200, 212) is an active BEE based on the set of active triggers (202, 304, 306).

3. 3. The method (800) of claim 2, wherein querying the BEE database (114) to determine the set of active BEEs further comprises determining (810) one or more inactive triggers (202, 304, 306) in the set of active triggers (202, 304, 306) by comparing a corresponding condition for each active trigger (202, 304, 306) in the set of active triggers (202, 304, 306) with the simulation state updates (502, 504), and removing (810) the one or more inactive triggers (202, 304, 306) from the set of active triggers (202, 304, 306).

4. 2. The method (800) of claim 1, further comprising: comparing (822) a prior set of active BEEs to the set of active BEEs; and for each active BEE that is included in the prior set of active BEEs and is not included in the set of active BEEs, storing (822) the active BEE as an incomplete BEE; and storing (822) information regarding the training performance, the information including one or more untaken evoked actions (206, 308, 310) in response to the incomplete BEE.

5. 2. The method (800) of claim 1, further comprising: determining (816) whether the invoked behavior (206, 308, 310) has a timing condition (204, 208) when the behavior represented by the simulation behavior notification (506, 508) matches the invoked behavior (206, 308, 310) for the at least one active BEE in the set of active BEEs; and storing (816) information regarding the timing of the simulation behavior notification (506, 508).

6. 6. The method (800) of claim 5, further comprising: comparing (818) the timing of the simulation behavior notifications (506, 508) with timing of corresponding triggers (202, 304, 306) of the at least one active BEE; and storing (818) information regarding the comparison of the timing.

7. The method of claim 1 , wherein the simulation state updates include changes to one or more of a simulated flight phase, a simulated weather, or a simulated aircraft state.

8. The method of claim 1 , wherein the plurality of simulation action notifications are indicative of corresponding plurality of user inputs received by the training environment.

9. a logic system operably coupled to a training environment that simulates a physical environment; and A computing system (100, 900, 402) comprising a computer readable storage system (112), The computer readable storage system (112) includes a BEE database (114) including a plurality of Behavioral Calls (BEEs) (200, 212) and instructions executable by the logic system, each BEE (200, 212) of the plurality of BEEs (200, 212) including one or more evoked actions (206, 308, 310) and one or more observable behaviors (210, 312, 314), the instructions comprising: receiving (802) simulation state updates (502, 504) from computing systems (100, 900, 402) of the training environment; Querying (804) the BEE database (114) to determine, from the plurality of BEEs (200, 212), a set of active BEEs based on the simulation state updates (502, 504); receiving (812) a plurality of simulation behavior notifications (506, 508); for each simulation behavior notification (506, 508), determining (814) whether the behavior represented by the simulation behavior notification (506, 508) matches an evoked behavior (206, 308, 310) for at least one active BEE in the set of active BEEs, and storing information regarding the evoked behavior (206, 308, 310) as a response to the at least one active BEE in the set of active BEEs; determining (826) a set of completed BEEs, the set of completed BEEs including each active BEE for which all corresponding evoked behaviors (206, 308, 310) have been satisfied; and A computing system (100, 900, 402) that is executable to: for each completed BEE of the set of completed BEEs, store (828) information about the completed BEE.

10. each BEE (200, 212) of the plurality of BEEs (200, 212) further comprising one or more triggers (202, 304, 306), and the instructions executable to query the BEE database (114) to determine the set of active BEEs, comprising: Querying (806) the BEE database (114) for one or more triggers (202, 304, 306) having conditions that match the simulation state update (502, 504); storing (808) the one or more triggers (202, 304, 306) as a set of active triggers (202, 304, 306); and The computing system (100, 900, 402) of claim 9, further comprising instructions executable to: query (811) the BEE database (114) to determine whether any BEE (200, 212) is an active BEE based on the set of active triggers (202, 304, 306).

11. 11. The computing system (100, 900, 402) of claim 10, wherein the instructions executable to query the BEE database (114) to determine the set of active BEEs further comprise instructions executable to determine (810) one or more inactive triggers (202, 304, 306) in the set of active triggers (202, 304, 306) by comparing a corresponding condition for each active trigger (202, 304, 306) in the set of active triggers (202, 304, 306) with the simulation state updates (502, 504), and to remove (810) the one or more inactive triggers (202, 304, 306) from the set of active triggers (202, 304, 306).

12. 10. The computing system (100, 900, 402) of claim 9, wherein the instructions are further executable to: compare (822) a previous set of active BEEs to the set of active BEEs; and, for each active BEE that is in the previous set of active BEEs and is not in the set of active BEEs, store (822) the active BEE as a non-completing BEE; and also store information about the non-completing BEE.

13. 10. The computing system (100, 900, 402) of claim 9, wherein the instructions are further executable to determine (816) whether the invoked behavior (206, 308, 310) has a timing condition (204, 208) when the behavior represented by the simulation behavior notification (506, 508) matches the invoked behavior (206, 308, 310) for the at least one active BEE in the set of active BEEs, and to store (816) information regarding the timing of the simulation behavior notification (506, 508).

14. 10. The computing system of claim 9, wherein the simulation state updates include changes to one or more of a simulated flight phase, a simulated weather, or a simulated aircraft state.

15. 10. The computing system of claim 9, wherein the plurality of simulation behavior notifications (506, 508) are indicative of corresponding plurality of user inputs received by the training environment.

16. A method (800) for evaluating training performance of a training subject (104) against a competency framework, the training taking place in a training environment that simulates a physical environment, the method (800) comprising: receiving (802) a first simulation state update (502, 504) from a computing system (100, 900, 402) of the training environment; Querying (804) a behavioral call-to-action (BEE) database (114) to determine a first set of active BEEs based on the first simulation state updates (502, 504) from a plurality of BEEs (200, 212) stored in the BEE database (114), wherein each BEE (200, 212) of the plurality of BEEs (200, 212) includes one or more evoked behaviors (206, 308, 310) and one or more observable behaviors (210, 312, 314) when querying (804) the BEE database (114); receiving (812) a plurality of simulation action notifications (506, 508) from the computing systems (100, 900, 402) of the training environment; for each simulation behavior notification (506, 508), determining (814) whether the behavior represented by the simulation behavior notification (506, 508) matches an evoked behavior (206, 308, 310) for at least one active BEE in the first set of active BEEs, and storing information regarding the evoked behavior (206, 308, 310) and the corresponding observable behavior (210, 312, 314) in response to the at least one active BEE in the first set of active BEEs; receiving a second simulation state update (502, 504) from the computing system (100, 900, 402) of the training environment; Querying the BEE database (114) to determine a second set of active BEEs based on the second simulation state updates (502, 504); Comparing (822) the first set of active BEEs to the second set of active BEEs; and The method (800) includes, for each active BEE that is included in the first set of active BEEs and that is not included in the second set of active BEEs, storing (822) the active BEE as a non-completed BEE and also storing information about the non-completed BEE.

17. Each BEE (200, 212) of the plurality of BEEs (200, 212) further includes one or more triggers (202, 304, 306), and querying the BEE database (114) to determine the first set of active BEEs or the second set of active BEEs comprises: Querying (806) the BEE database (114) for one or more triggers (202, 304, 306) having conditions that match the simulation state update (502, 504); storing (808) the one or more triggers (202, 304, 306) as a set of active triggers (202, 304, 306); and 17. The method (800) of claim 16, comprising: querying (811) the BEE database (114) to determine whether any BEE (200, 212) is an active BEE based on the set of active triggers (202, 304, 306).

18. 20. The method (800) of claim 17, wherein querying the BEE database (114) to determine the first set of active BEEs or the second set of active BEEs further comprises determining (810) one or more inactive triggers (202, 304, 306) in the set of active triggers (202, 304, 306) by comparing a corresponding condition for each active trigger (202, 304, 306) in the set of active triggers (202, 304, 306) with the simulation state updates (502, 504), and removing (810) the one or more inactive triggers (202, 304, 306) from the set of active triggers (202, 304, 306).

19. 17. The method (800) of claim 16, further comprising: determining (816) whether the invoked behavior (206, 308, 310) has a timing condition (204, 208) when the behavior represented by the simulation behavior notification (506, 508) matches the invoked behavior (206, 308, 310) for the at least one active BEE in the set of active BEEs; and storing (816) information regarding the timing of the simulation behavior notification (506, 508).

20. determining (826) a set of completed BEEs, the set of completed BEEs including each active BEE for which all corresponding evoked behaviors (206, 308, 310) have been satisfied; and 20. The method (800) of claim 16, further comprising: for each completed BEE in the set of completed BEEs, storing (828) information regarding the training performance, the information including the one or more observable behaviors (210, 312, 314) of the completed BEE.

Images