Evaluating training performance for capability framework using behavior extraction events

Abstract

The invention relates to evaluating training performance for a capability framework by utilizing behavior extraction events. One example provides a method (800) of evaluating training performance of a training subject (104) and the method is performed in a training environment. The method (800) receives (802) a simulation state update (502, 504) from a computing system (100, 900, 402) of the training environment, and queries (804) a Behavior Event (BEE) database to determine a set of active BEEs based on the simulation state update (502, 504). The method (800) also receives (812) a plurality of simulated action notifications (506, 508) from the computing system (100, 900, 402), and determines (814), for each simulated action notification (506, 508), whether an action represented by the simulated action notification (506, 508) matches an extraction action (206, 308, 310) for the active BEE. The method (800) also stores (820) information about the extraction action (206, 308, 310) as a response to the active BEE. The method (800) determines (826) a set of completed BEEs, the set of completed BEEs including any active BEEs that satisfy all corresponding extraction actions (206, 308, 310), and stores (828), for each completed BEE, information about the completed BEE.

Description

Technical Field

[0001] The disclosed examples relate to automatic assessment of training subjects against a competency framework. Background Art

[0002] Some industries evaluate professionals for a specified competency framework during professional training. In addition to the hard skills performed during training performance in a training environment, the competency framework may also include soft skills that the training subject will demonstrate. Exemplary soft skills include communication, leadership, teamwork, situational awareness, information management, management of hard skill tasks, and understanding of hard skill procedures. For example, the training of commercial pilots uses competency-based training and evaluation (CBTA) to evaluate student pilots with respect to the core pilot competencies defined by the International Civil Aviation Organization (ICAO). Currently, ICAO evaluates trained pilots based on nine pilot competencies: application of knowledge, application of procedures and compliance with regulations, communication, flight path management for automated aircraft, flight path management for manually controlled aircraft, leadership and teamwork, problem solving and decision making, situational awareness and information management, and workload management. The current method for evaluating soft skills is to perform by observation of a coach during training performance. This evaluation conducted by observation may be difficult for a coach to objectively evaluate the training subject and / or provide feedback to the training subject. Summary of the invention

[0003] A disclosed example provides a method for evaluating a training performance of a training subject against a capability framework. The training performance is performed in a training environment that simulates a physical environment. The method includes receiving a simulation state update from a computing system of the training environment, and querying a behavior induced event (BEE) database to determine a set of active BEEs from a plurality of BEEs stored in the BEE database based on the simulation state update. Each of the plurality of BEEs includes one or more induced actions and one or more observable behaviors. The method also includes receiving a plurality of simulation action notifications from the computing system of the training environment. The method also includes, for each simulation action notification, determining whether the action represented by the simulation action notification matches an induced action for at least one active BEE in the set of active BEEs, and storing information about the induced action as a response to at least one active BEE in the set of active BEEs. The method also includes determining a set of completed BEEs. The set of completed BEEs includes each active BEE that satisfies all corresponding induced actions. The method also includes storing information about the training performance for each completed BEE in the set of completed BEEs. The information includes one or more observable behaviors of the completed BEE.

[0004] Another example provides a computing system, including: a logic system, the logic system is operably coupled to a training environment simulating a physical environment; and a computer-readable storage system. The computer-readable storage system includes: a behavior elicitation event (BEE) database and instructions, the BEE database includes multiple BEEs, each of the multiple BEEs includes one or more elicitation actions and one or more observable behaviors; the instructions are executable by the logic system. The instructions are executable to receive a simulation state update from the computing system of the training environment, query the BEE database to determine an active BEE set from the multiple BEEs based on the simulation state update, and receive multiple simulation action notifications. The instructions are further executable to determine, for each simulation action notification, whether the action represented by the simulation action notification matches an elicitation action for at least one active BEE in the active BEE set, and store information about the elicitation action as a response to at least one active BEE in the active BEE set. The instructions are further executable to determine a set of completed BEEs. The set of completed BEEs includes each active BEE that satisfies all corresponding elicitation actions. The instructions are further executable to, for each completed BEE in the set of completed BEEs, store information about the completed BEE.

[0005] Another example provides a method for evaluating a training performance of a training subject for a capability framework. The training performance is performed in a training environment that simulates a physical environment. The method includes receiving a first simulation state update from a computing system of the training environment, and querying a behavior induced event (BEE) database to determine a first set of active BEEs from a plurality of BEEs stored in the BEE database based on the first simulation state update. Each of the plurality of BEEs includes one or more induced actions and one or more observable behaviors. The method also includes receiving a plurality of simulation action notifications from a computing system of the training environment. The method also includes, for each simulation action notification, determining whether the action represented by the simulation action notification matches an induced action for at least one active BEE in the first set of active BEEs, and storing information about the induced action and the corresponding observable behavior as a response to at least one active BEE in the first set of active BEEs. The method also includes: receiving a second simulation state update from the computing system of the training environment, querying the BEE database to determine a second set of active BEEs based on the second simulation state update, and comparing the first set of active BEEs with the second set of active BEEs. The method further includes, for each active BEE that is in the first set of active BEEs and not in the second set of active BEEs, storing the active BEE as an unfinished BEE and also storing information about the unfinished BEE. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1A block diagram of an exemplary computing system configured to utilize a BEE database to evaluate training performance is shown.

[0007] Figure 2 Shown by Figure 1 Block diagram of an exemplary BEE database used by a computing system.

[0008] Figure 3 A block diagram of an exemplary engine anti-ice BEE is shown.

[0009] Figure 4A and Figure 4B Schematically shows the use of Figure 1 An exemplary algorithm for evaluating the training performance of test subjects using the BEE database.

[0010] Figure 5 Shown with Figure 3 Table of exemplary simulation updates corresponding to successful completion of the engine anti-ice BEE.

[0011] Figure 6 and Figure 7 Schematically shows Figure 4A and Figure 4B An exemplary sub-process of the algorithm.

[0012] Figure 8 A flow chart illustrating an exemplary method for evaluating a training subject's training performance against a competency framework is shown.

[0013] Fig. 9 A block diagram of an exemplary computing system is shown. DETAILED DESCRIPTION

[0014] As previously mentioned, the training of pilots uses competency-based training and evaluation (CBTA) to evaluate the training object (pilot) for the ICAO competency framework. The current method of using CBTA in training solutions is to let the coach evaluate the training object about the ICAO competency framework by observing the training object during the training performance (such as during the training in the flight simulator). In addition, the coach must also provide supporting evidence, which shows how the training object successfully or unsuccessfully demonstrated the observable behavior corresponding to each ability in the ICAO competency framework. This type of current evaluation method has brought several problems. First, the ICAO competency framework mainly includes soft skills. After successfully completing the ICAO competency framework, it is difficult for the coach to objectively evaluate the training object and provide feedback to the training object. For example, the evaluation performed by observation may be difficult to provide information on the traceability between the actions performed by the training object and the ICAO competency framework. Lack of traceability may make it difficult to provide suggestions to help the training object improve its training performance in the next training performance to exceed the current training performance. Another problem with the evaluation performed by observation is that similar training performance may receive different evaluations between different coaches. In addition, the coach needs to be present during the training performance to observe. This can increase the cost of pilot training for covering the coach's salary and training.

[0015] Thus, an example is disclosed involving utilizing a computer-implemented behavior eliciting event (BEE) to automate the evaluation of a training subject's training performance against a competency framework. A BEE is an event that elicits a specific behavior from a training subject to demonstrate a soft skill competency specified in a competency framework. The training performance is performed by the training subject in a training environment that simulates a physical environment. Exemplary training environments are a fixed training device (FTD) flight simulator, a full motion simulator (FMS) flight simulator, and a virtual training environment. As described in more detail below, the disclosed example utilizes a BEE database comprising a plurality of BEEs to evaluate the training performance. Each of the plurality of BEEs comprises one or more triggers, one or more eliciting actions, and one or more observable behaviors, such as reference Figure 2 The disclosed example receives a simulation state update from a training environment and determines a set of active BEEs based on the simulation state update by querying a BEE database. The simulation state update indicates a change in a simulation state of the training environment.

[0016] Next, the disclosed example receives a plurality of simulated action notifications from the training environment. The simulated action notifications indicate corresponding plurality of user inputs received by the training environment. When the action represented by the simulated action notification matches an eliciting action of an active BEE in the set of active BEEs, the disclosed example stores information about the eliciting action and the corresponding observable behavior as a response to the active BEE. In addition, when the active BEE has satisfied all corresponding eliciting actions, the disclosed example determines that the active BEE is a completed BEE.

[0017] The disclosed examples also store information about any completed BEEs and / or uncompleted BEEs. The information can be used as evidence to support how the training subject demonstrates various competencies in the competency framework during training performance. The information can also be used for teaching purposes and to customize training scenarios for individual training subjects. Using a BEE database can provide scenario-independent behavior detection for evaluation, which helps reduce the creation time of training scenarios compared to current methods where each scenario is independently created.

[0018] Figure 1 A block diagram of an exemplary computing system 100 in communication with a training environment, in this example, in the form of an aircraft training environment 102, is depicted. Here, the aircraft training environment 102 is configured to simulate an aircraft environment for training pilots. In some examples, the aircraft training environment 102 may be configured as an FTD, an FMS, or a virtual training environment. A training subject 104 performs a training performance in the aircraft training environment 102 by using a training interface 106 to interact with a training computing system 108. The training computing system 108 is configured to execute computer simulated training scenarios. In some examples, the training interface 106 and the aircraft training environment 102 may be configured to simulate a cockpit within the aircraft training environment 102 and receive user input from the training subject 104. Exemplary user inputs include audio input, button presses or hardware selections into the training interface 106, video input, and other suitable inputs from the training subject 104.

[0019] In some examples, for example, the training computing system 108 may be further configured to utilize machine learning algorithms to analyze various user inputs, such as using speech-to-text recognition to determine the content of an audio input or using a classifier to recognize a specified gesture in a video input. In addition, the machine learning algorithm may be used with physiological measurements to help determine whether the training subject 104 is paying attention to a desired portion of a cockpit in the aircraft training environment 102. For example, such a configuration may help generate a perception model that can be used to detect eliciting actions corresponding to various monitoring capabilities in the capability framework, such as cognitive workload management capabilities. For example, the training subject 104 may also input user input in response to a computer simulated training scenario, such as a change in one or more of a simulated flight phase, simulated weather, or a simulated aircraft state. In other examples, the aircraft training environment 102 may have another configuration. In yet another example, other training environments besides the aircraft training environment may be used.

[0020] For example, the computing system 100 is configured to evaluate the training performance of the training subject 104 against a competency framework, such as the ICAO pilot competency. The computing system 100 includes a processor 110 and a storage system 112 operatively coupled to the aircraft training environment 102. The storage system 112 includes a BEE database 114 including a plurality of BEEs, such as those described in reference Figure 2 Storage system 112 also includes a BEE monitor 116 configured to be executable by processor 110. Storage system 112 also includes a learning record storage (LRS) 118 configured to store information related to the evaluation of the performance of training subject 104. In some examples, LRS 118 may be configured to also store information from aircraft training environment 102. In other examples, information from the training environment and / or information related to the performance evaluation may be stored in another location or multiple locations. Fig. 9 Additional aspects of the processor 110 and the storage system 112 are discussed. In other examples, the computing system 100 may be configured to receive flight operations quality assurance (FOQA) data recorded during real-time flight and detect pilot behavior against the ICAO competency framework. Such a configuration may help determine training recommendations for an airline fleet and / or determine the effectiveness of a training program for an airline fleet. Although discussed herein with reference to the aircraft training environment 102, in other examples, the BEE database may be used to evaluate another training performance against any suitable competency framework.

[0021] Each BEE stored in the BEE database indicates an event that elicited a specific behavior from the training subject. Figure 2A block diagram of the BEE database 114 is shown in more detail. The BEE database 114 includes a first BEE 200. The first BEE 200 includes one or more triggers 202 that define conditions for the first BEE 200 to be an active BEE. More specifically, each trigger can be an active trigger when the corresponding condition satisfies a simulated state of the training environment. In the example of pilot training, the trigger may include a specified simulated air temperature, a specified flight phase of a simulated aircraft, and a specified fuel level, altitude, speed, and / or another state of a simulated aircraft. In addition, examples of flight phases include stationary, cruising, takeoff, landing, taxiing, ascending, descending, and approaching. The trigger 202 may include an optional timing condition 204 indicating a specified time to respond to the trigger 202. The timing condition 204 may also indicate whether the training object 104 is expected to respond before the specified time or wait until after the specified time to respond. As a specific example, a fuel leak BEE has a trigger whose timing condition is to wait until the desired time before taking action to increase the fuel flow. The first BEE 200 also includes one or more eliciting actions 206 that define expected user inputs as a response to when the first BEE 200 is the active BEE. Similar to the trigger 202, the eliciting action 206 may include a timing condition 208. In some examples, for example, the trigger 202 and / or the eliciting action 206 may include logic statements (not shown for clarity), such as Boolean logic (e.g., AND, OR, etc.), sequence statements, and conditional statements. The first BEE 200 also includes one or more observable behaviors 210 that indicate the capabilities demonstrated by the training subject 104 by performing the one or more eliciting actions 206. In addition, each observable behavior 210 may conform to a specified capability in a capability framework (such as a specified pilot capability in the ICAO capabilities). Figure 3 A specific example of BEE is provided in .

[0022] Similarly, the BEE database 114 includes a second BEE 212 having one or more triggers, one or more induced actions, and one or more observable behaviors. In this way, the BEE database can help align induced actions with observable behaviors for any suitable capability framework, so that the BEE database can help detect evidence of observable behaviors in another simulated training environment. Figure 2 Two BEEs are depicted for clarity, but in other examples, the BEE database 114 may include any suitable number of BEEs, such as a single BEE or more than two BEEs. The BEE database 114 is illustrative. In other examples, the BEE database 114 may include another configuration.

[0023] Figure 3A block diagram of an exemplary engine anti-ice BEE 300 is depicted. The engine anti-ice BEE 300 may be used to evaluate a pilot's training performance in a training environment, such as with reference to Figure 4A and Figure 4B discussed. Figure 3 In the example of FIG. 1 , engine anti-ice BEE 300 corresponds to an engine anti-ice scenario during a simulated stationary flight phase in aircraft training environment 102. Therefore, engine anti-ice BEE 300 includes a flight phase indicator 302 for the stationary flight phase, which indicates that engine anti-ice BEE 300 may be active during the simulated stationary flight phase. Engine anti-ice BEE 300 also includes a first trigger 304, which is configured to be an active trigger when the simulated air temperature of the training environment is less than 10° C. Similarly, second trigger 306 is configured to be an active trigger when the training environment has simulated visible air moisture.

[0024] The engine anti-ice BEE 300 also includes a first elicitation action (EA) 308 and a second elicitation action 310. Here, the first elicitation action 308 defines a user input to the aircraft training environment 102 that instructs the pilot to select the engine anti-ice procedure in the electronic sign package of the aircraft training environment 102. The second elicitation action 310 defines a user input that instructs the pilot to verbally communicate to his co-pilot that the engine anti-ice procedure has been selected.

[0025] The engine anti-ice BEE 300 also includes a first observable behavior (OB) set 312 that defines capabilities associated with the pilot's demonstrated behavior performing the first eliciting action 308. Here, the first observable behavior set 312 indicates that the training subject demonstrated knowledge of procedures, knew where to get the required information, situational awareness, communication, problem solving, following standard operating procedures, and planning, prioritizing, and scheduling appropriate tasks. Similarly, the engine anti-ice BEE 300 also includes a second observable behavior set 314 associated with the response to the second eliciting action 310. The second observable behavior set 314 indicates that the training subject demonstrated communicating messages clearly, accurately, and concisely. The first observable behavior set 312 and the second observable behavior set 314 correspond to pilot capabilities defined in the ICAO capabilities. Figure 3 In other examples, engine anti-ice BEE 300 may have another configuration.

[0026] For example, one or more computing systems may be configured to execute an algorithm that uses the BEE database to evaluate training performance of training subjects in a training environment, such as aircraft training environment 102 . Figure 4A and Figure 4BAn overview flow chart of an exemplary algorithm 400 using the BEE database 114 is shown. Here, the computing system 402 and the computing system 100 execute the algorithm 400. In some examples, the computing system 402 can be part of the training computing system 108. For example, the computing system 402 includes a learning record storage device (LRS) 404, which is configured to store information about user input from the training object (such as interaction with another individual participating in the training environment), and information about actions performed by the training object. For example, the LRS 404 is further configured to store information about the simulated state of the training environment, such as simulated weather. The computing system 402 also includes an experience application programming interface (xAPI) monitor 406 and an xAPI recorder 408. In short, the xAPI monitor 406 is configured to monitor the LRS 404 for any new statements received from the training environment. The xAPI recorder 408 is configured to transfer information from the BEE monitor 116 to the LRS 404 for storage. In other examples, another configuration of one or more computing systems including a BEE database may perform algorithm 400 .

[0027] refer to Figure 4A , the xAPI monitor 406 queries the session in the LRS 404 for an update statement including a simulated state update or a simulated action notification at 410. In response, the LRS 404 returns the update statement to the xAPI monitor 406 at 412. At 414, the xAPI monitor 406 sends the update statement to the BEE monitor 116. Alternatively, at Figure 4B In the example of , querying for update statements at 410 and returning update statements at 412 are omitted.

[0028] After the BEE monitor 116 receives the update statement and when the update statement includes a simulation action notification (as indicated at 416), the algorithm 400 moves to the reference Figure 7 Alternatively, when the statement update includes a simulation state update (as indicated at 418), the algorithm 400 moves to the reference state update. Figure 6 The corresponding sub-process discussed above can be selected based on the configuration of the training environment Figure 4A or Figure 4B For example, you can choose Figure 4A For use with FTD or FMS. Figure 4B for use with a virtual training environment. Figure 4A and Figure 4B In other examples, the algorithm 400 may use another overview flow chart.

[0029] For purposes of illustration, the sub-processes of algorithm 400 are discussed with reference to engine anti-ice BEE 300. Thus, Figure 5 A table of exemplary update statements 500 corresponding to successful performances by training subject 104 in response to engine anti-ice BEE 300 is depicted. Here, first simulated state update 502 includes a change in simulated weather reflecting a simulated air temperature of 0° C. Additionally, second simulated state update 504 includes a change in simulated weather reflecting the presence of simulated visible air moisture. Next, first simulated action notification 506 indicates a corresponding user input selecting an engine anti-ice procedure in an electronic flight bag of aircraft training environment 102. Additionally, second simulated action notification 508 indicates a corresponding user input communicating the selection of an engine anti-ice procedure. For example, such user input may include an audio input. In other examples, algorithm 400 may use another BEE and / or other update statements.

[0030] As previously described, the algorithm 400 is discussed using the example of simulating an engine anti-ice BEE 300 during a stationary flight phase. In the current example, the algorithm 400 will execute more than one loop through various sub-processes. During each loop, the algorithm 400 executes different sub-processes and / or skips different sub-processes. The first loop begins when the xAPI monitor 406 sends the first simulated state update 502 to the BEE monitor 116 at 414, and the algorithm 400 moves to Figure 6 , as discussed above. Figure 6 , the BEE monitor 116 queries the BEE database 114 at 600 for any triggers associated with the first simulated state update 502 (simulated air temperature is 0° C.). In response, the BEE database 114 returns the first trigger 304 (air temperature is less than 10° C.) of the engine anti-ice BEE 300 at 602. Here, the first trigger 304 is an active trigger. Therefore, the BEE monitor 116 stores the first trigger 304 in the active trigger set at 604. In this example, based on the first simulated state update 502, no active triggers become inactive triggers, so the BEE monitor 116 skips 606. In other examples, the BEE monitor 116 may remove any inactive triggers from the active trigger set at 606. An inactive trigger is a trigger that has a condition that is not satisfied by the current state of the training environment. As a specific example, a change in the simulated flight phase may cause one or more triggers to become inactive. In this way, the active trigger set can help achieve fast querying of any active BEE.

[0031] Next, the BEE monitor 116 queries the BEE database 114 for any active BEEs based on the active trigger set at 608. As previously described, an active BEE has all corresponding triggers as active triggers. Here, the engine anti-ice BEE 300 has one active trigger (the first trigger 304) and one inactive trigger (the second trigger 306). Therefore, at 610, the BEE database 114 responds to the query without sending an active BEE to the BEE monitor 116. Furthermore, since no BEE was in the previous active BEE set and is not in the active trigger set (since the first trigger 304 is in the active trigger set), the BEE monitor 116 skips 612 and returns to 420.

[0032] In other examples, when a BEE is in a previously active BEE set and is not in the active BEE set, the BEE monitor 116 determines an unfinished BEE set at 612, which includes one or more BEEs that were in the previously active BEE set and are not in the active BEE set. For each unfinished BEE in the unfinished BEE set, the algorithm 400 may store information about the unfinished BEE. Specifically, the BEE monitor 116 queries the BEE database 114 (at 614) for the observable behavior of the unfinished BEE for each corresponding trigger with an optional timing condition and / or queries for each corresponding induced action (at 616). In response, the BEE database 114 returns the corresponding observable behavior of the unfinished BEE at 618 and 620, respectively. Next, the BEE monitor 116 sends an observable behavior evaluation for each unfinished BEE to the xAPI recorder 408 based on the corresponding observable behavior of the unfinished BEE at 622. The observable behavior evaluation may indicate that a subset of the corresponding observable behaviors were presented and / or partially presented in response to the incomplete BEE. As a specific example, the evaluation may indicate that two elicitation actions were completed, but there was an opportunity to complete five elicitation actions. Continuing to 624, the xAPI recorder 408 then sends information about observable behaviors that were presented, partially presented, and / or failed to be presented for storage to the LRS 404. In this manner, the LRS 404 may store information about elicitation actions that were completed, as well as information about elicitation actions that were missed in response to the incomplete BEE.

[0033] Returning to the example of the engine anti-ice BEE 300, the algorithm 400 continues with the second loop when the xAPI monitor 406 sends the second simulated status update 504 (visible moisture in the air) at 414. The BEE monitor 116 skips 416 and moves to the sub-process related to the simulated status update at 418. Figure 6, the BEE monitor 116 again queries the BEE database 114 at 600 for any triggers associated with the second simulated state update 504. Here, the BEE monitor 116 receives the second trigger 306 (simulated visible air moisture) at 602 and stores the second trigger 306 in the active trigger set at 604. The BEE monitor 116 further queries the BEE database 114 for any active BEEs based on the active trigger set at 608. Here, the first trigger 304 (weather below 10°C) and the second trigger 306 (visible air moisture) of the engine anti-ice BEE 300 are active. Therefore, the BEE database 114 stores the engine anti-ice BEE 300 in the active BEE set. Similar to the first loop, no BEE becomes an unfinished BEE due to the second simulated state update 504, and therefore the algorithm 400 returns to 420.

[0034] In the next third loop of the algorithm 400, the xAPI monitor 406 sends the first simulated action notification 506 (user input selecting the engine anti-ice procedure) at 414. Here, the action represented by the first simulated action notification 506 matches the first derived action 308 of the engine anti-ice BEE 300, and therefore the algorithm 400 moves to 7 at 416. Now referring to Figure 7 , the first derived action 308 has no corresponding timing condition. Similarly, none of the triggers of the engine anti-ice BEE 300 has an optional timing condition. Therefore, the algorithm 400 skips 700 and 702.

[0035] In other examples, the induced action as indicated at 704 and / or the corresponding trigger as indicated at 706 may have an optional timing condition. In such examples, the BEE monitor 116 checks at 700 whether the timing of the simulated action notification is within the timing condition of the induced action. As a specific example, a fuel leak BEE has a fuel flow trigger with an increase fuel flow indication with a timing condition. In addition, the timing condition indicates that a specified time duration has passed before the corresponding induced action is performed in response to the fuel flow trigger. The BEE monitor 116 further checks at 702 whether the timing of the trigger corresponds to the induced action. Information about the timing of the simulated action notification and / or the timing of the trigger may be included in the BEE assessment message to be stored.

[0036] Returning to the example of the engine anti-ice BEE 300, the BEE monitor 116 queries the BEE database 114 for one or more observable behaviors (OBs) corresponding to the first eliciting action 308 at 708. The BEE monitor 116 receives the first observable behavior set 312 at 710. In addition, the BEE monitor 116 stores information about the first eliciting action 308 as a response to the engine anti-ice BEE 300 by sending the BEE evaluation to the xAPI recorder 408 at 712, and in response, the xAPI recorder 408 sends a response message to be stored to the LRS 404 at 714. For example, the information may include the first observable behavior set 312 as the behavior exhibited by the training subject performing the first eliciting action 308. In other examples, other information about the eliciting action may be stored in another suitable manner as a response to the active BEE.

[0037] Continuing with the current example, the engine anti-ice BEE 300 has satisfied the first derivation action 308 but has not satisfied the second derivation action 310. Therefore, at least one corresponding derivation action of the engine anti-ice BEE 300 has not been satisfied, and therefore the engine anti-ice BEE 300 is still the active BEE. Therefore, the algorithm 400 skips 716 and moves to Figure 6 The BEE database 114 is queried for active triggers at 600. In this example, the first simulated action notification 506 (user input selecting the engine anti-ice procedure) does not match the conditions of another trigger, and therefore no new active triggers are returned by the BEE database 114. In addition, no new BEE becomes the active BEE, and therefore the algorithm 400 returns to 420.

[0038] Similarly, when the BEE monitor 116 receives the second simulated action notification 508 at 414, the algorithm 400 begins a fourth loop and moves to 416. Figure 7 , the BEE monitor 116 queries at 708 for observable behaviors (OBs) corresponding to the second derived action 310 and receives the second observable behavior set 314 at 710. In addition, the BEE monitor 116 uses the xAPI recorder 408 to store information about the second derived action 310 as a response to the engine anti-ice BEE 300, as described above with reference to the first derived action 308.

[0039] Here, in the current example, the engine anti-ice BEE 300 satisfies all corresponding derived actions (e.g., both the first derived action 308 and the second derived action 310). Therefore, the BEE monitor 116 determines at 716 that the engine anti-ice BEE 300 is a completed BEE. As previously discussed, the engine anti-ice BEE 300 does not have a trigger with a timing condition, and therefore the BEE monitor 116 skips 718. Next, the BEE monitor 116 queries at 720 for observable behaviors that are exhibited by executing only each corresponding derived action of the completed BEE. In the current example of the engine anti-ice BEE 300, the BEE database 114 returns at 722 a first observable behavior set 312 for the first derived action 308 and a second observable behavior set 314 for the second derived action 310. In response, for each observable behavior in the first observable behavior set 312 and the second observable behavior set 314, the BEE monitor 116 sends an observable behavior evaluation message to the xAPI recorder 408 at 724. At 726, the xAPI recorder 408 sends an observable behavior display message to be stored to the LRS 404. In this way, the observable behaviors displayed by the pilot in the context of the engine anti-ice BEE 300 are stored. In other examples, the BEE may have a trigger with a timing condition, as indicated at 718. In such examples, the BEE monitor 116 queries at 728 for observable behaviors displayed due to the timing of the corresponding induced action relative to the timing of the trigger of the BEE. At 730, the BEE monitor 116 receives the observable action and also includes information about the timing in the observable behavior evaluation sent at 724.

[0040] Similar to the third loop, algorithm 400 next moves to Figure 6 400 to query the trigger at 600 in the example. Since no new trigger becomes active due to the training subject performing the second lead-out action 310 or completing the engine anti-ice BEE 300, the algorithm 400 skips and returns to 420 and waits for a new simulation update. In this way, the algorithm 400 using the BEE database 114 can evaluate the pilot's training performance against ICAO capabilities. Figure 6 and Figure 7 is illustrative. Although algorithm 400 and corresponding sub-processes are discussed with reference to evaluating a pilot for engine anti-ice BEE 300, in other examples, algorithm 400 may be used to evaluate the training performance of another training subject for another competency framework. Similarly, in further examples, another algorithm utilizing BEE database 114 may be used to evaluate any training subject for any suitable competency framework executed in any suitable training environment that simulates a physical environment.

[0041] Figure 8 A flow chart of an exemplary method 800 for evaluating a training subject's training performance against a capability framework is shown. For example, the training performance is performed in a training environment that simulates a physical environment, such as the aircraft training environment 102. In such examples, the capability framework may include, for example, the ICAO capability framework. In other examples, another training environment and / or another capability framework may be used. For example, the method 800 may be performed by any suitable computing system including a BEE database, such as the computing system 100.

[0042] For example, method 800 includes receiving a simulation state update from a computing system of a training environment, such as training computing system 108, at 802. In the example of a simulated aircraft environment, the simulation state update may include a change in one or more of a simulated flight phase, simulated weather, or a state of a simulated aircraft. In other examples, the simulation state update may include another change in a simulated state of a physical environment (e.g., simulated weather, simulated location, simulated alert, etc.).

[0043] Returning, method 800 includes querying a BEE database at 804 to determine an active BEE set from a plurality of BEEs stored in the BEE database based on a simulated state update. For example, the BEE database 114 may be used. As discussed above, each of the plurality of BEEs includes one or more triggers, one or more induced actions, and one or more observable behaviors. Querying the BEE database to determine an active BEE set includes querying the BEE database at 806 for one or more triggers having conditions that match the simulated state update. Determining the active BEE set also includes storing one or more triggers as an active trigger set at 808. In addition, determining the active BEE set includes determining one or more inactive triggers in the active trigger set by comparing the corresponding conditions of each active trigger in the active trigger set with the simulated state update at 810, and removing one or more inactive triggers from the active trigger set. Determining the active BEE set also includes querying the BEE database at 811 to determine whether any BEE is an active BEE based on the active trigger set. In other examples, the active BEE set may be determined in another suitable manner.

[0044] The method 800 also includes receiving a plurality of simulated action notifications from a computing system of the training environment at 812. The plurality of simulated action notifications indicate corresponding plurality of user inputs received by the training environment. Examples of user inputs include detected voice, selected input on a user interface of the training environment, detected eye movement, and / or detected gestures. In other examples, the method 800 may query the BEE database again to determine the active BEE set, as discussed with reference to 804. The method 800 also includes determining, for each simulated action notification at 814, whether the action represented by the simulated action notification matches the elicitation action of at least one active BEE in the active BEE set. The method 800 includes, when the action represented by the simulated action notification matches the elicitation action, determining, at 816, whether the elicitation action has a timing condition, and storing information about the timing of the simulated action notification. The timing condition may indicate that the expected timing of the simulated action notification is after the expected duration or at the expected duration. Alternatively, the timing condition may indicate that the expected timing of the simulated action notification is within the expected duration. The method 800 also includes comparing the timing of the simulated action notification with the timing of a corresponding trigger of the at least one active BEE and storing information about the timing comparison at 818. In other examples, the method 800 may omit 816 and / or 818.

[0045] The method 800 includes storing information about eliciting actions and corresponding observable behaviors as a response to at least one active BEE in the set of active BEEs at 820. The method 800 also includes comparing a previous set of active BEEs with the set of active BEEs at 822, and for each active BEE that is in the previous set of active BEEs and is not in the set of active BEEs, storing the active BEE as an uncompleted BEE. The method 800 includes storing information about the training performance at 824. The information includes one or more eliciting actions that were missed in response to the uncompleted BEE. In other examples, 822 and 824 may be omitted.

[0046] The method 800 includes determining a set of completed BEEs at 826. The set of completed BEEs includes each active BEE that satisfies all corresponding elicitation actions. At 828, the method 800 also includes storing information about the training performance for each completed BEE in the set of completed BEEs. The information includes one or more observable behaviors of the completed BEE. In some examples, for example, the information may also include any appropriate information about the completed BEE, such as timing information.

[0047] Using the BEE database to evaluate the training performance described herein can provide various benefits. The evaluation can provide data about the training performance to help the coach objectively evaluate soft skill capabilities, and increase the consistency of the inter-rater reliability evaluated by the coach's observation. The evaluation can also help provide detailed evidence for use when asserting the proficiency of the training subject who performs the training performance. In addition, the evaluation can help provide traceability between the specified abilities in the eliciting action and the ability framework. This traceability can help provide feedback to the training subject about improving the ability level during the next training performance to exceed the advice of the current training performance. In addition, the evaluation can help enable subsequent training scenarios to be customized for the training subject based on the current training performance and / or previous training performance.

[0048] Additionally or alternatively, using the BEE database to evaluate multiple training performances can provide data to enable analysis across training subjects to determine recommendations for updating training scenarios used in training environments. In addition, the data can also help evaluate the effectiveness of training environments, particularly virtual training environments. Similarly, such evaluations can help analyze operational data to determine the effectiveness of training scenarios and / or determine recommended training scenarios across training subjects.

[0049] In some examples, the methods and processes described herein may be associated with a computing system of one or more computing devices. Specifically, such methods and processes may be implemented in hardware as a computer application or service, an application programming interface (API), a library, and / or other computer program products as described above.

[0050] Fig. 9 A simplified representation of a computing system 900 configured to provide any or all of the computing functionality described herein is schematically shown. Computing system 900 may take the form of one or more personal computers, server computers, and computers integrated with an aircraft as examples. Computing system 100, training computing system 108, and computing system 402 are examples of computing system 900.

[0051] The computing system 900 includes a logic subsystem 902 and a storage subsystem 904. The computing system 900 can optionally include a display subsystem 906, an input subsystem 908, a communication subsystem 910, and / or Fig. 9 Other subsystems not shown.

[0052] The logic subsystem 902 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, implement a technical effect, or otherwise achieve a desired result.

[0053] The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logic subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The individual components of the logic subsystem can optionally be distributed in two or more independent devices that can be remotely located and / or configured for coordinated processing. Various aspects of the logic subsystem may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0054] The storage subsystem 904 includes one or more physical devices configured to temporarily and / or permanently store computer information, such as data and instructions executable by the logic subsystem. When the storage subsystem includes two or more devices, the devices may be co-located and / or remotely located. The storage subsystem 904 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices. The storage subsystem 904 may include removable and / or built-in devices. As the logic subsystem executes instructions, the state of the storage subsystem 904 may be transformed—for example, to store different data.

[0055] The storage subsystem 904 may include removable and / or built-in devices. The storage subsystem 904 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage, etc. The storage subsystem 904 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0056] Aspects of logic subsystem 902 and storage subsystem 904 may be integrated into one or more hardware logic components. For example, such hardware logic components may include program specific integrated circuits and application specific integrated circuits (PASIC / ASIC), program specific standard products and application specific standard products (PSSP / ASSP), systems on chips (SOC), and complex programmable logic devices (CPLD).

[0057] The logic subsystem and the storage subsystem may collaborate to instantiate one or more logic machines. As used herein, the term "machine" is used to collectively refer to a combination of hardware, firmware, software, instructions, and / or any other components that collaborate to provide computer functions. In other words, a "machine" is never an abstract concept and always has a tangible form. A machine may be instantiated by a single computing device, or a machine may include two or more subcomponents instantiated by two or more different computing devices. In some specific implementations, a machine includes a local component (e.g., a software application executed by a computer processor) that collaborates with a remote component (e.g., a cloud computing service provided by a server computer network). The software and / or other instructions that give a particular machine its functionality can optionally be stored on one or more suitable storage devices as one or more unexecuted modules.

[0058] When including a display subsystem 906, the display subsystem can be used to present a visual representation of the data stored by the storage subsystem 904. The visual representation can take the form of a graphical user interface (GUI). When the methods and processes described herein change the data stored by the storage subsystem, and thereby transform the state of the storage subsystem, the state of the display subsystem 906 can also be transformed to intuitively represent the changes in the underlying data. The display subsystem 906 may include one or more display devices utilizing almost any type of technology. Such display devices may be combined with the logic subsystem and the storage subsystem in a shared housing, or such display devices may be peripheral display devices.

[0059] When an input subsystem 908 is included, the input subsystem may include or interact with one or more input devices such as a keyboard and a touch screen. In some examples, the input subsystem may include or interact with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be performed on-board or off-board. Exemplary NUI components may include microphones for voice and / or speech recognition; and infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition.

[0060] When a communication subsystem 910 is included, the communication subsystem may be configured to communicatively couple the computing system 900 to one or more other computing devices. The communication subsystem 910 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured to communicate via a wireless telephone network or a wired or wireless local or wide area network. In some examples, the communication subsystem may allow the computing system 900 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0061] Furthermore, the present disclosure includes configurations according to the following clauses.

[0062] Clause 1. A method for evaluating a training performance of a training subject against a capability framework, the training performance being performed in a training environment that simulates a physical environment, the method comprising: receiving a simulation state update from a computing system of the training environment; querying a behavior induced event (BEE) database to determine a set of active BEEs from a plurality of BEEs stored in the BEE database based on the simulation state update, each of the plurality of BEEs comprising one or more induced actions and one or more observable behaviors; receiving a plurality of simulation action notifications from the computing system of the training environment; for each simulation action notification, determining whether the action represented by the simulation action notification matches an induced action of at least one active BEE in the set of active BEEs, and storing information about the induced action as a response to the at least one active BEE in the set of active BEEs; determining a set of completed BEEs, the set of completed BEEs comprising each active BEE that satisfies all corresponding induced actions; and for each completed BEE in the set of completed BEEs, storing information about the training performance, the information comprising the one or more observable behaviors of the completed BEE.

[0063] Clause 2. A method according to clause 1, wherein each of the multiple BEEs further includes one or more triggers, and wherein querying the BEE database to determine the set of active BEEs includes querying the BEE database for one or more triggers having conditions that match the simulated state update, storing the one or more triggers as a set of active triggers, and querying the BEE database to determine whether any BEE is an active BEE based on the set of active triggers.

[0064] Clause 3. A method according to Clause 2, wherein querying the BEE database to determine the active BEE set also includes determining one or more inactive triggers in the active trigger set by comparing the corresponding conditions of each active trigger in the active trigger set with the simulation state update, and removing the one or more inactive triggers from the active trigger set.

[0065] Clause 4. The method according to Clause 1 also includes comparing a previous active BEE set with the active BEE set, and for each active BEE that was in the previous active BEE set and not in the active BEE set, storing the active BEE as an incomplete BEE and also storing information about the training performance, the information including one or more eliciting actions that were missed in response to the incomplete BEE.

[0066] Clause 5. The method according to Clause 1 further includes, when the action represented by the simulated action notification matches the induced action of at least one active BEE in the active BEE set, determining whether the induced action has a timing condition, and storing information about the timing of the simulated action notification.

[0067] Clause 6. The method of clause 5, further comprising comparing the timing of the simulated action notification with the timing of a corresponding trigger of the at least one active BEE, and storing information about the comparison of the timing.

[0068] Clause 7. The method of Clause 1, wherein the simulated state update comprises a change in one or more of simulated flight phase, simulated weather, or a state of a simulated aircraft.

[0069] Clause 8. The method of clause 1, wherein the plurality of simulated action notifications are indicative of a corresponding plurality of user inputs received by the training environment.

[0070] Clause 9. A computing system comprising: a logic system operably coupled to a training environment simulating a physical environment; and a computer-readable storage system comprising: a behavior elicitation event (BEE) database, the BEE database comprising a plurality of BEEs, each of the plurality of BEEs comprising one or more elicitation actions and one or more observable behaviors; and instructions executable by the logic system to receive a simulation state update from a computing system of the training environment, query the BEE database to determine a set of active BEEs from the plurality of BEEs based on the simulation state update, receive a plurality of simulation action notifications, for each simulation action notification, determine whether an action represented by the simulation action notification matches an elicitation action of at least one active BEE in the set of active BEEs, and store information about the elicitation action as a response to the at least one active BEE in the set of active BEEs, determine a set of completed BEEs, the set of completed BEEs comprising each active BEE that satisfies all corresponding elicitation actions, and for each completed BEE in the set of completed BEEs, store information about the completed BEE.

[0071] Clause 10. A computing system according to clause 9, wherein each of the plurality of BEEs further comprises one or more triggers, and wherein the instructions executable to query the BEE database to determine the set of active BEEs comprise instructions executable to perform the following operations: query the BEE database for one or more triggers having conditions that match the simulated state update, store the one or more triggers as a set of active triggers, and query the BEE database to determine whether any BEE is an active BEE based on the set of active triggers.

[0072] Clause 11. A computing system according to clause 10, wherein the instructions executable to query the BEE database to determine the active BEE set also include instructions executable to perform the following operations: determine one or more inactive triggers in the active trigger set by comparing the corresponding conditions of each active trigger in the active trigger set with the simulation state update, and remove the one or more inactive triggers from the active trigger set.

[0073] Clause 12. A computing system according to clause 9, wherein the instructions are further executable to compare a previous set of active BEEs with the set of active BEEs, and for each active BEE that is in the previous set of active BEEs and not in the set of active BEEs, store the active BEE as an unfinished BEE and also store information about the unfinished BEE.

[0074] Clause 13. A computing system according to clause 9, wherein the instructions are further executable to determine whether the induced action has a timing condition when the action represented by the simulated action notification matches the induced action of at least one active BEE in the set of active BEEs, and store information about the timing of the simulated action notification.

[0075] Clause 14. The computing system of Clause 9, wherein the simulated state update comprises a change in one or more of simulated flight phase, simulated weather, or a state of a simulated aircraft.

[0076] Clause 15. The computing system of Clause 9, wherein the plurality of simulated action notifications are indicative of a corresponding plurality of user inputs received by the training environment.

[0077] Clause 16. A method for evaluating a training performance of a training subject against a capability framework, the training performance being performed in a training environment that simulates a physical environment, the method comprising: receiving a first simulation state update from a computing system of the training environment; querying a behavior induced event (BEE) database to determine a first set of active BEEs from a plurality of BEEs stored in the BEE database based on the first simulation state update, each of the plurality of BEEs comprising one or more induced actions and one or more observable behaviors; receiving a plurality of simulation action notifications from the computing system of the training environment; for each simulation action notification, determining whether the action represented by the simulation action notification is consistent with the first active BEE The invention relates to a method for training a training environment comprising: matching an induced action of at least one active BEE in the first set of active BEEs with an induced action of at least one active BEE in the first set of active BEEs, and storing information about the induced action and the corresponding observable behavior as a response to the at least one active BEE in the first set of active BEEs; receiving a second simulation state update from the computing system of the training environment; querying the BEE database to determine a second set of active BEEs based on the second simulation state update; comparing the first set of active BEEs with the second set of active BEEs; and for each active BEE in the first set of active BEEs and not in the second set of active BEEs, storing the active BEE as an unfinished BEE and also storing information about the unfinished BEE.

[0078] Clause 17. A method according to clause 16, wherein each of the multiple BEEs further includes one or more triggers, and wherein querying the BEE database to determine the first active BEE set or the second active BEE set includes querying the BEE database for one or more triggers having conditions that match the simulated state update, storing the one or more triggers as an active trigger set, and querying the BEE database to determine whether any BEE is an active BEE based on the active trigger set.

[0079] Clause 18. A method according to Clause 17, wherein querying the BEE database to determine the first active BEE set or the second active BEE set also includes determining one or more inactive triggers in the active trigger set by comparing the corresponding conditions of each active trigger in the active trigger set with the simulation state update, and removing the one or more inactive triggers from the active trigger set.

[0080] Clause 19. The method according to Clause 16 also includes, when the action represented by the simulated action notification matches the induced action of the at least one active BEE in the active BEE set, determining whether the induced action has a timing condition, and storing information about the timing of the simulated action notification.

[0081] Clause 20. The method according to Clause 16 also includes determining a set of completed BEEs, wherein the set of completed BEEs includes each active BEE that satisfies all corresponding induced actions; and for each completed BEE in the set of completed BEEs, storing information about the training performance, wherein the information includes the one or more observable behaviors of the completed BEE.

[0082] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered to have limiting significance, as there may be multiple variations. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above process may be changed.

[0083] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

[0084] Parts List:

[0085] Computing system 100, 900, 402

[0086] Aircraft Training Environment 102

[0087] Training object 104

[0088] Training interface 106

[0089] Training computing system 108

[0090] Processor 110

[0091] Storage system 112

[0092] BEE Database 114

[0093] BEE Monitor 116

[0094] LRS118, 404

[0095] BEE 200, 212

[0096] Triggers 202, 304, 306

[0097] Timing conditions 204, 208

[0098] Leading actions 206, 308, 310

[0099] Observable behavior 210, 312, 314

[0100] Engine Anti-Ice BEE 300

[0101] Flight Phase Indicator 302

[0102] Algorithm 400

[0103] Application Programming Interface (xAPI) Monitor 406

[0104] xAPI Logger 408

[0105] Steps 410, 412, 414

[0106] Update Statement 500

[0107] Simulate status updates 502, 504

[0108] Simulation action notification 506, 508

[0109] Steps 600, 602, 604, 606, 608, 610, 614, 616, 618, 620, 622, 624 Steps 700, 702, 708, 710, 712, 714, 720, 722, 724, 726, 728, 730 Method 800

[0110] Steps 802, 804, 806, 808, 810, 811, 812, 814, 816, 818, 820, 822, 824, 826, 828

[0111] Logic Subsystem 902

[0112] Storage subsystem 904

[0113] Display subsystem 906

[0114] Input Subsystem 908

[0115] Communication subsystem 910

Claims

1. A method (800) for evaluating a training performance of a training subject (104) against a competency framework, the training performance being performed in a training environment simulating a physical environment, the method (800) comprising: Receiving (802) a simulation state update (502, 504) from a computing system (100, 900, 402) of the training environment; querying (804) a behavior induced event (BEE) database (114) to determine a set of active BEEs from a plurality of BEEs (200, 212) stored in the BEE database (114) based on the simulation state update (502, 504), each BEE (200, 212) in the plurality of BEEs (200, 212) including one or more induced actions (206, 308, 310) and one or more observable behaviors (210, 312, 314); receiving (812) a plurality of simulated action notifications (506, 508) from the computing system (100, 900, 402) of the training environment; for each simulated action notification (506, 508), determining (814) whether the action represented by the simulated action notification (506, 508) matches an induced action (206, 308, 310) for at least one active BEE in the set of active BEEs, and storing (820) information about the induced action (206, 308, 310) as a response to the at least one active BEE in the set of active BEEs; determining (826) a set of completed BEEs, the set of completed BEEs including each active BEE that satisfies all corresponding induced actions (206, 308, 310); as well as For each completed BEE in the set of completed BEEs, information about the training performance is stored (828), the information including the one or more observable behaviors (210, 312, 314) of the completed BEE.

2. The method (800) of claim 1, wherein each BEE (200, 212) of the plurality of BEEs (200, 212) further comprises one or more triggers (202, 304, 306), and wherein querying the BEE database (114) to determine the set of active BEEs comprises, querying (806) the BEE database (114) for one or more triggers (202, 304, 306) having conditions matching the simulation state update (502, 504), storing (808) the one or more triggers (202, 304, 306) as an active trigger set (202, 304, 306), and The BEE database (114) is queried (811) to determine whether any BEE (200, 212) is an active BEE based on the set of activity triggers (202, 304, 306).

3. The method (800) of claim 2, wherein querying the BEE database (114) to determine the active BEE set further comprises determining (810) one or more inactive triggers (202, 304, 306) in the active trigger set (202, 304, 306) by comparing a corresponding condition of each active trigger (202, 304, 306) in the active trigger set (202, 304, 306) with the simulation state update (502, 504), and removing (810) the one or more inactive triggers (202, 304, 306) from the active trigger set (202, 304, 306).

4. The method (800) of claim 1, further comprising, comparing a previous set of active BEEs to the set of active BEEs (822), and For each active BEE that was in the previous set of active BEEs and is not in the set of active BEEs, storing (822) the active BEE as an uncompleted BEE and also storing (822) information about the training performance, the information including one or more eliciting actions (206, 308, 310) missed in response to the uncompleted BEE.

5. The method (800) of claim 1 further comprising, when the action represented by the simulated action notification (506, 508) matches the induced action (206, 308, 310) for the at least one active BEE in the set of active BEEs, determining (816) whether the induced action (206, 308, 310) has a timing condition (204, 208), and storing (816) information about the timing of the simulated action notification (506, 508).

6. The method (800) of claim 5, further comprising comparing (818) the timing of the simulated action notification (506, 508) with the timing of a corresponding trigger (202, 304, 306) of the at least one active BEE, and storing (818) information about the comparison of the timing.

7. The method (800) of claim 1, wherein the simulated state update (502, 504) includes a change in one or more of simulated flight phase, simulated weather, or a state of a simulated aircraft.

8. The method (800) of claim 1, wherein the plurality of simulated action notifications (506, 508) indicate a corresponding plurality of user inputs received by the training environment.

9. A computing system (100, 900, 402), comprising: a logic system operably coupled to a training environment simulating a physical environment; and A computer readable storage system (112), the computer readable storage system comprising: a behavior eliciting event (BEE) database (114), the BEE database comprising a plurality of BEEs (200, 212), each BEE (200, 212) of the plurality of BEEs (200, 212) comprising one or more eliciting actions (206, 308, 310) and one or more observable behaviors (210, 312, 314); and Instructions that can be executed by the logic system to receiving (802) a simulation state update (502, 504) from a computing system (100, 900, 402) of the training environment, querying (804) the BEE database (114) to determine a set of active BEEs from the plurality of BEEs (200, 212) based on the simulation state update (502, 504), receiving (812) a plurality of simulation action notifications (506, 508), for each simulated action notification (506, 508), determining (814) whether the action represented by the simulated action notification (506, 508) matches an induced action (206, 308, 310) for at least one active BEE in the set of active BEEs, and storing information about the induced action (206, 308, 310) as a response to the at least one active BEE in the set of active BEEs, determining (826) a set of completed BEEs, the set of completed BEEs comprising each active BEE that satisfies all corresponding induced actions (206, 308, 310), and For each completed BEE in the set of completed BEEs, information about the completed BEE is stored (828).

10. The computing system (100, 900, 402) of claim 9, wherein each BEE (200, 212) of the plurality of BEEs (200, 212) further comprises one or more triggers (202, 304, 306), and wherein the instructions executable to query the BEE database (114) to determine the set of active BEEs comprise instructions executable to: querying (806) the BEE database (114) for one or more triggers (202, 304, 306) having conditions matching the simulation state update (502, 504), storing (808) the one or more triggers (202, 304, 306) as an active trigger set (202, 304, 306), and The BEE database (114) is queried (811) to determine whether any BEE (200, 212) is an active BEE based on the set of activity triggers (202, 304, 306).

11. The computing system (100, 900, 402) of claim 10, wherein the instructions executable to query the BEE database (114) to determine the set of active BEEs further comprise instructions executable to: determine (810) one or more inactive triggers (202, 304, 306) in the set of active triggers (202, 304, 306) by comparing a corresponding condition of each active trigger (202, 304, 306) in the set of active triggers (202, 304, 306) to the simulation state update (502, 504), and remove (810) the one or more inactive triggers (202, 304, 306) from the set of active triggers (202, 304, 306).

12. The computing system (100, 900, 402) of claim 9, wherein the instructions are further executable to comparing a previous set of active BEEs to the set of active BEEs (822), and For each active BEE that was in the previous set of active BEEs and is not in the set of active BEEs, the active BEE is stored (822) as an unfinished BEE and information about the unfinished BEE is also stored.

13. The computing system (100, 900, 402) of claim 9, wherein the instructions are further executable to, when the action represented by the simulated action notification (506, 508) matches the derived action (206, 308, 310) for the at least one active BEE in the set of active BEEs, determine (816) whether the derived action (206, 308, 310) has a timing condition (204, 208), and store (816) information about the timing of the simulated action notification (506, 508).

14. The computing system (100, 900, 402) of claim 9, wherein the simulated state update (502, 504) comprises a change in one or more of simulated flight phase, simulated weather, or a state of a simulated aircraft.

15. The computing system (100, 900, 402) of claim 9, wherein the plurality of simulated action notifications (506, 508) indicate a corresponding plurality of user inputs received by the training environment.

16. A method (800) of evaluating a training performance of a training subject (104) against a competency framework, the training performance being performed in a training environment simulating a physical environment, the method (800) comprising: Receiving (802) a first simulation state update (502, 504) from a computing system (100, 900, 402) of the training environment; querying (804) a behavior induced event (BEE) database (114) to determine a first set of active BEEs from a plurality of BEEs (200, 212) stored in the BEE database (114) based on the first simulation state update (502, 504), each BEE (200, 212) in the plurality of BEEs (200, 212) including one or more induced actions (206, 308, 310) and one or more observable behaviors (210, 312, 314); receiving (812) a plurality of simulated action notifications (506, 508) from the computing system (100, 900, 402) of the training environment; for each simulated action notification (506, 508), determining (814) whether the action represented by the simulated action notification (506, 508) matches an induced action (206, 308, 310) for at least one active BEE in the first set of active BEEs, and storing information about the induced action (206, 308, 310) and corresponding observable behavior (210, 312, 314) as a response to the at least one active BEE in the first set of active BEEs; receiving a second simulation state update (502, 504) from the computing system (100, 900, 402) of the training environment; querying the BEE database (114) to determine a second set of active BEEs based on the second simulation state update (502, 504); Comparing the first set of active BEEs with the second set of active BEEs (822); as well as For each active BEE that is in the first set of active BEEs and is not in the second set of active BEEs, the active BEE is stored (822) as an unfinished BEE and information about the unfinished BEE is also stored.

17. The method (800) of claim 16, wherein each BEE (200, 212) of the plurality of BEEs (200, 212) further comprises one or more triggers (202, 304, 306), and wherein querying the BEE database (114) to determine the first set of active BEEs or the second set of active BEEs comprises, querying (806) the BEE database (114) for one or more triggers (202, 304, 306) having conditions matching the simulation state update (502, 504), storing (808) the one or more triggers (202, 304, 306) as an active trigger set (202, 304, 306), and The BEE database (114) is queried (811) to determine whether any BEE (200, 212) is an active BEE based on the set of activity triggers (202, 304, 306).

18. The method (800) of claim 17, wherein querying the BEE database (114) to determine the first active BEE set or the second active BEE set further comprises determining (810) one or more inactive triggers (202, 304, 306) in the active trigger set (202, 304, 306) by comparing a corresponding condition of each active trigger (202, 304, 306) in the active trigger set (202, 304, 306) with the simulation state update (502, 504), and removing (810) the one or more inactive triggers (202, 304, 306) from the active trigger set (202, 304, 306).

19. The method (800) of claim 16, further comprising, when the action represented by the simulated action notification (506, 508) matches the induced action (206, 308, 310) for the at least one active BEE in the set of active BEEs, determining (816) whether the induced action (206, 308, 310) has a timing condition (204, 208), and storing (816) information about the timing of the simulated action notification (506, 508).

20. The method (800) of claim 16, further comprising, determining (826) a set of completed BEEs, the set of completed BEEs comprising each active BEE that satisfies all corresponding induced actions (206, 308, 310); and For each completed BEE in the set of completed BEEs, information about the training performance is stored (828), the information including the one or more observable behaviors (210, 312, 314) of the completed BEE.

Images