Pilot training evaluation system

Abstract

To provide a system and a method for utilizing a pilot training evaluation system in order to improve the utilization of an automated learning system by a student (or a group of students) to learn one or more lessons.SOLUTION: A method for providing a pilot training evaluation system includes: receiving a first training performance data set; analyzing the first training performance data set to determine a correlation between the first training performance data set and a training data comparison set; generating a training modification recommendation for an automated training system on the basis of at least the correlation; and communicating the training modification recommendation to the automated training system.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The primary disclosure relates generally to the control of an automated pilot training evaluation system. [Background technology]

[0002] In order to improve the efficiency and effectiveness of automated learning materials, numerous industries are incorporating automated learning systems into their training and education curricula. As such systems become more prevalent, it becomes increasingly important to improve the effectiveness and efficiency of these automated learning systems.

[0003] One way to improve the effectiveness and efficiency of automated learning systems is to increase the speed of feedback provided to the automated learning system regarding student performance, and to improve the quality of the feedback. In certain current learning systems, there is no data-driven approach to improving training courses and / or training performance. Certain current methods for providing automated learning (particularly pilot training) are poorly measured in numerous areas, leaving many unknowns regarding actual student performance. Summary of the Invention

[0004] In one particular embodiment, a method includes receiving a first training performance data set. The method also includes analyzing the first training performance data set to identify correlations between the first training performance data set and a training data comparison set. The method also includes generating training modification recommendations for an automated training system based at least on the correlations. The method also includes communicating the training modification recommendations to the automated training system.

[0005] In another particular embodiment, a system includes a memory configured to store instructions and one or more processors configured to receive a first training performance data set. The one or more processors are also configured to analyze the first training performance data set to identify correlations between the first training performance data set and the training data comparison set. The one or more processors are also configured to generate training modification recommendations for the automated training system based at least on the correlations. The one or more processors are also configured to communicate the training modification recommendations to the automated training system.

[0006] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to initiate, perform, or control operations including receiving a first training performance data set. The operations also include analyzing the first training performance data set to identify correlations between the first training performance data set and a training data comparison set. The operations also include generating training modification recommendations for an automated training system based at least on the correlations. The operations also include communicating the training modification recommendations to the automated training system.

[0007] In some embodiments, a device includes means for receiving a first training performance data set. The device also includes means for analyzing the first training performance data set to identify correlations between the first training performance data set and a training data comparison set. The device also includes means for generating training modification recommendations for an automated training system based at least on the correlations. The device also includes means for communicating the training modification recommendations to the automated training system.

[0008] The features, functions, and advantages described herein can be realized alone in various implementations or can be combined in yet other implementations, further details of which can be understood with reference to the following specification and drawings. [Brief description of the drawings]

[0009] [Figure 1] 1 illustrates an exemplary system for pilot training evaluation in accordance with at least one embodiment of the subject disclosure. [Diagram 2] 1 illustrates an exemplary architecture for a pilot training evaluation system in accordance with the present disclosure. [Diagram 3] 3 is a flow chart of one embodiment of a method 300 for providing an automated pilot training evaluation system in accordance with the subject disclosure. [Figure 4] 4 is a block diagram of a computing environment 400 including a computing device 410 configured to support aspects of computer-implemented methods and computer-executable program instructions (or code) in accordance with the subject disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Aspects disclosed herein use a pilot training evaluation system to improve the efficiency and effectiveness of an automated learning system, particularly by improving the way in which the automated learning system provides feedback to an automated learning provider. Rather than providing ad-hoc feedback to one part of the automated learning system (e.g., a particular portion of a curriculum, a particular instructor or collection of instructors, etc.), the pilot training evaluation system automatically and systematically applies advanced analytical techniques to a myriad of different data sources to provide the automated learning providers with a robust data set that they can use to evaluate their pilot training programs.

[0011] For example, by providing dynamic comparisons of instructors across various training locations, the automated learning system can evaluate the performance of an individual instructor against other instructors, instructors at other training locations, instructors at other automated learning providers, instructors specializing in other competencies, etc. The pilot training evaluation system can also identify differences in scoring patterns within a scored lesson to aid in improving the automated learning curriculum. Additionally, the pilot training evaluation system may enable an automated learning provider to track the progress of one or more instructors toward a given goal. The pilot training evaluation system may enable curriculum updates, further training requirements for a particular automated learning provider, further training requirements for a particular domain (e.g., geographic region, capability, operating area, etc.), further instructor training, etc., among other improvements to the automated learning system.

[0012] Varying the type of feedback provided to the automated learning system can improve the rate at which trainees learn the curriculum. For example, alerting the automated learning provider that a particular instructor and / or portion of the curriculum is not up to a particular standard. The pilot training evaluation system can identify declining performance and provide the automated learning provider with recommended action(s) to improve performance.

[0013] In addition to pilot training, other types of automated learning systems may be improved without departing from the scope of the primary disclosure. For example, aircraft mechanic training, aircraft maintenance training, flight crew training, and other types of automated learning systems may be improved through the pilot training evaluation system. The primary disclosure shows a system and method for using the pilot training evaluation system to improve a student's (or a group of students') use of the automated learning system to learn one or more lessons.

[0014] The drawings and the following description show several specific exemplary embodiments. Those skilled in the art will recognize that, even if not explicitly described or shown in the present specification, they may devise various configurations that embody the principles described herein and are included in the scope of the claims that follow this specification. Furthermore, any examples described herein are intended to aid in the understanding of the principles of the present disclosure and are not intended to be limiting. As a result, the present disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0015] Certain implementations are described herein with reference to the drawings. In the description, common features are provided with common reference numbers throughout the drawings. As may be used herein, various terms are used for the purpose of describing particular embodiments only and are not intended to be limiting. For example, the singular forms "a," "an," and "the" are intended to include the plural, unless the context clearly indicates otherwise. Furthermore, some features described herein may be present in the singular in some embodiments and in the plural in other embodiments. To illustrate, FIG. 1 depicts a system 100 including one or more processors ("(one or more) processors 106" in FIG. 1). It illustrates that in some embodiments, the system 100 includes a single processor 106 and in other embodiments, the system 100 includes multiple processors 106. For ease of reference herein, such features are generally introduced as "one or more" features, and hereafter, the singular feature will be referred to unless an aspect relating to multiple such features is described.

[0016] The terms "comprise", "comprises", and "comprising" may be used interchangeably with "include", "includes", or "including". Additionally, the term "wherein" is used interchangeably with the term "where". As used herein, "exemplary" indicates an example, an implementation, and / or an aspect, and should not be construed as limiting or as indicating a preferred or preferred embodiment. As used herein, orthogonal terms (e.g., "first", "second", "third", etc.) used to modify elements such as structures, components, operations, etc., do not in themselves indicate a priority or order of the element relative to another element, but rather merely distinguish the element from another element having the same name (except for the use of orthogonal terms). As used herein, the term "set" refers to a grouping of one or more elements, and the term "plurality" refers to a plurality of elements.

[0017] As used herein, "generating," "calculating," "using," "selecting," "accessing," and "determining" are interchangeable unless the context indicates otherwise. For example, "generating," "calculating," or "determining" a parameter (or signal) can refer to actively generating, calculating, or determining a parameter (or signal), or can refer to using, selecting, or accessing a parameter (or signal) that has already been generated, for example, by another component or device. As used herein, "connected" or "coupled" can include "communicatively coupled," "electrically coupled," or "physically coupled," or can also (or alternatively) include any combination thereof. Two devices (or components) can be directly or indirectly coupled (communicatively coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (wired networks, wireless networks, or combinations thereof), etc. Two devices (or components) that are electrically coupled can be included in the same device or different devices and can be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as by electrical communication, can send and receive electrical signals (digital or analog signals) directly or indirectly via one or more wires, buses, networks, etc. As used herein, "directly coupled" is used to describe two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without any intervening components.

[0018] 1 illustrates an exemplary system 100 for pilot training evaluation in accordance with at least one embodiment of the subject disclosure. In some embodiments, the system 100 includes a computing device 102 configured to communicate with an automated training system 104 via first training performance data 124, second training performance data 126, and / or training modification recommendation data 136. The automated training system 104 may be configured to communicate the first training performance data 124 and / or the second training performance data 126 to the computing device 102. The first training performance data 124 and the second training performance data 126 are associated with training data associated with one or more users of the automated training system 104. The computing device 102 may be configured to automatically analyze the first training performance data 124 and / or the second training performance data 126 to provide an automated evaluation of certain aspects of the automated training system 104.

[0019] For example, the automated pilot training system may be implemented at multiple pilot training centers. In one particular embodiment, the first training performance data 124 may be associated with one or more users of the automated training system 104 at the first location, while the second training performance data 126 may be associated with one or more users of the automated training system 104 at the second location. The automated training system 104 may be configured to communicate the first and second training performance data 124, 126 to the computing device 102. The computing device 102 may be configured to automatically analyze the first and second training performance data 124, 126 to generate one or more training modification recommendations 134 and transmit data associated therewith back to the automated training system 104 as training modification recommendation data 136. As described in more detail below, the training modification recommendation(s) 134 may include, among other things, a warning indicating that the training performance fails to meet a performance threshold, a recommendation to update training materials, instructions for corrective actions associated with one or more users of the automated training system, a training performance report, an indication of an instructor performance associated with a particular instructor, and / or any combination thereof. An example training modification recommendation 138 is shown in FIG. 1 and described in more detail below.

[0020] In some implementations, the computing device 102 may include one or more processors 106 coupled to the memory 108. The processor(s) 106 are configured to receive at least first training performance data 124 associated with training data of one or more users of the automated training system 104. In one particular implementation, the first training performance data 124 may include one or more data sets. The computing device 102 may be configured to store the one or more data sets in the memory 108 (e.g., as the first training performance data set 128). In some aspects, the first training performance data set 128 may include specific data associated with a particular group of users of the automated training system 104, a particular portion of an automated training curriculum, a particular location where the automated training system 104 is deployed, and / or some other suitable set of training performance data. For example, the first training performance data set 128 may include a plurality of scoring measures (e.g., individual scores, cumulative scores, etc.) assigned to one or more users of the automated training system 104.

[0021] In the same or alternative aspects, the processor(s) 106 may also be configured to receive second training performance data 126 associated with training data of one or more users of the automated training system 104. In a particular embodiment, the second training performance data 126 may include one or more data sets. The computing device may be configured to store the one or more data sets in the memory 108. Like the first training performance data set 128, the data set of the second training performance data 126 may include specific data associated with another particular group of users of the automated training system 104, another particular portion of the automated training curriculum, another particular location where the automated training system 104 is deployed, and / or some other suitable set of training performance data.

[0022] For example, the first training performance dataset 128 may include training data associated with a first group of users of the automated training system 104. Meanwhile, the second training performance dataset may include training data associated with a second group of users of the automated training system 104. In one particular embodiment, the first group of users may be different from the second group of users. In another particular embodiment, the first group of users may be associated with a group of users at a particular time. Meanwhile, the second group of users may be associated with the same group of users at another particular time. In yet another particular embodiment, the first group of users may be associated with a group of users training in a first curriculum competency (e.g., flight training curriculum). Meanwhile, the second group of users may be associated with the same and / or a different group of users training in a second curriculum competency (e.g., pre-flight curriculum). In yet another particular embodiment, the first group of users may be associated with a group of users in a first geographic area (e.g., city, state, country, etc.). Meanwhile, the second group of users may be associated with a second geographic area. In yet another particular embodiment, the first group of users may be associated with a group of users taught by a first instructor. Meanwhile, the second group of users may be associated with the same and / or another group of users taught by a second instructor. In yet another particular embodiment, the first group of users may be associated with a group of users at a first training location (e.g., a training facility owned and / or operated by a particular company). Meanwhile, the second group of users may be associated with a second training location (e.g., another training facility owned and / or operated by the same and / or another company).

[0023] In some implementations, the processor(s) 106 may be further configured to analyze the first training performance data set 128 to identify correlations 132 between the first training performance data set 128 and one or more training data comparison sets 130. In one particular implementation, the training data comparison set(s) 130 may be stored in memory 108 of the computing device 102.

[0024] In one particular embodiment, the processor(s) 106 may be configured to analyze the first training performance data set 128 by analyzing the first training performance data set 128 to identify a concordance correlation coefficient associated with the first training performance data set 128 and the training data comparison set(s) 130. The correlations 132 identified by the processor(s) 106 may also be stored in the memory 108. The concordance correlation coefficient is a statistical analysis tool used to measure the degree of concordance between two variables, and in particular to assess how well the first training performance data set 128 reproduces the training data comparison set(s) 130. In other embodiments, other methods of identifying the correlations 132 may be used without departing from the scope of the present disclosure. For example, the processor(s) 106 may be configured to identify the correlations 132 by identifying a Pearson correlation coefficient between the first training performance data set 128 and the training data comparison set(s) 130.

[0025] In some implementations, the computing device 102 may be configured to store in the memory 108 the training data comparison set(s) 130 for use by the processor(s) 106. In some aspects, the training data comparison set 130 may include a second training performance data set, as described in more detail above. In one such aspect, the processor(s) 106 may be configured to analyze the first training performance data set 128 against the second training performance data set to identify correlations 132 between the two training performance data sets. In the same or alternative aspects, the training data comparison set 130 may include a control performance data set. The control performance data set may include, for example, data associated with a particular criterion or threshold against which the actual training performance may be measured. Such a criterion may be an industry standard, a corporate standard, or some other suitable criterion or threshold for which training performance is acceptable.

[0026] In some implementations, the computing device 102 may be configured to generate training modification recommendations 134 for the automated training system 104 based at least on the correlation 132. As described in more detail above, the training modification recommendations 134 may include various presentations including a graphical display, alert(s), curriculum update recommendation(s), and the like. In one particular implementation, the computing device 102 may be configured to generate the training modification recommendations 134 based on one or more results of the analysis of the distribution of values ​​based on the first training performance dataset 128. In some aspects, the processor(s) 106 of the computing device 102 may be configured to identify a first distribution of values ​​139 based on the first training performance dataset 128. The first distribution of values ​​139 may be, for example, a value distribution curve associated with the first training performance dataset 128. In one particular aspect, the computing device 102 may be further configured to store the first distribution of values ​​139 in the memory 108.

[0027] In some aspects, the processor(s) 106 may be further configured to determine one or more measures associated with the first distribution of values ​​139. For example, the processor(s) 106 may be configured to determine a skewness measure associated with the first distribution of values ​​139. The skewness measure may, for example, measure the symmetry (or asymmetry) of a particular distribution curve. In one particular embodiment, if the first training performance dataset 128 includes data associated with a group of users of the automated training system 104 taught by a particular instructor, the distribution of user scores within the particular curve taught by the particular instructor may be represented by the first distribution of values ​​139. The skewness measure 140 may, for example, measure the extent to which a particular scoring distribution deviates from a desired scoring distribution. For example, if the ideal scoring distribution is a normal distribution, a skewness measure with a value of zero may indicate that the particular instructor distributes scores along a normal distribution and is consistent with the ideal scoring distribution. If the skewness measure is a positive value, it may indicate that the particular instructor distributes scores along a higher-tailed distribution. Thus, the particular instructor gives a more negative score than the ideal score. If the skewness measure is negative, it may indicate that the particular instructor distributes the scores along a lower tailed distribution. Thus, the particular instructor gives a more positive score than the ideal score.

[0028] In the same or alternative aspects, the processor(s) 106 may be further configured to determine a kurtosis measure associated with the distribution 139 of the first values. The kurtosis measure may, for example, measure the kurtosis of a particular distribution curve. In one particular example above, a kurtosis measure of zero value may indicate that the particular instructor distributes marks along a normal distribution, consistent with an ideal marking distribution. A positive kurtosis value may indicate that the particular instructor distributes marks more toward the center along the distribution. Thus, the particular instructor assigns more mid-level marks than the ideal marking. A negative kurtosis measure may indicate that the particular instructor distributes marks more toward the tail of the distribution. Thus, the particular instructor assigns less mid-level marks than the ideal marking.

[0029] In some implementations, the processor(s) 106 of the computing device 102 may be configured to generate training modification recommendations 134 based at least on one or both of the skewness measure 140 and the kurtosis measure 142, and the correlation 132. For example, if data in the first training performance data set 128 associated with a particular instructor has negative values ​​for the skewness measure 140 and the correlation 132 indicating a low correlation between the first training performance data set 128 and the training data comparison set 130, this may indicate that the particular instructor is too lenient in scoring. The processor(s) 106 may be configured to generate the training modification recommendations 134 based on the analysis of the various measures and communicate training modification recommendation data 136 associated with the training modification recommendations 134 to the automated training system 104. The automated training system 104 may be configured to perform one or more automated actions based on the training modification recommendation data 136, including generating a warning (e.g., alerting a curriculum supervisor) indicating that the training performance will fail to meet the performance threshold 144. The performance thresholds 144 stored in memory 108 may indicate one or more performance criteria associated with a group of users of the automated training system 104 associated with the first training performance data set 128. For example, the performance thresholds 144 may indicate that a particular percentage of students (e.g., 50%) meet a particular scoring threshold (e.g., 70% or greater).

[0030] As another example, if data in the first training performance dataset 128 associated with a particular instructor has positive values ​​for the skewness measure 140 and correlation 132 indicating a high correlation between the first training performance dataset 128 and the training data comparison set 130, this may indicate that an update may be needed to the curriculum of the automated training system. The processor(s) 106 may be configured to generate training modification recommendations 134 based on an analysis of the various measures and communicate training modification recommendation data 136 associated with the training modification recommendations 134 to the automated training system 104. The automated training system 104 may be configured to perform one or more automated actions based on the training modification recommendation data 136, including generating a warning indicating that the training performance fails to meet the performance threshold 144 (e.g., alerting a curriculum supervisor) and / or generating instructions for corrective action (e.g., automatically generating a curriculum update, etc.) associated with one or more users of the automated training system 104. The performance thresholds 144 stored in memory 108 may indicate one or more performance criteria associated with a group of users of the automated training system 104 associated with the first training performance data set 128. For example, the performance thresholds 144 may indicate that a particular percentage of students (e.g., 50%) meet a particular scoring threshold (e.g., 70% or greater).

[0031] As yet another example, if data in the first training performance dataset 128 associated with a particular instructor has a negative value for the kurtosis measure 142, this may indicate that a potential difference in student performance exists. The processor(s) 106 may be configured to generate training modification recommendations 134 based on the analysis of the various measures and communicate training modification recommendation data 136 associated with the training modification recommendations 134 to the automated training system 104. The automated training system 104 may be configured to perform one or more automated actions based on the training modification recommendation data 136, including generating a warning indicating that the training performance may not meet the performance threshold 144 (e.g., alerting a curriculum supervisor) and / or generating instructions for corrective action (e.g., automatically generating additional training material, such as exercises) associated with one or more users of the automated training system 104.

[0032] In some aspects, the training modification recommendation 134 may include a training performance report. The training performance report may include various data presented in various suitable forms configured to communicate one or more aspects of the training performance without departing from the scope of the main disclosure. For example, the training performance report may include a graphical display based at least on the correlation 132. In one particular embodiment, the graphical display may include data presented along multiple axes. A first axis may be associated with a first training measure, while a second axis may be associated with a second training measure. The processor(s) 106 may be configured to identify one or more values ​​of the first and second training measures by analyzing the first training performance data set 128. For example, a particular graphical display may include data presented along a first horizontal axis associated with a skewness measure 140 and a second vertical axis associated with a kurtosis measure 142. The example training modification recommendation 138 described below illustrates one such example graphical display. In the same or alternative embodiments, one or more of the training measures may be another mathematical moment measure.

[0033] The example training modification recommendation 138 illustrates a number of data points plotted along a first horizontal axis 146 associated with a skewness measure 140 of the first training performance data set 128 and a second vertical axis 148 associated with a kurtosis measure 142 of the first training performance data set 128. The example training modification recommendation 138 also illustrates a number of performance regions 150, 152. The first performance region 150 illustrates a cluster of training performance data points 154 within a region of acceptable performance 156. The first performance region 150 may indicate, for example, a current measured performance in the automated training system 104 within a performance threshold 144. The second performance region 152 illustrates a region within the example training modification recommendation 138 associated with an ideal clustering of the training performance data points. The second performance region 152 may indicate, for example, a desired performance in the automated training system 104.

[0034] The example training modification recommendation 138 also illustrates a number of outlying training performance data points 158. The outlying training performance data points 158 may indicate, for example, training performance that exceeds the performance threshold 144. In one particular embodiment, the outlying training performance data points 158 may be presented in different graphical representations depending on the distance of the particular outlying training performance data point 158 ​​from the center of the example training modification recommendation 138. For example, the outlying training performance data points 158B, 158C may be presented in a first color (e.g., orange) to indicate that they are outlying data but may not be of great concern to the automated training system 104, while the outlying training performance data points 158A, 158D may be presented in a second color (e.g., red) to indicate that they are significant outliers that may require more direct and / or more significant corrective action.

[0035] In some implementations, the processor(s) 106 of the computing device 102 may be further configured to communicate the training modification recommendations 134 to the automated training system 104. For example, the processor(s) 106 may be configured to generate training modification recommendation data 136 for communication to the automated training system 104.

[0036] In operation, system 100 may be configured to provide an exemplary automated pilot training evaluation system that may be used industry-wide to automatically and rigorously evaluate pilot training quality. For example, system 100 may be configured to generate training modification recommendations 134, where the training modification recommendations 134 include an indication of instructor performance associated with one or more pilot instructors based at least on correlation 132, skewness measure 140, and kurtosis measure 142. System 100 may be configured to compare different instructors of the same airline, a first instructor associated with a first airline and a second instructor associated with a second airline, instructors in different geographic regions, instructors within a particular geographic region, instructors across multiple curriculum competencies, instructors within a particular curriculum competencies, etc.

[0037] The automated training system 104 may be configured to provide the first training performance data 124 and / or the second training performance data 126 to the computing device 102 to enable the system 100 to generate the training modification recommendations 134. The automated training system 104 may include one or more processors 110 coupled to a memory 112. The automated training system 104 may be implemented as a stand-alone computing device (e.g., a flight simulator kiosk) and / or as a component of another computing device (e.g., as an app or program running on a smartphone or laptop computer). The automated training system 104 may also include components not shown in FIG. 1 . For example, to monitor input from and / or interaction with a user, the automated training system 104 may also include one or more input / output interfaces, one or more displays, one or more network interfaces, etc. Additionally, although FIG. 1 illustrates memory 112 of automated training system 104 as storing particular data, as described below, more, less, and / or different data may be present in memory 112 without departing from the scope of the subject disclosure.

[0038] The automated training system 104 may store performance data 114, user(s) 116, airline(s) 122, instructor(s) 118, training location(s) 120, etc. in memory 112. This data may be used to generate first and / or second training performance data 124, 126.

[0039] Performance data 114 may include data associated with one or more user performance of the automated training system 104 on one or more curriculum competencies. User(s) 116 may include data associated with the identity of one or more users of the automated training system 104. The identities may be anonymized, semi-anonymized, pseudo-anonymized, etc. Similarly, airline(s) 122, instructor(s) 118, and / or training location(s) 120 may include data associated with the airlines for which training performance is measured, the identities of those instructors, and / or identifiers of the various training locations. Depending on a particular implementation of the automated training system 104, there may be more, less, and / or different data between or within the automated training system 104. For example, a first automated training system may include user data (e.g., within user(s) 116) listing the user's first and last name, birth date, etc. Meanwhile, the second automated training system may include user data (e.g., in user(s) 116) that lists only the user's identification number. The processor(s) 110 of the automated training system 104 may be configured to collect, analyze, and / or communicate the data in memory 112 for communication to the computing device 102.

[0040] In some implementations, the processor(s) 106 of the computing device 102 may be configured to generate a first training performance data set 128 from the first training performance data 124 and generate a training data comparison set 130 from the second training performance data 126. This may include receiving a number of performance data requirements. The performance data requirements may, for example, describe what types of data in the first training performance data 124 will actually be used in identifying training modification recommendations 134, ensuring data integrity, and maintaining data standardization when receiving portions of the first training performance data 124 from multiple implementations of the automated training system 104. In some aspects, different implementations of the automated training system 104 may include different types of data from other implementations, more or less data on a particular topic than other implementations, stored data in a different format than other implementations, etc.

[0041] The processor(s) 106 of the computing device 102 may be configured to process the multiple performance data requirements to generate a single data mapping for the multiple data requirements. This may enable the processor(s) 106 to receive data from disparate sources and analyze data from the disparate sources, for example. The processor(s) 106 may be configured to generate the first training performance data set 128 using the single data mapping. In some aspects, the computing device 102 may also be configured to store the correlations 132 and training modification recommendations 134 in the memory 108.

[0042] In some implementations, the processor(s) 106 of the computing device 102 may be configured to generate a performance dashboard based at least on the first training performance dataset 128, the correlation 132, and the training modification recommendations 134. The performance dashboard may include analytical tools that may enable users of the automated training system to access and track the training modification recommendations. For example, the performance dashboard may include an indication that the first training performance dataset indicates that the performance of a first group of users of the automated training system 104 associated with the first training performance dataset 128 fails to meet the performance threshold 144.

[0043] 1 illustrates certain operations occurring within computing device 102 or automated training system 104, certain operations may be performed by other components of system 100 without departing from the scope of the subject disclosure. For example, automated training system 104 may be configured to generate a first training performance data set 128 from first training performance data 124 and / or generate a training data comparison data set 130 from second training performance data 126.

[0044] 1 depicts computing device 102 and automated training system 104 as separate entities, other configurations are possible without departing from the scope of the subject disclosure. For example, computing device 102 and automated training system 104 may be integrated into an electronic device. As a further example, some or all of computing device 102 may be integrated into some or all of the components of automated training system 104. As a further example, one or more components of computing device 102 and / or one or more components of automated training system 104 may be distributed across multiple computing devices (e.g., a cluster of servers).

[0045] FIG. 2 illustrates an exemplary architecture for a pilot training evaluation system 200 according to the subject disclosure. In general, the system 200 corresponds to some or all of the exemplary system 100 of FIG. 1. In some embodiments, the system 200 includes a number of data requirements 202, 204. As described in more detail above with reference to FIG. 1, these data requirements may describe, for example, what types of data in the first training performance data 124 will actually be used in identifying training modification recommendations 134, ensuring data integrity, and maintaining data standardization when receiving a portion of the first training performance data 124 from multiple embodiments of the automated training system 104. In some aspects, different embodiments of the automated training system 104 may include different types of data from other embodiments, more or less data on a particular topic than other embodiments, stored data in a different format than other embodiments, etc. For example, the file 202 of FIG. 2 may result from various different embodiments of the automated training system 104 of FIG. 1. Similarly, the various databases 204A, 204B, 204C may be part of the pilot training evaluation systems or may be implemented as repositories for training performance data (e.g., performance data 114 of FIG. 1) from the various pilot training evaluation systems.

[0046] In some implementations, the system 200 may also include a data processing module 206 coupled to the plurality of data requirements 202, 204. The data processing module may be one or more electronic devices and / or components of one or more electronic devices configured to extract, transform, and load data from the plurality of data requirements 202, 204. As described in more detail above with reference to FIG. 1, the system 200 may be configured to receive and analyze data from heterogeneous sources. For example, the processor(s) 106 of FIG. 1 may be configured to generate the first training performance data set 128 using a single data mapping for all of the data from the plurality of data requirements 202, 204. In some implementations, the system 200 may also include a database 208 coupled to the data processing module 206. The database 208 may be any suitable data storage device(s) configured to store the processed data generated by the data processing module 206.

[0047] In some implementations, system 200 may also include an analytics engine 210 coupled to database 208. Generally, analytics engine 210 corresponds to processor(s) 106 of FIG. 1 and is configured to analyze data stored in database 208 to identify correlations 132 between first training performance data set 128 and training data comparison set 130, and to generate training modification recommendations 134 for automated training system 104 based at least on correlations 132, as described in more detail above with reference to FIG.

[0048] In some implementations, system 200 may also include a database 212 coupled to analytics engine 210. Database 212 may be any suitable data storage device(s) configured to store data generated by analytics engine 210. Generally, database 212 corresponds to memory 108 of FIG. 1 and stores first training performance data set 128, training data comparison set 130, skewness measure 140, kurtosis measure 142, and performance threshold 144.

[0049] As described in more detail above with reference to FIG. 1, the pilot training evaluation system may be configured to generate training modification recommendations 214 (e.g., training modification recommendations 134 of FIG. 1) that are communicated to the automated training system. The training modification recommendations 214 may take many forms, including warnings, recommendations for corrective action, and the like. The training modification recommendations 214 also include, for example, curriculum update recommendations(s). In one particular aspect, the curriculum update recommendations may include automatically adding particular training modules to particular groups, such as instructors or students (for one or more locations), automatically removing particular training modules, and the like.

[0050] 1, the pilot training evaluation system may also be configured to generate one or more dashboards 216 based at least on the first training performance data set, the correlations, and the training modification recommendations 214. In one embodiment of the system 200, the dashboard(s) 216 may be communicated to one or more users of the system 200, one or more other components of the system 200, or some combination thereof. For example, data associated with one or more of the dashboards may be communicated back to the analytics engine 210 for further training evaluation and improvement.

[0051] 2 illustrates certain operations occurring within various components of the system 200, certain operations may be performed by other components of the system 200 without departing from the scope of the subject disclosure. For example, operations performed by the data processing module 206 may be decentralized and co-located with each individual data requirement 202, 204. As another example, the dashboard(s) 216 and the training modification recommendation(s) 214 may be generated in the analytics engine 210.

[0052] 2 depicts various components of the system 200 separately, other configurations are possible without departing from the scope of the main disclosure. For example, the data processing module 206 may be decentralized and co-located with each individual data requirement 202, 204. As another example, the dashboard(s) 216 and the training modification recommendation(s) 214 may be co-located with the analytics engine 210.

[0053] 3 is a flow chart of one embodiment of a method 300 for providing an automated pilot training evaluation system in accordance with the subject disclosure. Method 300 may be initiated, performed, or controlled by one or more processors executing instructions, such as by processor(s) 106 of FIG. 1 executing instructions from memory 108.

[0054] In some implementations, the method 300 receives 302 a first training performance data set. For example, the processor(s) 106 of FIG. 1 may receive the first training performance data set 128 associated with the performance of one or more users undergoing a curriculum with the automated training system 104.

[0055] In one embodiment of Figure 3, the method 300 also includes analyzing the first training performance data set to identify correlations between the first training performance data set and the training data comparison data set, at 304. For example, the processor(s) 106 of Figure 1 may analyze the first training performance set 128 to identify correlations 132 between the first training performance data set 128 and the training data comparison set 130.

[0056] In one embodiment of Figure 3, the method 300 also includes generating, at 306, training modification recommendations for the automated training system based at least on the correlation. For example, the processor(s) 106 of Figure 1 may generate training modification recommendations 134 for the automated training system 104 based at least on the correlation 132. In one embodiment of Figure 3, the method 300 also includes communicating, at 308, the training modification recommendations to the automated training system. For example, the processor(s) 106 of Figure 1 may communicate the training modification recommendations 134 to the automated training system 104.

[0057] Although method 300 is illustrated as including a particular number of steps, more, fewer, and / or different steps may be included within method 300 without departing from the scope of the subject disclosure. For example, method 300 may vary depending on the number and variety of data requirements available for processing, as described in more detail above with reference to FIG. 2. For example, method 300 may communicate training modification recommendations to one automated training system (and / or portion of an automated training system) prior to or simultaneously with generating training modification recommendations for another automated training system (and / or another portion of an automated training system).

[0058] 4 is a block diagram of a computing environment 400 including a computing device 410 configured to support aspects of the computer-implemented methods and computer-executable program instructions (or code) according to the subject disclosure. For example, the computing device 410, or a portion thereof, is configured to execute instructions to initiate, perform, or control one or more of the operations described in more detail above with reference to FIGS. 1-3. In one particular aspect, the computing device 410 may include the computing device 102 and / or the automated training system 104 of FIG. 1, the data processing module 206 and / or the analysis engine 210 of FIG. 2, one or more servers, one or more virtual devices, or a combination thereof.

[0059] Computing device 410 includes one or more processors 420. In one particular aspect, processor(s) 420 correspond to processor(s) 106 of FIG. 1. Processor(s) 420 are configured to communicate with a system memory 430, one or more storage devices 450, one or more input / output interfaces 440, one or more communication interfaces 460, or any combination thereof. System memory 430 includes volatile memory devices (e.g., random access memory (RAM) devices), non-volatile memory devices (e.g., read-only memory (ROM) devices, programmable read-only memory, and flash memory), or both. System memory 430 stores operating system 432, which may include a basic input / output system for booting computing device 410 as well as a full operating system for enabling computing device 410 to interact with a user, other programs, and other devices. The system memory 430 stores system (program) data 438, such as instruction instructions 436, training modification recommendations 134, correlations 132, first value distributions 139, first training performance data set 128 of FIG. 1, or combinations thereof.

[0060] The system memory 430 includes one or more applications 434 (e.g., collections of instructions) executable by the processor(s) 420. As an example, the one or more applications 434 include instructions 436 executable by the processor(s) 420 to initiate, control, or perform one or more of the operations described with reference to Figures 1-3. By way of example, the one or more applications 434 include instructions 436 executable by the processor(s) 420 to initiate, control, or perform one or more of the operations described with reference to the training modification recommendations 134, the correlations 132, or a combination thereof.

[0061] In one particular implementation, the system memory 430 includes a non-transitory computer-readable medium (e.g., a computer-readable storage device) having stored thereon instruction instructions 436 that, when executed by the processor(s) 420, cause the processor(s) 420 to initiate, execute, or control operations for an automated pilot training evaluation system. The operations also include analyzing the first training performance data set to identify correlations between the first training performance data set and the training data comparison set. The operations also include generating training modification recommendations for the automated training system based at least on the correlations. The operations also include communicating the training modification recommendations to the automated training system.

[0062] The one or more storage devices 450 include non-volatile storage devices, such as magnetic disks, optical disks, or flash memory devices. In one particular embodiment, the storage devices 450 include both removable and non-removable memory devices. The storage devices 450 are configured to store an operating system, images of the operating system, applications (e.g., one or more applications 434), and program data (e.g., program data 438). In one particular aspect, the system memory 430, the storage devices 450, or both, include tangible computer-readable media. In one particular aspect, one or more of the storage devices 450 are external to the computing device 410.

[0063] The one or more input / output interfaces 440 enable the computing device 410 to communicate with one or more input / output devices 470 to facilitate interaction with a user. For example, the one or more input / output interfaces 440 may include a display interface, an input interface, or both. For example, the input / output interface 440 may be adapted to receive input from a user, input from other computing devices, or a combination thereof. In some implementations, input / output interface 440 conforms to one or more standard interface protocols, including a serial interface (e.g., a universal serial bus (USB) interface or (IEEE (Institute of Electrical and Electronics Engineers) interface standard), a parallel interface, a display adapter, an audio adapter, or a custom interface. ("IEEE" is a registered trademark of the Institute of Electrical and Electronics Engineers, Inc. of Piscataway, NJ). In some embodiments, input / output device(s) 470 includes one or more user interface devices and a display, including any combination of buttons, keyboards, pointing devices, displays, speakers, microphones, touch screens, and other devices.

[0064] The processor 420 is configured to communicate with a device or controller 480 via one or more communication interfaces 460. For example, the one or more communication interfaces 460 may include a network interface. The device or controller 480 may include, for example, the automated training system 104 of FIG.

[0065] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device) stores instructions that, when executed by one or more processors, cause the one or more processors to initiate, perform, or control operations to perform some or all of the functions described above. For example, the instructions may be executable to perform one or more of the operations or methods of FIGS. 1-3. In some implementations, some or all of one or more of the operations or methods of FIGS. 1-3 may be performed by one or more processors (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs)) executing instructions through dedicated hardware circuitry, or any combination thereof.

[0066] The diagrams of the examples described herein are intended to provide a general understanding of the structure of various implementations. These diagrams are not intended to exhaustively describe all elements and features of apparatuses and systems utilizing the structures or methods described herein. Many other embodiments may become apparent to those skilled in the art upon review of the present disclosure. Other embodiments may be utilized and derived from the present disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. For example, the method operations may be performed in a different order than shown in the drawings, or one or more method operations may be omitted. The present disclosure and the figures should therefore be considered illustrative rather than restrictive.

[0067] Moreover, while specific examples have been shown and described herein, any subsequent configurations designed to achieve the same or similar results may be substituted for the specific embodiment shown. The present disclosure is intended to include any and all subsequent adaptations or variations of the various embodiments. Combinations of the above-described embodiments, as well as other embodiments not specifically described herein, will be apparent to those of skill in the art upon review of this specification.

[0068] The "Summary" of the disclosure is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above detailed description, various features may be grouped together or described within a single embodiment for the purpose of brevity of the disclosure. The above examples are illustrative of the disclosure, but are not limiting. Also, many modifications and variations are possible in accordance with the principles of the disclosure. As reflected in the following claims, the claimed subject matter may not be directed to all of the features of any of the disclosed embodiments. Thus, the scope of the disclosure is defined by the following claims and their equivalents.

[0069] Furthermore, the present disclosure includes embodiments according to the following clauses.

[0070] According to clause 1, a method includes receiving a first training performance data set and analyzing the first training performance data set to identify correlations between the first training performance data set and a training data comparison set. The method also includes generating training modification recommendations for an automated training system based at least on the correlations. The method also includes communicating the training modification recommendations to the automated training system.

[0071] Clause 2 includes the method of clause 1, wherein the training data comparison set includes a second training performance data set.

[0072] Clause 3 includes the method of clause 2, wherein the training data comparison set includes a control performance data set.

[0073] Clause 4 includes the method of clause 2 or 3, in which the first training performance data set includes training data associated with a first group of users of the automated training system, and the second training performance data set includes training data associated with a second group of users of the automated training system.

[0074] Clause 5 includes the method of clause 4, wherein the first group of users is distinct from the second group of users.

[0075] Clause 6 includes the method of clause 4 or 5, in which the first group of users is associated with training within a first training curriculum and the second group of users is associated with training within a second training curriculum.

[0076] Clause 7 includes the method of any one of clauses 4 to 6, where the first group of users is associated with a first geographic area and the second group of users is associated with a second geographic area.

[0077] Clause 8 includes the method of any one of clauses 4 to 7, where the first group of users is associated with a first instructor and the second group of users is associated with a second instructor.

[0078] Clause 9 includes the method of any one of clauses 4 to 8, wherein the first group of users is associated with a first training location and the second group of users is associated with a second training location.

[0079] Clause 10 includes the method of any one of clauses 1 to 9, wherein the method also includes determining a distribution of first values ​​based on the first training performance data set. The method also includes determining a skewness measure based on the distribution of first values, and generating the training modification recommendation for the automated training system is further based at least on the skewness measure.

[0080] Clause 11 includes the method of any one of clauses 1 to 10, wherein the method also includes determining a distribution of first values ​​based on the first training performance data set. The method also includes determining a kurtosis measure based on the distribution of first values, and generating the training modification recommendation for the automated training system is further based at least on the kurtosis measure.

[0081] Clause 12 includes the method of any one of clauses 1 to 11, in which the training modification recommendation includes a warning indicating that training performance will fail to meet a performance threshold.

[0082] Clause 13 includes the method of any one of clauses 1 to 12, in which the training modification recommendations include recommendations for updating training materials.

[0083] Clause 14 includes the method of any one of clauses 1 to 13, wherein the training modification recommendations include corrective action instructions associated with one or more users of the automated training system.

[0084] Clause 15 includes the method of any one of clauses 1 to 14, in which the training modification recommendations include a training performance report.

[0085] Clause 16 includes the method of clause 15, in which the training performance report includes a graphical display based at least on the correlation.

[0086] Clause 17 includes the method of clause 16, wherein the method further includes analyzing the first training performance dataset to identify one or more values ​​of a first training measure based on the first training performance dataset. The method also includes analyzing the first training performance dataset to identify one or more values ​​of a second training measure based on the first training performance dataset. In this case, a first axis of the graphical representation is associated with the first training measure and a second axis of the graphical representation is associated with the second training measure.

[0087] Clause 18 includes the method of clause 17, wherein the method further includes determining a distribution of first values ​​based on the first training performance data set, and the first training measure is a skewness measure based on the distribution of first values.

[0088] Clause 19 includes the method of clause 18, in which the second training measure is a kurtosis measure based on a distribution of the first values.

[0089] Clause 20 includes the method of any one of clauses 17 to 19, in which case the method further includes determining a distribution of first values ​​based on the first training performance data set, and wherein the first training measure or the second training measure is a mathematical moment measure based on the distribution of first values.

[0090] Clause 21 includes the method of any one of clauses 1 to 20, in which analyzing the first training performance data set to identify the correlation includes analyzing the first training performance data set to identify a match correlation coefficient associated with the first training performance data set and the training data comparison set.

[0091] Clause 22 includes the method of any one of clauses 1 to 21, wherein the first training performance data set includes a plurality of scoring measures assigned to users of the automated training system.

[0092] Clause 23 includes the method of any one of clauses 1-22, where the training data comparison set includes a second training performance data set, the first training performance data set including training data associated with a first group of users of the automated training system, the first group of users assigned to a first instructor, and the second training performance data set including training data associated with a second group of users of the automated training system, the second group of users assigned to one or more second instructors, the method further includes determining a distribution of first values ​​based on the first training performance data set, determining a skewness measure based on the distribution of first values, and determining a kurtosis measure based on the distribution of first values, and the training modification recommendation includes an indication of instructor performance associated with the first instructor, the indication based at least on the skewness measure and the kurtosis measure.

[0093] Clause 24 includes the method of clause 23, where the first instructor is associated with a first airline and the one or more second instructors are associated with a second airline.

[0094] Clause 25 includes the method of clause 23 or 24, in which the first instructor is associated with a first airline and the one or more second instructors are associated with the first airline, and the first instructor is different from the one or more second instructors.

[0095] Clause 26 includes the method of any one of clauses 23 to 25, where the first instructor is associated with a first geographic region and the one or more second instructors are associated with a second geographic region.

[0096] Clause 27 includes the method of any one of clauses 23 to 26, where the first instructor is associated with a first geographic region and the one or more second instructors are associated with the first geographic region, and the first instructor is different from the one or more second instructors.

[0097] Clause 28 includes the method of any one of clauses 23-27, where the first instructor is associated with a first curriculum competency and the one or more second instructors are associated with a second curriculum competency.

[0098] Clause 29 includes the method of any one of clauses 23-28, where the first instructor is associated with a first curricular competency and the one or more second instructors are associated with the first curricular competency, and the first instructor is different from the one or more second instructors.

[0099] Clause 30 includes the method of any one of clauses 1 to 29, in which receiving the first training performance data set includes receiving a plurality of performance data requirements, processing the plurality of performance data requirements to generate a single data mapping for the plurality of performance data requirements, and generating the first training performance data set.

[0100] Clause 31 includes the method of clause 30, wherein the method further includes storing the correlation and the training modification recommendation in a memory.

[0101] Clause 32 includes the method of clause 30 or 31, in which case the method further includes generating a performance dashboard based at least on the first training performance data set, the correlation, and the training modification recommendation.

[0102] Clause 33 includes the method of clause 32, in which the performance dashboard includes an indication that the first training performance dataset indicates that performance of a first group of users of the automated training system associated with the first training performance dataset fails to meet a performance threshold.

[0103] According to clause 34, a system includes a memory configured to store instructions and one or more processors configured to receive a first training performance data set. The one or more processors are also configured to analyze the first training performance data set to identify correlations between the first training performance data set and a training data comparison set. The one or more processors are also configured to generate training modification recommendations for an automated training system based at least on the correlations. The one or more processors are also configured to communicate the training modification recommendations to the automated training system.

[0104] Clause 35 includes the system of clause 34, wherein the training data comparison set includes a second training performance data set.

[0105] Clause 36 includes the system of clause 35, wherein the training data comparison set includes a control performance data set.

[0106] Clause 37 includes the system of clause 35 or 36, where the first training performance data set includes training data associated with a first group of users of the automated training system, and the second training performance data set includes training data associated with a second group of users of the automated training system.

[0107] Clause 38 includes the system of clause 37, wherein the first group of users is different from the second group of users.

[0108] Clause 39 includes the system of clause 37 or 38, in which the first group of users is associated with training within a first training curriculum and the second group of users is associated with training within a second training curriculum.

[0109] Clause 40 includes the system of any one of clauses 37 to 39, where the first group of users is associated with a first geographic area and the second group of users is associated with a second geographic area.

[0110] Clause 41 includes the system of any one of clauses 37 to 40, where the first group of users is associated with a first instructor and the second group of users is associated with a second instructor.

[0111] Clause 42 includes the system of any one of clauses 37 to 41, in which the first group of users is associated with a first training location and the second group of users is associated with a second training location.

[0112] Clause 43 includes the system of any one of clauses 34 to 42, where the one or more processors are further configured to determine a distribution of first values ​​based on the first training performance data set and determine a skewness measure based on the distribution of first values, and the one or more processors are further configured to generate the training modification recommendation for the automated training system based at least on the skewness measure.

[0113] Clause 44 includes the system of any one of clauses 34 to 43, where the one or more processors are further configured to determine a distribution of first values ​​based on the first training performance data set and determine a kurtosis measure based on the distribution of first values, and the one or more processors are further configured to generate the training modification recommendation for the automated training system based at least on the kurtosis measure.

[0114] Clause 45 includes the system of any one of clauses 34 to 44, in which the training modification recommendation includes a warning indicating that training performance will fail to meet a performance threshold.

[0115] Clause 46 includes the system of any one of clauses 34 to 45, in which the training modification recommendations include recommendations for updating training materials.

[0116] Clause 47 includes the system of any one of clauses 34 to 46, in which the training modification recommendations include corrective action instructions associated with one or more users of the automated training system.

[0117] Clause 48 includes the system of any one of clauses 34 to 47, in which the training modification recommendations include a training performance report.

[0118] Clause 49 includes the system of clause 48, wherein the training performance report includes a graphical display based at least on the correlation.

[0119] Clause 50 includes the system of clause 49, wherein the one or more processors are further configured to analyze the first training performance dataset to identify one or more values ​​of a first training measure based on the first training performance dataset, and to analyze the first training performance dataset to identify one or more values ​​of a second training measure based on the first training performance dataset, wherein a first axis of the graphical representation is associated with the first training measure and a second axis of the graphical representation is associated with the second training measure.

[0120] Clause 51 includes the system of clause 50, in which the one or more processors are further configured to determine a distribution of first values ​​based on the first training performance data set, and the first training measure is a skewness measure based on the distribution of first values.

[0121] Clause 52 includes the system of clause 51, wherein the second training measure is a kurtosis measure based on a distribution of the first values.

[0122] Clause 53 includes the system of any one of clauses 50 to 52, in which the one or more processors are further configured to determine a distribution of first values ​​based on the first training performance dataset, and the first training measure or the second training measure is a mathematical moment measure based on the distribution of first values.

[0123] Clause 54 includes the system of any one of clauses 34 to 53, in which the one or more processors are further configured to analyze the first training performance data set to identify the correlation by analyzing the first training performance data set to identify a match correlation coefficient associated with the first training performance data set and the training data comparison set.

[0124] Clause 55 includes the system of any one of clauses 34 to 54, wherein the first training performance data set includes a plurality of scoring measures assigned to users of the automated training system.

[0125] Clause 56 includes the system of any one of clauses 34 to 55, where the training data comparison set includes a second training performance data set, the first training performance data set including training data associated with a first group of users of the automated training system, the first group of users assigned to a first instructor, and the second training performance data set including training data associated with a second group of users of the automated training system, the second group of users assigned to one or more second instructors, the one or more processors are further configured to: determine a distribution of first values ​​based on the first training performance data set; determine a skewness measure based on the distribution of first values; and determine a kurtosis measure based on the distribution of first values, and the training modification recommendation includes an indication of instructor performance associated with the first instructor, the indication being based at least on the skewness measure and the kurtosis measure.

[0126] Clause 57 includes the system of clause 56, where the first instructor is associated with a first airline and the one or more second instructors are associated with a second airline.

[0127] Clause 58 includes the system of clause 56 or 57, in which the first instructor is associated with a first airline, the one or more second instructors are associated with the first airline, and the first instructor is different from the one or more second instructors.

[0128] Clause 59 includes the system of any one of clauses 56 to 58, where the first instructor is associated with a first geographic region and the one or more second instructors are associated with a second geographic region.

[0129] Clause 60 includes the system of any one of clauses 56-59, where the first instructor is associated with a first geographic region and the one or more second instructors are associated with the first geographic region, and the first instructor is different from the one or more second instructors.

[0130] Clause 61 includes the system of any one of clauses 56 to 60, where the first instructor is associated with a first curriculum competency and the one or more second instructors are associated with a second curriculum competency.

[0131] Clause 62 includes the system of any one of clauses 56-61, where the first instructor is associated with a first curricular competency and the one or more second instructors are associated with the first curricular competency, and the first instructor is different from the one or more second instructors.

[0132] Clause 63 includes the system of any one of clauses 34 to 62, in which the one or more processors are further configured to receive the first training performance data set by receiving a plurality of performance data requirements, processing the plurality of performance data requirements to generate a single data mapping for the plurality of performance data requirements, and generating the first training performance data set.

[0133] Clause 64 includes the system of clause 63, wherein the one or more processors are further configured to store the correlation and the training modification recommendation in a memory.

[0134] Clause 65 includes the system of clause 63 or 64, in which case the one or more processors are further configured to generate a performance dashboard based at least on the first training performance data set, the correlation, and the training modification recommendation.

[0135] Clause 66 includes the system of clause 65, where the performance dashboard includes an indication that the first training performance data set indicates that performance of a first group of users of the automated training system associated with the first training performance data set fails to meet a performance threshold.

[0136] According to clause 67, a non-transitory computer readable medium having stored thereon instructions executable by one or more processors to perform operations including receiving a first training performance data set. The operations also include analyzing the first training performance data set to identify correlations between the first training performance data set and a training data comparison set. The operations also include generating training modification recommendations for an automated training system based at least on the correlations. The operations also include communicating the training modification recommendations to the automated training system.

[0137] Clause 68 includes the non-transitory computer-readable medium of clause 67. In this case, the training data comparison set includes a second training performance data set.

[0138] Clause 69 includes the non-transitory computer readable medium of clause 68. In this case, the training data comparison set includes a control performance data set.

[0139] Clause 70 includes the non-transitory computer readable medium of clause 68 or 69, in which the first training performance data set includes training data associated with a first group of users of the automated training system and the second training performance data set includes training data associated with a second group of users of the automated training system.

[0140] Clause 71 includes the non-transitory computer readable medium of clause 70. In this case, the first group of users is different from the second group of users.

[0141] Clause 72 includes the non-transitory computer readable medium of clause 70 or 71, where the first group of users is associated with training within a first training curriculum and the second group of users is associated with training within a second training curriculum.

[0142] Clause 73 includes the non-transitory computer readable medium of any one of clauses 70 to 72, where the first group of users is associated with a first geographic area and the second group of users is associated with a second geographic area.

[0143] Clause 74 includes the non-transitory computer readable medium of any one of clauses 70 to 73, where the first group of users is associated with a first instructor and the second group of users is associated with a second instructor.

[0144] Clause 75 includes the non-transitory computer readable medium of any one of clauses 70 to 74, where the first group of users is associated with a first training location and the second group of users is associated with a second training location.

[0145] Clause 76 includes the non-transitory computer-readable medium of any one of clauses 67 to 75, in which the operations further include determining a distribution of first values ​​based on the first training performance data set, and determining a skewness measure based on the distribution of first values, and generating the training modification recommendation for the automated training system is further based at least on the skewness measure.

[0146] Clause 77 includes the non-transitory computer-readable medium of any one of clauses 67 to 76, in which the operations further include determining a distribution of first values ​​based on the first training performance data set, and determining a kurtosis measure based on the distribution of first values, and generating the training modification recommendation for the automated training system is further based at least on the kurtosis measure.

[0147] Clause 78 includes the non-transitory computer readable medium of any one of clauses 67 to 77. In this case, the training modification recommendation includes a warning indicating that training performance will fail to meet a performance threshold.

[0148] Clause 79 includes the non-transitory computer readable medium of any one of clauses 67 to 78, in which the training modification recommendations include recommendations for updating training materials.

[0149] Clause 80 includes the non-transitory computer-readable medium of any one of clauses 67 to 79, in which the training modification recommendations include corrective action instructions associated with one or more users of the automated training system.

[0150] Clause 81 includes the non-transitory computer-readable medium of any one of clauses 67 to 80. In this case, the training modification recommendations include a training performance report.

[0151] Clause 82 includes the non-transitory computer readable medium of clause 81. In this case, the training performance report includes a graphical display based at least on the correlation.

[0152] Clause 83 includes the non-transitory computer readable medium of clause 82. In that case, the operations further include analyzing the first training performance dataset to identify one or more values ​​of a first training measure based on the first training performance dataset, and analyzing the first training performance dataset to identify one or more values ​​of a second training measure based on the first training performance dataset, wherein a first axis of the graphical representation is associated with the first training measure and a second axis of the graphical representation is associated with the second training measure.

[0153] Clause 84 includes the non-transitory computer-readable medium of clause 83. In this case, the operations further include determining a distribution of first values ​​based on the first training performance data set, and the first training measure is a skewness measure based on the distribution of first values.

[0154] Clause 85 includes the non-transitory computer-readable medium of clause 84. In this case, the second training measure is a kurtosis measure based on a distribution of the first values.

[0155] Clause 86 includes the non-transitory computer-readable medium of any one of clauses 83 to 85, in which the operations further include determining a distribution of first values ​​based on the first training performance data set, and the first training measure or the second training measure is a mathematical moment measure based on the distribution of first values.

[0156] Clause 87 includes the non-transitory computer readable medium of any one of clauses 67 to 86, in which analyzing the first training performance data set to identify the correlation includes analyzing the first training performance data set to identify a match correlation coefficient associated with the first training performance data set and the training data comparison set.

[0157] Clause 88 includes the non-transitory computer-readable medium of any one of clauses 67 to 87, in which the first training performance data set includes a plurality of scoring measures assigned to users of the automated training system.

[0158] Clause 89 includes the non-transitory computer readable medium of any one of clauses 67-88, where the training data comparison set includes a second training performance data set, the first training performance data set including training data associated with a first group of users of the automated training system, the first group of users assigned to a first instructor, and the second training performance data set including training data associated with a second group of users of the automated training system, the second group of users assigned to one or more second instructors. The operations further include determining a distribution of first values ​​based on the first training performance data set, determining a skewness measure based on the distribution of first values, and determining a kurtosis measure based on the distribution of first values, and the training modification recommendation includes an indication of instructor performance associated with the first instructor, the indication based at least on the skewness measure and the kurtosis measure.

[0159] Clause 90 includes the non-transitory computer readable medium of clause 89. In that case, the first instructor is associated with a first airline and the one or more second instructors are associated with a second airline.

[0160] Clause 91 includes the non-transitory computer readable medium of clause 89 or 90, in which case the first instructor is associated with a first airline, the one or more second instructors are associated with the first airline, and the first instructor is different from the one or more second instructors.

[0161] Clause 92 includes the non-transitory computer readable medium of any one of clauses 89 to 91, where the first instructor is associated with a first geographic region and the one or more second instructors are associated with a second geographic region.

[0162] Clause 93 includes the non-transitory computer readable medium of any one of clauses 89 to 92, where the first instructor is associated with a first geographic region and the one or more second instructors are associated with the first geographic region, and the first instructor is different from the one or more second instructors.

[0163] Clause 94 includes the non-transitory computer readable medium of any one of clauses 89 to 93, where the first instructor is associated with a first curricular competency and the one or more second instructors are associated with a second curricular competency.

[0164] Clause 95 includes the non-transitory computer readable medium of any one of clauses 89-94, where the first instructor is associated with a first curricular competency and the one or more second instructors are associated with the first curricular competency, and the first instructor is different from the one or more second instructors.

[0165] Clause 96 includes the non-transitory computer readable medium of any one of clauses 67 to 95, in which receiving the first training performance data set includes receiving a plurality of performance data requirements, processing the plurality of performance data requirements to generate a single data mapping for the plurality of performance data requirements, and generating the first training performance data set.

[0166] Clause 97 includes the non-transitory computer-readable medium of clause 96. In this case, the operations further include storing the correlation and the training modification recommendation in a memory.

[0167] Clause 98 includes the non-transitory computer-readable medium of any one of clauses 96 or 97. In that case, the operations further include generating a performance dashboard based at least on the first training performance data set, the correlation, and the training modification recommendation.

[0168] Clause 99 includes the non-transitory computer readable medium of clause 98. In that case, the performance dashboard includes an indication that the first training performance data set indicates that performance of a first group of users of the automated training system associated with the first training performance data set fails to meet a performance threshold.

[0169] According to clause 100, a device includes means for receiving a first training performance data set, the device also includes means for analyzing the first training performance data set to identify correlations between the first training performance data set and a training data comparison set. The device also includes means for generating training modification recommendations for an automated training system based at least on the correlations. The device also includes means for communicating the training modification recommendations to the automated training system.

[0170] Clause 101 includes the device of clause 100. In this case, the training data comparison set includes a second training performance data set.

[0171] Clause 102 includes the device of clause 101, wherein the training data comparison set includes a control performance data set.

[0172] Clause 103 includes the device of clause 101 or 102, where the first training performance data set includes training data associated with a first group of users of the automated training system and the second training performance data set includes training data associated with a second group of users of the automated training system.

[0173] Clause 104 includes the device of clause 103, wherein the first group of users is different from the second group of users.

[0174] Clause 105 includes the device of clause 103 or 104, where the first group of users is associated with training within a first training curriculum and the second group of users is associated with training within a second training curriculum.

[0175] Clause 106 includes the device of any one of clauses 103 to 105, where the first group of users is associated with a first geographic area and the second group of users is associated with a second geographic area.

[0176] Clause 107 includes the device of any one of clauses 103 to 106, where the first group of users is associated with a first instructor and the second group of users is associated with a second instructor.

[0177] Clause 108 includes the device of any one of clauses 103 to 107, where the first group of users is associated with a first training location and the second group of users is associated with a second training location.

[0178] Clause 109 includes the device of any one of clauses 100 to 108, in which the device further includes means for determining a distribution of first values ​​based on the first training performance data set and means for determining a skewness measure based on the distribution of first values, and generating the training modification recommendation for the automated training system is further based at least on the skewness measure.

[0179] Clause 110 includes the device of any one of clauses 100 to 109, wherein the device further includes means for determining a distribution of a first value based on the first training performance data set and means for determining a kurtosis measure based on the distribution of the first values, and wherein generating the training modification recommendation for the automated training system is further based at least on the kurtosis measure.

[0180] Clause 111 includes the device of any one of clauses 100 to 110, in which the training modification recommendation includes a warning indicating that training performance will fail to meet a performance threshold.

[0181] Clause 112 includes the device of any one of clauses 100 to 111, in which case the training modification recommendations include recommendations for updating training materials.

[0182] Clause 113 includes the device of any one of clauses 100 to 112, in which case the training modification recommendations include corrective action instructions associated with one or more users of the automated training system.

[0183] Clause 114 includes the device of any one of clauses 100 to 113, in which case the training modification recommendations include a training performance report.

[0184] Clause 115 includes the device of clause 114. In this case, the training performance report includes a graphical display based at least on the correlation.

[0185] Clause 116 includes the device of clause 115, wherein the device further includes means for analysing the first training performance data set to identify one or more values ​​of a first training measure based on the first training performance data set, and means for analysing the first training performance data set to identify one or more values ​​of a second training measure based on the first training performance data set, wherein a first axis of the graphical representation is associated with the first training measure and a second axis of the graphical representation is associated with the second training measure.

[0186] Clause 117 includes the device of clause 116, wherein the device further includes means for determining a distribution of first values ​​based on the first training performance data set, and the first training measure is a skewness measure based on the distribution of first values.

[0187] Clause 118 includes the device of clause 117. In this case, the second training measure is a kurtosis measure based on a distribution of the first values.

[0188] Clause 119 includes the device of clause 118, wherein the device further includes means for determining a distribution of first values ​​based on the first training performance data set, and wherein the first training measure or the second training measure is a mathematical moment measure based on the distribution of first values.

[0189] Clause 120 includes the device of any one of clauses 100 to 119, where analyzing the first training performance data set to identify the correlation includes analyzing the first training performance data set to identify a match correlation coefficient associated with the first training performance data set and the training data comparison set.

[0190] Clause 121 includes the device of any one of clauses 100 to 120, wherein the first training performance data set includes a plurality of scoring measures assigned to users of the automated training system.

[0191] Clause 122 includes the device of any one of clauses 100 to 121, where the training data comparison set includes a second training performance data set, the first training performance data set including training data associated with a first group of users of the automated training system, the first group of users assigned to a first instructor, and the second training performance data set including training data associated with a second group of users of the automated training system, the second group of users assigned to one or more second instructors. The device further includes means for determining a distribution of first values ​​based on the first training performance data set, means for determining a skewness measure based on the distribution of first values, and means for determining a kurtosis measure based on the distribution of first values. The training modification recommendation includes an indication of instructor performance associated with the first instructor, the indication being based at least on the skewness measure and the kurtosis measure.

[0192] Clause 123 includes the device of clause 122, where the first instructor is associated with a first airline and the one or more second instructors are associated with a second airline.

[0193] Clause 124 includes the device of clause 122 or 123, where the first instructor is associated with a first airline, the one or more second instructors are associated with the first airline, and the first instructor is different from the one or more second instructors.

[0194] Clause 125 includes the device of any one of clauses 122 to 124, where the first instructor is associated with a first geographic region and the one or more second instructors are associated with a second geographic region.

[0195] Clause 126 includes the device of clauses 122 through 125, where the first instructor is associated with a first geographic region and the one or more second instructors are associated with the first geographic region, and the first instructor is different from the one or more second instructors.

[0196] Clause 127 includes the device of any one of clauses 122 to 126, where the first instructor is associated with a first curriculum competency and the one or more second instructors are associated with a second curriculum competency.

[0197] Clause 128 includes the device of any one of clauses 122-127, where the first instructor is associated with a first curriculum competency and the one or more second instructors are associated with the first curriculum competency, and the first instructor is different from the one or more second instructors.

[0198] Clause 129 includes the device of any one of clauses 100 to 128, in which receiving the first training performance data set includes receiving a plurality of performance data requirements, processing the plurality of performance data requirements to generate a single data mapping for the plurality of performance data requirements, and generating the first training performance data set.

[0199] Clause 130 includes the device of clause 129. In that case, the device further includes means for storing the correlation and the training modification recommendation in a memory.

[0200] Clause 131 includes the device of clause 129 or 130, in which case the device further includes means for generating a performance dashboard based at least on the first training performance data set, the correlation, and the training modification recommendation.

[0201] Clause 132 includes the device of clause 131. In which case, the performance dashboard includes an indication that the first training performance data set indicates that performance of a first group of users of the automated training system associated with the first training performance data set fails to meet a performance threshold.

Claims

1. 1. A method (300) comprising: receiving (302) a first training performance data set (128); analyzing (304) the first training performance data set to identify correlations (132) between the first training performance data set and a training data comparison set (130); generating (306) training modification recommendations (134) for an automated training system (104) based at least on the correlation; and communicating (308) the training modification recommendations to the automated training system.

2. The method of claim 1 , wherein the training data comparison set comprises a second training performance data set.

3. the first training performance data set includes training data associated with a first group of users (116) of the automated training system; The method of claim 2 , wherein the second training performance data set includes training data associated with a second group of users (116) of the automated training system.

4. The first group of users comprises: a first group of users associated with training within a first training curriculum, and the second group of users associated with training within a second training curriculum; a first group of users associated with a first geographic area, and the second group of users associated with a second geographic area; associated with a first instructor (118), and the second group of users is associated with a second instructor (118); or 4. The method of claim 3, wherein the second group of users is at least one of: associated with a first training location (120); and the second group of users is associated with a second training location (120).

5. determining a first distribution of values ​​(139) based on the first training performance dataset; and determining a skewness measure (140) based on the distribution of the first values; The method of claim 1 , wherein generating the training modification recommendations for the automated training system is further based at least on the skewness measure.

6. determining a distribution of first values ​​based on the first training performance data set; and determining a kurtosis measure (142) based on the distribution of the first values; The method of claim 1 , wherein generating the training modification recommendations for the automated training system is further based at least on the kurtosis measure.

7. 10. The method of claim 1, wherein the training modification recommendations include a warning indicating that training performance fails to meet a performance threshold (144), a recommendation to update training materials, or instructions for corrective action associated with one or more users of the automated training system.

8. The method of claim 1 , wherein the training modification recommendations include a training performance report.

9. The training performance report includes a graphical representation (138) based at least on the correlation; The method comprises: analyzing the first training performance data set to identify one or more values ​​of a first training measure based on the first training performance data set; analyzing the first training performance data set to identify one or more values ​​of a second training measure based on the first training performance data set; 9. The method of claim 8, wherein a first axis (146) of the graphical display is associated with the first training measure and a second axis (148) of the graphical display is associated with the second training measure.

10. 10. The method of claim 9, further comprising determining a distribution of first values ​​based on the first training performance data set, wherein the first training measure is a skewness measure based on the distribution of first values.

11. The method of claim 10 , wherein the second training measure is a kurtosis measure based on the distribution of the first values.

12. 10. The method of claim 9, further comprising determining a distribution of first values ​​based on the first training performance data set, wherein the first training measure or the second training measure is a mathematical moment measure based on the distribution of first values.

13. 2. The method of claim 1 , wherein analyzing the first training performance data set to identify the correlation comprises analyzing the first training performance data set to identify a match correlation coefficient associated with the first training performance data set and the training data comparison set.

14. A computer program comprising computer program instructions which, when executed by a computer processor, cause the computer processor to perform a method according to any one of claims 1 to 13.

15. a memory (108) configured to store instructions; 1. A system comprising one or more processors (106), the one or more processors: receiving (302) a first training performance data set (128); analyzing (304) the first training performance data set to identify correlations (132) between the first training performance data set and a training data comparison set (130); generating (306) training modification recommendations (134) for an automated training system (104) based at least on the correlation; and The system is configured to communicate (308) the training modification recommendations to the automated training system.

16. The training modification recommendation includes a graphical representation (138) based at least on the correlation, and the one or more processors: analyzing the first training performance data set to identify one or more values ​​of a first training measure (140) based on the first training performance data set; analyzing the first training performance dataset to identify one or more values ​​of a second training measure (142) based on the first training performance dataset; 16. The system of claim 15, wherein a first axis (146) of the graphical display is associated with the first training measure and a second axis (148) of the graphical display is associated with the second training measure.

Images