Civil aviation flight student multi-dimensional evaluation method

Abstract

The invention relates to the field of flight training evaluation and skill classification, and discloses a civil aviation flight student multi-dimensional evaluation method, which comprises the following steps: collecting original evaluation data of each check point in a flight training process, performing chi-square test on a relationship between the data of each check point and a final evaluation category of a student, and obtaining a final evaluation category of the student; and calculating the Phi correlation between each check point and the final classification of the trainee, performing normalization processing on the Phi correlation coefficient in each dimension, calculating the classification distinction degree of each check point, and multiplying the normalized Phi coefficient by the corresponding classification distinction degree to obtain the dynamic weight of each check point. Through constructing a scoring mechanism based on check point failure marking and weighted score deduction, refined quantitative evaluation of trainee training performance is realized. And a weight model fusing a normalized Phi coefficient, a category discrimination degree and a basic score is adopted, so that the discrimination capability and the constitutive property of the score are improved. The scoring result has good interpretability, and is convenient for identifying the ability weakness and the training improvement direction.

Description

Technical Field

[0001] The present invention relates to the field of flight training evaluation and skill classification, and specifically to a multi-dimensional evaluation method for civil aviation flight trainees. Background Art

[0002] In the current skill training and ability evaluation system, the performance of trainees is generally evaluated by a linear statistical method based on a scoring table. The system usually accumulates scores with fixed weights or uses the project completion rate as the evaluation basis, resulting in an average role of different checkpoints in the final score and making it difficult to reflect the actual contribution of each checkpoint in ability differentiation.

[0003] During the scoring process, for the checkpoints that fail, only marking or setting a unified deduction value is often carried out. This processing method lacks a hierarchical expression of the importance of failure items, making it difficult for some high-risk and high-weight operation mistakes to reflect their due impact in the final result and making it difficult to truly reflect the ability gap of trainees.

[0004] The scoring results are mostly presented as total scores, lacking a structured analysis of the composition of sub-item scores. It is difficult for system managers to identify specific problem points through the results, thus restricting the implementation of personalized training feedback and differentiated teaching strategies.

[0005] The classification method of evaluation results is mostly binary or single-level interval division, lacking a continuous grading mechanism. This setting cannot meet the dynamic hierarchical management needs of trainees at different levels and restricts the ability to optimize training paths based on data-driven.

[0006] To achieve more targeted and structured training evaluation, a comprehensive scoring and classification model based on behavior differences and task discrimination is needed to improve the adaptability and feedback efficiency of the training evaluation system in complex task environments. Summary of the Invention

[0007] Aiming at the deficiencies of the prior art, the present invention provides a multi-dimensional evaluation method for civil aviation flight trainees, which solves the problems in the existing evaluation methods that the discrimination of key checkpoints cannot be accurately reflected, the scoring results lack structure, and the classification granularity of trainee abilities is insufficient.

[0008] To achieve the above objectives, the present invention is realized through the following technical solutions: A multi-dimensional evaluation method for civil aviation flight trainees, comprising the following steps: S1. Collect the original evaluation data of each checkpoint during the flight training process, and the evaluation result of the checkpoint is a binary value of pass or fail; S2. Conduct a chi-square test on the relationship between the data of each checkpoint and the final evaluation category (category A or category B) of the trainee, and calculate the chi-square value of each checkpoint; S3. Calculate the Phi correlation coefficient between each checkpoint and the final classification of the trainee according to the chi-square value; S4. Normalize the Phi correlation coefficients within each dimension to obtain the normalized Phi coefficients; S5. Calculate the class discrimination of each checkpoint, defined as the difference in the passing rates of Class A and Class B trainees at this checkpoint; S6. Multiply the normalized Phi coefficient by the corresponding class discrimination to obtain the dynamic weight of each checkpoint; S7. Calculate the score of each checkpoint according to the basic score, dynamic weight of each checkpoint and the performance of the trainee; S8. Accumulate the scores of all checkpoints that are not passed and subtract them from 100 to obtain the final total score of the trainee; S9. Judge the comprehensive ability level of the trainee based on the final score to achieve multi-dimensional classification and evaluation of flight trainees.

[0009] Preferably, the calculation formula of the Phi correlation coefficient is: where is the Phi correlation coefficient, is the chi-square value, is the total number of samples.

[0010] Preferably, the calculation method of the normalized Phi correlation coefficient is: ; where represents the normalized Phi coefficient of the th checkpoint within its dimension, represents the original Phi correlation coefficient of the th checkpoint, represents the sum of the Phi correlation coefficients of all checkpoints within the same dimension, is the index of the checkpoint within this dimension, is the number of checkpoints within the dimension.

[0011] Preferably, the calculation formula of the class discrimination CD is: ; where and are the passing rates of Class A and Class B trainees at a certain checkpoint respectively.

[0012] Preferably, the dynamic weight of each checkpoint is calculated by the following formula: Among them, represents the dynamic weight of the th checkpoint, represents the normalized Phi coefficient of the th checkpoint, represents the class discrimination of the th checkpoint.

[0013] Preferably, the calculation formula of the checkpoint score is: ; Among them, represents the score or potential deduction value of the th checkpoint, represents the dynamic weight of the th checkpoint, is the basic score of the dimension to which it belongs, and 100 is to standardize the score to a 100-point system.

[0014] Preferably, the calculation formula of the final score of the trainee is: ; Among them, represents the final total score of the trainee, represents the initial full score or basic score of the trainee, is the failure identification variable of the checkpoint , is the score of the checkpoint .

[0015] Preferably, the dimension includes the core ability dimensions of flight trajectory management, communication, procedure execution, problem solving, and situational awareness.

[0016] Preferably, the evaluation method is applicable to any flight training stage, and the data used in this evaluation method includes data samples from the initial training of 9 hours and the intensive training stage of 13 hours.

[0017] The present invention provides a multi-dimensional evaluation method for civil aviation flight trainees. It has the following beneficial effects: 1. The present invention adopts a total score calculation method of "integrating checkpoint failure marking and weighted deduction", which realizes the dynamic response ability of the scoring mechanism to the operation errors of trainees. Compared with the existing method of simply accumulating passed items or percentage scoring, this scheme can clearly quantify the specific impact of unpassed items and solves the problem of insufficient discrimination of failed items in traditional methods.

[0018] 2. The present invention constructs a weight model of "normalized Phi coefficient × category discrimination degree × basic score" and embeds it into the final scoring logic, achieving an organic unity of ability discrimination and evaluation weight. Traditional solutions often fail to consider the differences in checkpoint discrimination ability, resulting in flat scoring results and lack of structure; this solution overcomes the technical shortcoming of such insensitive scoring.

[0019] 3. The "Total Score model" constructed by using the superposition calculation of failure items has strong interpretability. Managers can clearly identify the deduction points and influence weights of trainees at a glance. Compared with the black-box models or fuzzy scoring rules often used in existing methods, this solution effectively solves the problems that the training evaluation results are difficult to trace and are not conducive to corrective guidance.

[0020] 4. The multi-level classification mechanism based on score intervals provided by the present invention supports the docking of subsequent modules such as automatic hierarchical training and training feedback push. Different from the existing solutions that only provide a single judgment of passing or not, this mechanism has better adaptability and scalability, significantly improving the management efficiency and intelligent level of the training system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is the flow chart of the improved scoring of the present invention; Figure 2 is the original scoring flow chart; Figure 3 is the original score chart of 9-hour flight training; Figure 4 is the adjusted score chart of 9-hour flight training; Figure 5 is the original score chart of 13-hour flight training; Figure 6 is the adjusted score chart of 13-hour flight training. DETAILED DESCRIPTION OF THE INVENTION

[0022] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the specification of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0023] In order to more accurately reflect the comprehensive ability performance of flight cadets during the training process, the present invention provides a multi-dimensional evaluation method for civil aviation flight cadets based on statistical correlation analysis, which combines the correlation between checkpoint performance and the final classification (A / B category) to achieve scientific evaluation of trainees' key abilities and dynamic weight allocation. The following is a detailed description of the implementation manner of this method in combination with specific steps.

[0024] Please refer to the appendix Figure 1 - Appendix Figure 6 Embodiments of the present invention provide a multi-dimensional evaluation method for civil aviation flight trainees, including the following steps: S1. Collect the original evaluation data of each checkpoint during the flight training process, and the evaluation result of the checkpoint is a binary value of pass or fail; During the evaluation of the abilities of flight training trainees, to ensure that the scoring mechanism can truly and objectively reflect the performance of trainees during the training process, high-quality original data collection is required first. This step, as the basic link of the overall evaluation method of the present invention, directly determines the accuracy and applicability of subsequent correlation analysis and weight calculation.

[0025] As a prerequisite, the collected data should cover all operation behavior nodes with evaluation significance in the flight training tasks, and the paired data relationship between these nodes and the trainee evaluation results should be included to support subsequent multi-dimensional modeling work based on statistics.

[0026] In the present invention, the core objective of step S1 is to construct an original data structure system for evaluation analysis, and this data system needs to meet the following requirements: on the one hand, it can completely reflect all the performance details of each trainee during the training process; on the other hand, it must have a standardized data organization format to facilitate the direct call and processing of statistical analysis models.

[0027] In this embodiment, the original data collection object is the checkpoint results and the final evaluation results of the tasks performed by trainees at fixed flight training time points.

[0028] Checkpoints are various observation points preset in the flight training task process according to the civil aviation Competency-Based Training and Assessment (CBTA) framework, and each checkpoint corresponds to a skill or behavior requirement.

[0029] Each flight trainee will encounter multiple stages in the training task, and each stage includes several checkpoints. For example, during the 9-hour initial training, each trainee will experience 22 stages and a total of 108 checkpoints; during the 13-hour intensive training, it involves 36 stages and a total of 158 checkpoints.

[0030] It should be noted that each checkpoint is clearly defined as a binary state of "whether it meets the standard" in the training. The following coding rules are adopted in the present invention: If the trainee's performance at this checkpoint meets the predetermined operation standard, it is recorded as 0 (pass); If the performance does not meet the evaluation standard, it is recorded as 1 (fail).

[0031] To support the Phi correlation coefficient analysis method adopted in the present invention, all checkpoint data must be binary variables. This provides a data basis for constructing a 2×2 contingency table and performing a statistical independence test subsequently.

[0032] In this embodiment, in addition to collecting the performance data of the trainees at each checkpoint, it is also necessary to synchronously record the final evaluation results of each trainee during the training in this stage. This result also adopts a binary classification form: Class A: Indicates that the training is up to standard and the trainee is eligible for subsequent stage training; Class B: Indicates that the training is not up to standard and there is a risk of grounding.

[0033] In a possible implementation, each sample record needs to contain the following fields: Unique trainee number (student_id, used for data collation and not participating in scoring); Numbers of each checkpoint (such as FPM.2, SAW.1, etc., referring to the CBTA code specification); Pass status of each checkpoint (0 / 1); Final classification result of the trainee (A or B); Core competency dimension to which the checkpoint belongs (such as FPM, COM, PRO, etc.); Flight phase to which it belongs (such as engine start, cruise flight, landing, etc.).

[0034] As an option, to ensure the standardization of the data structure, the present invention organizes all data into a training record table in the form of a table structure. Each row represents a complete training performance sample of a trainee, and each column represents the evaluation result of a checkpoint and the final evaluation label.

[0035] Specifically, the following is a typical data structure style: Table 1: It can be understood that each "0 / 1" mark in the table indicates whether the trainee's performance at the corresponding checkpoint is up to standard, and the "evaluation result" is the comprehensive evaluation conclusion of the overall performance during the training stage. Such a structure not only supports human viewing but also facilitates statistical analysis of large sample data by machine programs.

[0036] In some embodiments, the dimension attribution information of each checkpoint can also be recorded to support subsequent normalization processing and dimension-level authority control.

[0037] For example, during 9 hours of training, 108 checkpoints can be respectively mapped to the following five competency dimensions: Table 2: 9-Hour Dimension Details Table During the 13-hour training, the sixth dimension KNO (Knowledge Application, with a total of 10 items) was added. The above mapping information can be referred to in the following table: Table 3: 13-hour Dimension Details Table As a further technical process, the following standardization and cleaning steps are performed on the collected data in the present invention: Delete all fields containing trainee identity information (such as name, gender, ID card, etc.) to comply with regulations; Use a missing value handling algorithm to eliminate incomplete records; Uniformly standardize the final assessment result labels into binary variables, with class A being 1 and class B being 0, or vice versa, depending on the implementation; Ensure that all checkpoint data types are boolean or numerical 0 / 1 for subsequent statistics.

[0038] Data processing and statistical analysis (Phi correlation coefficient) of step S1 After the data collection is completed, the performance of each checkpoint is analyzed for correlation by calculating the Phi correlation coefficient. The following are the Phi correlation coefficient analysis results for the 9-hour and 13-hour flight training phases. The table shows the degree of correlation of each checkpoint with the trainee's final assessment.

[0039] Table 4: Phi Correlation Coefficient Table for 9-hour Flight Training As can be seen from Table 4, for some checkpoints, such as checkpoint 20.1.5 (SAW) and checkpoint 18.4.2 (FPM), the Phi coefficients are relatively high, being 0.408 and 0.34 respectively, indicating that these checkpoints have a strong differentiating effect on the trainee's final classification. The checkpoints with high Phi coefficients are mainly concentrated in the SAW and FPM dimensions, suggesting that the trainee's information management and manual flight capabilities in complex environments have a significant impact on the final performance.

[0040] Table 5: Phi Correlation Coefficient Table for 13-hour Flight Training As can be seen from Table 5, checkpoint 21.3 (FPM) and checkpoint 32.2.1 (PRO) have relatively high Phi coefficients, being 0.298 and 0.275 respectively, indicating that these checkpoints have a strong indicative effect on trainee classification. In the 13-hour dataset, some checkpoints in the PSD and PRO dimensions also show relatively high Phi coefficients, reflecting the increasing importance of decision-making and regulation execution capabilities during longer training.

[0041] From the above Phi correlation coefficient table, the correlation between different checkpoints and the final evaluation results can be seen. For example, the Phi value of checkpoint 20.1.5 (SAW dimension) in the 9-hour flight training is 0.408, indicating a strong correlation with the final evaluation results of the trainees.

[0042] In summary, step S1 provides a data basis guarantee for the dynamic weighted scoring method of the present invention based on the Phi correlation coefficient and category discrimination by collecting standardized and structured flight training checkpoint data. After having complete checkpoint records and corresponding classification labels, subsequent steps can carry out checkpoint analysis and scoring modeling work.

[0043] S2. Perform a chi-square test on the relationship between the data of each checkpoint and the final evaluation category (category A or category B) of the trainees, and calculate the chi-square value of each checkpoint; After step S1 completes the collection of flight training data, the main purpose of step 2 is to quantify the correlation between each checkpoint and the final evaluation results (category A or category B) of the trainees through chi-square test and Phi correlation coefficient. The Phi correlation coefficient is a statistic that measures the strength of the relationship between binary classification variables and can help analyze the importance of each checkpoint in flight training evaluation.

[0044] Chi-square test and Phi correlation coefficient formula: The chi-square test is used to detect the difference between the observed frequency and the expected frequency. If the difference is significant, it indicates that there is a certain correlation between the variables. The calculation formula of the chi-square test is: Where: is the chi-square value, is the observed frequency, that is, the frequency in the actual sample data; is the expected frequency, the frequency calculated based on the independent hypothesis.

[0045] In binary classification data, the contingency table will show the distribution of trainees who passed and did not pass each checkpoint among category A and category B trainees. The specific 2×2 contingency table form is as follows: Table 6: According to this contingency table, the calculation formula of the expected frequency is as follows: Among them, a represents the number of students finally rated as Class A who passed the specific checkpoint; b represents the number of students finally rated as Class B who passed the specific checkpoint; c represents the number of students finally rated as Class A who did not pass the specific checkpoint; d represents the number of students finally rated as Class B who did not pass the specific checkpoint; , , , are the expected frequencies calculated based on these observed frequencies: The expected frequency corresponding to cell a; The expected frequency corresponding to cell b; The expected frequency corresponding to cell c; The expected frequency corresponding to cell d.

[0046] After obtaining the chi-square value, we can further calculate the Phi correlation coefficient to measure the strength of the relationship between two variables. The formula for the Phi correlation coefficient is: ; Where: is the Phi correlation coefficient, is the chi-square value, is the total sample size.

[0047] The value range of the Phi correlation coefficient is from -1 to 1. The closer the value is to 1, the stronger the correlation between the checkpoint and the final evaluation result of the students; the closer the value is to 0, the weaker the correlation.

[0048] Implementation process of Step 2 Data sorting and contingency table construction: According to the student evaluation data collected in Step S1, construct a 2×2 contingency table corresponding to each checkpoint, and record the distribution of passing and non-passing students among Class A and Class B students.

[0049] Calculation of the chi-square value: For each checkpoint, use the chi-square test formula to calculate the chi-square value between it and the final evaluation result of the students .

[0050] Calculation of the Phi correlation coefficient: Based on the chi-square value, calculate the Phi correlation coefficient to quantify the strength of the relationship between the checkpoint and the final evaluation result of the students.

[0051] Statistical analysis and result display: Organize the calculation results of the chi-square value and the Phi correlation coefficient into a table to display the correlation strength of each checkpoint, and help further analyze the impact of each checkpoint on the student evaluation result.

[0052] To improve the calculation efficiency, especially when dealing with a large amount of checkpoint data, the following extended solutions can be considered: Data preprocessing and distribution optimization: Before calculating the Phi correlation coefficient, preprocess and normalize the data to remove outliers and improve the accuracy of the chi-square test.

[0053] Use parallel computing: In the case of large amounts of data, parallel computing can be used to accelerate the calculation process of the chi-square test and the Phi correlation coefficient, improving efficiency.

[0054] Apply cross-validation: To verify the reliability of the Phi correlation coefficient, the robustness of the model can be checked through cross-validation.

[0055] Therefore, in step 2, the Phi correlation coefficient between each checkpoint and the final assessment result of the trainee is quantified by using the chi-square test. This process can effectively evaluate the importance of each checkpoint in flight training and provide a scientific basis for trainee scoring. Through the chi-square test and Phi correlation coefficient analysis, we can identify the checkpoints that have the greatest impact on the final assessment result of the trainee, thereby further optimizing the training evaluation system.

[0056] S3. Calculate the Phi correlation coefficient between each checkpoint and the final classification of the trainee according to the chi-square value; normalize the Phi correlation coefficient within each dimension to obtain the normalized Phi coefficient; In step 2, the Phi correlation coefficients of each checkpoint are obtained through the chi-square test. To eliminate the influence of the difference in the number of checkpoints in each dimension and avoid scoring imbalance, this step will normalize the Phi coefficients within each dimension to ensure that the sum of the Phi coefficients in each dimension is 1. After normalization, the relative importance of the checkpoints within each dimension can be standardized, providing a more fair basis for the subsequent weighted scoring model.

[0057] To avoid scoring imbalance caused by uneven numbers of checkpoints within each dimension, a normalization method is used to standardize the Phi values within each dimension so that their sum is 1. The normalization formula is as follows: ; where represents the normalized Phi coefficient of the th checkpoint within its respective dimension, represents the original Phi correlation coefficient of the th checkpoint, represents the sum of the Phi correlation coefficients of all checkpoints within the same dimension, is the index of the checkpoint within this dimension, is the number of checkpoints within the dimension.

[0058] The normalized Phi coefficient Indicates the importance of the checkpoint within its respective dimension, and the sum of the normalized Phi coefficients of all checkpoints within this dimension is 1. In this way, the normalized Phi coefficient of any checkpoint can intuitively reflect its relative contribution within this dimension.

[0059] Implementation process of Step 3: Determine the Phi coefficients of the checkpoints under each dimension; Based on the calculation results of Step 2, list the Phi values of all checkpoints within each dimension Calculate the total sum of the Phi coefficients under each dimension; Sum up the Phi values within each dimension to obtain the total sum of the Phi coefficients for that dimension: .

[0060] Normalization process: Normalize the Phi coefficients of each checkpoint, and use Formula 3 to convert the Phi coefficient of each checkpoint into a normalized Phi coefficient.

[0061] Structured display: Organize the normalized Phi coefficients into a table to show the relative importance of each checkpoint in each dimension.

[0062] Example: Normalization of the Phi coefficients of 13-hour flight training data Table 7: Normalized Phi coefficients of 13-hour flight training As can be seen from Table 7, compared with the traditional equal-weight scoring method, the scoring method adjusted by the Phi coefficient can significantly improve the discrimination of the true abilities of trainees. Taking checkpoint 30.3.3 as an example, its normalized Phi value is 0.73, and the final score is increased from the original score of 6.7 to 26.8, showing a strong discrimination effect. Similarly, the score of checkpoint 32.2.1 is increased to 12.8 after normalization, further proving the effective discrimination of this checkpoint for Class A and Class B trainees. Some checkpoints with relatively low original scores but high Phi coefficients (such as checkpoint 21.3) have a significant increase in scores after weight adjustment, reflecting the emphasis of the improved scoring method on key skill points. On the contrary, for checkpoints with low category discrimination (such as checkpoint 5.1), even though the normalized Phi coefficient is high, the score increase is still small, avoiding overemphasis on ineffective discrimination points. Overall, the improved scoring method effectively highlights key ability indicators, improves the discrimination of the true abilities of trainees, and provides a more scientific and reasonable scoring standard for the selection and training of flight trainees.

[0063] To ensure the calculation efficiency and accuracy of this step, the following optimization measures can be considered: Parallel computing: Perform parallel processing on the normalization calculation of the Phi values for each dimension to improve the calculation efficiency, especially when dealing with a large amount of data.

[0064] Visualization tool: To better display the normalized Phi coefficient, a corresponding visualization tool can be developed to graphically display the results, facilitating analysts to quickly identify important checkpoints.

[0065] Weighted normalization: For cases where certain dimensions have higher importance, a weighting factor can be introduced so that the normalized Phi coefficient of a specific dimension has a higher weight in the overall score.

[0066] Step 3 ensures a fair comparison of the importance of checkpoints within each dimension by normalizing the Phi coefficients within each dimension and avoids the influence of the difference in the number of checkpoints in different dimensions on the scoring results. After normalization, the obtained Phi coefficient can be used as the input for the subsequent scoring model, providing a scientific basis for further training and evaluation.

[0067] S4. Calculate the class discrimination degree of each checkpoint, defined as the difference in the passing rates of Class A students and Class B students at this checkpoint; In the evaluation method of the present invention, to enhance the discrimination ability of the evaluation model for different categories of students, an index of class discrimination degree (CD) is introduced. The class discrimination degree aims to reflect the effectiveness of this checkpoint in distinguishing different categories of students by measuring the difference in the passing rates of Class A students and Class B students at the same checkpoint. The larger the class discrimination degree, the stronger the ability of this checkpoint to distinguish students. Therefore, calculating the class discrimination degree is a key step in the evaluation model of the present invention, ensuring that the model can more sensitively identify the ability differences of different students.

[0068] As an option, the class discrimination degree is measured by comparing the difference in the passing rates of Class A and Class B students at each checkpoint. By calculating the class discrimination degree of each checkpoint, the discrimination effect of different evaluation dimensions on student categories can be evaluated, providing a basis for the subsequent weighted scoring model.

[0069] Class discrimination degree calculation formula: In this embodiment, the class discrimination degree (CD) is calculated using the following formula: ; Where: represents the passing rate of Class A students at the th checkpoint; represents the passing rate of Class B students at the th checkpoint; represents the class discrimination degree of the th checkpoint.

[0070] Specifically, the category discrimination degree is quantified by comparing the passing rates of Class A students and Class B students at the same checkpoint. The absolute value in the formula represents the difference in passing rates of the two types of students at this checkpoint. The larger this value is, the stronger the ability of this checkpoint to distinguish between Class A students and Class B students.

[0071] Calculation steps of the category discrimination degree: In a possible implementation, the calculation process of step 4 includes the following key links: First of all, it is necessary to obtain the passing rate data of Class A and Class B students at each checkpoint. This data usually comes from the actual assessment performance of students at each checkpoint. Specifically, calculate the passing rate of Class A students at a certain checkpoint and the passing rate of Class B students , which can be achieved in the following ways: It should be noted that the definitions of Class A and Class B students are determined according to specific assessment criteria, and may be divided according to the students' ability levels, training times or other criteria.

[0072] Next, calculate the category discrimination degree , that is, the absolute value of the difference in passing rates shown in the formula. For each checkpoint, by comparing the difference in passing rates between Class A and Class B students, the category discrimination degree of each checkpoint is obtained.

[0073] In this embodiment, the calculation result of the category discrimination degree is used to evaluate the importance of each checkpoint in the discrimination of students' abilities. Specifically, the higher the category discrimination degree of a checkpoint is, the stronger the ability to distinguish between Class A and Class B students, reflecting the contribution degree of this checkpoint in the evaluation model.

[0074] In some embodiments, each evaluation dimension can be weighted according to the level of the category discrimination degree. For example, a checkpoint with a higher category discrimination degree can be given a higher weight so that more attention is paid to these checkpoints with stronger discrimination degrees during the comprehensive scoring.

[0075] In order to further improve the discrimination degree of the evaluation model, in an extended embodiment, if the category discrimination degree of some checkpoints is low, that is smaller, it means that the role of this checkpoint in distinguishing between Class A and Class B students is not obvious. At this time, the following optimization methods can be used: Adjust the checkpoint settings: For checkpoints with low category discrimination degrees, consider adjusting their evaluation criteria, or increasing new dimensions to improve the discrimination degree of this checkpoint.

[0076] Add new category - distinguishing checkpoints: Some new checkpoints can be added, which can provide stronger differentiating information when distinguishing between Class A and Class B students.

[0077] It should be understood that by optimizing the checkpoints with relatively low category - distinguishing ability, the sensitivity of the entire evaluation model in distinguishing different student categories can be improved, thereby enhancing the accuracy and fairness of student scoring.

[0078] For example, in a flight training evaluation, if a checkpoint (such as a simulated flight operation) shows no obvious ability to distinguish between beginners (Class A) and students with some experience (Class B), the category - distinguishing ability of this checkpoint may be relatively low. In this case, it can be considered to enhance the category - distinguishing ability of this checkpoint by adding more discriminative items such as flight operation skills and emergency responses.

[0079] In another possible implementation, if the category - distinguishing ability in a certain training dimension (such as flight plan and management ability) is relatively low, more refined evaluation criteria can be introduced, such as evaluating students' decision - making performance in specific situations, to improve the category - distinguishing ability of this dimension.

[0080] Through the calculation of category - distinguishing ability, the effectiveness of each checkpoint in distinguishing student categories can be clarified, thereby optimizing the sensitivity and accuracy of the evaluation model. The introduction of category - distinguishing ability not only enhances the discrimination of the evaluation model but also provides a scientific basis for subsequent scoring weighting and evaluation dimension optimization. S5. Multiply the normalized Phi coefficient by the corresponding category - distinguishing ability to obtain the dynamic weight of each checkpoint; In the evaluation method of the present invention, step 5 aims to further optimize the role of checkpoints in the scoring model, specifically by calculating the final weight of the checkpoints. The final weight is the result of comprehensively considering the normalized Phi coefficient and category - distinguishing ability (CD) of each checkpoint, aiming to ensure that each checkpoint can fairly and effectively reflect the ability level of students during the evaluation process.

[0081] In the foregoing steps, we have calculated the normalized Phi coefficient and category - distinguishing ability of each checkpoint. The former measures the relative importance of the checkpoint within the evaluation dimension, and the latter measures the effectiveness of the checkpoint in distinguishing different categories of students. The core of step 5 is to combine these two factors and calculate the final weight of each checkpoint through a weighted method. In this way, the present invention can ensure that the scoring model more accurately reflects the ability performance of students and has strong discrimination.

[0082] In this embodiment, the final weight of the checkpoint is determined by two main factors: Normalized Phi Coefficient , which reflects the weight of this checkpoint within the evaluation dimension; Category Discrimination Degree , which measures the effectiveness of this checkpoint in differentiating between different categories of trainees.

[0083] As an option, the calculation formula for the final weight is as follows: Where: is the normalized Phi coefficient of the th checkpoint; is the category discrimination degree of the th checkpoint; and are the weight coefficients that adjust the relative importance of the normalized Phi coefficient and the category discrimination degree.

[0084] Specifically, the weight coefficients and can be adjusted according to the requirements of actual applications. For example, when the ability discrimination degree of the evaluation dimension has a higher impact on the trainee's performance, the weight of can be appropriately increased to emphasize the importance of the category discrimination degree. On the contrary, if the normalized Phi coefficient has a greater impact on the evaluation result, the weight of can be increased.

[0085] In this embodiment, the process of calculating the final weight of each checkpoint includes the following steps: First, calculate the normalized Phi coefficient of each checkpoint . This process has been completed in step 3. The normalized Phi coefficient reflects the relative importance of this checkpoint within the dimension, and the calculation method is to normalize the original Phi coefficient so that the sum of the Phi coefficients of all checkpoints under each dimension is 1.

[0086] Then, calculate the category discrimination degree of each checkpoint . This process has been completed in step 4. The category discrimination degree reflects the discrimination effect of this checkpoint between Class A and Class B trainees, and the calculation formula is: Where, and are the passing rates of Class A and Class B trainees at the th checkpoint, respectively.

[0087] Next, use the formula to combine the normalized Phi coefficient and the category discrimination degree to calculate the final weight of each checkpoint. Through the adjustment of the weight coefficients and , the final weight It can be flexibly changed according to the actual application requirements.

[0088] In a possible implementation, the weight coefficient can be adjusted according to the actual situation and , so as to better meet the specific evaluation objectives. For example, in some evaluation scenarios (such as the comprehensive ability evaluation of trainees), more attention may be paid to the category discrimination, while in other scenarios (such as the evaluation of specific skills), the normalized Phi coefficient may be more important.

[0089] Specifically, in some embodiments, when the category discrimination is low, the weight of the normalized Phi coefficient can be considered to be increased to reduce the influence of the category discrimination. On the contrary, when the category discrimination is high, the contribution of the weight coefficient to the final weight can be enhanced, so as to better reflect the importance of the checkpoint in differentiating the trainee categories.

[0090] Suppose in a 9-hour flight training evaluation, some checkpoints (such as flight plans) have a high normalized Phi coefficient but low category discrimination. At this time, the weight of the normalized Phi coefficient can be appropriately increased , while reducing the weight of the category discrimination , so as to emphasize the evaluation role of this checkpoint more in the final scoring.

[0091] For example, in the evaluation of flight operation skills, some skills have a strong differentiating effect between class A trainees (beginners) and class B trainees (experienced), and the category discrimination is high. Therefore, the weight of the category discrimination can be increased to highlight the contribution of this checkpoint in differentiating the trainee capabilities.

[0092] By calculating the final weight of the checkpoint, the solution of the present invention can effectively combine the normalized Phi coefficient with the category discrimination, improving the discrimination and accuracy of the scoring model. The calculation of the final weight can not only be flexibly adjusted according to the actual evaluation requirements, but also provide a more scientific basis for the subsequent comprehensive scoring of trainees. This weighting method provides great flexibility for the optimization and adjustment of the evaluation model, can meet the needs of different trainee groups and different evaluation objectives, thereby improving the fairness and accuracy of the evaluation.

[0093] S6. Calculate the scores of each checkpoint according to the basic scores, dynamic weights and trainee performances of each checkpoint; accumulate the scores of all unpassed checkpoints and deduct them from 100 to obtain the final total score of the trainee; In the foregoing steps, a quantitative basis has been provided for the evaluation contributions of each checkpoint by calculating the normalized Phi coefficient, category discrimination, and final weight. To further optimize the trainee evaluation process and ensure the fairness and reasonableness of the scores of each checkpoint, this step aims to calculate the final score of each checkpoint.

[0094] The core idea of this step is to calculate the score of each checkpoint by combining the normalized Phi coefficient, category discrimination, and basic score of each checkpoint. This process ensures that each checkpoint can fully reflect its role in ability discrimination when evaluating the trainee's ability, and the final score is made more representative through weight adjustment.

[0095] Formula for calculating the checkpoint score: In this embodiment, the score calculation of the checkpoint is achieved through the following two steps: Calculate the weight of the checkpoint The weight of each checkpoint is based on its normalized Phi coefficient and category discrimination The specific calculation formula is as follows: Where: represents the dynamic weight of the th checkpoint, represents the normalized Phi coefficient of the th checkpoint, represents the category discrimination of the th checkpoint; is the category discrimination of the th checkpoint.

[0096] This calculation method combines the importance of the checkpoint in the evaluation dimension (reflected by the Phi coefficient) and its contribution to the trainee ability discrimination (reflected by the category discrimination) to obtain the weight of each checkpoint.

[0097] Calculate the final score of the checkpoint After calculating the weight of each checkpoint , it is necessary to combine the basic score of the checkpoint to calculate the final score . The calculation formula is as follows: ; Where: represents the score or potential deduction value of the th checkpoint, is the weight of the th checkpoint; is the basic score of this checkpoint; 100 is for normalizing the score to a 100-point scale.

[0098] It should be noted that the basic score is determined according to the preliminary evaluation results of each checkpoint and is usually calculated dynamically based on the performance of the trainee at this checkpoint.

[0099] The implementation method of the calculation process In this embodiment, the process of calculating the score of each checkpoint includes the following steps: First, according to the normalized Phi coefficient obtained in step 5 and the category discrimination , use the formula to calculate the weight of each checkpoint .

[0100] Next, combine the basic score of each checkpoint , and calculate the final score of the checkpoint according to the formula . The basic score is usually based on the operation or test results of the trainee at this checkpoint. In some embodiments, the basic score can also be dynamically adjusted according to factors such as the training time and actual performance of the trainee.

[0101] The final score not only takes into account the weight of each checkpoint but also incorporates the actual ability performance of the trainee (reflected by the basic score) into the evaluation, making the evaluation result more accurate and reasonable.

[0102] The basic score is a quantification of the actual performance of the trainee at each checkpoint and is usually determined in the following ways: Static score: For some standardized checkpoints (such as basic flight operations), a fixed basic score can be preset for them.

[0103] Dynamic score: According to the real-time performance of the trainee during training, the basic score can be calculated dynamically. For example, in flight training, the basic score can be dynamically adjusted according to indicators such as the time and accuracy of the trainee completing the task.

[0104] Suppose in the flight training evaluation, the score calculations for checkpoint 1 (such as takeoff operation) and checkpoint 2 (such as landing operation) are as follows: The normalized Phi coefficient of checkpoint 1 is 0.8, the category discrimination is 0.6, and the basic score is 80; The normalized Phi coefficient of checkpoint 2 is 0.5, the category discrimination is 0.9, and the basic score is 90.

[0105] First, use the formula to calculate the weight of the checkpoint: ; ; Then, calculate the final score according to the formula: In this case, although the basic score of checkpoint 2 is higher, due to its lower normalized Phi coefficient, the final score is slightly lower than that of checkpoint 1. This indicates that checkpoint 1 plays a more prominent role in the assessment of trainees' abilities.

[0106] To further improve the accuracy and fairness of the evaluation model, the weight coefficient or the calculation method of the basic score can be adjusted according to the specific application scenario. For example: Adjust the basic score: When some checkpoints have less effect in distinguishing trainees with different abilities, the weight of the basic score can be adjusted to optimize the scoring.

[0107] Dynamically adjust the weights: Among different groups of trainees, the weights of the normalized Phi coefficient and the category discrimination degree can be dynamically adjusted according to the overall performance of the trainees, so as to improve the adaptability of the model.

[0108] By calculating the scores of the checkpoints, the present invention provides a method that comprehensively considers the importance, discrimination degree, and actual performance of each checkpoint in the assessment of trainees' abilities. The final score can not only accurately reflect the actual evaluation effect of each checkpoint, but also be adjusted according to the specific performance of the trainees to ensure the accuracy and fairness of the evaluation results. In addition, the score calculation method of the present invention has strong adaptability and can be optimized and adjusted according to the requirements of different evaluation scenarios to further improve the effect of the evaluation model.

[0109] S7. Judge the comprehensive ability level of the trainees based on the final score to achieve multi-dimensional classification evaluation of flight trainees.

[0110] After completing the quantization process of the scores of each checkpoint, to achieve a systematic evaluation of the comprehensive training performance of the trainees, it is necessary to further construct a total score calculation method based on the failure situation of the checkpoints and conduct classification determination based on this score. This step is closely connected with the aforementioned step 6, and its core goal is to form an evaluation mechanism for dividing the ability level of trainees by constructing a total score function model and combining the failure information and score weights of each checkpoint.

[0111] It should be noted that this mechanism depends on the checkpoint scores calculated in step 6 , and introduce a checkpoint failure identification variable To establish the "failed - deduction" mapping relationship, so as to realize the systematic total score rollback based on individual operation errors. This method can accurately reflect the training deficiencies of trainees at key points, and form hierarchical classification results based on this, which are used to support subsequent training suggestions and stratified training strategies.

[0112] In this embodiment, the total score of the trainee, Total Score, is calculated according to the following functional relationship: ; Wherein, represents the final total score of the trainee, represents the initial full score or basic score of the trainee, is the failure identification variable of checkpoint , and its value is: ; is the score of checkpoint . The specific calculation method is shown in Step 6, and its value is obtained by combining the product of the normalized Phi coefficient and the category discrimination degree with the basic score, that is: ; The above formula comprehensively reflects the weight of the checkpoint in the ability evaluation system and retains the differences in training performance. The deduction model only deducts scores for the unpassed items through the product relationship with , constituting a structured scoring method of "weighted deduction based on failed items".

[0113] In this embodiment, the trainees are classified according to the calculation result of Total Score.

[0114] In the standard classification mode, a fixed threshold is introduced to divide the score range into different ability level ranges. The exemplary classification rules are as follows: When , the trainee is classified as Class A, indicating a high training achievement; When , the trainee is classified as Class B, indicating the existence of ability short - boards or insufficient performance.

[0115] Specifically, the present invention is not limited to the two - level classification mode, and can also be extended to a multi - level classification scheme according to application requirements in the actual evaluation system. For example, as an option, more refined interval divisions can be set as follows: : Class S (excellent performance); : Class A (meeting the standard); : Class B (to be strengthened); : Class C (unqualified).

[0116] The specific values of the classification boundary can be adjusted according to the score distribution curve of a large sample of trainees collected during the system debugging phase, or can be set based on the normal distribution mean or percentile method to adapt to the complexity and tolerance requirements of different training tasks.

[0117] In a possible implementation, the classification mechanism and the training feedback module are linked for processing.

[0118] For example, when the system identifies that a trainee's Total Score is 78 and their main failure points are concentrated on low-weight inspection items, it can be marked as "marginally qualified" through the "boundary fault tolerance mechanism" and allowed to enter the re-evaluation link. As an option, this re-evaluation can be manually confirmed by the training administrator in combination with the specific content of the failure items and the classification suggestions can be adjusted.

[0119] It should be noted that this mechanism is essentially based on the strategy of "failure item superposition and rebate", so it has good scalability and fault tolerance adaptability. As an extended solution, additional evaluation dimensions (such as task completion time, reaction delay, etc.) can also be introduced and added to the basic score calculation item in Formula 7 through a linear fusion coefficient, making the classification result more complete in terms of dimensions.

[0120] Exemplarily, the following is an application case of a specific calculation process: The number of checkpoints for which a certain trainee participated in the evaluation is , among which the unpassed checkpoints are the 3rd, 5th, and 9th items, corresponding to scores: S3 = 4.2, S5 = 3.8, S9 = 2.7 = 2.7.

[0121] Substitute into the formula for calculation: ; Then according to the standard classification rules, this trainee can be classified as Class A, indicating that the training requirements are basically met.

[0122] It can be understood that the integrated scoring-classification mechanism of the present invention takes into account both scoring transparency and evaluation accuracy, and while based on the failure statistics of key checkpoints, effectively integrates the ability characterization of multiple dimensions of weights.

[0123] The above method is particularly applicable to comprehensive ability evaluation scenarios such as flight training, industrial skills assessment, medical simulation, etc. that require combining process performance and error analysis, and is also applicable to the basic logic interface for constructing subsequent training plan scheduling and feedback push modules.

[0124] Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-dimensional evaluation method for civil aviation flight cadets, characterized in that Including the following steps: S1. Collect the original evaluation data of each checkpoint during the flight training process, and the evaluation result of the checkpoint is a binary value of pass or fail; S2. Conduct a chi-square test on the relationship between the data of each checkpoint and the final evaluation category (category A or category B) of the trainee, and calculate the chi-square value of each checkpoint; S3. Calculate the Phi correlation coefficient between each checkpoint and the final classification of the trainee according to the chi-square value; S4. Normalize the Phi correlation coefficient within each dimension to obtain the normalized Phi coefficient; S5. Calculate the category discrimination degree of each checkpoint, which is defined as the difference in the passing rates of category A trainees and category B trainees at this checkpoint; S6. Multiply the normalized Phi coefficient by the corresponding category discrimination degree to obtain the dynamic weight of each checkpoint; S7. Calculate the score of each checkpoint according to the basic score, dynamic weight and trainee performance of each checkpoint; S8. Accumulate the scores of all unpassed checkpoints and subtract them from 100 to obtain the final total score of the trainee; S9. Judge the comprehensive ability level of the trainee based on the final score, and realize the multi-dimensional classification evaluation of flight trainees.

2. The multi-dimensional evaluation method for civil aviation flight cadets according to claim 1, wherein The calculation formula of the Phi correlation coefficient is: wherein, is the Phi correlation coefficient, is the chi-square value, is the total number of samples.

3. The multi-dimensional evaluation method for civil aviation flight cadets according to claim 1, wherein, The calculation method of the normalized Phi correlation coefficient is: ； Among them, represents the normalized Phi coefficient of the th checkpoint within its respective dimension, represents the original Phi correlation coefficient of the th checkpoint, represents the summation of the Phi correlation coefficients of all checkpoints within the same dimension, is the index of the checkpoint within this dimension, is the number of checkpoints within the dimension.

4. The multi-dimensional evaluation method for civil aviation flight cadets according to claim 1, wherein The calculation formula of the category discrimination degree CD is: ； Among them, and are the passing rates of Class A and Class B students at a certain checkpoint respectively.

5. A multi-dimensional evaluation method for civil aviation flight cadets according to claim 3, characterized in that Dynamic weights of each checkpoint Calculated by the following formula: Among them, represents the dynamic weight of the th checkpoint, represents the normalized Phi coefficient of the th checkpoint, represents the class discrimination of the th checkpoint.

6. The multi-dimensional evaluation method for civil aviation flight cadets according to claim 5, wherein Checkpoint score The calculation formula is as follows: ； Among them, represents the score or potential deduction value of the th checkpoint, represents the dynamic weight of the th checkpoint, is the basic score of the dimension to which it belongs.

7. The multi-dimensional evaluation method for civil aviation flight cadets according to claim 1, characterized in that The calculation formula of the final score of the trainee is: ； Among them, represents the final total score of the trainee, represents the initial full score or basic score of the trainee, is the failure identification variable of the checkpoint , is the score of the checkpoint .

8. A multi-dimensional evaluation method for civil aviation flight cadets according to claim 1, characterized in that The dimensions include the core ability dimensions of flight trajectory management, communication, procedure execution, problem solving, and situational awareness.

9. A multi-dimensional evaluation method for civil aviation flight cadets according to claim 1, characterized in that The evaluation method is applicable to any flight training stage, and the data used in this evaluation method includes the data samples of the 9-hour initial training and the 13-hour intensive training stage.

Images