Fine-grained ksa classification and behavior frequency analysis training comment evaluation method, system and device

Abstract

The application discloses a fine-grained KSA classification and behavior frequency analysis training comment evaluation method, system and device, and relates to the technical field of artificial intelligence. By acquiring a training comment text, preprocessing is performed on the training comment text to generate a structured text feature vector; fine-grained KSA extraction is performed based on the structured text feature vector; the obtained fine-grained KSA extraction data is input into a KSA-observable behavior knowledge graph, and based on semantic similarity, a matching associated standard observable behavior is calculated to generate an observable behavior set; different types of fine-grained KSA extraction data have different mapping weights for the same observable behavior; the display frequency of each observable behavior in the observable behavior set and the display number of different observable behaviors under each competence dimension are counted; based on the display frequency and the display number, a competence rating is calculated by comparing with a preset threshold interval; the evaluation process is transparent, the ability short board is accurately positioned, and the closed-loop linkage of evaluation and training is realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and specifically discloses a training evaluation method, system and device for fine-grained KSA classification and behavior frequency analysis. Background Technology

[0002] Competency-Based Training and Assessment (CBTA) is a modern training concept promoted by the International Civil Aviation Organization (ICAO). Its core lies in shifting from the traditional "time / task-based" training model to an "outcome / ability-based" model. Within the CBTA framework, trainee assessment follows a core logical chain: competence is defined and represented by a series of observable behaviors (OBs), and behind each OB lies a comprehensive reflection of corresponding knowledge, skills, and attitudes (collectively known as KSAs). Traditional CBTA assessment relies on assessors observing OBs and subjectively judging KSA compliance to determine competence, resulting in strong subjectivity and a lack of standardized procedures.

[0003] In the existing technology, the method represented by patent CN115587693B adopts a standardized work order mode to construct a static correlation matrix of "observation item-OB". The evaluator selects scores based on the preset observation items and obtains the rating through mathematical vector operations.

[0004] While this method achieves quantitative assessment within the CBTA framework, it has significant drawbacks: First, it heavily relies on on-site selection by assessors, and the results are influenced by immediate judgment and the completeness of records, resulting in strong subjectivity; Second, it simplifies the complex cognitive process into mechanical selection, with assessors only completing the "correspondence" operation and unable to conduct in-depth analysis of "observation-thinking-recording," leading to coarse granularity and insufficient depth in the assessment.

[0005] Liang et al.'s 2025 study used deep learning models such as RoBERTa+GCN to directly classify comment texts into OB labels, achieving high F1 scores on specific datasets. While efficient, this technique has key shortcomings: it is a "black box" text classification method, directly establishing a "comment text-OB label" mapping, completely skipping the "fact → KSA → OB" reasoning process, failing to explain the correspondence between comments and OBs, and lacking interpretability; at the same time, it cannot distinguish the knowledge, skills, and attitude deficiencies in KSA, making it difficult to support precise training guidance.

[0006] Systems like Project ORCA capture behavior through AI audio and video analysis, construct an "observation-recording-classification-evaluation" architecture, directly map OB (observation skills) to competencies, and automatically calculate results according to the CBTA (Competency and Assessment) scoring criteria.

[0007] Although the system is automated, it has significant drawbacks: First, the assessment logic is "behavior → OB → competence", ignoring the bridging role of KSA and failing to pinpoint deep-seated capability shortcomings; second, it is highly dependent on audio and video acquisition equipment, resulting in high system complexity and expensive deployment costs. It has not yet been successfully applied and validated in the domestic civil aviation sector, making its widespread adoption difficult.

[0008] Based on the above analysis, the existing technology has the following problems:

[0009] First, the assessment lacks interpretability; the competency ratings are not based on factual assessments, and the reasons for the ratings cannot be traced. Second, the assessment granularity is too coarse, skipping the KSA analysis step, and it cannot distinguish whether the trainees' shortcomings belong to the knowledge, skills or attitude dimensions. Third, the KSA analysis has a single dimension, only staying at the level of the knowledge, skills and attitudes triad, without distinguishing between knowledge types and skill types, and the assessment results cannot directly guide the design of training courses. Fourth, the evaluation lacks consistency; the comments from different evaluators vary greatly in expression, detail, and subjective bias, and there is a lack of standardized calibration mechanism. Fifth, it places high demands on evaluators, and the lack of supporting tools for the "facts → KSA → OB" thought process transformation leads to inconsistent quality of comments and low evaluation efficiency. Summary of the Invention

[0010] To overcome the shortcomings of existing technologies, this invention proposes a training evaluation method, system, and device for fine-grained KSA classification and behavior frequency analysis. This solves problems such as opaque evaluation logic, inaccurate weakness identification, inconsistent evaluation standards, and inability to guide subsequent training. It ensures that each evaluation rating has corresponding KSA factual basis, making the evaluation process explainable.

[0011] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a training evaluation method for fine-grained KSA classification and behavior frequency analysis, the method comprising: Obtain the training comment text, perform preprocessing on the training comment text, and generate a structured text feature vector; The structured text feature vector is input into a pre-trained multi-task classification model to extract fine-grained KSA data; the multi-task classification model includes: knowledge extraction branch, skill extraction branch, and attitude extraction branch; the fine-grained KSA data includes knowledge classification results, skill classification results, and attitude tendency results; The obtained fine-grained KSA extracted data is input into the KSA-observable behavior knowledge graph. Based on the semantic similarity between the semantic vector of each KSA element and the pre-stored semantic vector of each standard observable behavior node in the KSA-observable behavior knowledge graph, as well as the mapping weight corresponding to the fine classification type of each KSA element, the mapping confidence is calculated, and standard observable behaviors that reach the preset threshold are selected to generate an observable behavior set. The frequency of each observable behavior in the observable behavior set and the number of observable behaviors under each competency dimension are statistically analyzed. Based on the frequency and number of observable behaviors, a competency rating is calculated by comparing the results with a preset threshold range.

[0012] Optionally, the step of obtaining training comment text and performing preprocessing on the training comment text to generate structured text feature vectors includes: receiving input raw training comment text data; The original training comment text data is segmented to generate word sequence data; Part-of-speech tagging is performed on the word sequence data to generate word sequence data with part-of-speech tags; Dependency parsing is performed on the word sequence data with part-of-speech tags to generate syntactic structure data; The word sequence data, the word sequence data with part-of-speech tags, and the syntactic structure data are vectorized and concatenated to generate a fixed-dimensional structured text feature vector. The word sequence data is mapped to a first vector through a pre-trained word embedding model, the part-of-speech tags are mapped to a second vector through one-hot encoding, and the syntactic structure data is mapped to a third vector through dependency relation encoding. The first vector, the second vector, and the third vector are concatenated or weighted and summed to obtain the structured text feature vector.

[0013] Optionally, the step of inputting the structured text feature vector into a pre-trained multi-task classification model to extract fine-grained KSA data includes: identifying knowledge-related text fragments from the structured text feature vector through the knowledge extraction branch of the multi-task classification model, and outputting knowledge classification results; the knowledge classification results include descriptive knowledge, procedural knowledge, strategic knowledge, and adaptive knowledge; The skill extraction branch of the multi-task classification model identifies skill-related text fragments from the structured text feature vector and outputs skill classification results; the skill classification results include motor skills, cognitive skills, and metacognitive skills. The attitude extraction branch of the multi-task classification model is used to identify attitude-related text fragments from the structured text feature vector and output attitude tendency results. By integrating the knowledge classification results, the skill classification results, and the attitude tendency results, fine-grained KSA extraction data is generated.

[0014] Optionally, the knowledge extraction branch is used to extract knowledge elements; the skill extraction branch is used to extract skill elements; and the attitude extraction branch is used to extract attitude elements and determine their tendency.

[0015] Optionally, the display frequency of each observable behavior in the statistical observable behavior set, and the display quantity of different observable behaviors under each competency dimension, include: traversing the observable behavior set, performing frequency accumulation statistics on each observable behavior, and generating observable behavior display frequency data; Obtain preset competency dimension segmentation data and establish an association index between competency dimensions and observable behaviors; based on the association index, classify each observable behavior in the observable behavior set into the corresponding competency dimension; under each competency dimension, count the number of different types of observable behaviors and generate display quantity data for each competency dimension. By integrating the display frequency data and the display quantity data, basic data for competency assessment is generated.

[0016] Optionally, after calculating the competency rating based on the display frequency and display quantity, and comparing it with a preset threshold range, the method further includes: Extract KSA weakness dimension data from the fine-grained KSA extraction data; the KSA weakness dimension data includes: knowledge elements, skill elements, or missing attitude elements that are classified as negative. Retrieve pre-stored KSA-course mapping library data; The KSA weakness dimension data is matched with the KSA-course mapping library data to generate candidate training course data; Based on the severity parameter of the KSA weakness dimension data, the candidate training course data is prioritized and sorted to generate a recommended list of training courses.

[0017] Optionally, after generating the recommended list of training courses, the method further includes: Retrieve the competency rating data, the KSA weakness dimension data, the observable behavior mapping link information data in the observable behavior set, and the training course recommendation list data; The competency rating data, the KSA weakness dimension data, the observable behavior mapping link information data, and the training course recommendation list data are structured and integrated to generate a comprehensive evaluation report; the comprehensive evaluation report is then output.

[0018] Secondly, this invention provides a training evaluation system for fine-grained KSA classification and behavior frequency analysis, the system comprising: The data layer is used to store the fact-based evaluation base, the fine-grained KSA knowledge base, and the competency-observable behavior standard base; A fine-grained KSA extraction layer is used to perform fine-grained KSA extraction based on structured text feature vectors; The observable behavior mapping layer is used to input the obtained fine-grained KSA extracted data into the KSA-observable behavior knowledge graph. Based on the semantic similarity between the semantic vector of each KSA element and the pre-stored semantic vector of each standard observable behavior node in the KSA-observable behavior knowledge graph, as well as the mapping weight corresponding to the fine classification type of each KSA element, the mapping confidence is calculated, and standard observable behaviors that reach the preset threshold are selected to generate an observable behavior set. The competency assessment layer is used to count the frequency of each observable behavior in the observable behavior set, as well as the number of times different observable behaviors are displayed under each competency dimension; based on the frequency and number of displays, and compared with a preset threshold range, the competency rating is calculated. The training course matching layer is used to generate a recommended list of training courses based on KSA weakness dimension data. The application layer is used to generate comprehensive evaluation reports.

[0019] Thirdly, the present invention provides an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of the first aspects.

[0020] Compared with the closest prior art, the present invention has the following beneficial effects: This invention provides a training evaluation method, system, and device for fine-grained KSA classification and behavior frequency analysis. It fully replicates the expert thought process chain from observing facts to analyzing fine-grained KSA, mapping it to observable behaviors, and finally assessing competence. Each rating has corresponding factual basis, solving the problem of lack of interpretability in existing black-box evaluation technologies. Through fine-grained extraction of knowledge and skill categories, it accurately identifies the types of knowledge and skills lacking in trainees, directly guiding personalized training.

[0021] This invention automatically matches targeted training courses based on fine-grained KSA analysis results, forming a complete closed loop of assessment to identify problems, course recommendation to solve problems, and re-evaluation to verify effectiveness, thereby improving the relevance and efficiency of training. It replaces subjective judgment with a standardized fine-grained KSA knowledge base and dynamic mapping rules, eliminating assessment bias caused by differences in instructors' writing styles and ensuring comparability of results. Furthermore, it requires no changes to existing training processes or additional hardware, operating solely based on text-based feedback, resulting in extremely low implementation costs. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0023] Figure 1 This is a flowchart of a training evaluation method for fine-grained KSA classification and behavior frequency analysis provided by the present invention; Figure 2 This is a schematic diagram of the fine-grained KSA extraction process provided by the present invention; Figure 3 This is a schematic diagram of the mapping relationship of the KSA-observable behavior knowledge graph provided by the present invention; Figure 4 This is a schematic diagram of the training comment evaluation system for fine-grained KSA classification and behavior frequency analysis provided by the present invention; Figure 5 This is an internal structure diagram of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore merely examples, and should not be construed as limiting the scope of protection of the present invention.

[0025] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by those skilled in the art to which this invention pertains.

[0026] This invention provides a training evaluation method and system based on fine-grained Knowledge, Skill, Attitude (KSA) classification and behavior frequency analysis. Specifically, it relates to a training evaluation method and system based on fine-grained Knowledge, Skill, Attitude (KSA) classification, Observable Behaviors (OB) mapping, and behavior frequency analysis. This method and system is applicable to aviation fields such as flight training and air traffic control training, and can also be extended to competency-oriented training evaluation scenarios in other industries. The embodiments of this invention are described below with reference to the accompanying drawings.

[0027] Example 1: As Figure 1 As shown, Embodiment 1 of the present invention provides a training evaluation method for fine-grained KSA classification and behavior frequency analysis. Taking the application of this method to a server as an example, the method includes the following steps: Step S101: Obtain the training comment text, perform preprocessing on the training comment text, and generate a structured text feature vector.

[0028] The training feedback text consists of unstructured textual comments generated by instructors during flight / air traffic control training, used to record trainees' operational performance, skill level, and behavioral details. The preprocessing process, a basic operation of natural language processing, converts the raw feedback text into structured features recognizable by the model. Specifically, it includes receiving the input raw training feedback text data; The original training comment text data is segmented to generate word sequence data; Part-of-speech tagging is performed on the word sequence data to generate word sequence data with part-of-speech tags; Dependency parsing is performed on the word sequence data with part-of-speech tags to generate syntactic structure data; The word sequence data, the word sequence data with part-of-speech tags, and the syntactic structure data are vectorized and concatenated to generate a fixed-dimensional structured text feature vector. The word sequence data is mapped to a first vector through a pre-trained word embedding model, the part-of-speech tags are mapped to a second vector through one-hot encoding, and the syntactic structure data is mapped to a third vector through dependency relation encoding. The first vector, the second vector, and the third vector are concatenated or weighted and summed to obtain the structured text feature vector.

[0029] The structured text feature vector contains semantic, syntactic, and contextual information of the comments, providing a basic input for subsequent fine-grained KSA extraction.

[0030] Step S102: Input the structured text feature vector into a pre-trained multi-task classification model to extract fine-grained KSA data; the multi-task classification model includes: knowledge extraction branch, skill extraction branch and attitude extraction branch; the fine-grained KSA data includes knowledge classification results, skill classification results and attitude tendency results.

[0031] Fine-grained KSA extraction is the core step of this embodiment, aiming to accurately analyze the three elements of knowledge, skills, and attitudes from the comments, and to further subdivide knowledge and skills. In one embodiment, such as Figure 2 As shown, the fine-grained KSA extraction process includes: inputting structured text feature vectors into a pre-trained multi-task classification model; identifying knowledge-related text fragments through the knowledge extraction branch of the multi-task classification model and outputting four categories of knowledge classification results: expressive, procedural, strategic, and adaptive; identifying skill-related text fragments through the skill extraction branch and outputting three categories of skill classification results: motor, cognitive, and metacognitive; identifying attitude-related text fragments through the attitude extraction branch and outputting positive, negative, or missing tendency results; and integrating the above results to generate fine-grained KSA extraction data.

[0032] KSA - Observable Behavior Mapping is used to establish the association between capability elements and standard behaviors.

[0033] In one embodiment, the multi-task classification model uses BERT-base as a shared encoding layer, followed by two task heads: an attitude classification head (3-class classification) and a quality rating head (regression task). The training dataset consists of 100,000 manually annotated training comments, including 35,000 positive attitude samples, 40,000 neutral attitude samples, and 25,000 negative attitude samples. Each sample is also labeled with a quality score of 0-10. Loss function: A multi-task joint loss function is used, and the specific formula is as follows:

[0034] in, For cross-entropy loss, For mean square error loss, =0.6, =0.4 is the task weight; Hyperparameter settings: Learning rate 2×10 -5 The batch size is 32, the number of iterations is 10, the AdamW optimizer is used, and the weight decay coefficient is 0.01. Complete training process: a. Load pre-trained BERT-base weights; b. Freeze the encoder parameters for the first 6 layers, and fine-tune the parameters for the last 6 layers and the task head. c. Divide the training set, validation set, and test set in an 8:1:1 ratio; d. Evaluate performance on the validation set after each training round and save the best model; e. Perform final performance testing on the test set.

[0035] Step S103: Input the obtained fine-grained KSA extraction data into the KSA-observable behavior knowledge graph. Based on the semantic similarity between the semantic vector of each KSA element and the pre-stored semantic vector of each standard observable behavior node in the KSA-observable behavior knowledge graph, as well as the mapping weight corresponding to the fine classification type of each KSA element, calculate the mapping confidence and filter the standard observable behaviors that reach the preset threshold to generate an observable behavior set.

[0036] In the above embodiments, the obtained fine-grained KSA extracted data is input into the KSA-observable behavior knowledge graph, and the standard observable behaviors that are matched and associated based on semantic similarity are calculated to generate a set of observable behaviors, specifically including: Extract the semantic vector of each KSA element from the fine-grained KSA extraction data; Input the semantic vectors of each KSA element into the KSA-observable behavior knowledge graph; based on semantic similarity calculation, retrieve standard observable behavior nodes whose correlation with the semantic vectors of each KSA element reaches a preset threshold; Based on the sub-classification type of each KSA element, retrieve the corresponding mapping weight parameters; Based on the mapping weight parameters, calculate the mapping confidence of each standard observable behavior node; Filter out standard observable behaviors whose mapping confidence reaches a preset confidence threshold, and generate the observable behavior set.

[0037] Specifically, from the fine-grained KSA extraction data, each knowledge, skill, or attitude classification result unit is extracted as a KSA element; each KSA element contains at least the following fields: type field (knowledge / skill / attitude), sub-category field (such as descriptive knowledge, motor skills, etc.), tendency field (positive / negative / missing), and original text fragment; each KSA element is converted into a fixed-dimensional semantic vector through a pre-trained semantic encoder (such as Sentence-BERT).

[0038] The semantic similarity calculation uses the cosine similarity formula: im ; in, This is a semantic vector extracted from fine-grained KSA elements (encoded via Sentence-BERT). This is a pre-stored semantic vector for each standard observable behavior node in the KSA-Observable Behavior Knowledge Graph. The preset similarity threshold is 0.75, and only mapping relationships with similarity ≥ 0.75 are retained.

[0039] In one embodiment, the mapping process includes: extracting semantic vectors from data extracted from fine-grained KSA; inputting the semantic vectors into the KSA-observable behavior knowledge graph; retrieving standard observable behavior nodes with semantic relevance reaching a preset threshold; retrieving mapping weights according to the fine-grained KSA classification; calculating the mapping confidence score and filtering out qualified nodes to generate an observable behavior set. Different KSA types correspond to different weights; for example, procedural knowledge has a higher mapping weight than metacognitive skills to ensure mapping accuracy.

[0040] In one embodiment, the method for constructing the KSA-observable behavior knowledge graph includes the following steps: (1) Defining the ontology model: Constructing a KSA ontology library, defining three types of nodes: knowledge, skills, and attitudes. Knowledge nodes include expressive knowledge, procedural knowledge, strategic knowledge, and adaptive knowledge. Skill nodes include motor skills, cognitive skills, and metacognitive skills. Attitude nodes include three types: positive, negative, and missing. Constructing an OB ontology library, defining standard observable behavior nodes, and establishing an association between each OB node and the ICAO standard competency dimension. (2) Establishing initial mapping relationships: Based on industry standards, expert experience, and historical evaluation data, establishing initial associations between each KSA node and OB node, and setting initial mapping weights according to the strength of the association. For example, the initial weight of procedural knowledge and the "incoherent program execution" OB is set to 0.7, and the initial weight of metacognitive skills and the "failure to identify deviations in a timely manner" OB is set to 0.6. (3) Labeling weak association relationships: For KSA and OB that are not directly related, such as procedural knowledge and the "flight path deviation" OB, setting a lower initial weight (e.g., 0.3) and labeling them as weak association relationships. (4) Iteratively update the map: Based on the feedback of actual evaluation data, the mapping weights are optimized regularly. For example, the weight values ​​are adjusted according to the matching accuracy of KSA and OB in historical evaluations to improve the accuracy of map mapping.

[0041] In the above embodiments, the mapping confidence is calculated as follows: The mapping confidence of the i-th KSA element and the j-th standard observable behavior node The calculation formula is:

[0042] in, The initial mapping weights between this KSA type and OB. KSA element semantic vector With OB node pre-stored semantic vector Semantic similarity between them.

[0043] Step S104: Calculate the display frequency of each observable behavior in the observable behavior set, and the display quantity of different observable behaviors under each competency dimension.

[0044] Frequency and quantity statistics provide objective evidence for competency quantification. In one embodiment, the statistical process includes: traversing the set of observable behaviors and accumulating the number of occurrences of each OB to obtain the display frequency; establishing an index linking competencies and OBs, and classifying OBs to their corresponding dimensions; under each competency dimension, counting the number of OB types under each dimension to obtain the display quantity; and integrating frequency and quantity to generate basic assessment data. Frequency reflects the severity of the problem, while quantity reflects the breadth of problem coverage.

[0045] Step S105: Based on the display frequency and display quantity, calculate the competency rating by comparing with the preset threshold range.

[0046] Competency rating uses a 1-5 level quantitative standard, determined by comparing frequency and quantity against threshold ranges. For example, reaching the high threshold for both frequency and quantity results in Level 5 (Excellent), reaching the medium threshold in Level 4 (Good), reaching the low threshold in Level 3 (Compliant), falling below the low threshold in Level 2 (Basic Compliant), and being severely deficient in Level 1 (Needs Improvement). The rating results directly reflect the trainee's ability level.

[0047] In one embodiment, the preset threshold interval is determined based on percentile statistics of historical evaluation data, and the specific steps are as follows: Collect at least 1,000 training comments that have been rated, and count the total frequency of observable behaviors and the number of different OB types under each competency dimension; Sort the frequency and quantity in ascending order, and take the 20th percentile as the low threshold (corresponding to level 2), the 50th percentile as the medium threshold (corresponding to level 3), and the 80th percentile as the high threshold (corresponding to level 4). Frequency or quantity below the low threshold is classified as Level 1, above the high threshold as Level 5, and the rest are mapped according to combination rules.

[0048] For example: Frequency ≥ High Threshold and Quantity ≥ High Threshold → Level 5 (Excellent); Frequency ≥ Medium Threshold and Quantity ≥ Medium Threshold → Level 4 (Good); Frequency ≥ Low Threshold and Quantity ≥ Low Threshold → Level 3 (Compliant); Frequency < Low Threshold or Quantity < Low Threshold → Level 2 (Basically Compliant); Frequency and Quantity are both significantly lower than the Low Threshold (e.g., less than 10%) → Level 1 (Needs Improvement).

[0049] In one embodiment, the calculation of competency rating is performed, followed by: Extract KSA weakness dimension data from the fine-grained KSA extraction data; the KSA weakness dimension data includes: knowledge elements, skill elements, or missing attitude elements that are classified as negative. Retrieve pre-stored KSA-course mapping library data; The KSA weakness dimension data is matched with the KSA-course mapping library data to generate candidate training course data; Based on the severity parameter of the KSA weakness dimension data, the candidate training course data is prioritized and sorted to generate a recommended list of training courses.

[0050] The severity parameter is calculated as follows: For knowledge or skill deficiencies, severity = frequency of the observable behavior corresponding to the deficiency within the observable behavior set × average confidence level of the deficiency mapped to the observable behavior; for attitude deficiencies, severity = preset base weight (e.g., 1.0) × (1 + percentage of times the attitude deficiency is mentioned in the comments). A higher severity parameter value indicates a more urgent deficiency, and the corresponding training course should have a higher priority.

[0051] In one embodiment, after generating the training course recommendation list, the method further includes: Integrate rating, weakness, mapping link, and course data to generate and output a comprehensive evaluation report. The report includes visualized ratings, weakness analysis, reasoning links, and course recommendations to facilitate training management decisions.

[0052] The method flow will be explained in detail below with specific examples: Example 1: Evaluation comments for a single flight training session Input training comment text: During VOR approach training, trainees were able to recite the checklist clauses fluently, but their execution order was chaotic during actual operation; they failed to correct course deviations in a timely manner after they occurred during flight; instructors observed that trainees' operating actions were stiff and their throttle corrections were excessive; they were unaware that they had deviated from the planned course throughout the entire process; and their explanations of emergency response logic were unclear when asked.

[0053] Execution process: 1. Preprocessing: Word segmentation, part-of-speech tagging, and syntactic analysis of comments are performed to generate structured feature vectors and extract core semantics: memorization checklist, disordered operation sequence, uncorrected course deviation, stiff actions, excessive correction, undetected deviation, and unclear emergency response strategy.

[0054] 2. Fine-grained KSA extraction: Knowledge: Expressive knowledge (positive), procedural knowledge (negative), strategic knowledge (negative); Skills: Motor skills (negative), Metacognitive skills (negative); Attitude: Neutral.

[0055] 3. KSA-OB mapping, such as Figure 3 As shown: Procedural knowledge (negative) → Inconsistent program execution (confidence level 0.7); Motor skills (negative) → Inaccurate flight control (confidence level 0.5); Metacognitive skills (negative) → Delay in bias recognition (confidence level 0.3); Generate a set containing 3 OBs.

[0056] 4. Statistics: The frequency of each OB is 1; there is 1 type of OB for each of the program execution, decision-making, manipulation, and recognition dimensions.

[0057] 5. Rating: Program execution level 2, decision-making level 2, manipulation level 3, identification level 2.

[0058] 6. Course Recommendations: For weaknesses in procedural knowledge, implement procedural simulation exercises; for weaknesses in motor skills, conduct manipulation feedback training; for weaknesses in metacognition, conduct situational awareness training.

[0059] 7. Output a comprehensive report: including rating charts, detailed list of weaknesses, reasoning process, and course priority list.

[0060] Example 2: Cumulative evaluation across multiple training sessions Input: Comments on a trainee's 5 flight training sessions within 3 months.

[0061] Execution process: Perform preprocessing, KSA extraction, and OB mapping sequentially, and record the OB data for each step. Cumulative statistics: Total frequency of OBs, total number of OBs in each dimension; Calculate the overall competence rating; Analyze the changing trends of weaknesses and evaluate the effectiveness of previous training; Generate the training plan for the next stage.

[0062] Cumulative assessments can track skill development, verify training effectiveness, and support dynamic adjustments to training plans.

[0063] Example 3: Evaluating Consistency Calibration To eliminate differences in instructor comment styles, calibration is implemented: 1. Collect instructor historical comments and standard results, and extract features such as length, terminology density, and rating variance; 2. Generate a custom calibration factor; 3. Adjust the confidence level and weights in the KSA extraction and OB mapping stages; 4. Update factors regularly to ensure consistency in cross-faculty assessments.

[0064] The calibration mechanism effectively reduces subjective bias and improves the comparability of assessments.

[0065] Example 2: Based on the same technical concept, Example 2 of this invention also provides a training evaluation system for fine-grained KSA classification and behavior frequency analysis, such as... Figure 4 As shown, it includes: a data layer 210, a fine-grained KSA extraction layer 220, an observable behavior mapping layer 230, a competency assessment layer 240, a training course matching layer 250, and an application layer 260, wherein: I. Data Layer 210 The data layer is used to store a fact-based commentary base, a fine-grained KSA knowledge base, and a competency-observable behavior standards base.

[0066] Specifically, the fact-based feedback database stores historical training feedback text data, including original feedback written by instructors, corresponding trainee information, training time, training subject, and other metadata. The feedback text in the fact-based feedback database serves as the system's raw input data, supporting subsequent fine-grained KSA extraction processing.

[0067] The fine-grained KSA knowledge base stores semantic templates and keywords related to four categories of knowledge, three categories of skills, and attitudes. The four categories of knowledge include descriptive knowledge, procedural knowledge, strategic knowledge, and adaptive knowledge; the three categories of skills include motor skills, cognitive skills, and metacognitive skills; and attitudes include professional ethics, safety awareness, and teamwork tendencies. The fine-grained KSA knowledge base provides the classification criteria and semantic matching benchmarks for the fine-grained KSA extraction layer.

[0068] The Competency-Observable Behavior Standard Library stores predefined competency dimension classification data and the association index between each competency dimension and observable behaviors. Specifically, the library records which standard observable behaviors are included under each competency dimension, as well as the definition and evaluation criteria for each observable behavior. The Competency-Observable Behavior Standard Library provides the basis for OB classification and rating at the competency assessment level.

[0069] II. Fine-grained KSA extraction layer 220 The fine-grained KSA extraction layer is used to input structured text feature vectors into a pre-trained multi-task classification model to extract fine-grained KSA data; the multi-task classification model includes: a knowledge extraction branch, a skill extraction branch, and an attitude extraction branch; the fine-grained KSA data includes knowledge classification results, skill classification results, and attitude tendency results.

[0070] Specifically, the fine-grained KSA extraction layer receives structured text feature vectors from the preprocessing module and inputs these feature vectors into a pre-trained multi-task classification model. The multi-task classification model uses a BERT-based pre-trained model as a shared encoding layer, and its backend has three independent fully connected branches in parallel: a knowledge extraction branch, a skill extraction branch, and an attitude extraction branch.

[0071] The knowledge extraction branch identifies knowledge-related text fragments from structured text feature vectors and outputs knowledge classification results, including descriptive knowledge, procedural knowledge, strategic knowledge, and adaptive knowledge. The skill extraction branch identifies skill-related text fragments from structured text feature vectors and outputs skill classification results, including motor skills, cognitive skills, and metacognitive skills. The attitude extraction branch identifies attitude-related text fragments from structured text feature vectors and outputs attitude bias results, including positive attitudes, negative attitudes, or absent attitudes. Absent attitudes refer to comments that do not mention any relevant attitude or that do not explicitly describe an insufficient attitude.

[0072] The fine-grained KSA extraction layer integrates the knowledge classification results, skill classification results, and attitude bias results to generate fine-grained KSA extraction data. Each fine-grained KSA extraction data includes a KSA identifier, KSA type, confidence level, original text fragment, and bias.

[0073] III. Observable Behavior Mapping Layer 230 The observable behavior mapping layer is used to input the obtained fine-grained KSA extracted data into the KSA-observable behavior knowledge graph. Based on the semantic similarity between the semantic vector of each KSA element and the pre-stored semantic vector of each standard observable behavior node in the KSA-observable behavior knowledge graph, as well as the mapping weight corresponding to the fine classification type of each KSA element, the mapping confidence is calculated, and standard observable behaviors that reach the preset threshold are selected to generate an observable behavior set.

[0074] Specifically, the observable behavior mapping layer includes a KSA-observable behavior knowledge graph and a semantic similarity retrieval module. The KSA-observable behavior knowledge graph stores many-to-many mappings between fine-grained KSA elements and standard observable behaviors, where each KSA node and OB node are connected by weighted edges; different types of KSAs have different mapping weights for the same OB. The semantic similarity retrieval module calculates the similarity between the semantic vector of each KSA element in the fine-grained KSA extracted data and the semantic vector of the OB node in the KSA-observable behavior knowledge graph, retrieving standard observable behavior nodes whose relevance reaches a preset threshold.

[0075] The observable behavior mapping layer performs the following processing: First, it extracts the semantic vectors of each KSA element from the fine-grained KSA data; second, it inputs the semantic vectors of each KSA element into the KSA-observable behavior knowledge graph, and calculates the standard observable behavior nodes whose semantic similarity with each KSA element's semantic vector reaches a preset threshold based on semantic similarity; then, it retrieves the corresponding mapping weight parameters according to the fine classification type of each KSA element; next, it calculates the mapping confidence of each standard observable behavior node based on the mapping weight parameters; finally, it filters the standard observable behaviors whose mapping confidence reaches a preset confidence threshold, generating an observable behavior set.

[0076] IV. Competency Assessment Level 240 The competency assessment layer is used to count the frequency of each observable behavior in the observable behavior set, as well as the number of different observable behaviors under each competency dimension; based on the frequency and number of displays, and by comparing with a preset threshold range, a competency rating is calculated.

[0077] Specifically, the competency assessment layer includes an observable behavior statistics module and a competency calculation engine.

[0078] The observable behavior statistics module iterates through the set of observable behaviors, accumulates the frequency of each behavior, and generates display frequency data for each observable behavior. Simultaneously, this module acquires preset competency dimension segmentation data, establishes a correlation index between competency dimensions and observable behaviors, and categorizes each observable behavior in the set to its corresponding competency dimension based on this index. Under each competency dimension, it counts the number of different types of observable behaviors, generating display quantity data for each competency dimension. The observable behavior statistics module integrates the display frequency data and the display quantity data to generate basic data for competency assessment.

[0079] The competency calculation engine is used to calculate the rating for each competency dimension based on the competency assessment data and by comparing it with preset threshold ranges. Specifically, for each competency dimension, the engine compares the total frequency and number of different OB types under that dimension with preset frequency and quantity threshold ranges, respectively, to comprehensively determine a competency rating of 1-5. Frequency reflects the severity of the problem, while quantity reflects the breadth of the problem.

[0080] V. Training Course Matching Layer 250 The training course matching layer is used to generate a recommended list of training courses based on KSA weakness dimension data.

[0081] Specifically, the training course matching layer includes a course recommendation engine and a KSA-course mapping library. The KSA-course mapping library pre-stores a correlation matrix between fine-grained KSA types and training courses. For example: insufficient descriptive knowledge corresponds to online theoretical learning and knowledge tests; insufficient procedural knowledge corresponds to simulator program practice and process drills; insufficient strategic knowledge corresponds to case teaching and scenario simulations; insufficient adaptive knowledge corresponds to abnormal procedure training and emergency drills; insufficient motor skills correspond to repetitive operation practice and closed-loop feedback training; insufficient cognitive skills correspond to cognitive task analysis and problem-oriented learning; insufficient metacognitive skills correspond to reflective practice and self-assessment training; and attitude issues correspond to safety culture education and teamwork training, etc.

[0082] The course recommendation engine extracts KSA (Knowledge, Skills, and Attitudes) weakness dimension data from fine-grained KSA extraction data. This KSA weakness dimension data includes knowledge elements, skill elements, or missing attitude elements that are classified negatively. The engine matches the KSA weakness dimension data with the KSA-course mapping library data to generate candidate training course data. Based on the severity parameter of the KSA weakness dimension data, the engine prioritizes the candidate training course data and generates a recommended list of training courses.

[0083] VI. Application Layer 260 The application layer is used to generate comprehensive evaluation reports.

[0084] Specifically, the application layer retrieves competency rating data from the competency assessment layer, KSA weakness dimension data from the fine-grained KSA extraction layer, observable behavior mapping link information data from the observable behavior mapping layer, and a training course recommendation list from the training course matching layer. The application layer performs structured integration processing on the above data to generate a comprehensive evaluation report containing competency ratings, fine-grained KSA weakness analysis, observable behavior mapping link information, and a training course recommendation list, and then outputs this comprehensive evaluation report.

[0085] The comprehensive assessment report can be presented in the form of visual charts, including competency radar charts, KSA weakness distribution charts, OB trigger frequency statistics charts, and course recommendation lists, making the assessment results intuitive and readable.

[0086] System workflow example: Taking the analysis of a single flight training evaluation as an example, the system's workflow is as follows: First, the system obtains training comment texts from the fact comment database in the data layer, and generates structured text feature vectors after preprocessing.

[0087] Secondly, the fine-grained KSA extraction layer performs fine-grained KSA extraction based on structured text feature vectors, and outputs fine-grained KSA extraction data containing knowledge, skills and attitude classification results.

[0088] Then, the observable behavior mapping layer inputs the fine-grained KSA extracted data into the KSA-observable behavior knowledge graph, and generates an observable behavior set through semantic similarity retrieval and weight mapping.

[0089] Next, the competency assessment layer counts the frequency of each OB in the observable behavior set and the number of OBs displayed under each competency dimension, and calculates the competency rating by comparing it with the preset threshold range.

[0090] Subsequently, the training course matching layer generates a recommended list of training courses based on the KSA weakness dimension data.

[0091] Finally, the application layer integrates all the above data to generate and output a comprehensive evaluation report.

[0092] The system described above enables an automated and interpretable assessment loop, from unstructured training comments to competency ratings and course recommendations.

[0093] The present invention will be further described in detail below with reference to specific embodiments, but the scope of protection of the present invention is not limited to the following embodiments.

[0094] In one embodiment, the present invention also provides an electronic device, which may be a terminal, and its internal structure diagram may be as follows. Figure 5 As shown. The electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the training evaluation method for fine-grained KSA classification and behavior frequency analysis as described in any one of steps S101 to S104. The display screen can be a liquid crystal display or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0095] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0096] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A training comment evaluation method for fine-grained KSA classification and behavior frequency analysis, characterized in that, The method includes: Obtain the training comment text, perform preprocessing on the training comment text, and generate a structured text feature vector; The structured text feature vectors are input into a pre-trained multi-task classification model to extract fine-grained KSA data. The multi-task classification model includes a knowledge extraction branch, a skill extraction branch, and an attitude extraction branch. The fine-grained KSA data includes knowledge classification results, skill classification results, and attitude bias results. The knowledge classification results include descriptive knowledge, procedural knowledge, strategic knowledge, and adaptive knowledge. The skill classification results include motor skills, cognitive skills, and metacognitive skills. The attitude bias results include positive attitudes, negative attitudes, and missing attitudes. The obtained fine-grained KSA extracted data is input into the KSA-observable behavior knowledge graph. Based on the semantic similarity between the semantic vector of each KSA element and the pre-stored semantic vector of each standard observable behavior node in the KSA-observable behavior knowledge graph, as well as the mapping weight corresponding to the fine classification type of each KSA element, the mapping confidence is calculated, and standard observable behaviors that reach the preset threshold are selected to generate an observable behavior set. The frequency of each observable behavior in the observable behavior set and the number of observable behaviors under each competency dimension are statistically analyzed. Based on the frequency and number of observable behaviors, a competency rating is calculated by comparing the results with a preset threshold range.

2. The method according to claim 1, characterized in that, The step of obtaining training comment text and performing preprocessing on the training comment text to generate structured text feature vectors includes: receiving input raw training comment text data; The original training comment text data is segmented to generate word sequence data; Part-of-speech tagging is performed on the word sequence data to generate word sequence data with part-of-speech tags; Dependency parsing is performed on the word sequence data with part-of-speech tags to generate syntactic structure data; The word sequence data, the word sequence data with part-of-speech tags, and the syntactic structure data are vectorized and concatenated to generate a fixed-dimensional structured text feature vector. The word sequence data is mapped to a first vector through a pre-trained word embedding model, the part-of-speech tags are mapped to a second vector through one-hot encoding, and the syntactic structure data is mapped to a third vector through dependency relation encoding. The first vector, the second vector, and the third vector are concatenated or weighted and summed to obtain the structured text feature vector.

3. The method according to claim 1, characterized in that, The knowledge extraction branch is used to extract knowledge elements; the skill extraction branch is used to extract skill elements; and the attitude extraction branch is used to extract attitude elements and determine their tendency.

4. The method according to claim 1, characterized in that, The process of statistically analyzing the display frequency of each observable behavior in the observable behavior set and the display quantity of different observable behaviors under each competency dimension includes: traversing the observable behavior set, accumulating and statistically analyzing the frequency of each observable behavior, and generating observable behavior display frequency data. Obtain preset competency dimension segmentation data and establish an association index between competency dimensions and observable behaviors; based on the association index, classify each observable behavior in the observable behavior set into the corresponding competency dimension; under each competency dimension, count the number of different types of observable behaviors and generate display quantity data for each competency dimension. By integrating the display frequency data and the display quantity data, basic data for competency assessment is generated.

5. The method according to claim 4, characterized in that, The process of calculating the competency rating based on the display frequency and number of displays, and comparing it with a preset threshold range, further includes: Extract KSA weakness dimension data from the fine-grained KSA extraction data; the KSA weakness dimension data includes: knowledge elements, skill elements, or missing attitude elements that are classified as negative. Retrieve pre-stored KSA-course mapping library data; The KSA weakness dimension data is matched with the KSA-course mapping library data to generate candidate training course data; Based on the severity parameter of the KSA weakness dimension data, the candidate training course data is prioritized and sorted to generate a recommended list of training courses.

6. The method according to claim 5, characterized in that, Following the generation of the recommended training course list, the following is also included: Retrieve competency rating data, KSA weakness dimension data, observable behavior mapping link information data in the observable behavior set, and training course recommendation list data; The competency rating data, the KSA weakness dimension data, the observable behavior mapping link information data, and the training course recommendation list data are structured and integrated to generate a comprehensive evaluation report; the comprehensive evaluation report is then output.

7. A training evaluation system for fine-grained KSA classification and behavior frequency analysis, characterized in that, The system includes: The data layer is used to store the fact-based evaluation base, the fine-grained KSA knowledge base, and the competency-observable behavior standard base; A fine-grained KSA extraction layer is used to input structured text feature vectors into a pre-trained multi-task classification model to extract fine-grained KSA data. The multi-task classification model includes a knowledge extraction branch, a skill extraction branch, and an attitude extraction branch. The fine-grained KSA data includes knowledge classification results, skill classification results, and attitude bias results. The knowledge classification results include descriptive knowledge, procedural knowledge, strategic knowledge, and adaptive knowledge. The skill classification results include motor skills, cognitive skills, and metacognitive skills. The attitude bias results include positive attitudes, negative attitudes, and missing attitudes. The observable behavior mapping layer is used to input the obtained fine-grained KSA extracted data into the KSA-observable behavior knowledge graph. Based on the semantic similarity between the semantic vector of each KSA element and the pre-stored semantic vector of each standard observable behavior node in the KSA-observable behavior knowledge graph, as well as the mapping weight corresponding to the fine classification type of each KSA element, the mapping confidence is calculated, and standard observable behaviors that reach the preset threshold are selected to generate an observable behavior set. The competency assessment layer is used to count the frequency of each observable behavior in the observable behavior set, as well as the number of times different observable behaviors are displayed under each competency dimension; based on the frequency and number of displays, and compared with a preset threshold range, the competency rating is calculated. The training course matching layer is used to generate a recommended list of training courses based on KSA weakness dimension data. The application layer is used to generate comprehensive evaluation reports.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-6.

Images