Method for evaluating training data of air operations control personnel

Abstract

The present application relates to the technical field of data processing and machine learning, and particularly relates to an evaluation processing method for aviation operation controller training data, comprising: obtaining multi-modal operation data submitted by a trainee in a training process; uniformly converting data of different modalities into text data and structured behavior event sequences; based on a pre-constructed behavior observation point knowledge base, performing semantic analysis on the text data to identify a recognition result of a behavior observation point associated with a standard operation procedure from the text data; determining a state identifier of each evaluation dimension according to the recognition result; if the structured behavior event sequence satisfies any rule condition in a pre-set deterministic rule base, directly outputting a corresponding state identifier; otherwise, aggregating the recognition result of the behavior observation point into a state identifier through an aggregation function. The present application improves the reliability of training data processing and the efficiency of the evaluation process through unified processing of multi-modal data and a two-layer judgment architecture with rule priority.

Description

Technical Field

[0001] This invention relates to the fields of data processing and machine learning technology, and in particular to an evaluation and processing method for training data of aviation operations control personnel. Background Technology

[0002] In the field of skills assessment for aviation operations control personnel, manual assessment is currently the most common method. This involves qualified assessors subjectively scoring the responses of those being assessed based on pre-set assessment criteria. While this method has accumulated experience over time, its inherent technical shortcomings are becoming increasingly apparent: the assessment standards lack a unified quantitative scale, and different assessors may give significantly different scores for the same responses; for assessment items with free expression, manual methods struggle to fully extract and compare all key decision-making elements in the responses, easily overlooking necessary operational items stipulated in the procedures. Furthermore, under high-load and abnormal scenarios, assessors may exhibit inconsistencies between their verbal statements and operational actions—that is, their verbal expression may conform to the procedures, but during actual operation, due to cognitive resource overload, they may miss key information points. Traditional assessment methods struggle to capture such behavioral deviations.

[0003] In recent years, some studies have attempted to introduce multi-source data fusion techniques to improve the objectivity of assessments. For example, literature CN115994713B proposes a method for evaluating the effectiveness of operational training based on multi-source data. This method acquires operational videos, equipment logs, and manually entered information during the operational training process, inputting these into three single-source effectiveness evaluation models based on these data, respectively, to obtain three independent scores. The scores are then evaluated for data source consistency, and the scores that pass the consistency evaluation are input into a multi-source decision fusion model to obtain the final evaluation score. This method achieves comprehensive utilization of multi-source data, but its technical solution has the following limitations: First, data fusion occurs at the decision-making level, and each modality of data is still processed independently, failing to achieve unified representation and fusion of multi-source data at the feature or semantic level; second, the evaluation process relies on the black-box output of three independent evaluation models, lacking a deterministic judgment mechanism for key operational behaviors, and cannot ensure that mandatory safety operations are not missed or misjudged by the model; third, the evaluation results are presented as continuous scores, making it difficult to trace back to specific operational behaviors and regulatory provisions.

[0004] Another paper, CN121786764B, proposes a pilot training operation evaluation method based on multimodal temporal fusion. This method acquires video streams, audio signals, and multidimensional operational state parameter sequences, achieving frame-level temporal alignment. Multimodal feature vectors are extracted, and a cross-modal attention mechanism is used to generate fused features. An operational causal logic topology graph containing physical boundary constraints is constructed. Logical bias operators are applied to the fused features based on real-time calculated physical state boundary parameters to increase the judgment weight of target operational nodes. Finally, a weighted score is calculated, and an evaluation report is output. This method introduces physical laws to correct the multimodal evaluation results, but its technical solution has the following limitations: First, its core driving logic is based on physical laws such as aircraft energy state and physical boundary parameters, rather than operational specifications such as standard operating procedures, making it impossible to directly evaluate based on a job competency framework. Second, its knowledge base is built on physical state trigger vectors rather than behavioral observation points, lacking a structured mapping mechanism from specific operational behaviors to capability dimensions. Third, its output is a weighted score, which is a continuous value, making the correspondence between the evaluation results and specific operational behaviors unclear. Summary of the Invention

[0005] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0006] This invention provides a method for evaluating and processing training data of aviation operations control personnel, including:

[0007] S100: Acquire the multimodal operation data submitted by the trainee during the aviation operation control training process. The multimodal operation data includes voice data, text data, image data, and video data.

[0008] S200, Modality standardization transformation is performed on the multimodal operation data to uniformly convert data of different modalities into text data and structured behavioral event sequences.

[0009] S300, based on a pre-built behavioral observation point knowledge base, performs semantic parsing on the text data to identify the recognition results of behavioral observation points associated with standard operating procedures from the text data; the behavioral observation point knowledge base contains multiple behavioral observation points, each behavioral observation point corresponds to a specific observable operational behavior, and each behavioral observation point is pre-assigned to one of multiple evaluation dimensions; the evaluation dimension is a category used to measure the operational skill characteristic dimension of the trainee.

[0010] S400, based on the identification results of the behavior observation points, determine the status identifiers for each evaluation dimension; the status identifiers are used to characterize whether the training object meets the preset operation requirements in each evaluation dimension; wherein, the status identifiers for each evaluation dimension are determined in the following way: if the structured behavior event sequence satisfies any rule condition in the preset deterministic rule base, the corresponding status identifier is directly output; otherwise, the status identifiers are aggregated by the identification results of the behavior observation points through an aggregation function.

[0011] S500 outputs structured data containing status identifiers for each evaluation dimension.

[0012] This invention achieves the fusion processing of heterogeneous training data within the same framework by uniformly converting multimodal operational data into text data and structured behavioral event sequences; it avoids the uncertainty of probabilistic models in key judgments by assigning fixed values ​​to mandatory key behaviors through a pre-set deterministic rule base, thereby improving the reliability of evaluation results; and it makes the evaluation results traceable to specific operational behaviors and regulatory basis by outputting binary status identifiers and associating them with specific behavioral observation points, thus meeting the requirements of traceability and verifiability of evaluation results in high-security scenarios.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating the evaluation and processing method for training data of aviation operations control personnel provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0018] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0019] First Embodiment

[0020] To address the shortcomings of existing technologies, such as the difficulty in uniformly processing multimodal operational data within a single framework, the lack of collaborative judgment mechanisms for behavioral and verbal data, the lack of deterministic judgment of key operational behaviors, and the untraceability of evaluation results, this invention provides an evaluation and processing method for training data of aviation operations control personnel, such as... Figure 1 As shown, the method includes the following steps:

[0021] S100: Acquire multimodal operational data submitted by the trainee during aviation operation control training.

[0022] In this embodiment, the term "trainee" broadly refers to personnel who need to undergo assessments of operational skills, decision-making abilities, or procedural execution standards. As a specific example and not a limitation, the trainee can be personnel in aviation operations control positions, such as civil aviation dispatch trainees or dispatchers.

[0023] The multimodal operational data refers to the raw response data submitted by the trainee during training or assessment, which includes multiple modalities and can be presented in the form of speech, text, images, and video. Specifically:

[0024] Recording of verbal statements, such as verbally explaining the reasons for the decision to release the vehicle;

[0025] Written descriptions, such as emergency response plans;

[0026] Operation logs are saved in image format, such as screenshots of flight plan messages and route planning interfaces;

[0027] Screen operation recordings, such as recording mouse and keyboard operation sequences in an electronic release system.

[0028] S200, Modality standardization transformation is performed on the multimodal operation data to uniformly convert data of different modalities into text data and structured behavioral event sequences.

[0029] To achieve unified processing of the aforementioned multimodal operational data, this step performs targeted modal standardization conversion based on the format characteristics of each modal data. Specifically:

[0030] For speech data: A speech recognition model based on deep neural networks is used to convert speech signals into text data and retain timestamp information so that key expressions identified later can be located to specific moments in the original speech stream, providing traceable evidence for subsequent analysis.

[0031] For text data: it can be used directly as a text data stream, or it can be cleaned and then used as a text data stream.

[0032] For image data: The optical character recognition engine is invoked to extract the text content contained in the image as text data, supporting mixed recognition of handwritten and printed text, such as extracting decision speed values ​​from photos of paper performance charts.

[0033] For video data: Target detection and action recognition are performed on the video data to detect the trainee's actions on the user interface and generate a structured sequence of behavioral events. First, a target detection algorithm identifies key areas in the user interface, including the flight plan editing area, the NOTAM list, the weather layer, the fuel calculation panel, and the alternate landing site selection dropdown menu. Then, a temporal action recognition model trained with pre-defined operation category labels, such as a dual-path video recognition network based on the SlowFast architecture, is used to detect the trainee's actions on the user interface and generate a structured sequence of behavioral events. Each event record is formatted as (timestamp, operation type, target object, confidence score), for example ("03:22", "click", "alternate landing site selection button", 0.89). The operation type refers to the type of action performed by the trainee on elements of the user interface, including at least one of clicking, dragging, selecting, and text input; the target object refers to the trainee's identification information on the user interface. The training samples for the temporal action recognition model include operation type labels such as click, drag, selecting, and text input.

[0034] After completing the above modal standardization transformation, the obtained text data is further processed by domain terminology standardization: non-standard expressions are mapped to a preset standard terminology system to obtain standardized text data. For example, the colloquial expression "check wx" is standardized to "check the weather conditions", and "min fuel" is standardized to "announce the minimum fuel level", ensuring that subsequent semantic parsing is based on a unified industry language system.

[0035] Through the above modality standardization transformation, data from different modalities are uniformly converted into text data and structured behavioral event sequences, providing standardized input for subsequent semantic parsing and state identification.

[0036] S300, based on a pre-built behavioral observation point knowledge base, performs semantic parsing on the text data and identifies the recognition results of behavioral observation points associated with standard operating procedures from the text data.

[0037] (I) Construction of the knowledge base for behavioral observation points

[0038] The behavioral observation point knowledge base is constructed based on industry operational standard frameworks such as ICAO Doc 9868 "Training Manual" or equivalent job competency frameworks. The behavioral observation point knowledge base stores multiple behavioral observation points in a structured format. Each behavioral observation point corresponds to a specific observable operational behavior, and each behavioral observation point is pre-assigned to one of multiple assessment dimensions. The specific operational behavior refers to the behavioral unit in which the trainee's actions, objects, and results during training or assessment can be directly or indirectly perceived and recorded by the computer system through technologies such as speech recognition, image recognition, or log analysis. Examples include clicking a release button, inputting fuel quantity values, and verbally stating the reasons for a decision. Each behavioral observation point contains at least the following fields: a unique identifier for the observation point (e.g., OB 4.8), the assessment dimension it belongs to (e.g., "communication standardization"), a standard operating procedure description (e.g., "use standard communication language in emergency situations"), a list of triggering keywords (e.g., "minimum fuel", "Mayday", "declare emergency"), and an optional semantic template (e.g., "[declare][minimum fuel]").

[0039] The assessment dimensions are categories used to measure the operational skill characteristics of trainees, including but not limited to communication standardization, information processing ability, decision-making ability, risk management ability, teamwork and leadership.

[0040] Taking the "Low Visibility Flight Departure" training subject for aviation operations control positions as an example, the following related behavioral observation points are selected from the knowledge base based on the subject scenario: OB 4.8 "Using standard communication language in emergency situations", OB 3.2 "Accurately interpreting and integrating multi-source intelligence such as NOTAMs, weather, and airport status", and OB 5.4 "Developing reasonable diversion plans based on threat assessment (such as runway visual range below 550 meters)". The above set of OBs constitutes the available behavioral observation points to be matched under this subject.

[0041] (ii) Identification strategy for behavioral observation points

[0042] For each behavioral observation point, a two-layer strategy of "keyword matching + semantic similarity matching" is used for identification:

[0043] First layer: Keyword matching

[0044] The text data is segmented and keywords are extracted, and a list of preset trigger keywords for the behavior observation point is retrieved. If any trigger keyword corresponding to the behavior observation point, or its synonym or abbreviation, appears in the text data, the recognition result of the behavior observation point is marked as recognized. For example, if the text data contains "announce minimum fuel consumption" or "Minimum fuel", then the behavior observation point (OB 4.8) is recognized as recognized. To improve robustness, keyword matching supports fuzzy matching and word form restoration.

[0045] Second layer: Semantic similarity matching

[0046] If the text data does not contain the preset trigger keywords for the behavior observation point, the semantic similarity between the text data and the standard operating procedure description text of the behavior observation point is calculated. The semantic similarity calculation uses a pre-trained sentence embedding model (e.g., Sentence-BERT) to encode the text to be matched and the standard operating procedure description text into fixed-dimensional semantic vectors, and then calculates the cosine similarity between them. If the similarity exceeds a preset threshold, such as 0.75, the recognition result of the behavior observation point is marked as recognized.

[0047] (III) Context Enhancement and Negation Detection

[0048] During the matching process, context enhancement and negation detection mechanisms are also introduced:

[0049] Centered on the statement to be matched, the system extracts several statements before and after it, or several structured behavioral events that are temporally adjacent, such as operation events occurring within 10 seconds of each other under the same training subject. This information is then concatenated with the statement to be matched to form an expanded text window. This expanded text window serves as input, and the dual-layer strategy of keyword matching and semantic similarity matching is re-executed. Introducing a context window avoids missed identifications due to scattered expressions. For example, a trainee might verbally state "Fuel may be insufficient." This statement alone might not identify OB4.8 as recognized; however, subsequent operation logs record a click event for the "Request priority landing" button. The system associates these two events and concatenates them into "Fuel may be insufficient. Operation: Click Request priority landing," then sends this information to the dual-layer strategy for a comprehensive judgment on whether it has been recognized.

[0050] Negation detection: If the text data contains explicit negative words, such as "not checked," "no," "not executed," or "not announced," the recognition result for that behavior observation point is reversed. Specifically, when a negative word is detected, if the current recognition result is "recognized," it is reversed to "not recognized," and the confidence level is set to 0; if the current recognition result is already "not recognized," no further adjustment is needed. Through the above processing, it is ensured that negative expressions are not misjudged as positive behaviors.

[0051] (iv) Output of recognition results

[0052] The identified behavioral observations are used as the identification results. These results can be represented binary, with a first value indicating "identified" and a second value indicating "not identified." In one specific implementation, the first value is 1 and the second value is 0. For each behavioral observation, an identification result is output, including a Boolean value indicating whether it was identified or not, and a matching confidence score, which is a continuous value between 0 and 1. Through the above two-layer strategy and context enhancement and negative detection mechanisms, it is possible to accurately identify the various behavioral observations exhibited or missing by the trainee during the assessment process from free text and operation logs, providing reliable basic data for the subsequent determination of status identifiers.

[0053] (v) Traceability Recording of Identification Results

[0054] At the same time as each behavioral observation point identification result is generated, the key evidence that triggered that result is recorded.

[0055] For behavioral observations identified through keyword matching, key evidence includes the triggering keyword, the rule number to which the keyword belongs, the key behavioral feature description defined in the original rule text, and the original fragment in the input data that triggered the keyword, such as specific keywords in text data or specific events in operation logs.

[0056] For behavioral observations identified through semantic similarity matching, key evidence includes the standard operating procedure description text on which the matching is based, the calculated similarity value, and the most influential lexical units or operation timestamps in the input text data located through the attention mechanism.

[0057] The aforementioned evidence, along with the behavioral observation point identification results, is stored together for subsequent tracing and verification of status identifiers.

[0058] S400, based on the identification results of the behavior observation points, determine the status identifier of each evaluation dimension.

[0059] The status identifier is used to characterize whether the trainee meets the preset operational requirements in each evaluation dimension. The status identifier is a binary status value or a finite discrete status value, such as "met / not met" or "identified / not identified," used to replace traditional continuous quantitative scores, making the evaluation results more explicit and operable, and reducing subjective interpretation bias caused by small differences in scores. In this embodiment, the status identifier can be represented binaryly, with a third value representing "met" and a fourth value representing "not met." In one specific implementation, the third value is 1, and the fourth value is 0.

[0060] The status identifiers for each evaluation dimension are determined using the following dual-path method:

[0061] First path: Rule matching path

[0062] The structured behavioral event sequence is compared with a preset deterministic rule base. If the structured behavioral event sequence satisfies any rule condition in the preset deterministic rule base, the state identifier corresponding to that rule is directly output.

[0063] Second path: Semantic aggregation path

[0064] If the structured behavioral event sequence does not meet any rule condition in the deterministic rule base, the identification results of the behavioral observation points are aggregated into a state identifier by an aggregation function.

[0065] The two paths described above constitute a collaborative judgment logic that prioritizes rules and supplements semantics, ensuring that critical operational behaviors are directly determined by deterministic rules, while non-critical behaviors are derived by aggregating semantic recognition results.

[0066] The deterministic rule base is used to make deterministic judgments on mandatory safety-related operations. The deterministic rule base contains at least one key behavioral feature and its corresponding fixed quantification value. The key behavioral feature refers to a mandatory safety-related operation explicitly stipulated in operating regulations or standard operating procedures, such as "declaring an emergency" or "performing a minimum fuel level check." The key behavioral feature has the following three characteristics: mandatory (the action must be performed, based on mandatory modal words such as "must" or "shall" in the regulations); severe consequences (failure to perform the action or incorrect performance will significantly increase operational safety risks); and observability (the action can be clearly identified through operational data submitted by the trainee). If the operational data triggers any rule condition in the deterministic rule base, i.e., a corresponding key behavioral feature is identified from the operational data, the fixed quantification value corresponding to that rule is directly output as the mapping result, and it is not further used in subsequent model inference. Once set, the fixed quantification value remains constant throughout the rule base's lifecycle and is unaffected by changes in model parameters or data distribution.

[0067] The construction of the deterministic rule base includes the following steps:

[0068] (1) Obtaining the source of the rules

[0069] The rules can be derived from one or more of the following normative documents: industry operating regulations, such as the Civil Aviation Administration of China (CAAC) regulations, ICAO standards and recommended practices, operations manuals, standard operating procedures, training syllabi, and safety bulletins. These documents collectively define the key behaviors that operators must perform or are absolutely prohibited from performing in various operating scenarios.

[0070] (2) Identification of key behavioral features

[0071] For the aforementioned normative documents, identify the behavioral items explicitly defined by mandatory modal words such as "must," "strictly prohibited," and "must not." For each identified behavioral item, extract its key semantic elements to form a structured description of "triggering condition - key behavioral feature." Here, the key behavioral feature refers to the mandatory operation defined by the item itself, i.e., "what must be done" or "what is prohibited"; the triggering condition refers to the preconditions that must be met to trigger the key behavioral feature, i.e., "under what circumstances must it be performed or prohibited."

[0072] For example, from the clause "When the fuel level is below the final reserve fuel level, the captain must declare a minimum fuel level status," the trigger condition is extracted as "fuel level is below the final reserve fuel level," and the key behavioral feature is "declaring a minimum fuel level status." This key behavioral feature and the behavioral observation point OB 4.8 described in S300 refer to the same operational behavior in content, but they belong to different levels of representation: the behavioral observation point is a standardized entry used to evaluate the dimensional mapping, focusing on whether it is identified, while the key behavioral feature is a trigger condition in the rule base, focusing on whether the rule is triggered.

[0073] In this way, each key behavioral feature is associated with one or more behavioral observation points, thereby connecting the rule determination with the subsequent evaluation dimensions.

[0074] (3) Status identifier mapping

[0075] Each identified key behavioral feature is assigned a corresponding status identifier. If the behavioral feature is a mandatory positive behavior, it is assigned a "met" status identifier when it is met, and a "not met" status identifier when it is not met. If the behavioral feature is a prohibited negative behavior, it is assigned a "not met" status identifier when it occurs. Once set, the status identifier remains constant throughout the rule base's lifecycle and is unaffected by changes in model parameters or data distribution.

[0076] (4) Rule condition coding

[0077] The aforementioned key behavioral features and their corresponding status identifiers are encoded into computer-executable rule conditions. Each rule condition consists of a trigger condition, a behavioral requirement, and a status identifier, expressed in an "If-Then" format. The "If" part is the matching condition for semantic features in the structured behavioral event sequence, corresponding to the trigger condition; the "Then" part is the output status identifier and the corresponding behavioral observation point identifier, corresponding to the behavioral requirement and status identifier. The matching conditions support keyword matching, such as detecting whether the event sequence contains a specific operation type, regular expression matching, or semantic template matching, or one or more other methods.

[0078] (5) Rule verification and maintenance

[0079] The completed rule base needs to be tested on a pre-defined validation dataset to ensure that the triggering logic of each rule is consistent with expectations and that there are no conflicts or redundancies between rules. When the normative documents are updated, the rule base is updated synchronously through manual review or natural language processing technology, deleting invalid rules, modifying and adjusting existing rules, or adding new rules.

[0080] Example rule entries (taking the aviation operations control position as an example):

[0081] Rule R-01 for the "Declare Minimum Fuel" operation: The trigger condition is that the structured behavioral event sequence contains an equivalent operation event of "Declare Minimum Fuel" or "Minimum fuel", or an equivalent expression appears in the text data. When this condition is met, the "Met" status flag is output, and the behavioral observation point flag OB 4.8 is associated. Rule R-02 for the "Check Runway Visual Range" operation: The trigger condition is that no operation event of checking the weather panel is detected in the structured behavioral event sequence, and no expression related to "Runway Visual Range" or "RVR" is mentioned in the text data. When this condition is met, the "Not Met" status flag is output, and the behavioral observation point flag OB 3.2 is associated. Rule R-03 for the "Develop Alternate Landing Plan" operation: The trigger condition is that no operation event of selecting an alternate landing site is detected in the structured behavioral event sequence. When this condition is met, the "Not Met" status flag is output, and the behavioral observation point flag OB 5.4 is associated.

[0082] The aggregation function is used to aggregate the identification results of multiple behavioral observations under the same evaluation dimension into a state identifier for that dimension. The aggregation function can be any one of weighted summation, minimum value taking, or majority voting.

[0083] When using weighted summation, the weight coefficients for each behavioral observation point are predetermined according to one or more of the following methods:

[0084] (1) Importance level based on job competency: Senior evaluators compare the contribution of behavioral observation points to the evaluation dimension pairwise, and determine the normalized weights using the analytic hierarchy process or the Delphi method. For example, under the "decision-making ability" dimension, the weight of OB 5.4 "developing alternative landing plans" can be set to 0.6, and the weight of OB 5.2 "identifying threats" can be set to 0.4.

[0085] (2) Based on historical data statistics: By analyzing the sample library that has been labeled, the correlation coefficient between the identification result of each behavior observation point and the final state of that dimension is calculated, and the correlation coefficient is normalized and used as the weight.

[0086] (3) Based on security sensitivity classification: For key behavior observation points involving operational security, assign higher weights, for example, set the weight of the key behavior observation point to twice that of other observation points, so as to ensure that its contribution to the dimension state identification is amplified.

[0087] In one specific implementation, assuming there are multiple behavioral observation points under a certain evaluation dimension, the identification result of each behavioral observation point (taking either a first value or a second value) is multiplied by its corresponding weight coefficient, and all products are summed to obtain the weighted score for that evaluation dimension. The weighted score is then compared with a preset activation threshold. If the weighted score is greater than or equal to the activation threshold, the status indicator takes the third value ("satisfied"); otherwise, it takes the fourth value ("not satisfied"). In one specific implementation, the activation threshold can be set to 0.5.

[0088] When using the minimum value approach, no weighting coefficients are involved; the minimum value among the identification results of all behavioral observation points under that dimension is directly taken as the state identifier for that dimension. That is, when the identification results of all behavioral observation points under that dimension are all the first value (all "identified"), the state identifier is the third value ("satisfied"); if there is any behavioral observation point with an identification result of the second value ("not identified"), the state identifier is the fourth value ("not satisfied"). This method is suitable for strict scenarios that require all sub-behaviors to meet the standard.

[0089] When majority voting is used, no weighting coefficients are involved. The state corresponding to the identification results of more than half of the behavioral observation points under the evaluation dimension is taken as the state identifier of that dimension. Specifically, the number of identification results of each behavioral observation point under the evaluation dimension with the first value ("identified") is counted. If this number exceeds half, the state identifier is the third value ("satisfied"); otherwise, the fourth value ("not satisfied") is taken. In a specific implementation, when the first value is 1 and the second value is 0, this method is equivalent to: counting the number of identification results with a value of 1, if it exceeds half, the state identifier is 1 ("satisfied"); otherwise, it is 0 ("not satisfied").

[0090] In one specific implementation, when weighted summation is used as the aggregation function, the weight coefficients are pre-set and stored in a mapping table, and are directly read and used during state identifier calculation. The weights of each behavioral observation point within each evaluation dimension are independently assigned, and the weights between different dimensions are not compared or weighted across dimensions.

[0091] To achieve a traceable association between the status identifiers of each evaluation dimension and the identification results of the behavior observation points, this embodiment pre-constructs a mapping table between status identifiers and behavior observation points. This mapping table records which behavior observation points' identification results jointly determine the status identifier for each evaluation dimension.

[0092] The assessment dimensions refer to different categories used to measure the comprehensive abilities of trainees, based on industry competency frameworks such as ICAO Doc 9868. These categories include, but are not limited to, communication standardization, information processing ability, decision-making ability, risk management ability, teamwork, and leadership. Each assessment dimension has a fixed hierarchical relationship with its multiple subordinate behavioral observation points, which is defined during the construction phase of the behavioral observation point knowledge base.

[0093] When the status identifier is determined, for any evaluation dimension, the identification results of each behavioral observation point associated with that dimension are merged into a status identifier for that dimension through an aggregation function. This ensures that each status identifier corresponds to a specific set of behavioral observation points and corresponding regulatory basis. This mapping table and the behavioral observation point knowledge base described in S300 are different application layers of the same knowledge base. The knowledge base defines the membership relationship between behavioral observation points and evaluation dimensions, and the mapping table reads this membership relationship during the status identifier determination stage and performs aggregation calculations accordingly.

[0094] S500 outputs structured data containing status identifiers for each evaluation dimension.

[0095] The structured data includes status identifiers for each evaluation dimension, identification results for each behavior observation point, and behavior observation point identifiers associated with each status identifier, thereby enabling complete traceability from evaluation results to specific behavior observation points and judgment criteria.

[0096] In one specific implementation, the structured data is presented in the form of a radar chart, with each dimension of the radar chart corresponding to an evaluation dimension; clicking on any dimension expands a secondary menu, displaying the identification results of each behavioral observation point under that dimension, the judgment reason such as "no mention of fuel check", the cited regulatory clause number and the corresponding behavioral observation point identifier.

[0097] In another specific implementation, the structured data further includes key evidence fragments upon which the identification results of each behavioral observation point are based, to support post-event tracing and verification. Key evidence fragments include the original text of the keywords triggering the rule, timestamps, lexical positions located through attention mechanisms, or sequences of operational events. The structured data can be exported to JSON or PDF format for archiving or post-event tracing.

[0098] The first embodiment provides a method for evaluating and processing training data of aviation operations control personnel, which has the following beneficial effects:

[0099] 1. Unified processing of multimodal data

[0100] By standardizing the modality transformation, operational data from different modalities such as speech, text, images, and videos are uniformly converted into text data and structured behavioral event sequences. This solves the problem that multimodal heterogeneous data is difficult to process and represent uniformly within the same framework, enabling training data from different sources and formats to be effectively utilized in subsequent semantic parsing and state identification steps.

[0101] 2. Deterministic determination of key operational behaviors

[0102] By assigning fixed values ​​to mandatory safety-related operations using a pre-defined deterministic rule base, this method achieves deterministic determination of key behavioral characteristics. Compared with purely model-driven evaluation schemes, this method does not rely on probabilistic models to determine mandatory operations such as "announcing minimum fuel levels" and "checking runway visibility," thus avoiding missed or false judgments of key behaviors due to model uncertainty.

[0103] 3. Structured output of status indicators

[0104] By replacing traditional continuous quantized scores with binary or finitely discrete state identifiers as evaluation results, the evaluation results become more explicit and operable, reducing subjective interpretation bias caused by minor differences in scores. Simultaneously, the structured data is linked to behavioral observation point identifiers and identification criteria, enabling the evaluation results to be traced back to specific operational behaviors and regulations, meeting the requirements of traceability and verifiability of evaluation results in high-security scenarios.

[0105] 4. Co-judgment of behavioral and verbal data

[0106] By converging the structured behavioral event sequence with the behavioral observation point identification results in the state identification determination step, the collaborative determination of operational behavior and verbal expression is realized, which can identify behavioral patterns where the verbal statement meets the requirements but the actual operation deviates.

[0107] The first embodiment described above describes the data processing flow during the inference phase. The second embodiment below will describe the training process of the evaluation model, which, after training, is used to replace the sentence embedding model in the semantic matching step of the first embodiment.

[0108] It's important to note that the training and inference phases employ different technical approaches. The fundamental reason for this lies in their differing objectives: the training phase aims to enable the model to achieve semantic generalization, understanding free-form expressions not present in the training data. Therefore, a purely data-driven multi-task learning strategy is used, without introducing rule base constraints; otherwise, the model would only be able to fit a limited set of rules, failing to achieve generalization. The inference phase aims to achieve comprehensive recognition while ensuring evaluation reliability. Therefore, a collaborative decision-making architecture prioritizing rules and supplementing them with the model is adopted. The rule base ensures deterministic determination of mandatory key behaviors, while the model completes semantic parsing tasks not covered by the rule base. These two phases work together to form a complete training-inference-feedback optimization technical loop.

[0109] Second Embodiment

[0110] In this embodiment, semantic parsing of the text data is performed based on the current evaluation model, which is obtained in the following way:

[0111] S10, Obtain training samples, and use the current evaluation model to perform feature numerical transformation on the training samples to obtain the transformation result.

[0112] The training samples are multimodal operational data submitted by the trainees during aviation operation control training. In this embodiment, the current evaluation model refers to the version of the evaluation model currently deployed for semantic parsing. Upon initial deployment, the current evaluation model is the basic evaluation model obtained through initial training; in subsequent operations, whenever model parameters are adjusted and verified, the updated model replaces the original model, becoming the new current evaluation model.

[0113] In this embodiment, the evaluation model is a machine learning model trained for semantic parsing. Its training objective is to output quantified scores for each evaluation dimension and matching probabilities for each behavioral observation point, given input training samples. The feature numerical transformation refers to inputting unstructured data from the training samples into the evaluation model, which then processes it into a structured quantified feature representation. The transformation result is the output of the evaluation model.

[0114] In one illustrative embodiment, the evaluation model is built on the Transformer architecture and includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The input embedding layer converts a standardized text sequence into a vector representation, the text sequence being composed of speech recognition results, text extracted by optical character recognition, and a structured behavioral event sequence concatenated in chronological order. The multi-head self-attention layer captures long-range semantic dependencies in the input sequence. The feedforward neural network layer performs a non-linear transformation on the output of the attention layer to further extract high-level semantic features. The multi-head self-attention layer contains H parallel attention heads, where H is a positive integer greater than or equal to 4. In one specific embodiment, H is set to 12.

[0115] The output layer comprises a first output branch and a second output branch. The first output branch is a scoring regression branch, which maps the semantic feature vector output by the Transformer encoder to a K-dimensional quantized score vector, where K is the preset number of evaluation dimensions. In one specific implementation, K is set to 9, corresponding to the 9 core dimensions in the job competency framework. The second output branch consists of M independent binary classifiers, each outputting the matching probability of a behavior observation point. M equals the total number of behavior observation points in the pre-built behavior observation point knowledge base. In one specific implementation, for the aviation operations control position, the behavior observation point knowledge base is constructed according to the ICAO Doc 9868 framework, containing 9 competency dimensions and a total of 53 behavior observation points, so M is set to 53. Each binary classifier uses a logistic regression structure, outputting the matching probability by transforming the [CLS]-labeled hidden layer vector through a linear transformation and a sigmoid activation function.

[0116] The training process of the evaluation model employs a multi-task joint learning strategy. The joint loss function comprises two components: the first is the rating regression loss, which measures the deviation between the model's predicted quantitative score and the correction information provided by the evaluator, calculated using mean squared error; the second is the behavior observation point classification loss, which measures the accuracy of the model's prediction of the matching probability of each behavior observation point, calculated using binary cross-entropy. These two losses are weighted and summed using a preset balancing coefficient to form the final joint loss value. During training, the model parameters are iteratively updated using a mini-batch gradient descent algorithm until the joint loss value converges on the validation set.

[0117] S20, obtain correction information provided by at least one evaluator for the same training sample, wherein the evaluator has a level identifier.

[0118] The evaluation subject refers to a qualified person who manually evaluates training samples and provides correction information, such as an instructor or inspector in an aviation operations control position. The evaluation subject has a level identifier, which includes at least a first-level identifier and a second-level identifier, with the second-level identifier having higher evaluation authority than the first-level identifier. The level identifier is predetermined based on at least one of the evaluation subject's years of experience, certification qualifications, or historical scoring consistency indicators. Wherein:

[0119] Years of experience: This refers to the cumulative number of years the evaluator has been engaged in teaching or assessment work in this position. For example, those with more than 5 years of experience can be classified as Level 2 (Advanced), while those with less than 5 years can be classified as Level 1 (Junior).

[0120] Certification qualifications: refers to the industry qualifications or instructor certification levels held by the assessor. For example, those holding advanced instructor certification or inspector qualifications are classified as Level 2, while those holding basic instructor certification are classified as Level 1.

[0121] Historical scoring consistency index: This refers to the average deviation between the historical scoring records of the evaluator and the consensus scores of multiple evaluators. Specifically, it involves calculating the average absolute error between the evaluator's scores on the same batch of answers in past assessments and the average scores of at least three other evaluators. If this error is lower than a preset threshold, such as 1.0 point, it is classified as Level 2; otherwise, it is classified as Level 1.

[0122] The above indicators can be used individually or in a weighted combination. In one specific implementation, an initial level is first determined based on years of experience and certification qualifications. Then, the level is dynamically adjusted using a historical scoring consistency indicator. For example, if a judge's historical scoring consistency indicator falls below a preset threshold multiple times consecutively, its level is upgraded from Level 1 to Level 2.

[0123] The correction information refers to the quantitative evaluation results or correction opinions given by the evaluation subject to the training samples according to preset evaluation standards and industry regulations. Each correction information includes at least the following: the quantitative score of each evaluation dimension, and the label of whether the optional behavioral observation points are satisfied. The correction information serves as a benchmark for judging whether the model transformation results are acceptable and as a source of supervision signals for the samples to be optimized in the incremental sample set. In one specific implementation, the first level is identified as a primary evaluation subject, such as a junior instructor, and the second level is identified as a senior evaluation subject, such as a senior instructor or inspector. The correction information provided by the senior evaluation subject is given higher weight in subsequent model parameter adjustments.

[0124] S30, determine the deviation tolerance range based on the level identifier of the evaluation subject.

[0125] The deviation tolerance range is used to determine whether the deviation between the transformation result of the current evaluation model and the correction information provided by the evaluation subject is acceptable. Determining this range includes two stages: initial setting and dynamic adjustment.

[0126] (i) Initial Range Setting: Based on the level of the judging body, an initial deviation tolerance range width is preset for different levels of judging bodies. Specifically, the initial range width for the second-level (advanced) judging body is smaller than that for the first-level (basic) judging body. For example, the initial range width for the second-level judging body is set to ±1.0 points, while the initial range width for the first-level judging body is set to ±1.5 points. This difference reflects the stricter tolerance for deviation for the advanced judging body.

[0127] (ii) Dynamic Decrease Mechanism: As the number of iterations or version updates of the current evaluation model increases, the width of the deviation tolerance range dynamically decreases. Specifically, after each adjustment of the model parameters, the current deviation tolerance range width is multiplied by a preset decay factor, such as 0.95, to gradually narrow the range width. To prevent excessive narrowing that would fail to meet practical application requirements, a lower limit value is set for the range width, such as ±0.5 points. When the calculated width is lower than the lower limit value, the lower limit value is used as the current deviation tolerance range width.

[0128] For example, for the second-level identification assessment subject, the initial tolerance range for deviation is ±1.0 points. After three adjustments to the model parameters, the tolerance range decreases to ±1.0 × 0.95. 3 The initial score was approximately ±0.86 points; after 10 adjustments, it approached the lower limit of ±0.5 points. For the first-level identification assessment subject, the initial range was ±1.5 points, which decreased proportionally, with the lower limit remaining at ±0.5 points.

[0129] This dynamic decay mechanism reduces the tolerance for bias as the model's scoring ability improves, thereby driving the model to evolve towards higher accuracy. It should be noted that the decay factor and lower limit are adjustable based on the specific application scenario and are not fixed parameters; the dynamic decay mechanism is also an optional feature and can be enabled or disabled during implementation.

[0130] S40, calculate the deviation value between the conversion result and the correction information, and determine whether the deviation value exceeds the deviation tolerance range; if it does, use the training sample and the corrected data of the correction information as the sample to be optimized.

[0131] The deviation value is calculated as follows: For each evaluation dimension, the quantitative score of that dimension in the transformation result is calculated (denoted as s). map ) and the quantized score of that dimension in the correction information (denoted as s)ref The absolute difference between d and |s map -s ref |; Sum or average the absolute differences of all dimensions to obtain the overall deviation value. If the overall deviation value is greater than the deviation tolerance range corresponding to the current evaluation subject, it is judged as exceeding the tolerance range.

[0132] When the judgment exceeds the limit, the following information is packaged into a sample to be optimized: the training sample, the correction information, the correction data provided by the evaluation body, and the level identifier of the evaluation body. The correction data refers to the specific changes made by the evaluation body to the parts of the conversion result that are inconsistent with the correction information, including the modified quantitative scores of each dimension and the reasons for the modification. For example, the score of the "communication standardization" dimension is modified from 8.0 points to 7.5 points, and the reason for the modification is marked as "not using standard terminology".

[0133] When the same training sample receives correction information from multiple evaluators that are inconsistent with each other, an arbitration rule is activated: the correction information and corrected data provided by the second-level (high-level) evaluator take precedence, overriding the correction information and transformation results of the first-level evaluator. Simultaneously, the reasons for the conflict are recorded, including but not limited to the specific numerical differences in scores from each evaluator, the identifiers of the behavioral observation points involved, the key evidence fragments where disagreement exists, and the respective level identifiers of the evaluators. The recorded reasons for the conflict are stored in a log system for subsequent analysis of model evaluation blind spots, optimization of the behavioral observation point knowledge base, or targeted training for the evaluators.

[0134] S50, construct an incremental sample set based on the sample to be optimized.

[0135] Maintain an incremental sample pool to store all historically accumulated samples to be optimized. Each sample to be optimized must contain at least the following fields: training sample, correction data provided by the evaluator including the corrected quantitative scores for each evaluation dimension and the reasons for the correction, the level identifier of the evaluator, and the correction timestamp.

[0136] When constructing the incremental sample set, the samples to be optimized in the incremental sample pool are first deduplicated and denoised:

[0137] Deduplication: If the same training sample has correction data provided by multiple evaluators, according to the arbitration rules, only the correction data of the highest-level evaluator shall be retained, or all correction data shall be retained but each record shall be appended with a weight identifier corresponding to its evaluator level.

[0138] Denoising: Verify the completeness and validity of each sample to be optimized, checking whether the corrected data contains correction information for all evaluation dimensions, whether the correction reason is empty, or whether there are obvious logical errors. For samples that fail the verification, mark them as invalid and remove them from the sample pool, and record the exception log for administrator review.

[0139] In the samples to be optimized, the corrected data provided by the second-level (advanced) evaluator is assigned a first weight, and the corrected data provided by the first-level (primary) evaluator is assigned a second weight, with the first weight being greater than the second weight. This weight will be used in the weighted calculation of the loss function during subsequent model parameter adjustments, making the feedback signal from the advanced evaluator have a greater influence on model optimization.

[0140] After deduplication and denoising, all valid samples to be optimized in the current incremental sample pool are combined to form an incremental sample set. This incremental sample set can be used alone for model parameter tuning, or it can be combined with the initial training set for model updates. Through these steps, a high-quality, multi-sample, and traceable incremental sample set is constructed, providing a reliable source of supervision signals for subsequent model parameter tuning.

[0141] In one specific implementation, the incremental sample set is also periodically checked for category distribution, and the sample collection strategy is adjusted based on the check results.

[0142] S60, when the incremental sample set meets a preset triggering condition, the parameters of the current evaluation model are adjusted using the incremental sample set to obtain an updated evaluation model, and the updated evaluation model is used as the current evaluation model for the next iteration. In this invention, the preset update condition includes at least one of the following two situations:

[0143] (1) The number of valid samples in the incremental sample set reaches a preset threshold, such as 200; valid samples refer to the samples to be optimized that have passed the verification and are retained in the progressive sample pool after deduplication and noise reduction in S50.

[0144] (2) The time elapsed since the last model update exceeds the preset time interval, for example, 7 days.

[0145] When any condition is met, the progressive parameter adjustment process is automatically triggered.

[0146] During the progressive parameter tuning phase, the evaluation model is optimized using a parameter-efficient fine-tuning technique. Specifically, the parameters of the pre-trained backbone network are kept frozen, and low-rank trainable parameters are added only next to the attention weight matrix of the Transformer layer. Taking LoRA (Low-Rank Adaptation) as an example, for the original weight matrix W0, two low-rank matrices A and B are introduced, and the updated weight matrix is ​​represented as Wc = W0 + BA. During training, only the parameters of A and B are optimized. The number of trainable parameters in this way is usually much smaller than the total number of parameters in the original model, for example, no more than 1% of the total number of parameters in the original model. This significantly reduces computational and storage overhead while avoiding the catastrophic forgetting problem caused by small-sample progressive training.

[0147] It should be noted that LoRA is only an exemplary implementation of efficient parameter fine-tuning in this invention. Those skilled in the art can also use other equivalent techniques, such as adapters, prefix fine-tuning, or IA3. The proportion of the aforementioned low-rank parameters does not exceed 1%, the update trigger threshold is, for example, 200 valid samples, the time interval is 7 days, and the training time is, for example, 10 to 20 minutes. These are all examples. In practical applications, they can be flexibly set according to factors such as the scale of training data, model complexity, and computing resources. These alternative solutions and parameter adjustments all fall within the protection scope of this invention.

[0148] The progressive parameter adjustment process also employs a dynamic learning rate scheduling strategy. The specific steps are as follows:

[0149] (1) After each training round, calculate the loss value of the current evaluation model on the independent validation set. The independent validation set is a sample set extracted from historical response data that does not participate in model training or progressive parameter adjustment. Each sample in this set contains a training sample and its corresponding ground truth label. The ground truth label refers to a unified label formed through a consensus mechanism after multiple judges with second-level identifiers independently evaluate the same training sample, and is used as a benchmark answer to evaluate the model performance. The independent validation set is used to: monitor the trend of loss value changes during model parameter adjustment, and evaluate the scoring consistency and key behavior observation point identification accuracy of the updated model after the model update is completed, in order to decide whether to deploy or roll back.

[0150] The loss value can be, for example, expressed as a joint loss function L = α × L. score +β×L OB Among them, L score For the score regression loss, L OB The classification loss for behavioral observation points is denoted by α and β, which are balance coefficients satisfying α + β = 1. In one specific implementation, α = 0.7 and β = 0.3. This scheduling strategy helps the model escape local optima and achieve more refined parameter convergence.

[0151] The rating regression loss is used to measure the deviation between the mapping results output by the evaluation model (i.e., the quantified scores of each evaluation dimension) and the correction information provided by the evaluator (i.e., the quantified scores of the same dimension). In this invention, mean squared error (MSE) is used as the calculation method for the rating regression loss.

[0152] Specifically, for a batch of training data containing N samples, each sample corresponds to K evaluation dimensions, let:

[0153] The mapping result is the quantitative score of the i-th sample in the j-th evaluation dimension of the evaluation model output;

[0154] y ij The quantitative score of the i-th sample on the j-th evaluation dimension in the correction information provided to the evaluation subject.

[0155] Then the score regression loss L score Defined as:

[0156] .

[0157] The smaller the loss value, the closer the quantitative score output by the model is to the human-supervised label, meaning the higher the scoring accuracy of the model.

[0158] The behavior observation point classification loss is used to measure the accuracy of the evaluation model's prediction of the matching results for each behavior observation point. In this invention, the matching result of each behavior observation point is regarded as an independent binary classification problem, i.e., "satisfied" or "not satisfied", so the binary cross-entropy loss function is used for calculation.

[0159] Specifically, for a batch of training data containing N samples, each sample corresponding to M behavioral observation points, let:

[0160] To evaluate the matching probability of the r-th behavior observation point in the i-th sample output by the model, the value ranges from 0 to 1;

[0161] z ir The true label of the r-th behavior observation point in the i-th sample in the correction information provided to the evaluation subject, with a value of 0 indicating that it does not meet the requirements and 1 indicating that it does meet the requirements.

[0162] The classification loss for behavioral observation points is L. OB Defined as:

[0163] .

[0164] The smaller the loss value, the more accurate the model's judgment on whether the behavioral observation points meet the requirements.

[0165] (2) Compare the current loss value with the historical best loss value, which is the minimum loss value in all previous rounds. If the current loss value is greater than or equal to the historical best loss value, it is considered that the loss value in this round has not decreased, and the continuous non-decreasing counter is incremented by 1; if the current loss value is less than the historical best loss value, the historical best loss value is updated to the current loss value, and the continuous non-decreasing counter is reset to 0.

[0166] When the value of the non-decreasing counter reaches a preset threshold D (e.g., D=5), the current learning rate is decayed to a preset proportion γ of the original value (e.g., γ=0.5), and the non-decreasing counter is reset to 0, allowing training to continue. This scheduling strategy helps the model escape local optima and achieve finer parameter convergence.

[0167] After the progressive parameter tuning is complete, evaluate the performance of the updated model on an independent validation set, including the following two metrics:

[0168] Scoring consistency metric: This measures the consistency between the quantitative score output by the updated model and the consensus label of multiple evaluators (e.g., senior evaluators). The scoring consistency metric can be calculated using the intragroup correlation coefficient (ICC). The closer the metric is to 1, the more consistent the model score is with the expert consensus.

[0169] The consensus label refers to a unified label obtained through statistical aggregation after at least two evaluators with second-level identifiers independently provide correction information for the same training sample. This label serves as the ground truth for evaluating model performance. The specific construction method is as follows:

[0170] At least three evaluators with second-level identifiers are selected to independently provide correction information for the same training sample, including quantitative scores for each evaluation dimension and labels indicating whether behavioral observations are satisfied. For the quantitative scores of each evaluation dimension, the arithmetic mean or median of all evaluators' scores is calculated. For each behavioral observation, majority voting is used to determine whether it is satisfied; that is, if more than half of the evaluators label it as satisfied, it is considered satisfied. To eliminate individual scores with large deviations, abnormal scores that deviate from the group mean by more than a preset threshold (e.g., 2.0 points) can be removed first, and then the mean of the remaining scores is calculated. The final unified label is the consensus label, which is used as a benchmark to verify the performance of the centralized evaluation model.

[0171] Key behavioral feature recognition accuracy: This measures the updated model's ability to correctly identify preset key behavioral features, and can be evaluated using the F1-score. This metric combines precision and recall; a higher value indicates a more reliable judgment of mandatory key operations. If both metrics meet preset performance standards, such as ICC ≥ 0.85 and F1-score ≥ 0.90, the updated model is considered to be performing well and can be deployed as the new current evaluation model, or used as the current evaluation model for the next iteration, for semantic mapping of subsequent text data. If either metric fails to meet the standard, the update is abandoned, the model is restored to the previous version, and a failure log is recorded, including the name of the failed metric, its specific value, and the update round information, for subsequent analysis and optimization.

[0172] The deployed evaluation model is used in the semantic parsing step of the first embodiment. Specifically, during the inference phase, when the text data in S300 fails to identify a behavior observation point through keyword matching, the evaluation model is invoked to replace the calculation process of the sentence embedding model: the text to be matched and the standard operating procedure description text of the behavior observation point are input into the evaluation model, and the evaluation model directly outputs the matching probability of the behavior observation point; if the matching probability exceeds a preset threshold (e.g., 0.75), the identification result of the behavior observation point is marked as identified. That is, in one specific embodiment, semantic similarity matching can be performed using a pre-trained sentence embedding model such as Sentence-BERT; in another specific embodiment, a specially trained evaluation model can be used to directly output the matching probability to replace the calculation process of the sentence embedding model.

[0173] The above verification and deployment mechanisms ensure the robustness of the model update process and avoid the risk of performance regression or missed detection of critical behaviors.

[0174] Furthermore, the method also includes: performing post-event asynchronous verification on the samples to be optimized that have been included in the incremental sample set.

[0175] The post-event asynchronous verification includes: after any sample to be optimized has accumulated at least Q retrospective supervision labels independently provided by evaluation subjects with second-level identifiers, calculating the deviation between the original correction information in the sample to be optimized and the consensus value of the retrospective supervision labels. Here, the retrospective supervision labels refer to the correction information obtained by the evaluation subjects afterward (i.e., after the sample to be optimized has been added to the incremental sample set) by independently re-evaluating the sample; the consensus value is the statistical aggregation result of the retrospective correction information provided by multiple evaluation subjects, for example, using the arithmetic mean for scores and majority voting for behavioral observation point labeling. Here, Q is a positive integer greater than or equal to 2, for example, Q=3.

[0176] If the deviation exceeds a preset deviation threshold—for example, for the rating dimension, the mean absolute error is greater than 2.0 points; for behavioral observation points, the inconsistency rate exceeds 30%—then the sample to be optimized is determined to be an anomalous sample, and its weight in the incremental sample set is reduced (e.g., by multiplying the weight by a decay factor of 0.5) or it is directly removed from the incremental sample set. The preset deviation threshold can be adjusted according to the strictness of the application scenario; the smaller the threshold, the stricter the filtering.

[0177] Through the above-mentioned post-event asynchronous verification mechanism, abnormal monitoring signals caused by systematic bias or accidental errors of individual evaluation subjects can be filtered out, preventing them from polluting the incremental sample set in the long term. This forms a dual protection system with real-time authoritative decision-making and long-term data purification, together with the real-time arbitration rules.

[0178] Furthermore, the method also includes: performing hash calculations on the behavioral observation point identifiers, rule conditions, and key evidence fragments upon which each parameter adjustment is based to generate an immutable verification log. The hash calculation uses a cryptographic hash function such as SHA-256. Each log record is bound to corresponding structured data and stored in an independent verification database. When post-event data integrity verification is required, the hash value of the current data is recalculated and compared with the originally stored hash value: if they match, it proves that the log has not been tampered with since its generation; otherwise, an alarm is triggered and an abnormal event is recorded. Through the above mechanism, end-to-end traceability and tamper-proofing are achieved from the original training samples to the final structured data, meeting the data integrity assurance requirements in high-security scenarios.

[0179] Based on the first embodiment, the second embodiment further achieves the following technical effects by introducing a training and optimization mechanism for the evaluation model:

[0180] 1. Continuous improvement in semantic parsing accuracy

[0181] In the first embodiment, semantic similarity matching can be performed using a general sentence embedding model (such as Sentence-BERT) to calculate text similarity. However, this general model is not specifically trained for the behavior observation point matching task in this invention, leaving room for improvement in matching accuracy. The second embodiment employs a multi-task joint learning strategy, simultaneously optimizing both rating regression and behavior observation point classification tasks. This allows for specialized training of the evaluation model, further improving its semantic matching accuracy during the inference phase compared to the general sentence embedding model. Consequently, it outputs more accurate recognition results when replacing the general model for semantic parsing. Both approaches can be implemented in the first embodiment, with the general model providing the basic solution and the trained evaluation model offering a more precise optimization solution.

[0182] 2. Feedback-driven continuous model optimization

[0183] Building upon the first embodiment, the second embodiment introduces a hierarchical feedback mechanism, enabling the model to continuously optimize based on correction information from the evaluator. The difference between the status flags output during the inference phase (first embodiment) and the results of the manual review by the evaluator is fed back to the training phase (second embodiment) as samples to be optimized, driving model parameter adjustments and forming a continuous evolutionary cycle of "inference-feedback-optimization," making the model increasingly accurate in practical use.

[0184] 3. The dynamic balance between security and automation

[0185] Combining the rule-based fallback mechanism of the first embodiment and the self-assessment mode of the second embodiment, self-assessment is enabled for low-risk routine subjects when the model performance reaches a preset safety threshold, with the model directly outputting results. Simultaneously, a sliding window monitors model performance in real time; if performance deteriorates, the system automatically exits and resumes manual review. This mechanism reduces the workload of manual review while ensuring assessment security.

[0186] 4. Computational efficiency of model optimization

[0187] The second embodiment employs a parameter-efficient fine-tuning technique, updating only some low-rank parameters while keeping the backbone network frozen, reducing the computational and storage overhead of model adjustment and enabling the model to be continuously optimized under limited computing resources.

[0188] 5. Quality assurance of training data

[0189] By deduplicating and denoising the samples to be optimized, performing asynchronous post-processing, and attenuating or removing the weights of abnormal samples, abnormal supervision signals caused by systematic biases or accidental errors of individual evaluators are effectively filtered out, ensuring the quality of incremental training data.

[0190] The first and second embodiments work together to form a complete "training-inference-feedback optimization" technical closed loop: the evaluation model trained by the second embodiment is used in the semantic parsing step of the first embodiment; the difference data between the status identifiers output by the first embodiment and the manual review results serves as samples to be optimized, driving the adjustment of the model parameters in the second embodiment. This closed-loop mechanism enables the evaluation system to be continuously optimized in practical applications. While ensuring the deterministic determination of key behaviors and the traceability of evaluation results, it achieves continuous improvement in evaluation accuracy, meeting the dual requirements of reliability and continuous evolution capability of the training and evaluation system in high-security scenarios.

[0191] Third Embodiment

[0192] The first embodiment described above provides a method for evaluating and processing training data, and the second embodiment provides a method for training and optimizing the evaluation model. Building upon this, this embodiment further provides a method for performance monitoring and switching between autonomous evaluation modes. This method enables autonomous evaluation for low-risk subjects when model performance reaches a preset safety threshold, and automatically exits the mode when performance declines, thereby achieving a dynamic balance between safety and automation.

[0193] In this embodiment, the method further includes:

[0194] S70, obtain the first and second performance metrics of the updated evaluation model on the independent validation set.

[0195] The first performance index represents the consistency of the scores between the updated evaluation model and the consensus labels of multiple evaluators, and is measured by mean absolute error (MAE) or intragroup correlation coefficient (ICC). The second performance index represents the miss rate of the updated evaluation model for preset key behavioral features, that is, the proportion of key behavioral features not identified by the model to the total number of key behavioral features that should be identified by the evaluators.

[0196] In the independent validation set, the consensus label of each sample is predetermined and fixed in the aforementioned manner, and no longer changes with model updates.

[0197] Examples of key behavioral characteristics:

[0198] Declaring minimum fuel levels or a state of emergency corresponds to the behavioral observation point OB 4.8;

[0199] Check the runway visual range and confirm that it meets the minimum operational standards, corresponding to behavior observation point OB 3.2;

[0200] Develop an alternate landing plan that includes at least one alternative landing site, corresponding to behavior observation point OB 5.4;

[0201] It was confirmed that the closure of the runway or navigation facilities mentioned in the NOTAM did not directly affect the release of the aircraft, corresponding to the behavior observation point OB3.2.

[0202] The aforementioned list of key behavioral features is pre-defined during initialization and can be adjusted by the advanced evaluation body based on actual operation. In model performance evaluation, the false negative rate is calculated only for these clearly defined key behavioral features.

[0203] S80, when both the first performance indicator and the second performance indicator meet the preset security threshold, such as MAE≤0.7 and Miss Rate≤3%, the self-evaluation mode is enabled for the part of the input training sample of the training object that belongs to the preset security level category.

[0204] The preset safety level category refers to low-risk routine tasks pre-classified based on task type, emergency level, or deviation from standard operating procedures. Examples include standard flight departure tasks that do not involve emergency handling, fuel criticality, airport closures, or complex airspace coordination. In autonomous assessment mode, the conversion result is directly used as the final output and no longer triggers a review by the assessment body.

[0205] During the autonomous evaluation mode, the actual performance of the updated evaluation model in the current processing batch is monitored in real time using a sliding window. Specifically, for each preset number (e.g., 100) of new training samples processed, the first and second performance metrics are recalculated based on the most recently accumulated preset window size (e.g., 500) of reviewed samples. If any performance metric exceeds the preset safety threshold, the autonomous evaluation mode is immediately exited, and the full review process is resumed; that is, the transformation results of all training samples must trigger a review by the evaluation body. Simultaneously, the reason for exit and the timestamp are recorded, and the autonomous evaluation mode can only be reactivated after the model again meets the preset safety threshold.

[0206] Through the aforementioned self-evaluation mechanism, the system can automatically output evaluation results when the model performance meets the standards, while ensuring evaluation security. This reduces reliance on external manual evaluation and optimizes system operating efficiency.

[0207] The complete technical solution consisting of the first, second, and third embodiments provided by this invention can be deployed in an aviation operation control training management system or a similar job training and assessment platform, serving as the core assessment and optimization engine of the system.

[0208] In practical training scenarios, the method described in the first embodiment can be used to perform modality standardization, semantic parsing, and behavior observation point identification on the multimodal operation data submitted by trainees during the training process. Through a two-layer decision architecture of "rule priority, model supplementation," clear state labels are generated for each evaluation dimension, and structured evaluation data is output. In the semantic parsing stage, the evaluation model trained in the second embodiment can be used to replace the general sentence embedding model for semantic matching. Because this model is specifically trained for behavior observation point matching tasks, its semantic matching accuracy in the inference stage is further improved compared to the general model, thereby enhancing the accuracy of behavior observation point identification.

[0209] The evaluation model training and optimization mechanism described in the second embodiment runs in the background, enabling the model to continuously adjust its parameters based on feedback data from actual use. Specifically, the difference between the structured data output during the inference phase and the correction information from the judging subject, after being determined by the bias tolerance range, is selected as samples to be optimized and included in the incremental sample set, triggering the model parameter adjustment process. As incremental samples continue to accumulate, the semantic parsing accuracy of the model gradually improves in iterations, while the bias tolerance range dynamically narrows as the model performance improves, forming a convergence process where higher accuracy leads to more refined optimization, reducing misjudgments caused by model uncertainty.

[0210] The performance monitoring and self-evaluation mode switching method described in the third embodiment further provides a control mechanism for the model's running state. When the model's performance metrics (scoring consistency, key behavioral feature false negative rate) on the independent validation set meet the preset safety threshold, the self-evaluation mode can be enabled for low-risk routine training subjects, allowing the evaluation results to be directly output by the system without triggering manual review. During operation, the model performance is monitored in real time using a sliding window. Once the performance metrics exceed the safety threshold, the mode is automatically exited and manual review is resumed. This reduces the reliance on external manual evaluation and optimizes system operating efficiency while ensuring evaluation security.

[0211] This invention can be deployed independently in an airline's training management department, or integrated as a module into an existing flight training management system or operations control training platform. Through these deployment methods, this invention automates and intelligently processes training assessments, reduces subjective bias in manual assessments, improves the efficiency and data consistency of the assessment process, and provides systematic and standardized technical support for personnel competency assessments in high-safety positions.

[0212] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in this invention.

[0213] This invention also provides a computer-readable storage medium storing computer-executable instructions for performing the methods described in this invention.

[0214] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0215] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for evaluating and processing training data of aviation operations control personnel, characterized in that, include: S100, acquire the multimodal operation data submitted by the trainee during the aviation operation control training process, the multimodal operation data including voice data, text data, image data and video data; S200, Modality standardization transformation is performed on the multimodal operation data to uniformly convert data of different modalities into text data and structured behavioral event sequences; S300, based on a pre-built behavioral observation point knowledge base, performs semantic parsing on the text data to identify the recognition results of behavioral observation points associated with standard operating procedures from the text data; the behavioral observation point knowledge base contains multiple behavioral observation points, each behavioral observation point corresponds to a specific observable operational behavior, and each behavioral observation point is pre-assigned to one of multiple evaluation dimensions; the evaluation dimension is a category used to measure the operational skill feature dimension of the trainee; S400, based on the identification results of the behavior observation points, determine the status identifiers for each evaluation dimension; the status identifiers are used to characterize whether the training object meets the preset operation requirements in each evaluation dimension; wherein, the status identifiers for each evaluation dimension are determined in the following way: if the structured behavior event sequence satisfies any rule condition in the preset deterministic rule base, the corresponding status identifier is directly output; otherwise, the status identifiers are aggregated by the identification results of the behavior observation points through an aggregation function. S500 outputs structured data containing status identifiers for each evaluation dimension.

2. The method according to claim 1, characterized in that, The modal normalization conversion includes: Perform speech recognition on the speech data and convert it into text data; Optical character recognition is performed on image data to extract text content from the image data as text data; Target detection and action recognition are performed on video data to detect the operation behavior of the training object on the operation interface and generate a structured sequence of behavioral events.

3. The method according to claim 1, characterized in that, The behavior observation point knowledge base is built based on the industry operation standard framework, and each behavior observation point is associated with a standard operating procedure description text and a list of triggering keywords.

4. The method according to claim 3, characterized in that, Semantic parsing of the text data, identifying the recognition results of behavioral observation points associated with standard operating procedures from the text data, including: The text data is segmented and keywords are extracted. If the text data contains a preset trigger keyword for the behavior observation point, the recognition result of the behavior observation point is marked as recognized. If the text data does not contain the preset trigger keyword of the behavior observation point, the semantic similarity between the text data and the standard operating procedure description text of the behavior observation point is calculated. If the semantic similarity exceeds the preset threshold, the recognition result of the behavior observation point is marked as recognized. The identified behavioral observations are used as the identification results.

5. The method according to claim 4, characterized in that, The semantic similarity is calculated using a pre-trained sentence embedding model, which encodes the text data and the standard operating procedure description text into semantic vectors, and calculates the cosine similarity between the semantic vectors of the text data and the standard operating procedure description text.

6. The method according to claim 1, characterized in that, The construction of the deterministic rule base includes: extracting key behavioral features from operating regulations, operating manuals, standard operating procedures or safety notices, assigning a corresponding status identifier to each key behavioral feature, and encoding the key behavioral features and their corresponding status identifiers as rule conditions.

7. The method according to claim 1, characterized in that, The aggregation function can be any one of weighted summation, minimum value taking, or majority voting.

8. The method according to claim 1, characterized in that, The structured data also includes the identification results of each behavior observation point and the behavior observation point identifier associated with each state identifier.

9. The method according to claim 1, characterized in that, In S300, semantic parsing is performed on the text data based on the current evaluation model, which is obtained in the following way: Acquire training samples, and use the current evaluation model to perform feature numerical transformation on the training samples to obtain the transformation result; the training samples are multimodal operation data submitted by the trainee during the aviation operation control training process; Obtain correction information for the same training sample from at least one evaluator, wherein the evaluator has a level identifier; The tolerance range for deviation is determined based on the level identifier of the evaluation subject; Calculate the deviation between the conversion result and the correction information, and determine whether the deviation exceeds the deviation tolerance range; If the error exceeds the limit, the training samples and the corrected data of the correction information will be used as samples to be optimized. The correction data refers to the specific changes made by the evaluation body to the parts of the conversion result that are inconsistent with the correction information; Construct an incremental sample set based on the samples to be optimized; When the incremental sample set meets the preset triggering conditions, the parameters of the current evaluation model are adjusted using the incremental sample set to obtain the updated evaluation model, and the updated evaluation model is used as the current evaluation model for the next iteration.

10. The method according to claim 9, characterized in that, The level identifier of the evaluation subject includes at least a first level identifier and a second level identifier, and the evaluation authority of the second level identifier is higher than that of the first level identifier; in the sample to be optimized, the correction data provided by the evaluation subject with the second level identifier is assigned a higher weight than the correction data provided by the evaluation subject with the first level identifier.

Images