Digital navigation intelligence knowledge classification and structured analysis evaluation method
By constructing a NOTAM dataset and knowledge graph, and combining a large language model and a quality assessment model, the problems of low parsing efficiency and inaccurate assessment of NOTAMs were solved, achieving deep semantic understanding and automated processing of NOTAMs, thus improving aviation safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2025-12-09
- Publication Date
- 2026-04-28
AI Technical Summary
Existing technologies are insufficient for efficient and automated processing of Notices to Airmen (NOTAMs). They suffer from low parsing efficiency, strong subjectivity in manual parsing, and difficulty in in-depth correlation analysis and quantitative assessment, leading to potential aviation safety hazards.
By establishing a navigation notification dataset, we use multi-layer structured prompt words and a large language model to jointly extract entities and events, construct a knowledge graph, and use an entity-event quality assessment model for quantitative evaluation, including syntactic, logical, and domain-level verification. We also combine the AHP-entropy weight method to determine the weights of the evaluation indicators.
It achieves deep semantic understanding and association analysis of NOTAM, improves parsing efficiency, provides scientific and objective evaluation results, reduces the risk of human error, and outputs standardized machine-readable data.
Smart Images
Figure CN121936451A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent processing and knowledge graph technology for aeronautical information, specifically to a method for classifying and structured analyzing and evaluating digital aeronautical information knowledge. Background Technology
[0002] Notices to Airmen (NOTAMs) are a critical source of information for ensuring safe aviation operations, used to promptly disseminate temporary changes or hazardous situations regarding airspace, airport facilities, and service procedures. With the continued growth of global air traffic, the number of NOTAMs issued has surged, and their texts are characterized by semi-structured formats, numerous technical terms, strong semantic implicitness, and multiple event couplings. For example, a single NOTAM might simultaneously describe runway closure and lighting downgrades, accompanied by complex spatiotemporal constraints.
[0003] Currently, the automated processing of NOTAMs faces the following technical bottlenecks: Low efficiency: The massive amount of NOTAM information requires professionals to invest significant time in screening and interpreting it. In high-density operating environments, this can easily lead to excessive workload and errors. Human analysis is highly subjective, and fatigue, negligence, or misunderstanding of complex clauses can easily result in the omission or misjudgment of key information, posing security risks. Lack of deep correlation: Humans struggle to quickly and deeply correlate individual NOTAMs with airport operations, flight schedules, airspace status, etc., making it difficult to effectively assess their cascading impacts. Difficulty in quantitative evaluation: The lack of automated methods for evaluating the quality of NOTAM information itself, such as completeness, accuracy, and consistency, prevents the provision of reliable quantitative evidence for intelligence dissemination and use.
[0004] While some existing technologies attempt to resolve NOTAMs based on rules or simple machine learning models, these methods generally suffer from poor generalization ability, insufficient understanding of complex semantics, low accuracy in entity and event extraction, and isolated modules lacking coordination. In particular, existing technologies lack an integrated solution capable of simultaneously achieving high-precision information extraction, semantic association, and quality assessment throughout the entire flight phase.
[0005] Therefore, there is an urgent need in this field for a technical solution that can automatically, accurately, and deeply understand the semantics of NOTAMs and quantitatively evaluate their information quality in order to improve the efficiency of aviation intelligence utilization and flight safety. Summary of the Invention
[0006] In view of the above problems, the present invention provides a method for classifying and structured parsing and evaluating digital aeronautical information knowledge, which solves the technical problems of low generalization ability and accuracy of the parsing of aeronautical notices in the prior art.
[0007] This invention provides a method for classifying and structured analyzing and evaluating digital aeronautical information knowledge, comprising the following steps: Step S1: Establish a NOTAM dataset based on historical NOTAM messages. The NOTAM dataset has multiple samples, each sample including instructions, notification text, entities, and events. The entities include various types and attributes of airport facilities, and the events include the types and attributes of the operational status of airport facilities. Step S2: Perform entity-event joint extraction on the announcement text of the sample, including: establishing multi-layer structured prompt words, calling a large language model based on the prompt words, verifying the output results of the large language model, and obtaining the entity and event prediction values corresponding to the announcement text; Step S3: Construct a knowledge graph based on the predicted values of entities and events in the sample; Step S4: Establish an entity-event quality assessment model. The entity-event quality assessment model can evaluate multiple corresponding assessment indicators based on the weighted values of entity and event predictions to obtain a quality score. Step S5: Perform entity-event joint extraction, knowledge graph construction, and entity-event quality assessment model processing on the notification text to be evaluated to obtain a quality score.
[0008] Preferably, in step S1, the entity includes various types of airport facilities, specifically including: airport, runway, lighting system, RVR facility, terminal area program, ILS and navigation beacon; the event includes types of operational status of airport facilities, specifically including unavailability event, degraded event, restricted event, change event, recovery event, and alert event.
[0009] Preferably, in step S2, the step of establishing multi-layered structured prompt words and calling the large language model based on the prompt words specifically includes: (1) Establish a multi-layered structured prompt word template, including five layers, namely: The system role definition layer is used to identify the large language model as an aviation intelligence expert; Task description layer: Used to detail the requirements for entity recognition and event extraction; Domain knowledge injection layer: used to describe entity types, event types, and attribute specifications that include NOTAMs; Output format constraint layer: used to require output to be in standard JSON format; Example Demo Layer: Provides parsing examples of typical NOTAMs; (2) Fill the multi-layer structured prompt word template to obtain prompt words; combine the announcement text of the sample with the prompt words to form a complete input instruction, and then call the inference interface of the large language model.
[0010] Preferably, in step S2, the step of verifying the output result of the large language model specifically includes: (1) Syntax-level validation: The output is checked using a JSON parser and a JSON Schema validator to ensure that the JSON syntax is correct; (2) Logic-level verification, including checking the consistency of temporal logic and spatial logic; (3) Domain-level verification, including: obtaining the NOTAM knowledge base, which includes the specification attributes of standard entities and standard events; The domain-level validation confidence score is obtained using the following expression:
[0011] in, This represents the confidence level, and Z is the normalization constant. This represents the weight of the i-th attribute to be compared, where m is the total number of attributes. This is a function for calculating semantic similarity. These are the canonical attributes for standard entities and standard events in the knowledge base. Let i be the i-th attribute to be compared; (4) If the output of the large language model fails the syntax-level check, logic-level check, or domain-level check, then the large language model is used again for reasoning until it passes the check.
[0012] Preferably, step S3 specifically includes: Step S3-1: Construct the NOTAM domain ontology, including: determining the specific parameter entries of the NOTAM domain ontology based on the type and attributes of airport facilities and the type and attributes of the operational status of airport facilities; Step S3-2: Construct a knowledge graph corresponding to the predicted values of the entities and events based on the ontology of the navigation notice domain, including: treating each entity and event as a node, and establishing connections based on the relationship between the event node and the affected entity node.
[0013] Preferably, step S4 specifically includes: Step S4-1: Determine the evaluation metrics for the entity-event quality assessment model, specifically including: perfect match accuracy, micro F1 score, macro F1 score, key attribute accuracy, and illusion rate; Step S4-2: Use the analytic hierarchy process (AHP) to determine subjective weights and the entropy weight method to determine objective weights. Step S4-3: Weight the evaluation indicators based on the subjective and objective weights to obtain a quality score.
[0014] Preferably, in step S4-1, the expression for the perfect match accuracy is:
[0015] in, This indicates the exact match accuracy. This indicates the number of completely identical evaluation samples. Indicates the total number of evaluation samples; The micro F1 score is calculated based on the overall true positive, false positive, and false negative scores of all samples. The expression for the macroscopic F1 score is:
[0016] in, Indicates the macroscopic F1 score, The total number of all entities or event categories. Indicates the first F1 score for each entity or event category; The accuracy of the key attributes is determined based on the recognition accuracy of the core operational attributes of runway number, event type, CAT level, and ICAO airport code. The expression for the hallucination rate is:
[0017] in, Indicates the rate of hallucination. This indicates the number of assessment samples that experienced hallucinations.
[0018] Preferably, step S4-2 specifically includes: The evaluation indicators were subjectively compared, specifically including: pairwise comparisons of the five indicators to determine the relative importance of each pair, quantitative scoring using the 1-9 scale, and construction of a judgment matrix. ; Calculate the largest eigenvalue of the judgment matrix The corresponding feature vectors are normalized, and the resulting normalized feature vectors are the subjective weights of each indicator. ; Consistency ratio Make a judgment. If the condition is met, the judgment is passed; otherwise, a subjective evaluation is performed again to obtain subjective weights until the condition is met. condition.
[0019] Preferably, the consistency ratio The expression is:
[0020]
[0021] in, Indicates the consistency ratio. Indicators of consistency This represents the average random consistency index. This represents the largest eigenvalue of the judgment matrix. This indicates the total number of indicators.
[0022] Preferably, in step S4-3, The expression for quality rating is:
[0023] in, Indicates quality score. Indicates the exact match accuracy. Represents the microscopic F1 score, Indicates macroscopic F1 score, Indicates the accuracy of key attributes and Indicates the rate of hallucination; To evaluate the weight of indicators Each column; The weight of the evaluation indicators The expression is:
[0024] in, This indicates the weights of the evaluation indicators corresponding to the analytic hierarchy process (AHP). This indicates the weight of the evaluation index corresponding to the entropy weight method. This is the preference coefficient.
[0025] Compared with the prior art, the present invention has at least the following beneficial effects: (1) Deep semantic understanding and association analysis are realized: By constructing a knowledge graph of NOTAMs, this invention achieves a leap from information extraction to knowledge association and reasoning. It can automatically discover and establish complex semantic associations between different NOTAMs and between NOTAMs and elements such as airports, airspace, and flights, thereby supporting graph-based influence propagation analysis.
[0026] (2) The evaluation results are scientific, objective, and interpretable: This invention uses a multi-dimensional quantitative evaluation index system, especially the key indicator of illusion rate, which can effectively quantify and monitor the risk of model fabrication of information, providing a key guarantee for the reliability of the output results. The AHP-entropy weight method is used to determine the weight of the indicators, which scientifically balances the subjective judgment of domain experts' experience with the contribution of the objective distribution of evaluation data, so that the final comprehensive quality score is both in line with business cognition and respects data facts, and the evaluation conclusion is more convincing and interpretable.
[0027] (3) Forming an automated closed loop and significantly improving efficiency: This invention provides an end-to-end integrated solution from raw message input to the generation of final structured reports and quality scores. This solution automates the entire process of aeronautical information processing, effectively reducing safety risks caused by human fatigue and negligence.
[0028] (4) The system is highly practical and has broad application prospects: The output results of this invention, such as structured JSON, knowledge graph and quality score, are standardized machine-readable data and have strong downstream system integration capabilities. Attached Figure Description
[0029] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.
[0030] Figure 1 The flowchart illustrates the digital navigation information knowledge classification and structured analysis evaluation method provided by this invention. Detailed Implementation
[0031] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.
[0032] This invention first constructs a dataset based on historical NOTAMs (Notices to Airmen), with each sample containing the notice text, entities, and events. It then uses a multi-layered structured prompt word input to a large language model and validates the output to achieve joint entity-event extraction. The large language model's extraction process can obtain categories such as impact level and flight phase. Simultaneously, an entity-event quality assessment model is established to weighted evaluate multiple quality indicators to obtain a quality score. The notices to be evaluated are sequentially extracted, a knowledge graph is built, and quality assessment is performed, outputting a quality score. This invention can effectively improve the accuracy of NOTAM parsing.
[0033] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution is provided through a specific embodiment, such as... Figure 1 As shown, a method for classifying and structured analyzing digital aeronautical information knowledge is disclosed, and the specific implementation steps are as follows: Step S1: Establish a NOTAM dataset based on historical NOTAM messages. The NOTAM dataset has multiple samples, each sample including instructions, notification text, entities, and events. The entities include various types and attributes of airport facilities, and the events include the types and attributes of the operational status of airport facilities. In this step, the present invention first collects NOTAM (Notice to Airmen) messages stored over a period of time as the raw data source. NOTAM messages can be recorded by airports or aeronautical information centers, and contain several standard fields, including: lines Q, A, B, C, and E. Line E represents the notification text, which is in free text format and describes in detail specific facility status changes, restrictions, or other important information. The present invention extracts line E of the NOTAM messages as the notification text.
[0034] During the dataset construction process, the entities and events corresponding to the notification texts of each historical NOTAM message were labeled. This invention first determined the entity types of NOTAM messages, including seven categories: airports, runways, lighting systems, RVR facilities, terminal area programs, ILS, and navigation beacons.
[0035] Each entity has specific attributes, detailed below: Airport entity: ICAO code, status, related types, time attributes.
[0036] Runway entity: airport, runway number, status, location information, and published distance.
[0037] The lighting system entities include the unique attributes of the runway centerline lights, edge lights, approach lights, and ground lights, such as their identification color, intensity, and CAT rating.
[0038] RVR facility entities: corresponding runway, status, and unavailable location.
[0039] Terminal area program entities: program type, name, status.
[0040] ILS Entities: Device Type, Status, CAT Class, Program Name.
[0041] Navigation station entity: device type, status, identification frequency.
[0042] This invention defines the event types of NOTAM messages, including six categories: unavailability event, degradation event, restriction event, change event, recovery event, and notification event.
[0043] Each event has specific attributes, detailed below: Unavailable events: completely unavailable or partially unavailable, including the scope, degree, and reason for unavailability.
[0044] Degradation event: Performance degradation standards before and after degradation, and the scope of impact.
[0045] Restricted events: Distinguish between shutdown and operation restrictions, and specify the restriction conditions and the degree of impact.
[0046] Change Event: Change details, post-change information, and references.
[0047] Recovery event: recovery status, recovery time.
[0048] Warning events: warning messages, solutions, and alternatives.
[0049] In this step, the historical NOTAM message announcement text is analyzed to obtain all entity types described in the text, as well as the corresponding attribute information such as type, name, and identification code of the entity; and the event types described in the text and the corresponding attribute information of the event are also obtained.
[0050] The NOTAM dataset established by this invention contains 9,671 samples obtained from historical NOTAM messages, and their specific distribution is shown in Table 1.
[0051] Table 1
[0052] Through the above annotations, this invention establishes a structured NOTAM dataset. The dataset contains several samples, each consisting of three parts: instructions, notification text, entity sets, and event sets. The dataset includes high-impact events such as complete runway closures, medium-impact events such as partial downgrades of lighting systems, and low-impact events such as minor adjustments to navigation station parameters. This diverse data collection strategy allows the dataset to fully reflect the complexity and diversity of NOTAMs in real-world business scenarios. The dataset not only preserves the original text information but also establishes a structured knowledge representation through manual annotation.
[0053] The instructions in the dataset samples of this invention represent structured prompts for inputting large models, which will be described in detail in subsequent steps.
[0054] Step S2: Perform entity-event joint extraction on the announcement text of the sample, including: establishing multi-layer structured prompt words, inputting the prompt words into a large language model, verifying the output results of the large language model, and obtaining the entity and event prediction values corresponding to the announcement text; In this step, the present invention establishes prompt words through prompt word templates and announcement texts, and guides the large language model to directly output entity and event prediction values in a preset structured format through multi-layered structured prompt words, as described in detail below.
[0055] In this step, the present invention first determines the prompt word template, which includes a five-layer architecture, namely: (1) System role definition layer: Clarify the role of the large language model as an aviation intelligence expert. For example, it can require the meaning of professional terms in the field of NOTAM. (2) Task description layer: details the specific requirements for entity recognition and event extraction. For example, the model needs to identify seven types of entities and six types of events in a clear natural language description. It is required to output preset attribute fields for each identified entity and event, and to establish a clear binding relationship between the event and the entity it affects. (3) Domain knowledge injection layer: includes entity types, event types, and attribute specifications unique to NOTAM; (4) Output format constraint layer: Requires output in standard JSON format to ensure machine readability; (5) Example Demonstration Layer: Provides parsed examples of typical NOTAMs to enhance the model’s few-shot learning ability.
[0056] The following example illustrates how to determine the specific content of prompts based on a prompt template. Taking RVR facility retrieval as an example, the structured prompts are as follows: System Role Definition Layer: "You are an AI assistant specializing in handling Notices to Airmen (NOTAMs) related to a certain type. Your task is to return the results as an array of JSON objects, including the attributes corresponding to that type from the NOTAM text, the events that occurred, and the attributes of those events." Task description layer: "Extract the affected runway information from the NOTAM text and describe the availability status of its RVR devices"; Domain knowledge injection layer: Completely Unavailable Rule: When expressions such as "RVR U / S" appear in NOTAM, it is marked as completely unavailable. Segment unavailability rules: Identify specific unavailable segments such as TDZ, MID, STOP END, and THR. Multi-runway handling rules: Output separate rules for multi-runway scenarios such as RWY 10 / 28. "; Output format constraint layer: Runway: Runway identifier, such as '21', '01L' RVR Status: Unavailable section description, including "All Unavailable", "Grounding Strip Unavailable", etc. Data type: Fixed as 'rvr' "; Example demonstration layer: "A typical example of NOTAM parsing is as follows:" { Input: 'RWY 21 RVR U / S' Output: [ { "Runway": "21", "RVR Status": "All Unavailable", "datatype": "rvr"}], 'RVR Status': 'All Unavailable' 'datatype': 'rvr' }".
[0057] In this step, the present invention combines the notification text of the sample with the prompt words to form a complete input instruction. Then, it calls the inference interface of the large language model. After receiving the input, the large language model, based on its pre-trained language understanding ability and world knowledge, combined with the task description, domain constraints and example demonstrations provided in the prompt words, generates a JSON format output containing entity and event information.
[0058] To improve the reliability and accuracy of the extraction results, this embodiment performs multi-level automatic verification on the output of the large language model.
[0059] The first layer is syntax-level validation, which uses a JSON parser and a JSON Schema validator to check the output, ensuring that the JSON syntax is correct, all required fields are present, and the field data types meet the requirements.
[0060] The second layer is logic-level validation, which includes checking the consistency of spatial logic and whether the attributes of events conform to semantic expression. Specifically, in terms of spatial logic validation, the system checks whether the runway number matches the airport code. For example, if runway 36R does not exist in the runway configuration database of an airport, it is marked as an anomaly if the model outputs that runway. Additionally, it checks whether the location coordinates of the navigation beacons are within a reasonable geographical range and whether the ILS facilities are correctly associated with the corresponding runways.
[0061] The third layer is domain-level validation, which verifies the rationality of entity attributes and event parameters based on the NOTAM knowledge base. Specifically, the NOTAM knowledge base is first acquired. This knowledge base consists of structured and standardized NOTAM entries and corresponding entity models compiled from aviation intelligence sources and stored in a relational database or knowledge graph format. When validating a NOTAM to be verified or entity / event parameters extracted by the system, each parameter of the object to be verified is first mapped to the corresponding standardized attributes in the knowledge base. Then, the similarity of each attribute i is calculated, and a weight is assigned to each attribute to reflect its importance to the final confidence level. The expression is:
[0062] in, This represents the confidence level, and Z is the normalization constant. This represents the weight of the i-th attribute to be compared, where m is the total number of attributes. This is a function for calculating semantic similarity. These are the canonical attributes for standard entities and standard events in the knowledge base. Let i be the i-th attribute to be compared.
[0063] In practice, if the output of the large language model fails the automatic verification, the large language model will be used again for reasoning until it passes the automatic verification.
[0064] After the above processing, this step finally yields structured entity and event prediction values. For each input NOTAM text, the system generates a JSON data structure containing a complete list of entities and events. Each entity includes attributes such as type, name, and identifier code, while each event includes attributes such as type, target entity, time range, status description, and reason explanation. Furthermore, each extraction result is accompanied by a verification status and confidence score, providing a reliable data foundation for subsequent quality assessment.
[0065] Step S3: Construct a knowledge graph based on the predicted values of entities and events in the sample; In this step, the specific steps of the present invention include: (1) Constructing the NotaM domain ontology Based on the needs of aviation operations, a domain ontology is constructed that includes the following core concepts and relationships: Facilities include: airports, runways, lighting systems, RVR facilities, ILS, navigation beacons, etc. Flight phases are categorized into three phases: airport / ground operations, terminal area arrivals and departures, and en route. Event categories: unavailable, downgraded, restricted, changed, restored, prompts, and their hierarchical relationships; Attribute types: time attributes (start / end time, repetition period), spatial attributes (coordinates, height range), performance attributes (CAT level, operational capability), impact attributes, cause attributes, etc.
[0066] The ontology defines entity types, attribute types, and the types of relationships between entities and events, and between entities, to guide subsequent graph construction.
[0067] (2) Construct a knowledge graph corresponding to the predicted values of the entities and events based on the NOTAM domain ontology. The JSON output of the entity and event information generated in the preceding steps is converted into a knowledge graph instance. Specifically, each entity and event is first treated as a node.
[0068] The relationships between event nodes and affected entity nodes are established by creating edges. Specifically, this includes establishing "influence" or "constraint" relationships between event nodes and affected entity nodes, explicitly indicating which entity(s) each event affects. Further relationships such as "located in," "component," and "dependent" are established between entity nodes; for example, a runway is located at an airport, an ILS system serves a runway, and a lighting system is installed on a runway. Additionally, relationships such as "substitute" and "backup" are established between similar entity nodes; for example, a backup runway can be used when a runway is closed. Through these relationships, a multi-layered, multi-dimensional knowledge network is formed.
[0069] In addition to entities and event nodes, the knowledge graph also fully integrates multi-dimensional attribute information of events, including temporal attributes (such as start time, end time, and repetition period), spatial attributes (such as specific coordinates and altitude range), impact attributes, and causal attributes. These attributes are incorporated into the graph as supplementary information for event nodes, enriching the semantic expression of events. This allows the knowledge graph to not only display the direct relationship between events and entities but also accurately reflect the spatiotemporal context and specific causes of events, thereby supporting more granular queries and reasoning and enhancing the application value of the knowledge graph in actual aviation operations.
[0070] Step S4: Establish an entity-event quality assessment model. The entity-event quality assessment model can evaluate multiple corresponding quality indicators based on the weighted values of entity and event predictions to obtain a quality score. In this step, the present invention establishes an entity-event quality assessment model to provide quantitative assurance for the reliability of the output results. The assessment indicators include: perfect match accuracy, micro F1 score, macro F1 score, key attribute accuracy, and illusion rate, which are described in detail below.
[0071] Exact match accuracy evaluates the proportion of samples whose complete JSON structure output by the model is completely identical in content to the standard answer. The formula is as follows:
[0072] in, This indicates the exact match accuracy. This indicates the number of completely identical evaluation samples. This indicates the total number of samples evaluated.
[0073] The micro F1 score, calculated based on the total true positives, false positives, and false negatives of all samples, is used to evaluate the overall recognition accuracy of the model. Its expression is:
[0074] in, TP represents the micro F1 score, FP represents the number of correct answers, FN represents the number of fabricated answers, and FN represents the number of missing answers.
[0075] The macro F1 score is the arithmetic mean of the F1 scores across all entity or event categories. It is used to evaluate the model's average ability to identify each category. The calculation formula is:
[0076] in, Indicates the macroscopic F1 score, The total number of all entities or event categories. Indicates the first The F1 score for each entity or event category.
[0077] Key attribute accuracy refers to the accuracy rate in identifying core operational attributes such as runway number, event type, CAT level, and ICAO airport code. When calculating key attribute accuracy, the system checks whether all key attributes for each sample are correct. Only when all key attributes match the standard answer is the sample counted as having all key attributes correctly. Key attribute accuracy equals the number of samples with all key attributes correctly divided by the total number of samples.
[0078] The illusion rate assesses the extent to which the model generates false information not mentioned in the original NOTAM text, and is calculated using the following formula:
[0079] in, Indicates the rate of hallucination. This indicates the number of assessment samples that experienced hallucinations.
[0080] For the above evaluation indicators, this embodiment uses a combination of the Analytic Hierarchy Process (AHP) and the entropy weight method to determine the weight of each indicator, as expressed in the following expression:
[0081] in, Indicates the weight of the evaluation indicators. This indicates the weights of the evaluation indicators corresponding to the analytic hierarchy process (AHP). This indicates the weight of the evaluation index corresponding to the entropy weight method. This is a preference coefficient used to adjust the weighting of subjective experience and objective data in the weighting distribution. It can be adjusted according to the emphasis of the actual application scenario; for example, in scenarios emphasizing expert authority, it can be set to... .
[0082] This combined approach takes into account both the subjective judgment of domain experts and the objective characteristics of the data itself, resulting in a more reasonable and balanced weight allocation.
[0083] (1) Determining subjective weights using the analytic hierarchy process : In this step, a subjective comparison of the evaluation indicators is required. Specifically, this includes: comparing each of the five indicators pairwise to determine the relative importance of each pair, and using a 1-9 scale for quantitative scoring. Construct a judgment matrix :
[0084] Among them, matrix elements Representing indicators relative to indicators Importance, indicators relative to indicators Importance, indicators relative to indicators Importance and Indicators relative to indicators The importance of.
[0085] For the judgment matrix The eigenvalue method is used to solve the problem and calculate the largest eigenvalue of the judgment matrix. The corresponding feature vectors are normalized, and the resulting normalized feature vectors are the subjective weights of each indicator. .
[0086] To ensure consistency in judgments, the consistency ratio needs to be calculated, expressed as:
[0087]
[0088] in, Indicates the consistency ratio. Indicators of consistency This represents the average random consistency index. This represents the largest eigenvalue of the judgment matrix. This indicates the total number of indicators.
[0089] This invention addresses the consistency ratio Make a judgment. If the judgment is correct, the assessment is passed; otherwise, a new subjective evaluation is required to obtain the subjective weight.
[0090] (2) Determining objective weights using the entropy weight method
[0091] In this step, the present invention first establishes a data matrix based on the evaluation indicators of each sample. , elements Indicates the first The sample at the th Scores on each indicator.
[0092] Then the data matrix Standardization is performed, based on a standardized data matrix. The information entropy of each indicator is calculated using the elements, expressed as:
[0093]
[0094] in, This represents the information entropy of the j'-th indicator. Represents the total number of samples. This represents the probability of the i'th sample on the j'th indicator. This represents the standardized value of the i'th sample on the j'th indicator. Finally, the weights of the indicators are determined by information entropy, expressed as:
[0095] in, This represents the weight of the j'-th indicator. This represents the information entropy of the k-th indicator. This indicates the total number of indicators.
[0096] All Obtaining objective weights by splicing The smaller the entropy value of an indicator, the greater its degree of variation, the more information it provides, and the greater its weight.
[0097] Finally, this invention uses a weighted evaluation of multiple quality indicators to obtain a quality score, expressed as:
[0098] in, Indicates quality score. Indicates the exact match accuracy. Represents the microscopic F1 score, Indicates macroscopic F1 score, Indicates the accuracy of key attributes and Indicates the rate of hallucination. The columns are for evaluating the weights of the indicators.
[0099] Through the above steps, this invention establishes a multi-dimensional quantitative evaluation index system that can balance the subjective judgment of domain experts' experience with the contribution of the objective distribution of evaluation data, so that the final comprehensive quality score is both in line with business understanding and respects data facts.
[0100] Step S5: Perform entity-event joint extraction, knowledge graph construction, and entity-event quality assessment model processing on the notification text to be evaluated to obtain a quality score.
[0101] In this step, the present invention processes the notification text to be evaluated to obtain a quality score and generate an evaluation report.
[0102] Specifically, the entity-event joint extraction process described above can be adopted. The system preprocesses the original message, including field parsing, text normalization, and abbreviation mapping. Then, the preprocessed text is concatenated with a predefined structured prompt word template to form a complete input command and call the large language model interface. After the large language model returns JSON output, the system performs syntax parsing and executes multi-level validation, including syntax-level validation, logic-level validation, domain-level validation, and automatic correction, finally obtaining the validated entity and event prediction values.
[0103] Then, a knowledge graph is built based on the predicted values of entities and events. The entity-event quality assessment model established in the previous steps is then activated. Multiple quality indicators are evaluated based on the weighted values of entity and event predictions to obtain a quality score.
[0104] Finally, a NOTAM structured analysis report is generated from the quality scores. In some embodiments, the analysis report may record the final quality score.
[0105] While the specific embodiments of the present invention depict actions or steps in a particular order, this should be understood as requiring such actions or steps to be performed in the shown specific order or sequential order, or requiring all illustrated actions or steps to be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations. The above descriptions are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention.
[0106] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for classifying and structured analyzing digital aeronautical information knowledge, characterized in that, Includes the following steps: Step S1: Establish a NOTAM dataset based on historical NOTAM messages. The NOTAM dataset has multiple samples, each sample including instructions, notification text, entities, and events. The entities include various types and attributes of airport facilities, and the events include the types and attributes of the operational status of airport facilities. Step S2: Perform entity-event joint extraction on the announcement text of the sample, including: establishing multi-layer structured prompt words, calling a large language model based on the prompt words, verifying the output results of the large language model, and obtaining the entity and event prediction values corresponding to the announcement text; Step S3: Construct a knowledge graph based on the predicted values of entities and events in the sample; Step S4: Establish an entity-event quality assessment model. The entity-event quality assessment model can evaluate multiple corresponding assessment indicators based on the weighted values of entity and event predictions to obtain a quality score. Step S5: Perform entity-event joint extraction, knowledge graph construction, and entity-event quality assessment model processing on the notification text to be evaluated to obtain a quality score.
2. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 1, characterized in that, In step S1, the entities include various types of airport facilities, specifically including: airports, runways, lighting systems, RVR facilities, terminal area procedures, ILS, and navigation beacons; the events include types of operational status of airport facilities, specifically including unavailability events, degraded events, restricted events, change events, recovery events, and alert events.
3. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 2, characterized in that, In step S2, the step of establishing multi-layered structured prompt words and calling the large language model based on the prompt words specifically includes: (1) Establish a multi-layered structured prompt word template, including five layers, namely: The system role definition layer is used to identify the large language model as an aviation intelligence expert; Task description layer: Used to detail the requirements for entity recognition and event extraction; Domain knowledge injection layer: used to describe entity types, event types, and attribute specifications that include NOTAMs; Output format constraint layer: used to require output to be in standard JSON format; Example Demo Layer: Provides parsing examples of typical NOTAMs; (2) Fill the multi-layer structured prompt word template to obtain prompt words; combine the announcement text of the sample with the prompt words to form a complete input instruction, and then call the inference interface of the large language model.
4. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 3, characterized in that, Step S2, specifically the step of validating the output of the large language model, includes: (1) Syntax-level validation: The output is checked using a JSON parser and a JSON Schema validator to ensure that the JSON syntax is correct; (2) Logic-level verification, including checking the consistency of temporal logic and spatial logic; (3) Domain-level verification, including: obtaining the NOTAM knowledge base, which includes the specification attributes of standard entities and standard events; The domain-level validation confidence score is obtained using the following expression: in, This represents the confidence level, and Z is the normalization constant. This represents the weight of the i-th attribute to be compared, where m is the total number of attributes. This is a function for calculating semantic similarity. These are the canonical attributes for standard entities and standard events in the knowledge base. Let i be the i-th attribute to be compared; (4) If the output of the large language model fails the syntax-level check, logic-level check, or domain-level check, then the large language model is used again for reasoning until it passes the check.
5. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 4, characterized in that, Step S3 specifically includes: Step S3-1: Construct the NOTAM domain ontology, including: determining the specific parameter entries of the NOTAM domain ontology based on the type and attributes of airport facilities and the type and attributes of the operational status of airport facilities; Step S3-2: Construct a knowledge graph corresponding to the predicted values of the entities and events based on the ontology of the navigation notice domain, including: treating each entity and event as a node, and establishing connections based on the relationship between the event node and the affected entity node.
6. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 5, characterized in that, Step S4 specifically includes: Step S4-1: Determine the evaluation metrics for the entity-event quality assessment model, specifically including: perfect match accuracy, micro F1 score, macro F1 score, key attribute accuracy, and illusion rate; Step S4-2: Use the analytic hierarchy process (AHP) to determine subjective weights and the entropy weight method to determine objective weights. Step S4-3: Weight the evaluation indicators based on the subjective and objective weights to obtain a quality score.
7. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 6, characterized in that, In step S4-1, the expression for the perfect match accuracy is: in, This indicates the exact match accuracy. This indicates the number of completely identical evaluation samples. Indicates the total number of evaluation samples; The micro F1 score is calculated based on the overall true positive, false positive, and false negative scores of all samples. The expression for the macroscopic F1 score is: in, Indicates the macroscopic F1 score, The total number of all entities or event categories. Indicates the first F1 score for each entity or event category; The accuracy of the key attributes is determined based on the recognition accuracy of the core operational attributes of runway number, event type, CAT level, and ICAO airport code. The expression for the hallucination rate is: in, Indicates the rate of hallucination. This indicates the number of assessment samples that experienced hallucinations.
8. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 7, characterized in that, Step S4-2 specifically includes: The evaluation indicators were subjectively compared, specifically including: pairwise comparisons of the five indicators to determine the relative importance of each pair, quantitative scoring using the 1-9 scale, and construction of a judgment matrix. ; Calculate the largest eigenvalue of the judgment matrix The corresponding feature vectors are normalized, and the resulting normalized feature vectors are the subjective weights of each indicator. ; Consistency ratio Make a judgment. If the condition is met, the judgment is passed; otherwise, a subjective evaluation is performed again to obtain subjective weights until the condition is met. condition.
9. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 8, characterized in that, The consistency ratio The expression is: in, Indicates the consistency ratio. Indicators of consistency This represents the average random consistency index. This represents the largest eigenvalue of the judgment matrix. This indicates the total number of indicators.
10. The method for classifying and structured analyzing digital aeronautical information knowledge according to claim 9, characterized in that, In step S4-3, The expression for quality rating is: in, Indicates quality score. Indicates the exact match accuracy. Represents the microscopic F1 score, Indicates macroscopic F1 score, Indicates the accuracy of key attributes and Indicates the rate of hallucination; To evaluate the weight of indicators Each column; The weight of the evaluation indicators The expression is: in, This indicates the weights of the evaluation indicators corresponding to the analytic hierarchy process (AHP). This indicates the weight of the evaluation index corresponding to the entropy weight method. This is the preference coefficient.