Avionics system evaluation index system construction method based on large language model agent

Abstract

The invention discloses an avionics system evaluation index system construction method based on a large language model agent. The avionics system evaluation index system construction method comprises the steps that a knowledge extraction module performs knowledge extraction on a design document of an avionics system by using the large language model agent and preliminarily constructs a knowledge graph; the reflection verification module automatically checks and corrects entities and relationships in the knowledge graph based on a large language model agent in combination with a learning normal form, and finally outputs an optimized knowledge graph; and the index generation module automatically generates an evaluation index for each entity node in the optimized knowledge graph based on a large language model agent to form an avionics system evaluation index system. According to the method, extraction, induction and system construction of avionics system evaluation indexes are automatically completed, and the method has the advantages that manual decomposition is not needed, the process is traceable, the coverage is wide, and the efficiency is high.

Description

Technical Field

[0001] This invention belongs to the field of avionics system evaluation technology, specifically relating to a method for constructing an avionics system evaluation index system based on a large language model intelligent agent. Background Technology

[0002] Currently, the construction of evaluation index systems for avionics systems mainly adopts an expert-experience-driven approach. The specific process generally includes expert discussions, manual summarization of system functions and failure modes, and definition of each level of index. This method relies on multiple rounds of manual meetings, document compilation, and human judgment, making the process complex and time-consuming. Because it is difficult for experts to reach a complete consensus, evaluation standards become inconsistent, and the standardization of the process and the traceability of the results are weak. Furthermore, as avionics systems become increasingly complex, traditional manual methods of constructing and maintaining index systems are difficult to expand efficiently and cannot guarantee comprehensiveness and depth.

[0003] To improve process efficiency, Model-Driven Systems Engineering (MBSE) was introduced, which designs a standardized indicator system through structured modeling. This method requires first building a global system model, and then decomposing and mapping evaluation indicators onto the model. While it improves process standardization and normalization to some extent, its process heavily relies on a high-quality initial model and continuous expert involvement, particularly in managing model accuracy and complexity. Practical applications have shown that MBSE is cumbersome and lacks flexibility when dealing with large-scale, unstructured system descriptions and multi-dimensional interactive coupling.

[0004] In summary, existing methods for constructing evaluation index systems for avionics systems mainly rely on expert experience and manual induction, resulting in complex processes, long processing times, and insufficient standardization and traceability. Model-driven systems engineering (MBSE) improves the standardization and normalization of processes to some extent, but it heavily depends on high-quality initial models and continuous expert participation. Furthermore, when dealing with large, complex, and multi-dimensional interactive avionics systems, it suffers from cumbersome operation, insufficient flexibility, and difficulty in meeting the needs of rapid system development and continuous expansion. Summary of the Invention

[0005] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a method for constructing an avionics system evaluation index system based on a large language model agent. Based on a large language model multi-agent, this method automatically completes the extraction, summarization, and system construction of avionics system evaluation indicators. It has advantages such as eliminating the need for manual decomposition, process traceability, broad coverage, and high efficiency. This method performs better when dealing with unstructured data, complex systems, and rapidly changing requirements. It effectively overcomes the limitations of existing expert-driven and model-driven methods and greatly improves the standardization and automation level of the evaluation index system.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for constructing an evaluation index system for avionics systems based on large language model intelligent agents includes:

[0008] The knowledge extraction module uses a large language model intelligent agent to extract knowledge from the design documents of the avionics system and initially construct a knowledge graph;

[0009] The reflection and verification module is based on a large language model agent and combines learning paradigms to automatically check and correct entities and relationships in the knowledge graph, and finally outputs an optimized knowledge graph.

[0010] The indicator generation module is based on a large language model intelligent agent, which automatically generates evaluation indicators for each entity node in the optimized knowledge graph, forming an evaluation indicator system for avionics systems.

[0011] To optimize the above technical solution, the specific measures also include:

[0012] In the aforementioned knowledge extraction module, the large language model agent first paginates the design document. For each page, based on a custom knowledge extraction paradigm, it extracts the content at the subsystem, capability, function, and module levels, filtering and summarizing it into subsystem knowledge blocks, capability knowledge blocks, function knowledge blocks, and module knowledge blocks. Then, it extracts entities for each knowledge block, obtaining new entities through entity recognition, naming standardization, and disambiguation of complex polysemous expressions. Each new entity is then registered in the knowledge graph management container. Within each knowledge block, it identifies the basic relationships between entities and extracts them as standard triples in the form of starting entity-relation type-ending entity as new relations. For each new relation, it checks whether the entities at both ends have been registered in the knowledge graph management container: if they have been registered, the corresponding edges are created directly on the knowledge graph; if they have not been registered, the new entities are registered first, and then the relation is established.

[0013] In the knowledge extraction module described above, whenever a new entity is obtained, its embedding vector is calculated to have semantic similarity with all existing entities in the current knowledge graph. If the similarity is higher than a set threshold, it is considered to be the same entity to prevent duplicate or redundant registration and storage; otherwise, the new entity is registered in the knowledge graph management container and stored in the knowledge graph in a standardized manner.

[0014] In the knowledge extraction module described above, synonym merging and redundancy removal are performed during the relation extraction process to ensure that the knowledge graph's topological structure is concise, closed-loop, and semantically clear.

[0015] The aforementioned reflection and verification module, based on a large language model agent, combines typical naming conventions and correct / incorrect hierarchical assignment cases provided by the learning paradigm to systematically check the naming conventions and accuracy of hierarchical labels of various entity nodes in the knowledge graph. It identifies abnormal entities with non-standard naming and improper hierarchical label assignment and corrects the identified abnormal entities.

[0016] The aforementioned reflection and verification module checks and corrects the relationships in the knowledge graph, including: automatically detecting whether there are unreasonable self-loop relationships between entities in the knowledge graph and removing abnormal edges in a timely manner; judging the logical rationality of each relationship; and eliminating, merging, and standardizing potential relationship duplication, directional confusion, or type conflict connections.

[0017] The aforementioned indicator generation module automatically traverses all entity nodes in the knowledge graph data structure. For each entity node, it automatically retrieves all relationships established in the knowledge graph. Centered on the current entity, it compiles a set of prompt words describing its structural and functional relationships with related nodes. These prompt words contain the entity's attribute information and characterize its context and specific role within the system. After the prompt words are compiled, the detailed information of the entity is input into the large language model agent for comprehensive reasoning, analysis, and automatic generation of a set of evaluation indicators.

[0018] The aforementioned evaluation indicators cover technical performance, functional requirements, and safety and reliability.

[0019] The present invention has the following beneficial effects:

[0020] This invention enables the automatic generation of efficient, multi-level, and scalable indicator systems for unstructured source documents, effectively improving the data value mining capability and intelligence level of avionics system evaluation. It overcomes the problems of existing avionics system evaluation indicator system construction methods, such as reliance on manual labor, cumbersome processes, strong subjectivity, insufficient standardization and traceability, difficulty in adapting to the development of system complexity, and difficulty in achieving automatic expansion and efficient iteration across scenarios and domains.

[0021] This invention organically combines the natural language understanding and reasoning capabilities of a large language model with the structured representation of a knowledge graph and an automatic reflection mechanism. This forms a "zero-sample, configurable, multi-agent" indicator generation paradigm that can adapt to different evaluation scenarios simply by extracting templates, without historical sample annotations or fixed ontology constraints. This significantly improves automation, compatibility, and scenario generalization flexibility. The invention uses a knowledge extraction module to achieve hierarchical entity and relationship intelligent identification and structured assembly of unstructured data such as various design documents of avionics systems. A reflection and verification module utilizes the reasoning and self-checking capabilities of the large language model's multi-agents to automatically verify and correct multi-dimensional indicators such as knowledge structure consistency, entity labeling, and relationship rationality. Based on the hierarchy and dependencies of the knowledge graph, a bottom-up aggregation generation strategy is adopted to intelligently synthesize a multi-level indicator system, supporting comprehensive multi-dimensional evaluations of system health, performance, and security. The overall process is modular and automated, requiring minimal manual intervention, possessing strong scalability and adaptability, and providing a general technical route for the intelligent generation of indicators for other complex systems.

[0022] High degree of automation: Based on the custom extraction paradigm, the entire process of knowledge identification, organization and indicator generation is automatically completed by the large language model and related functional modules, which greatly reduces the manual operation links.

[0023] Strong structure and standardization: The system filters and summarizes complex document knowledge in layers to ensure that the knowledge graph results are clear in structure and semantics, which facilitates subsequent indicator generation and system analysis;

[0024] It has good adaptability and scalability: through a configurable schema and modular chunk extraction mechanism, it can flexibly cope with domain documents of different formats and content structures, and adapt to the needs of multiple types of systems and new knowledge types.

[0025] Traceable and easy to review: The hierarchical structure of the system as a whole and the correlation between its components are fully considered. Each indicator and its source can be traced back to the corresponding fragment of the original document and the specific entities and relationships in the knowledge graph, which facilitates subsequent review, maintenance and continuous improvement of the system.

[0026] Reduce subjectivity and human error: Automated judgment and data fusion processes can reduce the bias caused by subjective human judgment, making the evaluation system more objective and consistent;

[0027] This invention can be widely applied in aerospace, electronic information, rail transportation, industrial equipment, energy and power, medical and other fields, and is especially suitable for industries and scenarios that require the construction of multi-level indicator systems, performance evaluation, quality review and knowledge management for complex products, systems and engineering. Attached Figure Description

[0028] Figure 1This is a flowchart illustrating the method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent, as described in this invention.

[0029] Figure 2 This is a flowchart illustrating the knowledge extraction process of this invention.

[0030] Figure 3 This is a flowchart illustrating the implementation process of the present invention through reflection and verification.

[0031] Figure 4 The flowchart for generating the indicators of this invention is shown below. Detailed Implementation

[0032] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0033] This invention provides a method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent. This method achieves structured extraction of document knowledge, reflective verification, and automatic generation of a multi-level index system. Figure 1 As shown, the specific implementation process is as follows:

[0034] I. System Composition and Overall Architecture

[0035] The system framework of this invention adopts a collaborative approach with three main modules: knowledge extraction, reflection and verification, and indicator generation. These modules are further subdivided into several functional components. Data flow is organized hierarchically, through inter-module interaction and multi-agent cooperation mechanisms. This enables indicator extraction for design documents, semantic consistency verification, multi-level indicator aggregation, and the output of a traceable and standardized indicator system. The main functions and interactions of each module are as follows:

[0036] 1. Knowledge Extraction Module

[0037] The knowledge extraction module of this invention extracts structured knowledge from multi-level, unstructured design documents in complex avionics systems or similar fields according to a clear operational process, greatly improving the accuracy, comprehensiveness, and automation of the subsequent automatic generation of evaluation indicator systems. For example... Figure 2 As shown, its implementation process includes the following five steps, each of which is equipped with specific core technology solutions and components.

[0038] (1) Define Schema: Specify extraction normal form and hierarchical entity types

[0039] The process begins with defining the schema. The schema, or knowledge extraction paradigm, is the foundation for structured parsing of the entire document. Users or the system flexibly configure the main entity categories and hierarchical order to be identified based on the actual situation of the avionics system documents being processed, such as subsystem level, capability level, functional level, and module level. The schema not only supports custom entity classifications but also allows for refined entity attributes and relationship types, ensuring flexible adaptation and standardized output across different domains / projects. If content at a certain level is missing in a specific document (page), the system outputs a null value at the corresponding location in the schema, avoiding falsification and redundant information addition, thus ensuring the high purity of the subsequent knowledge graph. Furthermore, when interacting with a large language model, the system can guide the model to extract only specific entities and attributes through custom schema templates, improving the targeting and accuracy of the extraction.

[0040] Specifically, this embodiment uses a typical design document from an actual avionics system as the entry point. For the avionics system design document to be processed, the user or development team first analyzes the information hierarchy and business requirements of the target document. Through the system configuration interface or schema script, they set the entity extraction categories and their hierarchical structure that are of key interest in the field. For example, this embodiment presets the schema to a four-level structure: subsystem level, capability level, function level, and module level. Each level can be configured with main attributes (such as name, number, description, interface relationships, etc.) and allowed typical relationship types (such as "belongs to," "contains," "depends on," etc.). If a certain level is missing in the specific content of the document, the schema outputs a null value for that item without performing any forgery or completion.

[0041] (2) Document pagination and block extraction knowledge extraction

[0042] Due to the limited context window length of large language models (e.g., 8K, 32K), complete design documents often require pagination. The system typically splits the document into pages, processing each page separately. This prevents knowledge loss due to context truncation and improves the efficiency and accuracy of subsequent chunking. For each paginated page, the system extracts relevant content from different levels, such as subsystems, capabilities, functions, and modules, based on the aforementioned schema framework. This content is then filtered and summarized into four distinct knowledge chunks. This multi-level, chunked extraction strategy effectively filters out redundant information irrelevant to metric generation while retaining and refining core knowledge points at each level, significantly increasing the density of pure knowledge signals.

[0043] Specifically, the complete design document (assuming it's A4 format and 100 pages) is imported into the system. It's automatically paginated (each page is a block). The block engine traverses each page and extracts content from four entity categories—"Subsystem," "Capability," "Function," and "Module"—based on the schema. For example, if the system locates the "Data Acquisition Subsystem" on page 18, it captures all relevant capability descriptions, functional decompositions, and included modules under that section according to the schema. Categories without descriptions are returned as empty blocks. Within each block, only the core information relevant to the current page and level is retained, filtering out irrelevant content to purify the system for subsequent entity modeling.

[0044] (3) Entity extraction and entity merging

[0045] For each knowledge chunk, the system employs a large language model for entity extraction. During extraction, the required entity categories are pre-defined in the extraction schema, and the powerful text understanding and abstraction capabilities of the large language model are leveraged to achieve entity recognition, naming standardization, and disambiguation of complex polysemous expressions. After entity extraction, the system adds each new entity to a knowledge graph management container, where the knowledge graph is a class. At this point, the system employs a uniqueness guarantee and synonym merging mechanism: whenever a new entity is encountered, its embedding vector (e.g., BERT semantic vectors) is calculated to have semantic similarity to all existing entities in the current knowledge graph. If the similarity exceeds a set threshold (e.g., 0.7), it is considered the same entity to prevent duplicate and redundant storage; otherwise, it is treated as a new entity node and stored in the knowledge graph. This mechanism ensures that the same object across documents and expressions can be uniquely identified in the knowledge graph, improving structural integrity and the efficiency of subsequent queries and aggregations.

[0046] Specifically, the system employs its built-in foundational large language model to batch process all knowledge blocks, automatically identifying entities with standardized naming conventions and attribute uniformity. For example, if "navigation module," "navigation function," and "navigation subsystem" appear on pages 7, 25, and 89 respectively, the system will determine whether they are the same object or have inclusion or hierarchical relationships based on the configured schema and naming standards, combined with embedded semantic similarity (e.g., BERT cosine similarity ≥ 0.7). If the two are highly similar, they are determined to be the same knowledge graph node, retaining only one standardized entity to avoid redundancy. This mechanism ensures that various abbreviations, contextual substitutions, and synonyms are effectively resolved in the project.

[0047] (4) Relation extraction and relation merging

[0048] Not only do entities need standardized management, but the multi-category relationships between entities also require structured extraction. Within each knowledge block, the system uses a large language model to identify basic relationships between entities (such as composition, dependency, inclusion, and belonging) and extracts them into standard triples (starting entity - relationship type - ending entity). For each new relationship, the system checks whether the two end entities have been registered in the container: if they already exist, the corresponding edge is created directly on the knowledge graph; if they do not exist, the new entities are registered first, and then the relationship is established. Simultaneously, the relationship extraction process also includes synonym merging and redundancy removal (such as detecting duplicate relationships, reverse relationships, or illogical relationships and merging or removing them), ensuring that the knowledge graph's topological structure is concise, closed-loop, and semantically clear.

[0049] Specifically, for each entity formed after the extraction of each block, the system further utilizes a large language model to extract key relationships between entities, such as "the navigation module belongs to the navigation subsystem" and "data processing capability depends on the sensor module," and temporarily stores all relationships as triples (starting entity - relationship type - target entity). Subsequently, the system automatically checks whether the endpoint entities of the triples have been registered in the current knowledge graph management container. If they exist, edges are added directly; if a new entity is present, it is automatically added to the graph and a connection is established. At the same time, the system detects redundancy such as relationship conflicts, duplications, and directional confusion, and intelligently merges similar semantic relationships to ensure the topological accuracy and semantic purity of the entire knowledge graph structure.

[0050] (5) Knowledge graph construction completed

[0051] Once the entities and relationships of all pages and knowledge blocks at all schema levels have been extracted, compared, standardized, and merged, the entire document-level structured knowledge graph is complete, and the system outputs a complete structured knowledge graph. This knowledge graph features clear layering, complete associations, and guaranteed node uniqueness. It accurately reflects the original multi-level system and dependencies of the document and serves as key intermediate data for the automatic generation of the indicator system. It is not merely a collection of entities and attributes but also fully preserves the original document's multi-level, upstream and downstream dependencies, and cross-level mappings, becoming the fundamental data basis for subsequent indicator reflection and verification, automated indicator aggregation, and systematic evaluation. Its high consistency, high standardization, and high traceability are the key technical support for this invention's automatic construction of a multi-level complex system indicator system.

[0052] 2. Reflection and Verification Module

[0053] After knowledge extraction is completed and a preliminary knowledge graph is constructed, this invention includes a reflection and verification module to ensure the accuracy and standardization of structured knowledge. For example... Figure 3As shown, this module leverages the powerful semantic understanding and reasoning capabilities of large language models, combined with a few-shot learning paradigm, to automatically perform multi-level and multi-dimensional in-depth checks and corrections on entities and relationships in the knowledge graph. The final output is a well-structured, semantically clear, and consistent knowledge graph, providing a solid foundation for the high-quality construction of subsequent indicator systems. The reflection and verification module specifically includes the following two aspects:

[0054] (1) Reflection and inspection of entity normativity and hierarchical labeling

[0055] This step primarily involves systematically verifying and automatically correcting the naming conventions and hierarchical labels of various entity nodes in the knowledge graph. The method is as follows:

[0056] —Utilizing the few-shot capability of large language models, we provide some typical naming conventions and cases of correct and incorrect hierarchical assignments to guide the model to better identify entities with non-standard names (such as using abbreviations, initials, spelling errors, or redundant modifiers) and improper hierarchical label assignments (such as subsystems being labeled as module-level).

[0057] —For detected abnormal entities, the system will automatically generate modification suggestions, such as standard naming templates and suggested correct hierarchical labels, and can automatically correct them under certain rules. The modification opinions can also be fed back to the manual review interface for final confirmation.

[0058] —After this step, all entities are processed to ensure a consistent naming style, accurate expression, and a scientific and reasonable hierarchical distribution, laying a consistent foundation for the reliable application of subsequent knowledge.

[0059] Specifically, during the review phase, the system automatically performs naming convention and hierarchical label correctness checks on all entity nodes in the knowledge graph. Utilizing a few-shot large language model with pre-defined standard and deviation cases, such as "the correct name should be 'xxx module', not 'xxm' or 'data123'," the model automatically identifies and labels non-compliant naming examples. For hierarchical misalignment behaviors like "a subsystem being labeled as a module," it promptly makes suggestions or adjustments to ensure that all nodes have a consistent naming style and accurate, complete hierarchical structure.

[0060] (2) Relationship rationality and self-loop investigation

[0061] Knowledge graphs not only require accurate entities, but also that the relationships between entities be reasonable and compliant. This section mainly aims to achieve the following objectives:

[0062] —Self-loop detection: Automatically detects whether there are unreasonable self-loop relationships between entities in the knowledge graph (such as self-referential relationships like A→A), and promptly removes abnormal edges to prevent loop errors from affecting the overall structure.

[0063] —Semantic rationality judgment: Based on the strong semantic reasoning ability of the large language model, the logical rationality of each relationship is judged. For example, it checks whether the association type is appropriate for the hierarchical relationship between the two entities (such as whether the "inclusion relationship" only exists between high and low level entities), and whether the relationship description violates common sense or physical laws.

[0064] —Identify confusion and redundancy: Eliminate, merge, and standardize potential duplicate relationships, directional confusion, or type conflicts to ensure the simplicity and semantic clarity of the relational network structure.

[0065] Specifically, during the traversal of knowledge graph relationships, the system automatically detects abnormal connections such as self-referential entities (e.g., "navigation module" pointing to itself) and non-compliant hierarchical crossings (e.g., "module contains subsystems"). Based on a few-shot LLM semantic reasoning model, it assesses the business logic thickness of "triples" and automatically filters for logically flawed, counterintuitive, and semantically conflicting relationships. Self-loop relationships are directly removed, while the rest are corrected, merged, or manually reviewed according to model suggestions.

[0066] The specific handling of confusing and redundant relationships is as follows: For redundant phenomena in the knowledge graph, such as directional conflicts (e.g., A contains B and B contains A) and ambiguous multiple edges (e.g., a large number of repeated edges that are difficult to distinguish between "dependency" and "association"), the model automatically merges similar types of relationships, sorts out and labels conflict situations, and ensures that the final structure of the knowledge graph is concise and orderly, and that the semantic expression is clear and consistent.

[0067] After multiple rounds of automatic reflection, standardization, and normalization of entities and relationships, the system finally outputs a high-quality knowledge graph with clear hierarchy, standardized naming, and semantic verifiability, providing a foundation for the generation of intelligent indicator systems.

[0068] After the two automated checks and corrections mentioned above, the reflection and verification module can output a high-quality knowledge graph data structure that is well-named, hierarchically clear, semantically consistent, and structurally sound. This structured knowledge not only provides highly reliable data support for subsequent indicator generation but also greatly improves the traceability and scalability of the knowledge system, fundamentally ensuring the accuracy and reliability of the entire automated indicator system construction process.

[0069] 3. Indicator Generation Module

[0070] like Figure 4As shown, this module uses a well-structured knowledge graph as input and targets all entity nodes in the knowledge graph (regardless of their level) to achieve efficient and intelligent automatic generation of evaluation indicators. Unlike the traditional manual design and bottom-up approach, this invention leverages the understanding and inductive capabilities of a large language model, allowing for independent indicator generation for each entity node, thus greatly improving the flexibility and comprehensiveness of the indicator system construction. The specific steps are as follows:

[0071] (1) Traversal and filtering of entity nodes

[0072] The system automatically traverses all entity nodes (such as each subsystem, module, capability, etc.) in the knowledge graph data structure, without distinguishing their hierarchy (whether it's module, function, capability, or subsystem level). Any node in the knowledge graph can be directly processed and generated as an object, without needing to differentiate between bottom-up priorities, demonstrating strong flexibility and complete coverage. This design ensures the comprehensiveness and flexibility of indicator generation, and also facilitates rapid adjustment and expansion of specific levels, types, or subsets of entities under different needs and scenarios.

[0073] (2) Structural relationship analysis and prompt word generation

[0074] For each entity node, the system automatically retrieves all established relationships within the knowledge graph, including its upstream (belonging objects), downstream (its subordinate components), peers (parallel collaborating objects), and other cross-layer or semantically related nodes. Centered on the current entity, the system compiles a set of prompts describing its structural and functional relationships with related nodes. These prompts not only contain the entity's attribute information but also clearly depict its context and specific role within the system, such as "Entity X belongs to capability Y, participates in function Z, and coexists in the same category as modules A and B," etc.

[0075] Specifically, centering on each entity, the system retrieves all its associated structural relationships (including superior affiliation, subordinate composition, parallel collaboration, dependency interfaces, etc.) and automatically integrates these structured contexts with the entity's own attributes. The system generates detailed prompts to construct a multi-perspective, multi-dimensional information description of the current entity. For example, "The navigation function belongs to the flight control system, operates in conjunction with the positioning module, and requires compatibility with ground station data."

[0076] (3) Evaluation indicators are automatically generated

[0077] Once the prompts are ready, the system inputs the entity's detailed information (including its attributes and relationships with all associated nodes) into the large language model. The model then performs comprehensive reasoning, analysis, and automatically generates a set of evaluation metrics. These metrics can cover aspects such as technical performance, functional requirements, and security and reliability. The model can combine industry rules, domain knowledge, entity contextual relationships, and other multi-dimensional factors to output metrics that conform to general standards and are consistent with the actual ontology structure. This process requires no manual intervention and relies entirely on the structural facts of the knowledge graph and the semantic understanding of the large language model.

[0078] Specifically, the complete information and structural relationships of each of the above entities are input into the large language model, which automatically outputs a batch of evaluation indicators covering functional requirements, performance metrics, and other content. For example, "The navigation module must meet the requirements of positioning accuracy better than 3 meters, have redundant data verification capabilities, and a data processing rate of ≥ N data entries per second."

[0079] This process supports parallel generation of indicators by any entity without manual intervention or specified review paths. The model generation results automatically record the associated sources of the indicators, providing a basis for subsequent traceability and system interoperability.

[0080] The final system outputs a structured evaluation index system covering all entity nodes and multiple granular levels. All indicators can be reconstructed from their original document context based on their knowledge graph associations, achieving standardized output, facilitating expansion and reuse. This solution has been demonstrated in field applications across multiple avionics systems, showcasing its automated coverage, traceability, and strong reusability, which is beneficial for subsequent unified evaluation and continuous upgrades of multiple systems.

[0081] Through the above steps, independent and efficient evaluation metrics for any entity node in the knowledge graph are generated. The entire process significantly improves the efficiency and accuracy of knowledge extraction, relationship analysis, and metric generation in complex system design documents. Simultaneously, the output metric system boasts a standardized structure, clear hierarchy, and rich detail, fully reflecting the overall architecture and key performance characteristics of the evaluated system. This greatly enhances the automation and adaptability of the metric system construction, enabling the system to quickly respond to changes in entity structure and flexibly support different scenarios and granular requirements. It lays a solid foundation for the automated evaluation of complex systems. Its advantages of automation, structuring, traceability, and ease of horizontal reuse bring tangible technical value and application prospects to application scenarios such as large-scale system R&D evaluation and standard system construction in key industries.

[0082] This invention can automatically and efficiently extract multi-level entities and their relationships from unstructured texts such as domain design documents, construct a structured knowledge graph, and then directly generate corresponding evaluation index systems for each entity node within the knowledge graph. This solution eliminates the need for manual annotation or pre-built ontology libraries; it only requires a custom extraction schema configuration template to adapt to different domains and document structures, thus improving the automation and versatility of index system construction.

[0083] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for constructing an evaluation index system for avionics systems based on large language model intelligent agents, characterized in that, include: The knowledge extraction module uses a large language model intelligent agent to extract knowledge from the design documents of the avionics system and initially construct a knowledge graph; The reflection and verification module is based on a large language model agent and combines learning paradigms to automatically check and correct entities and relationships in the knowledge graph, and finally outputs an optimized knowledge graph. The indicator generation module is based on a large language model intelligent agent, which automatically generates evaluation indicators for each entity node in the optimized knowledge graph, forming an evaluation indicator system for avionics systems.

2. The method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent according to claim 1, characterized in that, In the knowledge extraction module, the large language model agent first paginates the design document. For each page, based on a custom knowledge extraction paradigm, it extracts the content at the subsystem, capability, function, and module levels, filtering and summarizing them into subsystem knowledge blocks, capability knowledge blocks, function knowledge blocks, and module knowledge blocks. Then, for each knowledge block, entity extraction is performed. Through entity recognition, naming standardization, and disambiguation of complex polysemous expressions, new entities are obtained, and each new entity is registered in the knowledge graph management container. Within each knowledge block, the basic relationships between entities are identified, and standard triples in the form of starting entity-relationship type-ending entity are extracted as new relationships. For each new relationship, it is checked whether the entities at both ends have been registered in the knowledge graph management container: if they have been registered, the corresponding edges are created directly on the knowledge graph; if they have not been registered, the new entities are registered first, and then the relationship is established.

3. The method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent according to claim 2, characterized in that, In the knowledge extraction module, whenever a new entity is obtained, its embedding vector is calculated to have semantic similarity with all existing entities in the current knowledge graph. If the similarity is higher than a set threshold, it is identified as the same entity to prevent duplicate or redundant registration and storage; otherwise, the new entity is registered in the knowledge graph management container and stored in the knowledge graph in a standardized manner.

4. The method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent according to claim 2, characterized in that, In the knowledge extraction module, synonym merging and redundancy removal are performed during the relation extraction process to ensure that the topological structure of the knowledge graph is simple, closed-loop, and semantically clear.

5. The method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent according to claim 1, characterized in that, The reflection and verification module, based on the large language model agent, combines typical naming conventions and correct and incorrect hierarchical assignment cases provided by the learning paradigm to systematically check the naming conventions and accuracy of hierarchical labels of various entity nodes in the knowledge graph, identify abnormal entities with non-standard naming and improper hierarchical label assignment, and correct the identified abnormal entities.

6. The method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent according to claim 1, characterized in that, The reflection and verification module checks and corrects the relationships in the knowledge graph, including: automatically detecting whether there are unreasonable self-loop relationships between entities in the knowledge graph and removing abnormal edges in a timely manner; judging the logical rationality of each relationship; and eliminating, merging and standardizing potential relationship duplication, directional confusion or type conflict connections.

7. The method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent according to claim 1, characterized in that, The indicator generation module automatically traverses all entity nodes in the knowledge graph data structure. For each entity node, it automatically retrieves all relationships established in the knowledge graph. Centered on the current entity, it compiles a set of prompt words describing its structural and functional relationships with related nodes. These prompt words contain the entity's attribute information and characterize its context and specific role within the system. After the prompt words are compiled, the detailed information of the entity is input into the large language model agent for comprehensive reasoning, analysis, and automatic generation of a set of evaluation indicators.

8. The method for constructing an evaluation index system for avionics systems based on a large language model intelligent agent according to claim 1, characterized in that, The evaluation indicators cover technical performance, functional requirements, and safety and reliability.

Images