A method and system for evaluating large language models and optimizing prompt words based on RDFS ontology.

Abstract

This application relates to a method for evaluating large language models and optimizing prompt words for RDFS ontology. The method includes the following steps: Step 1: Constructing positive and negative example datasets; Step 2: Designing prompt word templates at multiple levels; Step 3: Combining test cases from the standardized test dataset with different types of prompt word templates to generate input sequences to the large language model to be evaluated, and obtaining the output judgment results and explanatory text; Step 4: Parsing the judgment results and explanatory text to generate a comprehensive evaluation report; Step 5: Based on the data analysis results of the comprehensive evaluation report, constructing a three-element mapping relationship of "prompt word type - evaluation dimension - axiom type" to form a system performance graph, and implementing the optimization process accordingly. This solution solves the problems of incomplete evaluation systems, incomplete test data, and single evaluation dimensions in existing systems, achieving a systematic and accurate evaluation and improvement of the model's RDFS ontology learning ability.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph and semantic computing technology, specifically relating to an evaluation and optimization method for ensuring the accuracy and consistency of the logical axioms of the generated RDFS ontology during the process of automatically constructing or completing knowledge graphs using large language models. Background Technology

[0002] With the deep integration of Semantic Web and Artificial Intelligence technologies, the Resource Description Framework Schema (RDFS), as a core semantic modeling language, provides a solid foundation for machine-understandable and reasonable knowledge representation. RDFS achieves a unified description of data semantics by defining classes, attributes, and the hierarchical and constraint relationships between them (such as subclasses, sub-attributes, domains, and ranges), and has become a core technology in fields such as building high-quality knowledge graphs, achieving accurate intelligent retrieval, and completing the semantic integration of heterogeneous data.

[0003] Faced with the ever-increasing volume of unstructured and semi-structured text data, how to automatically and accurately extract and generate ontology structures that conform to the RDFS semantic specification from natural language has become a key challenge in improving the intelligence level of semantic systems. Traditional methods (such as rule-based, statistical learning, or deep learning methods) generally suffer from bottlenecks such as strong dependence on labeled data, insufficient generalization ability, and high computational cost, making it difficult to meet the requirements of high accuracy and strong robustness in open domain scenarios.

[0004] The emergence of Large Language Models (LLMs) has provided a new solution to the aforementioned problems. Their powerful semantic understanding and generation capabilities have shown potential in tasks such as entity relation extraction and ontology draft generation. However, when applying LLMs to RDFS ontology learning, existing techniques suffer from three major shortcomings: First, the evaluation system is incomplete. Current research lacks a systematic evaluation framework tailored to the characteristics of RDFS ontology, failing to cover the full range of capabilities from basic concept recognition to complex logical relation reasoning. Second, the test data is incomplete. There is a lack of standardized test sets containing both positive examples (semantic validity axioms) and negative examples (semantic conflict axioms), making it impossible to effectively evaluate the model's semantic recognition and error correction capabilities. Third, the evaluation dimensions are singular. There is an over-reliance on traditional metrics such as accuracy and recall, neglecting comprehensive consideration of key dimensions such as the rationality of model interpretations (e.g., the "illusion" phenomenon) and the consistency of outputs (e.g., resilience to input perturbations). These shortcomings lead to one-sided evaluation results, failing to provide accurate and reliable guidance for model optimization.

[0005] The aforementioned deficiencies make it impossible to effectively control the quality of ontology axioms generated by large language models in the automated construction process of industrial-grade knowledge graphs. The direct consequence is that logically conflicting and semantically erroneous axioms pollute the knowledge base, undermining its inherent consistency. This leads to incorrect results in downstream applications such as semantic search, intelligent question answering, and reasoning decision-making based on the knowledge graph, significantly reducing system reliability. Therefore, there is an urgent need in this field for a technical solution that can systematically evaluate and optimize the quality of ontology generation from large language models (RDFS), ensuring the correct operation of the semantic system from the source, thereby fundamentally improving the quality of knowledge graphs. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing technologies and provide a large language model evaluation and prompt word optimization scheme for RDFS ontology. This scheme addresses the problems of incomplete evaluation system, inadequate test data, and single evaluation dimension in existing technologies, thereby improving the quality of ontology generated by large language models from the source and ensuring the accuracy and consistency of the knowledge graph constructed from them.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] Firstly, a method for evaluating large language models and optimizing prompt words for RDFS ontology is provided, characterized by the following steps:

[0009] Test dataset construction steps: Automatically extract logical axioms conforming to the RDFS specification from existing ontology or knowledge base to form a set of positive examples, and automatically generate a set of negative examples corresponding to the set of positive examples based on preset logical conflict rules, thereby forming a standardized test dataset for evaluation.

[0010] Prompt word engineering design steps: Design prompt word templates at multiple levels, including at least a basic type without semantic explanation, an instruction-enhanced type with axiomatic semantic explanation, and a context-guided type that provides relevant semantic context;

[0011] Model testing execution steps: Combine the test cases in the standardized test dataset with different types of prompt word templates to generate an input sequence to the large language model to be evaluated, and obtain the output judgment results and explanation text;

[0012] Multi-dimensional evaluation steps: The judgment results and explanatory text are analyzed, and quantitative evaluation is carried out from three dimensions: classification ability, explanation and understanding ability, and stability ability, and a comprehensive evaluation report is generated.

[0013] Prompt word optimization steps: Based on the comprehensive evaluation report, analyze the impact of different prompt word templates on the performance of the large language model in each evaluation dimension, and select or generate optimized prompt word strategies accordingly.

[0014] Furthermore, the RDFS logical axioms include axioms of hierarchical relationships between classes, axioms of hierarchical relationships between attributes, axioms of domain constraints for attributes, and axioms of range constraints for attributes. It should be noted that the evaluation focus of this invention is on the aforementioned logically binding RDFS axioms, rather than syntactic axioms that merely serve as declarations or annotations. This distinction ensures that the evaluation work targets the core part of the ontology responsible for semantic reasoning.

[0015] Furthermore, the construction of the positive example set is based on programmatic access and automated information extraction from authoritative structured knowledge sources. This process is implemented through standardized program interfaces, including a standard ontology processing interface for parsing local ontology files and a standard structured query interface for querying remote knowledge bases, thereby ensuring the authority of the data source and high coverage of data acquisition. From high-quality knowledge sources, the system automatically extracts various predefined RDFS logical axioms, including class hierarchy relationships, attribute hierarchy relationships, attribute domain constraints, and attribute value domain constraints, which are then integrated and standardized to form a standardized positive example set.

[0016] Furthermore, the generation of the negative example set is achieved through systematic semantic transformation, mainly including two strategies: logical orientation subversion and constraint consistency violation. Logical orientation subversion directly constructs fallacies by reversing the directionality in the hierarchical relation axioms. Constraint consistency violation utilizes existing logical assertions in the ontology and derives new axioms that contradict them through deductive reasoning, thereby constructing a deeper level of semantic conflict.

[0017] Furthermore, the constraint consistency violation is designed as a negative example generation mechanism based on logical reasoning, which procedurally accesses the target ontology, identifies and utilizes the existing logical assertions therein, and constructs deep semantic conflicts through automatic deductive reasoning. This mechanism is implemented in at least one of the following ways: 1) Violation of the axiom of mutual exclusion of concepts: Based on two classes declared as disjoint in the ontology, the domain or range of an attribute originally applied to one class is changed to the other class, which is disjoint, thereby generating a logical conflict; 2) Violation of the constraint of existence quantifier: Based on the inclusion axiom describing the existence of attribute constraints between classes (in the form of X ⊑ ∃PY), the domain or range of the attribute P is changed to a class Z that logically contradicts the axiom (e.g., X and Z are disjoint), thereby generating a logical conflict; 3) Violation of instantiated data: Based on the existing concrete instances and their attribute relationships in the knowledge base (in the form of P(a, b) and C(a)), the domain or range of the attribute P is changed to a class that logically conflicts with the instance type C, thereby generating a logical conflict.

[0018] Furthermore, the basic prompt word is characterized by using concise task instructions that do not contain specific axiom type descriptions or additional contextual information, in order to directly require the large language model to make a correctness judgment on the input axiom.

[0019] Furthermore, the enhanced prompt word is characterized by adding semantic definitions and logical rule descriptions of the judged axiom type to the basic prompt word, in order to guide the large language model to focus on the semantic constraints of a specific type.

[0020] Furthermore, the context-guided prompt is characterized by introducing background knowledge fragments related to the target axiom into the instruction. The background knowledge includes the hierarchical structure of classes or attributes, instantiation relationships, or existing axiom fragments, in order to construct a semantic context environment that assists reasoning.

[0021] Furthermore, during model testing, test cases from the standardized test dataset are combined with different types of prompt word templates, along with output format requirements, to generate an input sequence and submit it to the large language model to be evaluated. The judgment result and explanatory text returned by the large language model are then obtained. The judgment result is a binary decision by the model regarding the correctness of the axioms, while the explanatory text is the logical argument generated by the model to support its decision. Both of these outputs serve as the raw data for subsequent multi-dimensional evaluation and are indispensable.

[0022] Furthermore, the multi-dimensional evaluation step automatically analyzes and quantifies the judgment results and explanatory text from the following three core dimensions, generating a comprehensive evaluation report: 1) Classification ability: evaluating the accuracy of the large language model in correctly classifying the RDFS logical axioms into binary categories; 2) Explanation and comprehension ability: evaluating the semantic correctness and logical rationality of the explanatory text provided by the large language model for its judgment results; 3) Stability ability: evaluating the consistency of the output results of the large language model when faced with semantically equivalent input perturbations. These three dimensions, from the perspectives of judgment results, internal reasoning process, and external robustness, together constitute a hierarchical and systematic capability evaluation system.

[0023] Furthermore, the classification ability is evaluated by calculating one or more classification evaluation metrics. These metrics are based on the confusion matrix formed by the model's judgment results and the true labels, and encompass precision, recall, and F1 score. This set of metrics comprehensively reflects the model's overall performance in recognizing valid axioms and identifying logical conflicts, and allows for fine-grained analysis of different types of RDFS logical axioms to accurately reveal the model's cognitive boundaries in specific semantic relationships.

[0024] Furthermore, the evaluation of the explanation and comprehension ability is achieved by quantifying the explanation accuracy rate and the illusion rate. The explanation accuracy rate quantifies the proportion of examples where the model correctly judges the explanation text is consistent with the true semantics of the axioms; this metric aims to verify whether the model's judgment is based on correct logical grounds, rather than accidental guessing. The illusion rate quantifies the proportion of examples where the model correctly judges the explanation text contains fictitious information, unfounded inferences, or factual errors; this metric is specifically used to measure the content credibility problem unique to large language models in knowledge reasoning tasks, effectively exposing the "illusion" defect of the model fabricating or confusing concepts within the logical chain, and is a direct measure of the credibility of the model's internal thought process.

[0025] Furthermore, the stability capability is evaluated by constructing a semantically equivalent set of input perturbations and calculating a consistency index. The consistency index includes: hard consistency, which uses a strict all-or-nothing criterion to measure whether the model gives completely consistent outputs for multiple equivalent expressions of the same semantics, directly reflecting the accuracy and certainty of the model's semantic understanding; and soft consistency, which measures stability by statistically analyzing the dispersion of the model's output within the perturbation set, and can sensitively capture the ambiguity and uncertainty in the model's judgments. It is an effective supplement to the hard consistency index, and the combination of the two can comprehensively characterize the model's robustness against input changes.

[0026] Furthermore, the multi-dimensional evaluation step automatically parses and quantifies the judgment results and explanatory text, and performs calculations and analyses from three dimensions: classification ability, explanation and understanding ability, and stability ability, ultimately generating a structured comprehensive evaluation report.

[0027] The comprehensive evaluation report aims to systematically present a panoramic view of the model's performance, and its content includes at least: 1) Dimensional performance summary: In a quantifiable form, it focuses on showcasing the model's overall performance in the three core dimensions of classification, interpretation and understanding, and stability; 2) Fine-grained performance analysis: For different types of RDFS logical axioms, it lists detailed evaluation results for the three dimensions to reveal the model's specific performance and capability boundaries when processing various semantic relationships; 3) Prompt word effectiveness analysis: It compares and shows the impact of different prompt word strategies on model performance, providing direct data basis for subsequent prompt word optimization; 4) Model capability diagnosis and optimization suggestions: Based on the above analysis, it identifies the model's key performance bottlenecks and potential defects, and proposes targeted optimization directions accordingly.

[0028] Furthermore, the core of the prompt word optimization step lies in constructing a data-driven closed loop from performance evaluation to strategy generation. This step first transforms the comprehensive evaluation report into a structured and analyzable system performance graph by establishing a ternary mapping relationship between "prompt word type - evaluation dimension - axiom type". Based on this map, this step performs bidirectional diagnostic analysis: 1) Strategy performance profile construction: Based on the ternary mapping relationship, quantify the performance of each prompt word strategy on specific axiom types and evaluation dimensions, and generate a strategy capability matrix to clarify the advantageous scenarios and failure boundaries of each strategy; 2) Bidirectional diagnostic analysis: Based on the strategy capability matrix, perform the following analysis: Horizontal comparative analysis: Under the same axiom type and evaluation dimension, screen prompt word strategies with significantly better performance than the benchmark; Vertical bottleneck diagnosis: Identify the model's capability shortcomings on specific evaluation dimensions or axiom types; 3) Dynamic strategy generation: In response to the results of the bidirectional diagnostic analysis, when the existing strategy cannot meet the performance requirements, automatically generate optimized prompt word strategies for the identified capability shortcomings through semantic element deconstruction and recombination; 4) Optimization decision output: Output the optimized prompt word strategy set and automatically configure it into the model test execution step to guide the subsequent evaluation or application of the large language model; at the same time, generate a structured optimization decision report, which at least includes scenario-based strategy recommendations and their generation basis. This step, through the aforementioned data-driven closed-loop optimization mechanism, transforms the prompt word engineering from an experience-dependent process into an interpretable and reproducible systematic process, ultimately achieving targeted enhancement of the model ontology's learning capabilities.

[0029] Secondly, the present invention provides a large language model evaluation and prompt word optimization device for RDFS ontology, comprising:

[0030] The test dataset construction module is used to automatically extract logical axioms that conform to the RDFS specification from existing ontology or knowledge base to form a set of positive examples, and automatically generate a set of negative examples corresponding to the set of positive examples based on preset logical conflict rules, thereby forming a standardized test dataset for evaluation.

[0031] The prompt word engineering design module is used to design and store prompt word templates at multiple levels. The types include at least the basic type without semantic explanation, the instruction-enhanced type with axiomatic semantic explanation, and the context-guided type that provides relevant semantic context.

[0032] The model test execution module is used to combine test cases in the standardized test dataset with different types of prompt word templates to generate input sequences to the large language model to be evaluated, and to obtain the judgment results and explanation text output by the large language model.

[0033] The multi-dimensional evaluation module is used to parse the judgment results and explanatory text, and to conduct quantitative evaluation from three dimensions: classification ability, explanation and understanding ability, and stability ability, and generate a comprehensive evaluation report.

[0034] The prompt word optimization module is used to analyze the impact of different prompt word templates on the performance of the large language model in each evaluation dimension based on the comprehensive evaluation report, and to select or generate optimized prompt word strategies accordingly.

[0035] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0036] Thirdly, the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed on a computing device, cause the computing device to perform the large language model evaluation and prompt word optimization method for RDFS ontology as described in any of the claims of the first aspect.

[0037] Fourthly, the present invention provides an electronic device, comprising:

[0038] Memory is used to store executable instructions for a computer;

[0039] A processor is configured to execute computer-executable instructions stored in the memory to implement the large language model evaluation and prompt word optimization method for RDFS ontology as described in the first aspect.

[0040] Fifthly, the present invention provides a computer program product comprising computer program instructions that, when the computer program instructions are executed on a computer, cause the computer to perform the large language model evaluation and prompt word optimization method for RDFS ontology as described in the first aspect.

[0041] Compared with existing technologies, this invention achieves four fundamental breakthroughs by constructing an integrated "evaluation-optimization" technical paradigm: 1) In terms of evaluation mechanism, it pioneers an automatic negative example generation method based on logical conflict, combined with hierarchical prompt word design, solving the problem of insufficient evaluation of deep semantic understanding of RDFS logical axioms in existing technologies, and achieving full capability coverage from simple semantic recognition to complex logical reasoning; 2) In terms of evaluation system, it is the first to incorporate explanation and understanding capabilities into the core evaluation dimensions, and by introducing indicators such as illusion rate, it establishes a comprehensive evaluation standard covering classification ability, explanation credibility, and stability, overcoming the defect of existing technologies that ignore the credibility of the reasoning process; 3) In terms of optimization method, by establishing a precise mapping mechanism between prompt word strategy and multi-dimensional performance, it achieves a fundamental shift from manual trial and error to data-driven approach, forming an interpretable and iterative closed-loop optimization process; 4) In terms of technical universality, the evaluation framework and optimization process of this invention are independent of specific large language models and domain ontology, possessing high universality and scalability, solving the common industry problem of weak generalization ability and difficulty in horizontal comparison caused by the high degree of customization in existing solutions. The organic integration of these four innovations constitutes a complete technical solution that is not available in existing technologies. It provides a reliable general benchmark and tool for the systematic evaluation and performance improvement of large language models in the field of knowledge reasoning, and ultimately serves the core goal of building high-quality and highly reliable knowledge graphs. Attached Figure Description

[0042] To more intuitively demonstrate the technical solution and implementation process of the present invention, a brief description of the accompanying drawings referenced in this specification is provided below. The accompanying drawings are intended to assist in understanding the structural composition and operational flow of the present invention and are schematic in nature.

[0043] Figure 1 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be noted that the described embodiments are merely illustrative examples of the technical solutions of this invention and do not constitute a limitation on the scope of protection of this invention.

[0045] Example 1: This example introduces a method for evaluating large language models and optimizing prompt words for RDFS ontology, including:

[0046] Test dataset construction steps: Automatically extract logical axioms conforming to the RDFS specification from existing ontology or knowledge base to form a set of positive examples, and automatically generate a set of negative examples corresponding to the set of positive examples based on preset logical conflict rules, thereby forming a standardized test dataset for evaluation.

[0047] Prompt word engineering design steps: Design prompt word templates at multiple levels, including at least a basic type without semantic explanation, an instruction-enhanced type with axiomatic semantic explanation, and a context-guided type that provides relevant semantic context;

[0048] Model testing execution steps: Combine the test cases in the standardized test dataset with different types of prompt word templates to generate an input sequence to the large language model to be evaluated, and obtain the output judgment results and explanation text;

[0049] Multi-dimensional evaluation steps: The judgment results and explanatory text are analyzed, and quantitative evaluation is carried out from three dimensions: classification ability, explanation and understanding ability, and stability ability, and a comprehensive evaluation report is generated.

[0050] Prompt word optimization steps: Based on the comprehensive evaluation report, analyze the impact of different prompt word templates on the performance of the large language model in each evaluation dimension, and select or generate optimized prompt word strategies accordingly.

[0051] like Figure 1 As shown in this embodiment, the method for evaluating large language models and optimizing prompt words for RDFS ontology involves the following steps in its application process:

[0052] Step 1: Construct the dataset of positive and negative examples, which includes the following steps:

[0053] Step 1.1: Constructing a set of positive examples relies on programmatic access to and information extraction from structured knowledge sources. The technical implementation is based on two types of standard interfaces: standard ontology processing interfaces for parsing local ontology files (such as OWL API and JenaAPI), and standard structured query interfaces for querying remote knowledge bases (such as SPARQL endpoints), thereby ensuring the authority of the data source, the automation of data extraction, and high coverage. For knowledge sources that do not provide programmatic interfaces, the logical axioms are obtained by parsing data in their graphical user interface. In this embodiment, high-quality knowledge sources with broad acceptance are preferred: DBpedia, YAGO, and NELL.

[0054] 1) Automated extraction based on DBpedia and YAGO knowledge bases: For large knowledge bases such as DBpedia and YAGO that provide standard SPARQL endpoints, positive examples of the four core RDFS logical axioms can be automatically and in batches obtained by designing and executing specific SPARQL query statements.

[0055] (1) The query statement for obtaining the parent-child relationship is:

[0056] PREFIX rdfs:<http: / / www.w3.org / 2000 / 01 / rdf-schema#>

[0057] SELECT ?subclass ?superclass

[0058] WHERE {?subclass rdfs:subClassOf ?superclass .}

[0059] Here, "PREFIX rdfs:" defines "rdfs:" as a namespace prefix, pointing to the standard URI of the RDFS vocabulary. "http: / / www.w3.org / 2000 / 01 / rdf-schema#" represents the standard Uniform Resource Identifier (URI) of the RDFS vocabulary, used to define the RDFS namespace. "SELECT ?subclass ?superclass" indicates that we want to retrieve two variables from the query results: "?subclass" and "?superclass". "WHERE" contains the graphical schema, in which "?subclass rdfs:subClassOf ?superclass ." means that we want to find all data pairs that satisfy the relationship "a class is a subclass of another class".

[0060] (2) The query statement for obtaining the parent-child attribute relationship is:

[0061] PREFIX rdfs:<http: / / www.w3.org / 2000 / 01 / rdf-schema#>

[0062] SELECT ?subproperty ?superproperty

[0063] WHERE {?subproperty rdfs:subPropertyOf ?superproperty .}

[0064] The query "?subproperty rdfs:subPropertyOf ?superproperty ." searches for all data pairs that satisfy the condition "a property is a subproperty of a larger property".

[0065] (3) The query statement for obtaining the axiom of the attribute domain is:

[0066] PREFIX rdfs:<http: / / www.w3.org / 2000 / 01 / rdf-schema#>

[0067] SELECT ?property ?domain

[0068] WHERE {?property rdfs:domain ? domain .}

[0069] This query focuses on the relationship between an attribute and its domain. The query "?property rdfs:domain ?domain ." finds all data pairs that satisfy the description "a certain attribute applies to an object of a specific type".

[0070] (4) The query statement for obtaining the axiom of the attribute value range is:

[0071] PREFIX rdfs:<http: / / www.w3.org / 2000 / 01 / rdf-schema#>

[0072] SELECT ?property ?range

[0073] WHERE {?property rdfs:range ?range .}

[0074] This query involves the relationship between an attribute and its value range. The query "?property rdfs:range ?range ." finds all data pairs that satisfy the description "the value of a certain attribute must belong to an object of a specific type".

[0075] 2) Acquisition based on the NELL knowledge base: For continuous learning systems like NELL, ontology information can be obtained through the graphical interface provided on its official website. By browsing its "Categories" (class hierarchy) and "Relations" (attribute hierarchy) interfaces, positive examples of subclass relation axioms and subattribute relation axioms can be manually or randomly selected and recorded through simulated clicks. Furthermore, for the details interface of each attribute (Relation), NELL provides the domain and range information for that attribute, thereby allowing the construction of positive examples of the domain and range definition axioms.

[0076] Step 1.2: Construct a set of negative examples through systematic semantic transformation. This process is designed as a controllable "logic conflict injection" mechanism, mainly following two principles: first, "logical orientation subversion," which directly constructs fallacies by reversing the directionality in the axioms of hierarchical relationships; second, "constraint consistency violation," which utilizes existing logical assertions in the ontology (such as mutually exclusive concept declarations, existential quantifier constraints, and instantiated data) to derive new axioms that contradict them through deductive reasoning, thereby constructing a deeper level of semantic conflict.

[0077] Negative example generation based on the principle of "logical reversal": This principle applies to hierarchical relationship axioms (subclass, subattribute), constructing semantic fallacies by directly reversing the logical direction of elements in the axiom. Specific implementation: For any hierarchical relationship positive example, a negative example is generated by programmatically swapping the positions of its subject and object. For example, given a subclass relationship positive example: Student ⊑ Person (Student is a subclass of Person), the generated negative example is: Person ⊑ Student (Person is a subclass of Student). Here, ⊑ represents a parent-child class relationship, where the class on the left is a subclass of the class on the right.

[0078] Domain constraint negative example generation based on the "constraint consistency violation" principle: This principle applies to domain constraint axioms. By utilizing existing logical assertions in the ontology for deductive reasoning, new axioms that contradict known facts are actively constructed. Based on ontology reasoning tools such as the OWL API or Jena API, the following strategies can be executed automatically:

[0079] Strategy D1: Construct counterexamples using the mutual exclusion axiom of concepts and domain declarations. The reasoning logic of this strategy is as follows: Assume that the domain of attribute r is class X, and that class X and class Y are disjoint. Then, claiming that class Y is the domain of attribute r constitutes a negative example. For example, given that the domain of hasGraduatedFrom is Person, and Person and Location are disjoint, a negative example can be automatically generated: the domain of hasGraduatedFrom is Location.

[0080] Strategy D2: Construct counterexamples using the axiom of mutual exclusion of concepts and quantifier constraints. The reasoning logic of this strategy is as follows: Assume that classes X and Z are disjoint, and that instances of X are connected to instances of Y through the attribute r (i.e., X ⊑ ∃rY). Here, ∃ represents an existence constraint. Then, claiming that Z is the domain of r implies X ⊑ Z, which contradicts the known disjointness of X and Z; therefore, this claim can serve as a negative example. For example, given that Student and University are disjoint, and Student ⊑ ∃hasTeacher.Professor, where hasTeacher means there is a teacher and Professor means professor, a negative example can be generated: University is the domain of hasTeacher.

[0081] Strategy D3: Constructing counterexamples using instantiation contradictions. The reasoning logic of this strategy is as follows: Assume that individual a belongs to class Y (i.e., Y(a)), individual a is connected to individual b through attribute r (i.e., r(a, b)), and class X and class Y are disjoint. Then, claiming that X is the domain of r implies that a also belongs to X (i.e., X(a)), causing individual a to belong to two disjoint classes simultaneously, creating a contradiction. Therefore, this claim can serve as a negative example. For example, given that individual Alice is Professor, Alice wrote a book Book1 (i.e., hasWritten(Alice, Book1)), and Professor and Book are disjoint, a negative example can be generated: Book is the domain of hasWritten.

[0082] Negative instance generation based on the "constraint consistency violation" principle: This principle applies to range constraint axioms. By utilizing existing logical assertions in the ontology for deductive reasoning, new axioms that contradict known facts are actively constructed. Based on ontology reasoning tools such as the OWL API or Jena API, the following strategies can be executed automatically:

[0083] Strategy R1: Construct counterexamples using the mutual exclusion axiom of concepts and domain declarations. The reasoning logic of this strategy is as follows: Assume that class X is the domain of attribute r, and that class X and class Y are disjoint. Then, claiming that class Y is the domain of attribute r constitutes a negative example. For example, given that the domain of hasAuthor (hasAuthor) is Person, and Person and Book are disjoint, a negative example can be automatically generated: Book is the domain of hasAuthor.

[0084] Strategy R2: Constructing counterexamples using the axiom of mutual exclusion and quantifier constraints. The reasoning logic of this strategy is as follows: Assume that class Y and class Z are disjoint, and that instances of X are connected to instances of Y through the attribute r (i.e., X ⊑ ∃ rY). Then, claiming that Z is the range of r implies the existence of an individual that belongs to both Y and Z, contradicting the disjointness axiom; therefore, this claim can be a negative example. For example, given that Professor and Textbook are disjoint, and UniversityCourse ⊑ ∃ hasInstructor.Professor, where UniversityCourse represents a university course and hasInstructor indicates an instructor, a negative example can be generated: Textbook is the range of hasInstructor.

[0085] Strategy R3: Constructing counterexamples using instantiation contradictions. The reasoning logic of this strategy is as follows: Assume that individual b belongs to class Y (i.e., Y(b)), individual a is connected to individual b through attribute r (i.e., r(a, b)), and class X and class Y are disjoint. Then, claiming that X is the range of r implies that b also belongs to X, causing individual b to belong to two disjoint classes simultaneously, creating a contradiction. Therefore, this claim can serve as a negative example. For example, given that individual Book1 is Book (i.e., Book(Book1)), Alice wrote Book1 (i.e., hasWritten(Alice, Book1)), and Person is disjoint to Book, a negative example can be generated: Person is the range of hasWritten.

[0086] Step 2: Design prompt word templates at multiple levels, including at least a basic type without semantic explanation, an instruction-enhanced type with axiomatic semantic explanation, and a context-guided type that provides relevant semantic context.

[0087] The technical feature of the basic prompt words lies in using concise task instructions that do not contain specific axiom type descriptions or additional contextual information, directly requiring the large language model to judge the correctness of the input axioms. This type of prompt word relies entirely on the model's own intrinsic knowledge for zero-shot inference, aiming to evaluate the model's baseline semantic understanding and logical discrimination capabilities under the simplest instructions. The prompt word template can be designed as: "Please judge whether the following RDFS ontology axiom is correct: [Axiom content]". For example, when the input axiom is student ⊑ person, the complete prompt is: "Please judge whether the following RDFS ontology axiom is correct: student ⊑ person".

[0088] The technical feature of the enhanced prompt words lies in the addition of semantic definitions and logical rule descriptions of the axiom type being judged to the basic prompt words, guiding the large language model to focus on specific types of semantic constraints. This type of prompt word is suitable for evaluating the model's ability to distinguish different types of ontology axioms and its compliance with specific logical rules. Taking subclass relations as an example, the prompt word template could be: "Please judge whether the following subclass inclusion relation is correct. If a class A is a subclass of class B, then every individual belonging to A also belongs to B. Axiom: [Axiom content]". For example, for the axiom "Pianist ⊑ Musician", the complete prompt is: "Please judge whether the following subclass inclusion relation is correct. If a class A is a subclass of class B, then every individual belonging to A also belongs to B. Axiom: Pianist ⊑ Musician".

[0089] The technical feature of the context-guided prompts lies in further introducing background knowledge fragments related to the target axiom into the instructions. This background knowledge includes hierarchical structures of classes or attributes, instantiation relationships, or existing axiom fragments, to construct a semantic context environment to assist reasoning. This type of prompt can effectively examine the model's ability to judge semantic consistency in multi-hop reasoning, information integration, and complex contexts. For example, when judging a subclass relationship, relevant class hierarchy information is provided. The prompt template can be: "Given [relevant background knowledge]. Based on the above information, please determine whether the following axiom is correct: [axiom content]". For example, "Given: Artist ⊑ Person, Musician ⊑ Artist. Please determine whether the following axiom is correct: Pianist ⊑ Musician." By providing the context that "Artist is a subclass of Person, Musician is a subclass of Artist," the model is assisted in chain reasoning to determine whether "Pianist is a subclass of Musician."

[0090] Step 3: Combine the test cases in the standardized test dataset with different types of prompt word templates to generate an input sequence to the large language model to be evaluated, and obtain the output judgment result and explanation text.

[0091] Specifically, each test case in the standardized test dataset is combined with prompt word templates at different levels. A complete input sequence is generated by dynamically filling in the axioms and context placeholders in the templates, and structured output format requirements are added (such as "Judgment result: [Correct / Incorrect]; Explanation: [Reasoning process]"). The constructed input sequences are submitted in batches to the large language model to be evaluated (such as GPT-4, LLaMA, etc.). The binary judgment results returned by the model and their corresponding natural language explanation text are obtained through the API interface. Special attention should be paid to the possible "illusion" phenomenon in the explanation text. Finally, the model output is parsed and stored to form a raw data set containing judgment labels and reasoning processes, providing a foundation for subsequent multi-dimensional evaluation.

[0092] Step 4: Analyze the judgment results and explanation text, and conduct a quantitative evaluation from three dimensions: classification ability, explanation and understanding ability, and stability ability, and generate a comprehensive evaluation report.

[0093] In this embodiment, step 4 specifically includes the following steps:

[0094] Step 4.1: Multi-dimensional Evaluation. For the judgment results and explanatory text, automated parsing and quantitative statistics are performed from the following three core dimensions: 1) Classification ability: Evaluating the accuracy of the large language model in correctly classifying the RDFS logical axioms into binary categories; 2) Explanation and understanding ability: Evaluating the semantic correctness and logical rationality of the explanatory text provided by the large language model for its judgment results; 3) Stability ability: Evaluating the consistency of the output results of the large language model when faced with semantically equivalent input perturbations. These three dimensions, from the perspectives of judgment results, internal reasoning process, and external robustness, together constitute a hierarchical and systematic capability evaluation system. In specific implementation:

[0095] 1. Evaluation of classification ability: Based on the confusion matrix formed by the model's judgment results and the true labels, and covering accuracy, precision, recall, and F1 score:

[0096] 1) Classification Accuracy: This measures the proportion of correct judgments across all judgments made by the model. It can be formally defined as:

[0097] Accuracy Classify = .

[0098] In this context, TP stands for True Positive, representing the number of positive examples that are correctly identified as axioms; TN stands for True Negative, representing the number of negative examples that are incorrectly identified as axioms; FP stands for False Positive, representing the number of negative examples that are incorrectly identified as correct axioms; and FN stands for False Negative, representing the number of positive examples that are incorrectly identified as incorrect axioms.

[0099] 2) Precision: This measures the proportion of actual positive examples among those predicted as positive by the model. In other words, it measures how many of the axioms predicting "correctly" are actually true. It can be formally defined as:

[0100] Precision = .

[0101] 3) Recall: This measures the proportion of all true positive examples that the model identifies, i.e., how many of all "correct" axioms are judged as correct. It can be formally defined as follows:

[0102] Recall = .

[0103] 4) F1 score: A harmonic mean that comprehensively considers precision and recall, suitable for scenarios with imbalanced positive and negative samples, and can be formally defined as:

[0104] F1 = .

[0105] The above metrics can be calculated separately for the four types of RDFS axioms (subclass, subattribute, domain, and range) to achieve fine-grained performance analysis.

[0106] 2. Assessment of Explanation and Comprehension Ability: The reliability of the model's reasoning process is assessed by quantifying the accuracy of explanations and the rate of hallucination.

[0107] 1) Explanation Accuracy: This measures whether the model can accurately explain the reasoning behind its judgments while providing the "judgment result." It represents the proportion of correctly explained examples where the model provides the correct explanation. The specific definition is as follows:

[0108] Accuracy Explain = .

[0109] TP_EC and TN_EC represent the number of positive examples that are correctly judged and also correctly explained, and the number of negative examples that are correctly judged and also correctly explained, respectively.

[0110] 2) Illusion Rate: This refers to the proportion of illusory content in the model's explanation among all correctly judged examples. This metric reflects the frequency with which the model contains fabricated information, unfounded inferences, or factual errors in its output explanation. Specifically, illusory content may manifest as: referencing non-existent classes or attributes, fabricating undefined relationships in the ontology, misusing ontology semantic rules, etc. The illusion rate can be formally defined as follows:

[0111] Hallucination = .

[0112] This metric is used to quantify the content credibility of large language models in knowledge reasoning and is an important supplementary indicator for evaluating model stability.

[0113] It is worth noting that for correctly judged positive or negative examples, the correctness of the explanation is divided into three cases: the explanation is completely correct, the explanation is incorrect and contains illusions, and the explanation is incorrect and does not contain illusions. In the practical application of large language models, the last case is relatively rare; incorrect explanations are often due to illusions generated by the large language model. Furthermore, the judgment of the explanation correctness and illusion rate can be achieved through at least one of the following methods: a) manual annotation and verification by a person skilled in the art based on ontology knowledge; b) consistency comparison between the explanation text and a known ontology knowledge base based on rules or a pre-trained verification model.

[0114] 3. Stability Assessment: A consistency index is calculated by constructing a semantically equivalent set of input perturbations (e.g., adjusting the order of axiom statements or replacing synonyms).

[0115] 1) Hard consistency: refers to the ability of a large language model to maintain complete consistency in its output when faced with semantically invariant input perturbations. In the RDFS logical axiom judgment task, if the model gives different judgment results for multiple equivalent expressions of the same semantics (such as axiom order rearrangement, different forms of axiom representation, etc.) (e.g., some are "correct" and some are "incorrect"), it is considered to violate hard consistency. Hard consistency can be formally defined as follows: Let the total number of samples be N, where N-Hard samples satisfy complete output consistency, then the hard consistency metric is:

[0116] Consistency Hard = .

[0117] 2) Soft consistency: This can be measured using the mean squared error (MSE). A smaller MSE indicates a more stable model judgment across different statements; conversely, a larger MSE indicates greater fluctuation in model output and a lack of stability. Soft consistency evaluation can more precisely reflect the model's fluctuation trend when facing input changes, effectively complementing hard consistency. Soft consistency can be formally defined as follows: For each test case... Construct k semantically equivalent perturbation versions , , …, The discrete output corresponding to the model is denoted as: , , …, ,in It is either 0 or 1, where 0 indicates a large language model for the example. If the condition is incorrect, the condition is correct; otherwise, it is correct (1). Example mean Formalizable definition:

[0118] =

[0119] Sample The mean square error can be formally defined as:

[0120] =

[0121] If the test set has N samples, then overall soft consistency can be defined as:

[0122]

[0123] Step 4.2: Based on the above quantitative evaluation, generate a structured comprehensive evaluation report. Its content should include at least: 1) Dimensional performance summary: Presenting the model's overall performance across the three core dimensions of classification, interpretation and understanding, and stability in a quantifiable form; 2) Fine-grained performance analysis: Listing detailed evaluation results for each of the three dimensions for different types of RDFS logical axioms, revealing the model's specific performance and capability boundaries when processing various semantic relationships; 3) Prompt word effectiveness analysis: Comparing and demonstrating the impact of different prompt word strategies on model performance, providing direct data for subsequent prompt word optimization; 4) Model capability diagnosis and optimization suggestions: Based on the above analysis, identifying the model's key performance bottlenecks and potential defects, and proposing targeted optimization directions accordingly.

[0124] Step 5: Based on the data analysis results of the comprehensive evaluation report, a system performance graph is formed by constructing a ternary mapping relationship of "prompt word type - evaluation dimension - axiom type", and the following specific optimization process is implemented accordingly:

[0125] Step 5.1: Policy Performance Profile Construction. The system reads data from the comprehensive evaluation report and automatically constructs a three-dimensional mapping table. For example, given an axiom type (such as "value range constraint") and an evaluation dimension (such as "explanation accuracy"), the table can list the performance scores (such as 0.85, 0.92, 0.78) of all prompt word types (basic, instruction-enhanced, context-guided) under this combination, thereby generating a visualized policy capability matrix.

[0126] Step 5.2: Two-way diagnostic analysis. The system performs automated diagnosis based on the above strategy capability matrix:

[0127] Step 5.2.1: Horizontal Comparative Analysis. The system automatically selects prompt strategies that significantly outperform the benchmark (e.g., average performance) within the same axiom type and evaluation dimension. For example, the analysis reveals that for the "classification ability" of "domain axioms," the instruction-enhanced prompt strategy achieves an F1 score of 0.95, while the basic one is only 0.82. The system then records the "instruction-enhanced" strategy as the superior strategy in this scenario.

[0128] Step 5.2.2: Vertical Bottleneck Diagnosis. The system identifies its weaknesses by comparing the model's performance across different axiom types or evaluation dimensions. For example, longitudinal analysis revealed that the model's "interpretation accuracy" (0.65) was significantly lower than other axiom types (average 0.88) when dealing with the "range axiom," thus diagnosing "understanding the range axiom" as the model's key bottleneck.

[0129] Step 5.3: Dynamic Strategy Generation. When the diagnostics reveal fundamental limitations in the existing strategy (e.g., all existing prompts have an illusion rate exceeding 40% for the "value range axiom"), the system initiates a strategy generation mechanism. This mechanism deconstructs the semantic elements (such as logical rule descriptions and contextual structure) of existing high-performance prompts and reorganizes and semantically enhances elements to address identified capability gaps. For example, to address the weakness in understanding the value range axiom, the system automatically generates a new prompt template. This template, based on the instruction enhancement type, additionally inserts explicit definitions of value range constraints and counterexamples illustrating easily confused points.

[0130] Step 5.4: Optimize Decision Output. The system ultimately outputs two parts: 1) An executable strategy set: The newly generated prompt word strategies (such as the new template for the value range axiom mentioned above) are packaged into a strategy set and automatically updated and configured into the strategy library of the "Model Test Execution Module" through the application programming interface to guide the next round of model evaluation or practical application. 2) A structured report: An optimization decision report is also generated for R&D personnel to review. The report clearly lists in tabular form: scenario-based strategy recommendations (e.g., "It is recommended to use the newly generated 'value range-specific optimization' prompt word when handling the value range axiom"); the basis for generation (e.g., "This strategy was diagnosed as stemming from the model's widespread illusion of value range logic. The new strategy, by adding counterexamples, reduced the illusion rate by 22% in the initial verification").

[0131] This step, through the aforementioned data-driven closed-loop optimization mechanism, transforms the prompt word engineering from manual trial and error into an interpretable and iterative systematic process, ultimately achieving targeted and continuous enhancement of the model's RDFS ontology learning ability.

[0132] Example 2: This example provides a large language model evaluation and prompt word optimization device for RDFS ontology, including:

[0133] The test dataset construction module is configured to automatically extract logical axioms conforming to the RDFS specification from existing ontology or knowledge base to form a set of positive examples, and automatically generate a set of negative examples corresponding to the set of positive examples based on preset logical conflict rules, thereby forming a standardized test dataset for evaluation.

[0134] The prompt word engineering design module is configured to design and store prompt word templates at multiple levels. The types include at least a basic type without semantic explanation, an instruction-enhanced type with axiomatic semantic explanation, and a context-guided type that provides relevant semantic context.

[0135] The model test execution module is configured to combine test cases in the standardized test dataset with different types of prompt word templates to generate an input sequence to the large language model to be evaluated, and obtain the judgment result and explanation text output by the large language model.

[0136] The multi-dimensional evaluation module is configured to parse the judgment results and explanatory text, and to perform quantitative evaluation from three dimensions: classification ability, explanation and understanding ability, and stability ability, and generate a comprehensive evaluation report.

[0137] The prompt word optimization module is configured to analyze the impact of different prompt word templates on the performance of the large language model across various evaluation dimensions based on the comprehensive evaluation report, and thereby select or generate optimized prompt word strategies.

[0138] Example 3: This example provides a computer-readable storage medium storing computer instructions that, when executed on a computing device, cause the computing device to perform the large language model evaluation and prompt word optimization method for RDFS ontology as described in any of the first aspects.

[0139] Example 4: This example provides an electronic device, including:

[0140] Memory is used to store executable instructions for a computer;

[0141] A processor is configured to execute computer-executable instructions stored in the memory to implement the large language model evaluation and prompt word optimization method for RDFS ontology as described in any of the first aspects.

[0142] Example 5: This example provides a computer program product containing computer program instructions that, when executed on a computer, cause the computer to perform the large language model evaluation and prompt word optimization method for RDFS ontology as described in any of the first aspects.

[0143] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A method for evaluating large language models and optimizing prompt words for RDFS ontology, characterized in that, The method includes the following steps: Step 1: Automatedly construct a standardized test dataset containing positive and negative examples; Step 2: Design prompt word templates at multiple levels, including at least a basic type without semantic explanation, an instruction-enhanced type with axiomatic semantic explanation, and a context-guided type that provides relevant semantic context; Step 3: Combine the test cases in the standardized test dataset with the different types of prompt word templates to generate an input sequence to the large language model to be evaluated, and obtain the output judgment results and explanation text; Step 4: Analyze the judgment results and explanatory text, and conduct a quantitative evaluation from three dimensions: classification ability, explanation and comprehension ability, and stability ability, and generate a comprehensive evaluation report; Step 5: Based on the data analysis results of the comprehensive evaluation report, a system performance map is formed by constructing a three-element mapping relationship of "prompt word type - evaluation dimension - axiom type", and the optimization process is implemented accordingly; Specifically, step 1 includes the following steps: Step 1.1: Constructing the set of positive examples is achieved by calling a standardized program interface to automatically extract logical axioms that conform to the RDFS specification from the structured knowledge source; Step 1.2: Constructing the negative example set is achieved by applying a systematic semantic transformation to the positive example set. The semantic transformation algorithm includes at least two strategies: logical pointer subversion and constraint consistency violation. The aforementioned logical subversion directly constructs fallacies by reversing the directionality in the axiom of hierarchical relationships; The aforementioned constraint consistency violation utilizes existing logical assertions in the ontology to perform automatic deductive reasoning in order to generate negative examples of semantic conflicts. The various levels of prompt word templates in step 2 include at least the following types: Basic prompt template, using task instructions that do not include specific axiom type descriptions or additional contextual information. The enhanced prompt word template adds semantic definitions and logical rule descriptions of the axiom type being judged to the instructions in the basic prompt word template. Context-guided prompt templates introduce background knowledge fragments related to the target axiom into the instructions to construct a semantic context environment that assists reasoning.

2. The method for evaluating large language models and optimizing prompt words for RDFS ontology according to claim 1, characterized in that, The constraint consistency violation in step 1.2 is achieved by using existing logical assertions in the ontology for automatic deductive reasoning to generate negative examples of semantic conflicts, specifically including at least one of the following methods: 1) Violation of the axiom of mutual exclusion of concepts: Based on two classes declared as disjoint in the ontology, the domain or value range of an attribute originally applied to one class is changed to the other disjoint class, thereby generating a logical conflict; 2) Violation of quantifier constraints: Based on the inclusion axiom that there are attribute constraints between descriptive classes, the domain or range of attribute P is changed to a class Z that would logically contradict the axiom, thus generating a logical conflict; 3) Destruction based on instantiated data: Based on the specific instances and their attribute relationships that already exist in the knowledge base, the domain or value range of attribute P is tampered with to a class that logically conflicts with the instance type C, thereby generating a logical conflict.

3. The method for evaluating large language models and optimizing prompt words based on RDFS ontology according to claim 1, characterized in that, Step 3 specifically includes: Test cases from a standardized test dataset are combined with different types of prompt word templates, and a complete input sequence is generated through dynamic filling. The input sequences are submitted in batches to the large language model through a programmatic interface, and the output containing the judgment results and explanation text is obtained. The output is parsed to form a structured raw data set for subsequent evaluation.

4. The method for evaluating large language models and optimizing prompt words for RDFS ontology according to claim 1, characterized in that, Step 4 is as follows: Step 4.1: Multi-dimensional evaluation. Based on the judgment results and explanatory text, automated analysis and quantitative statistics are performed from the following three core dimensions: The classification ability is evaluated through one or more metrics based on the confusion matrix, including accuracy, precision, recall, and F1 score. The assessment of interpretation and comprehension ability is achieved through interpretation accuracy rate and illusion rate; interpretation accuracy rate is used to measure the proportion of correctly judged examples in which the interpretation text is also correct; illusion rate is used to measure the proportion of correctly judged examples in which the interpretation text contains fictitious or incorrect information. The stability capability is evaluated by constructing a semantically equivalent set of input perturbations and calculating a consistency index, which includes hard consistency and soft consistency. Step 4.2: Based on the above quantitative evaluation, generate a structured comprehensive evaluation report, which should include at least the following: 1) Dimensional performance summary: In a quantifiable form, show the overall performance of the model in the three core dimensions of classification, interpretation and understanding, and stability; 2) Fine-grained performance analysis: For different types of RDFS logical axioms, detailed evaluation results of the three dimensions are listed to reveal the specific performance and capability boundaries of the model when processing various semantic relationships; 3) Prompt word effectiveness analysis: The impact of different prompt word strategies on model performance is compared and shown to provide direct data basis for subsequent prompt word optimization; 4) Model capability diagnosis and optimization suggestions: Based on the above analysis, the key performance bottlenecks and potential defects of the model are identified, and targeted optimization directions are proposed accordingly.

5. The method for evaluating large language models and optimizing prompt words for RDFS ontology according to claim 1, characterized in that, Step 5 is as follows: 1) Strategy performance profile construction: Based on the ternary mapping relationship of "prompt word type - evaluation dimension - axiom type", the performance of each prompt word strategy on specific axiom type and evaluation dimension is quantified to generate a strategy capability matrix; 2) Two-way diagnostic analysis: The following analysis is performed based on the strategy capability matrix: Horizontal comparative analysis: Under the same axiom type and evaluation dimensions, the filtering performance of the prompt word strategy is significantly better than the benchmark. Vertical bottleneck diagnosis: Identifying the model's shortcomings in specific evaluation dimensions or axiom types; 3) Dynamic strategy generation: In response to the results of bidirectional diagnostic analysis, when the existing strategy cannot meet the performance requirements, an optimized strategy is automatically generated through semantic element deconstruction and recombination. 4) Optimize decision output: Based on the results generated by the dynamic strategy, output an optimized set of prompt word strategies and automatically configure them into the model test execution module to guide the subsequent evaluation or application of the large language model; at the same time, generate a structured optimization decision report, which includes at least scenario-based strategy recommendations and their generation basis.

6. A large language model evaluation and prompt word optimization device for RDFS ontology, characterized in that, The apparatus for implementing the method according to any one of claims 1-5, the apparatus comprising: The test dataset construction module is configured to automatically extract logical axioms conforming to the RDFS specification from existing ontology or knowledge base to form a set of positive examples, and automatically generate a set of negative examples corresponding to the set of positive examples based on preset logical conflict rules, thereby forming a standardized test dataset for evaluation. The prompt word engineering design module is configured to design and store prompt word templates at multiple levels. The types include at least a basic type without semantic explanation, an instruction-enhanced type with axiomatic semantic explanation, and a context-guided type that provides relevant semantic context. The model test execution module is configured to combine test cases in the standardized test dataset with different types of prompt word templates to generate an input sequence to the large language model to be evaluated, and obtain the judgment result and explanation text output by the large language model. The multi-dimensional evaluation module is configured to parse the judgment results and explanatory text, and to perform quantitative evaluation from three dimensions: classification ability, explanation and understanding ability, and stability ability, and generate a comprehensive evaluation report. The prompt word optimization module is configured to analyze the impact of different prompt word templates on the performance of the large language model across various evaluation dimensions based on the comprehensive evaluation report, and thereby select or generate optimized prompt word strategies.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a method for large language model evaluation and prompt word optimization for RDFS ontology as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, the computer instructions implement a method for large language model evaluation and prompt word optimization for RDFS ontology as described in any one of claims 1-5.

Images