Assessment and enhancement method and device for LLMs concept mutual exclusion recognition capability

Abstract

The invention discloses an evaluation and enhancement method and device for the concept mutual exclusion recognition capability of LLMs, and belongs to the technical field of evaluation of large language models. The method comprises the following steps: acquiring original data from a specified knowledge source, and constructing a structured concept hierarchy diagram; based on the original data and the concept hierarchy diagram, constructing an evaluation data set containing a mutual exclusion concept pair positive example and a non-mutual exclusion concept pair negative example; designing a multi-level cue word system, wherein the multi-level cue word system comprises a direct expression type cue word, a field limiting type cue word and a structure guiding type cue word; and combining the concept pair samples in the evaluation data set with the cue words, inputting the combined concept pair samples and cue words into the LLMs to be evaluated, and evaluating a judgment result output by the LLMs and a reasoning basis from three dimensions of judgment accuracy, interpretation reliability and cue word robustness and enhancement effect. According to the method, benchmark construction, prompt enhancement and multi-dimensional verification are organically integrated in one framework, so that the assessment result can directly guide and verify a capability enhancement strategy.

Description

Technical Field

[0001] This invention relates to the field of large language model evaluation technology, and more specifically, to a method and apparatus for evaluating and enhancing the mutual exclusion recognition capability of LLMs. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have demonstrated outstanding semantic understanding and knowledge reasoning capabilities in natural language processing tasks. Against this backdrop, the identification of conceptual disjointness, as a key task for measuring the deep semantic understanding and logical reasoning capabilities of models, is becoming increasingly important. Conceptual disjointness refers to two or more concepts that do not share common instances semantically; for example, although "cat" and "dog" both belong to the superordinate concept of "animal," a specific instance cannot be attributed to both "cat" and "dog," thus constituting mutually exclusive concepts. Accurately identifying the mutually exclusive relationships between concepts is crucial for improving the performance of LLMs in downstream tasks such as knowledge reasoning, automatic classification, question answering systems, and output consistency detection.

[0003] While identifying mutually exclusive relationships between concepts is crucial for knowledge engineering and high-quality reasoning, traditional methods primarily rely on rule-based ontology reasoning, statistical learning, or specialized representation learning techniques. These methods are generally limited by the completeness of the knowledge base, high annotation costs, and insufficient model generalization ability.

[0004] In recent years, LLMs have provided a new technological paradigm for the automated identification of semantic relationships. Leveraging their powerful natural language understanding and generation capabilities, they can perform zero-shot or few-shot reasoning on relationships between concepts based on context, without relying on explicit knowledge structures. This opens up a completely new path for the efficient and flexible discovery of conceptual mutual exclusivity, making them a promising supplement to, and even an alternative to, traditional methods.

[0005] However, current industry-wide systematic performance evaluations of LLMs for the specific task of identifying mutually exclusive concepts suffer from several fundamental limitations: First, the lack of a dedicated benchmark for deep logical reasoning makes it difficult for existing general evaluation methods (such as simple question-and-answer accuracy) to effectively expose the logical inconsistencies and interpretive illusions in mutual exclusion judgments—that is, the deep-seated defect of the model making correct judgments but having incorrect reasoning basis. Second, traditional prompting engineering methods are mostly general strategies that fail to explicitly integrate the ontological axioms and structured reasoning rules underlying mutually exclusive concepts into the prompting process, making it difficult to upgrade the model's reasoning mode from "intuitive semantic matching" based on surface associations to "structured logical reasoning" based on formal rules. Finally, existing technical solutions typically treat evaluation and enhancement as isolated steps, lacking a closed-loop technical framework that integrates benchmark construction, targeted prompt enhancement, and multi-dimensional effect verification, making it difficult to achieve systematic, reproducible, and verifiable improvements in the model's recognition capabilities. The aforementioned shortcomings together constitute the performance bottleneck of current LLMs in this critical reasoning task, severely restricting their application in downstream tasks requiring high reliability, such as knowledge graph completion and high-risk automated decision-making. Summary of the Invention

[0006] The purpose of this invention is to provide an evaluation and enhancement method and apparatus for the mutual exclusion recognition capability of LLMs, aiming to systematically measure and improve the ability of LLMs to distinguish mutual exclusion relationships between concepts by constructing standardized evaluation benchmarks and high-performance prompting strategies.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] In a first aspect, the present invention provides a method for evaluating and enhancing the mutual exclusion recognition capability of LLMs, comprising:

[0009] The system automatically acquires raw data from knowledge sources, performs concept extraction, relation parsing, and text annotation integration to form a unified knowledge representation and constructs a structured concept hierarchy graph. The knowledge sources include hierarchical recognition datasets with concept explanations and datasets with structural information. The datasets with structural information refer to datasets with clear concept hierarchy relationships and semantic connections between entities.

[0010] Based on the concept hierarchy diagram and / or the knowledge source, an evaluation dataset containing positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs is automatically constructed.

[0011] Design a multi-level prompting system, including direct expression prompts, domain-limiting prompts, and structure-guided prompts;

[0012] The concept pair samples in the evaluation dataset are combined with each prompt word and input into the LLMs to be evaluated to obtain the output judgment results and reasoning basis; the concept pair samples include positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs;

[0013] The judgment result and reasoning basis are evaluated, and the prompt words are enhanced based on the evaluation result.

[0014] Preferably, the concept extraction, relation parsing and text annotation integration refers to identifying and extracting independent concept units from a specified knowledge source through a data parser and natural language processing, parsing and establishing semantic relationships between concept units, and associating and fusing text descriptions or tag annotations of the same concept unit obtained from different sources to form a machine-readable and structurally unified knowledge representation.

[0015] The concept hierarchy diagram is a directed graph structure used to represent the hierarchical relationships between classes in a dataset;

[0016] In the conceptual hierarchy diagram, each node represents a class, or conceptual unit, and the directed edges between nodes point from the child conceptual node to the parent conceptual node.

[0017] Preferably, constructing the mutually exclusive concept pair includes:

[0018] Construct mutually exclusive concept pairs based on the aforementioned concept hierarchy diagram;

[0019] Positive examples of mutually exclusive relation concepts declared through explicit semantic axioms are directly extracted from the original data;

[0020] Based on known hierarchical and mutually exclusive relationship concepts, a new, implicit mutually exclusive relationship concept pair of positive examples is derived and generated through an automated rule engine.

[0021] Preferably, the step of constructing mutually exclusive concept pairs based on the concept hierarchy diagram includes:

[0022] The structural features of concept nodes are calculated using graph algorithms on the concept hierarchy graph, and candidate concept pairs are sampled using heuristic rules.

[0023] The sampled candidate concept pairs are mapped to vectors, and the similarity between the vectors is calculated. Candidate concept pairs with similarity below a set threshold are selected as positive examples of mutually exclusive concept pairs.

[0024] For concept pairs A and B that are determined to have a mutual exclusion relationship, a graph traversal algorithm is used to automatically mark all sub-concepts of A and all sub-concepts of B as mutually exclusive, generating positive examples of mutually exclusive concept pairs.

[0025] Preferably, the step of deriving and generating new, implicit, positive examples of mutually exclusive relationship concept pairs based on known hierarchical and mutually exclusive relationship concept pairs through an automated rule engine includes:

[0026] Traverse each parent concept node in the concept hierarchy graph, pair all direct child concept nodes of the parent concept node in pairs, and automatically generate mutually exclusive concept pairs.

[0027] Based on the declared mutually exclusive concept pairs A and B, obtain the sets of all descendant child concept nodes of A and B respectively, perform Cartesian product operation on the obtained child concept node sets, and mark all the resulting concept pairs as positive examples of mutually exclusive concept pairs;

[0028] Traverse the attribute set of the original data, extract the list of multi-value domains for each attribute, and when the domain of an attribute is specified as at least two concepts at the same time, pair the specified concepts to generate mutually exclusive concept pairs.

[0029] When two concepts X and Y are associated with two known mutually exclusive concept pairs through the same attribute, X and Y are automatically deduced to be positive examples of mutually exclusive concept pairs.

[0030] Preferably, constructing the negative example of the non-mutually exclusive concept pair includes:

[0031] The text description of the concept unit is represented as a high-dimensional vector, and the similarity between the high-dimensional vectors is calculated. Concept pairs with similarity greater than a set threshold are selected as negative examples of non-mutually exclusive concept pairs.

[0032] Traverse the concept hierarchy graph and mark all concept pairs that have direct or indirect hierarchical relationships as negative examples of non-mutually exclusive concept pairs;

[0033] Based on the semantic structure and descriptive logic axioms explicitly defined in the original data, concept pairs that do not logically satisfy the mutual exclusion condition are automatically identified and constructed as negative examples of non-mutually exclusive concept pairs.

[0034] Preferably, the judgment results and reasoning basis are evaluated, including establishing a judgment accuracy evaluation index system, an explanation reliability evaluation index system, and a prompt word robustness and enhancement effect evaluation index system;

[0035] The accuracy evaluation index system includes:

[0036] Accuracy is calculated as follows:

[0037] Accuracy = (TP + TN) / (TP + TN + FP + FN);

[0038] Among them, TP refers to positive examples of mutually exclusive concept pairs correctly identified by LLMs, TN refers to negative examples of non-mutually exclusive concept pairs correctly identified by LLMs, FP refers to negative examples that LLMs misclassified as mutually exclusive, and FN refers to positive examples that LLMs failed to classify as non-mutually exclusive.

[0039] Precision is calculated as follows:

[0040] Precision = TP / (TP + FP);

[0041] Recall is calculated as follows:

[0042] Recall = TP / (TP + FN);

[0043] The F1 score is calculated as follows:

[0044] F1 = 2 * (Precision * Recall) / (Precision + Recall);

[0045] Bias is calculated as follows:

[0046] Bias = |(TP+FN) / N - (TN+FP) / N|;

[0047] The explanatory reliability assessment index system includes:

[0048] ;

[0049] Explanatory hallucination rate = 1 - (number of reasonably explainable TP samples / total number of TP samples);

[0050] The evaluation index system for the robustness and enhancement effect of the prompt words includes:

[0051] The Prompt Consistency is calculated as follows:

[0052] Prompt Consistency = Number of prompts used for consistency assessment / Total number of prompts;

[0053] The sensitivity index is calculated as follows:

[0054] Sensitivity = -∑(p_i × log(p_i)),

[0055] Where p_i represents the frequency of the i-th judgment result under different prompt words;

[0056] The gain factor Gain is calculated as follows:

[0057] Gain = (Metric_enhanced - Metric_baseline) / Metric_baseline × 100%,

[0058] Wherein, Metric_enhanced represents the performance index value of the LLMs model under enhanced prompts, and Metric_baseline represents the performance index value of the LLMs model under baseline prompts; the enhanced prompts refer to domain-limited prompts and structure-guided prompts; the baseline prompts refer to direct expression prompts.

[0059] Secondly, the present invention provides an evaluation and enhancement apparatus for LLM concept mutual exclusion recognition capability, used to implement the above-mentioned evaluation and enhancement method for LLM concept mutual exclusion recognition capability, the apparatus comprising:

[0060] The preprocessing and knowledge acquisition module is used to automatically acquire raw data from knowledge sources, perform concept extraction, relation parsing, and text annotation integration to form a unified knowledge representation and construct a structured concept hierarchy graph. The knowledge sources include hierarchical recognition datasets with concept explanations and datasets with structural information. The datasets with structural information refer to datasets with clear concept hierarchy relationships and semantic connections between entities.

[0061] The evaluation dataset construction module is used to automatically construct an evaluation dataset containing positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs based on the concept hierarchy graph and / or the knowledge source.

[0062] The prompt word system design module is used to design a multi-level prompt word system, including direct expression prompt words, domain-limiting prompt words, and structure-guided prompt words;

[0063] An automated test execution module is used to combine concept pair samples from the evaluation dataset with each prompt word, input them into the LLMs to be evaluated, and obtain the output judgment results and reasoning basis; the concept pair samples include positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs;

[0064] The multi-dimensional effect evaluation module is used to evaluate the judgment results and reasoning basis, and to enhance the prompt words based on the evaluation results.

[0065] Preferably, the multi-dimensional effect evaluation module includes establishing a discrimination accuracy evaluation index system, an interpretation reliability evaluation index system, and a prompt word robustness and enhancement effect evaluation index system;

[0066] The accuracy evaluation index system includes:

[0067] Accuracy is calculated as follows:

[0068] Accuracy = (TP + TN) / (TP + TN + FP + FN);

[0069] Among them, TP refers to positive examples of mutually exclusive concept pairs correctly identified by LLMs, TN refers to negative examples of non-mutually exclusive concept pairs correctly identified by LLMs, FP refers to negative examples that LLMs misclassified as mutually exclusive, and FN refers to positive examples that LLMs failed to classify as non-mutually exclusive.

[0070] Precision is calculated as follows:

[0071] Precision = TP / (TP + FP);

[0072] Recall is calculated as follows:

[0073] Recall = TP / (TP + FN);

[0074] The F1 score is calculated as follows:

[0075] F1 = 2 * (Precision * Recall) / (Precision + Recall);

[0076] Bias is calculated as follows:

[0077] Bias = |(TP+FN) / N - (TN+FP) / N|;

[0078] The explanatory reliability assessment index system includes:

[0079] ;

[0080] Explanatory hallucination rate = 1 - (number of reasonably explainable TP samples / total number of TP samples);

[0081] The evaluation index system for the robustness and enhancement effect of the prompt words includes:

[0082] The Prompt Consistency is calculated as follows:

[0083] Prompt Consistency = Number of prompts used for consistency assessment / Total number of prompts;

[0084] The sensitivity index is calculated as follows:

[0085] Sensitivity = -∑(p_i × log(p_i)),

[0086] Where p_i represents the frequency of the i-th judgment result under different prompt words;

[0087] The gain factor Gain is calculated as follows:

[0088] Gain = (Metric_enhanced - Metric_baseline) / Metric_baseline × 100%,

[0089] Wherein, Metric_enhanced represents the performance index value of the LLMs model under enhanced prompts, and Metric_baseline represents the performance index value of the LLMs model under baseline prompts; the enhanced prompts refer to domain-limited prompts and structure-guided prompts; the baseline prompts refer to direct expression prompts.

[0090] Thirdly, the present invention provides a computer-readable storage medium for storing one or more programs, said one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any of the methods in the evaluation and enhancement methods for LLM concept mutual exclusion recognition capabilities according to claims 1 to 8.

[0091] Compared with the prior art, the above-mentioned technical solution of the present invention achieves the following significant beneficial effects:

[0092] (1) The present invention designs a multi-level prompt word system, which introduces structure-guided prompt words and explicitly injects ontological reasoning rules into the model reasoning process, so that the model shifts from relying on "semantic intuition" to reasoning based on "logical rules", thereby systematically improving its accuracy and robustness when dealing with complex or unfamiliar concepts.

[0093] (2) This invention organically integrates benchmark construction, prompt enhancement and multi-dimensional verification into a framework, forming a complete technical closed loop from evaluation to enhancement. This enables the evaluation results to directly guide and verify capability enhancement strategies, overcomes the drawback of the separation of evaluation and enhancement in traditional schemes, and lays the foundation for the application of the technology in high reliability scenarios.

[0094] (3) It achieves accurate diagnosis of deep reasoning defects in the model. The present invention introduces the explanatory reliability assessment into the evaluation system, especially the explanatory illusion rate, which is a targeted indicator. This enables the present invention to effectively identify the deep defects of the model that "the judgment is correct but the reasoning is wrong". This provides a diagnostic dimension for optimizing the model that cannot be achieved by traditional accuracy assessment. Attached Figure Description

[0095] Figure 1 This is a schematic diagram of the evaluation and enhancement method for the mutual exclusion recognition capability of LLMs provided by the present invention. Detailed Implementation

[0096] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0097] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0098] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0099] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0100] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0101] It should be emphasized here that the step markers mentioned below are not a limitation on the order of the steps, but should be understood as meaning that the steps can be executed in the order mentioned in the embodiments, or in a different order than in the embodiments, or several steps can be executed simultaneously.

[0102] Example 1

[0103] This embodiment 1 provides an evaluation and enhancement method for the mutual exclusion recognition capability of LLM concepts, such as... Figure 1 As shown, its application process specifically involves the following steps:

[0104] Step 1: Automatically retrieve raw data from designated authoritative knowledge sources, perform concept extraction, relation parsing, and text annotation integration, and construct a structured concept hierarchy diagram.

[0105] In this embodiment, authoritative knowledge sources mainly include hypothetical and superordinate recognition datasets with conceptual explanations and datasets with structural information, which are used to support LLMs' understanding of conceptual semantic boundaries.

[0106] Contextual recognition datasets with concept explanations can come from two types of data sources: (1) datasets containing concept text explanations, such as the 2016 Semantic Evaluation Competition Task 13, whose data is usually stored in a structured format (such as JSON); (2) hierarchical relationships with annotation information defined by semantically labeled words (such as rdfs:label and rdfs:comment in the RDFS language) in ontology or knowledge graph. RDFS (Resource Description Framework Schema) is an ontology language that supports hierarchical relationships between concepts and attributes.

[0107] Datasets with structured information refer to datasets that possess clearly defined hierarchical relationships between concepts and semantic connections between entities. These can include high-quality ontology or knowledge graphs, such as large-scale knowledge graphs like DBpedia and YAGO. DBpedia is a knowledge base project that extracts structured information from Wikipedia and publishes it in RDF format. By extracting information boxes, categories, and links, it constructs an open knowledge graph that can be used for semantic queries. YAGO is a large-scale ontology knowledge base built by combining information from multiple sources such as Wikipedia WordNet and GeoNames. It emphasizes semantic consistency and the rigor of the ontology structure, supporting stronger reasoning and conceptual hierarchy analysis. This knowledge can originate from publicly available ontology from academic research or open knowledge graphs widely adopted in industry. Such knowledge sources typically follow semantic web standards such as RDF and OWL, supporting complex semantic queries and logical reasoning.

[0108] In this embodiment, concept extraction, relation parsing, and text annotation integration refer to the process of identifying and extracting independent conceptual units from heterogeneous knowledge sources using a data parser and natural language processing technology, parsing and establishing semantic relationships (especially hyponyms and hypernyms) between conceptual units, and associating and fusing text descriptions or tag annotations of the same conceptual unit obtained from different sources to form a machine-readable and structurally unified knowledge representation. Specifically, for example, for JSON format datasets, extraction is completed by parsing their predefined fields; for RDF / OWL knowledge bases, structured concepts, relationships, and annotations are obtained through SPARQL queries or ontology parsing APIs (such as the OWL API or Jena API).

[0109] It should be noted that a concept hierarchy graph is a directed graph structure used to represent the hierarchical relationships between classes in a dataset. In the graph, each node represents a conceptual unit (class), and the directed edges between nodes point from the hyponym (child concept node) to the hypernym (parent concept node), representing the semantic inheritance relationship of "is a kind of".

[0110] In this embodiment, the concept hierarchy graph can be constructed using a graph structure construction module based on adjacency lists, etc. By inputting pre-extracted hierarchical relationship pairs, a directed acyclic graph can be automatically constructed. The completed graph can be persistently stored in a graph database for efficient querying and traversal.

[0111] For example, knowledge of system initialization includes the following axiom of subclass relationships (i.e., SubClassOf):

[0112] SubClassOf(Sedan, Car)

[0113] SubClassOf(SUV, Car)

[0114] SubClassOf(electric vehicle, car)

[0115] SubClassOf(bus, car)

[0116] SubClassOf(bicycle, non-motorized vehicle)

[0117] SubClassOf(electric bicycles, non-motorized vehicles)

[0118] SubClassOf(Automobile, Motor Vehicle)

[0119] SubClassOf(motorcycle, motor vehicle)

[0120] SubClassOf(motor vehicle, vehicle)

[0121] SubClassOf(non-motorized vehicle, vehicle)

[0122] For each of the parent-child class relationships mentioned above, construct a directed edge. For example, based on SubClassOf(Sedan, Car), there exists an edge in the graph pointing from the "Sedan" node to the "Car" node. The completed concept hierarchy graph is a directed acyclic graph, where the root concept is "Vehicle".

[0123] Step 2: Based on the above information and concept hierarchy diagram, and by applying ontological reasoning rules, automatically construct an evaluation dataset containing positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs. Specifically, this includes the following steps:

[0124] Step 2.1: Construct a dataset of positive examples for mutually exclusive concept pairs based on the above data sources. This mainly includes the following methods: constructing positive examples of mutually exclusive concept pairs based on the concept hierarchy graph, obtaining positive examples of mutually exclusive concept pairs with well-defined relationships, and generating positive examples of mutually exclusive concept pairs by expanding the examples based on structural information.

[0125] In this embodiment, positive examples of mutually exclusive concept pairs are constructed based on a concept hierarchy graph. An efficient strategy is employed to generate candidate mutually exclusive concept pairs and make preliminary judgments. The core of this strategy lies in utilizing the hierarchical structure of the concept hierarchy graph to achieve transitive reasoning of mutually exclusive relationships. The specific implementation includes the following sub-steps:

[0126] Step 2.1.1: Candidate Concept Pair Sampling: Using graph algorithms (such as shortest path, lowest common ancestor) to calculate the structural features (such as semantic distance, branch differences) of concept nodes, heuristic rules are used to sample candidate concept pairs. For example, concept pairs with large semantic distances and belonging to different main branches (i.e., high lowest common ancestor levels) are prioritized to efficiently identify potential mutually exclusive relationships. Taking semantic distance as an example, the shortest path length between any two concept nodes in the graph is calculated. The longer the path, the greater the semantic distance. Consider the concepts "car" and "bicycle". The path is: car -> automobile -> motor vehicle -> vehicle <- non-motor vehicle <- bicycle, with a path length of 4. According to the rule: if the path length is ≥ 4, the concept pair (car, bicycle) is included in the candidate set.

[0127] Step 2.1.2: Precise determination of mutual exclusion relationships based on concept embedding. This method uses pre-trained language models (such as BERT, GPT series) or knowledge graph embedding algorithms (such as TransE, ComplEx) to map the sampled concept pairs (especially high-level root node pairs) into vector representations for precise determination.

[0128] The technical process includes: First, constructing text based on concept labels or semantic annotations, preprocessing it, and inputting it into an embedding model to obtain semantic vectors; next, calculating the similarity between semantic vectors (such as cosine similarity) to measure semantic relevance; finally, selecting low-similarity concept pairs as positive examples of mutually exclusive concept pairs based on a preset similarity threshold. For example, for the concept pair (motor vehicle, non-motor vehicle), by calculating the cosine similarity of their vector representations (assuming a result of -0.85) and setting a threshold (such as <-0.5 for mutual exclusion), since -0.85 < -0.5, it can be determined that Disjoint (motor vehicle, non-motor vehicle) holds, meaning that an entity cannot be both a motor vehicle and a non-motor vehicle simultaneously. This method, as an effective supplement to graph methods, achieves automatic and accurate sample construction and, combined with graph methods, enhances the semantic coverage and diversity of the dataset.

[0129] Step 2.1.3: Transitive Labeling of Mutual Exclusion Relationships. Once a pair of high-level concepts (such as A and B) is determined to have a mutual exclusion relationship, a graph traversal algorithm is used to automatically label all sub-concepts of A and all sub-concepts of B as mutually exclusive, thereby expanding the dataset size and ensuring logical consistency. For example, through logical deduction, based on SubClassOf(car, motor vehicle) and Disjoint(motor vehicle, non-motor vehicle), the mutual exclusion between "car" and "non-motor vehicle" can be deduced, i.e., Disjoint(car, non-motor vehicle). Alternatively, the graph traversal algorithm can automatically perform the above deduction: traverse the set of all sub-class concepts of "motor vehicle" S_motor = {car, motorcycle, sedan, SUV, ...}; traverse the set of all sub-class concepts of "motor vehicle" S_non_motor = {bicycle, electric bicycle, ...}; and label the combination of each concept C1 in S_motor and each concept C2 in S_non_motor as mutually exclusive Disjoint(C1, C2). As a result, the system automatically generates and stores a large number of mutually exclusive axioms, such as Disjoint(car, bicycle), Disjoint(car, electric bicycle), and Disjoint(bus, bicycle).

[0130] In this embodiment, mutually exclusive concept pairs with clear definitions are obtained. From ontology or high-quality knowledge graph that follows strict logical norms, mutually exclusive concept pairs declared through explicit semantic axioms (such as the owl:disjointWith predicate in OWL language) are directly extracted. This method provides high-confidence "gold standard" data for model evaluation. Specific technical implementations include: (1) For large knowledge graphs such as DBpedia and YAGO, systematic extraction is performed using standard query languages ​​(such as SPARQL). By executing queries such as SELECT ?classA ?classB WHERE { ?classA owl:disjointWith ?classB.}, all explicitly declared mutually exclusive concept pairs can be retrieved efficiently. Here, SELECT means selecting the target variable, and WHERE means conditional constraint statement. (2) For ontology such as gene and galen constructed with ontology languages ​​such as OWL, the corresponding parsing tools (such as OWL API or Apache Jena) are used to directly parse the mutually exclusive axioms in the ontology file and accurately extract mutually exclusive concept pairs. For example, in the Galen medical ontology, there are explicit mutual exclusion declarations of concepts as follows: DisjointClasses(HeartValve, Bone) means that “heart valve” and “bone” are mutually exclusive, that is, an anatomical structure instance cannot belong to both classes at the same time; DisjointClasses(Artery, Vein) means that “artery” and “vein” are mutually exclusive, that is, any blood vessel instance is either an artery or a vein, and cannot belong to both at the same time.

[0131] In this embodiment, expanding the examples based on structural information refers to deriving and generating new, implicitly mutually exclusive concept pairs based on known hierarchical and mutually exclusive relationship pairs using an automated rule engine. This module, as a crucial part of dataset construction, can significantly improve the size and logical consistency of the sample set. Its core technology implementation includes the following four automated reasoning strategies:

[0132] Step 2.1.a: Automatic Sibling Concept Derivation Strategy: Traverse each parent concept node in the concept hierarchy graph, pairing all its direct child concept nodes (sibling concepts) together to automatically generate positive examples of mutually exclusive concept pairs. Specifically, for any parent concept node A, obtain its complete set of direct child concept nodes {B, C, D, …}, and then generate mutually exclusive pairs such as Disjoint(B, C), Disjoint(B, D), Disjoint(C, D), etc. This strategy can build basic mutually exclusive relationships in batches from ordinary classification structures without relying on explicit declarations.

[0133] Step 2.1.b: Mutual Exclusion Relationship Inheritance Transmission Strategy: Based on the declared mutual exclusion relationship (such as Disjoint(A,B)), obtain all descendant child concept sets Sub_A and Sub_B of A and B respectively, and then perform Cartesian product operation on these two sets, marking all concept pairs in them as positive examples of mutual exclusion relationship concept pairs.

[0134] Step 2.1.c: Attribute Constraint Conflict Detection Strategy: This strategy identifies type boundary divisions from attribute definitions. The system scans the domain declarations of all attributes in the ontology. When it finds that the domain of an attribute *r* is simultaneously specified as multiple concepts (such as A and B), it automatically determines that these concepts are mutually exclusive. The specific process includes: traversing the attribute set, extracting the multi-value domain list for each attribute, and pairing the concepts it contains into pairs to generate positive examples of mutually exclusive concept pairs. This method can uncover implicit type mutual exclusion from logical conflicts.

[0135] Step 2.1.d: Existence Constraint Conflict Derivation Strategy: This strategy is based on existence constraints in the descriptive logic for reasoning. The system identifies concept definitions with existence constraints. When two concepts X and Y are associated with two known mutually exclusive concepts A and B through the same attribute r, X and Y are automatically deduced to be mutually exclusive. The process includes: extracting the existence constraints of the concepts, matching attributes with target concepts, querying the known mutual exclusion relation database to confirm that Disjoint(A, B) holds, and then generating Disjoint(X,Y), which is marked as a positive example of a mutually exclusive relation concept pair. This strategy is suitable for discovering mutually exclusive relations under complex semantic definitions.

[0136] Step 2.2: Construct a dataset of negative examples for non-mutually exclusive concept pairs. This aims to systematically collect concept pairs that are semantically compatible or have a clear intersection, and therefore logically do not satisfy the mutual exclusion condition. Specifically, this is achieved through the following methods: a negative example construction method based on semantic embedding, a negative example construction method based on concept hierarchy graphs, and a negative example construction method based on structural information augmentation.

[0137] In this embodiment, negative example concept pairs are constructed based on semantic embedding. A pre-trained language model or knowledge graph embedding algorithm is used to represent the related text of each concept unit as a high-dimensional vector. By calculating the cosine similarity or Euclidean distance between the vectors, concept pairs that are semantically highly similar (e.g., cosine similarity greater than 0.8) are selected as negative examples. These concepts are adjacent in the semantic space and have significant semantic overlap, therefore they should not be judged as mutually exclusive.

[0138] In this embodiment, negative example concept pairs are constructed based on the concept hierarchy graph, directly utilizing the hierarchical relationships between concepts. The constructed concept hierarchy graph is traversed, and all concept pairs with direct or indirect hierarchical relationships are marked as negative examples. This is because concepts with inheritance relationships necessarily have an inclusion relationship semantically, rather than mutual exclusion.

[0139] In this embodiment, negative example concept pairs are constructed by expanding upon structural information. Based on the semantic structure and descriptive logic axioms explicitly defined in the knowledge graph or ontology, logically non-exclusive concept pairs are automatically identified and constructed as high-quality negative example samples. This method is implemented through a formal rule engine, and the core reasoning rules are as follows:

[0140] Step 2.2.1: Proof by contradiction based on the existence of intersection: If two concepts C and D have a common subconcept E (i.e., satisfying the axioms SubClassOf(E, C) and SubClassOf(E, D)), or a common instance i (i.e., satisfying ClassAssertion(C, i) and ClassAssertion(D, i)), then E or i constitutes the intersection of C and D, logically proving by contradiction that C and D are not mutually exclusive. Similarly, if concept E is explicitly defined as the intersection of C and D (i.e., EquivalentClasses(E, ObjectIntersectionOf(C, D))), it also directly indicates that C and D are not mutually exclusive. Here, SubClassOf represents the parent-child class relationship, ClassAssertion represents the class containing instance relationship, EquivalentClasses represents equivalent classes, and ObjectIntersectionOf represents the intersection of concepts.

[0141] Step 2.2.2: Compatibility rule based on semantic role sharing: If instances x and y belonging to C and D respectively are associated with the same instance z through the same object property R (i.e., satisfying ObjectPropertyAssertion(R, x, z) and ObjectPropertyAssertion(R, y, z)), it indicates that instances of C and D can share the same semantic role or relationship in a specific context, implying that there is semantic overlap between them, and they should not be judged as mutually exclusive. Here, ObjectPropertyAssertion indicates that a given property exists between the first instance and the second instance.

[0142] Step 2.2.3: Non-conflict rules based on descriptive logical constraints: The system detects logical constraints (such as existential-universal quantifier structures, domain / definition conditions) in the definitions of complex concepts. When the constraints do not constitute explicit mutual exclusion conflicts between concepts, the corresponding concept pair is considered a negative example. This mainly includes three scenarios:

[0143] Existence-Universal Restriction Structure Non-Conflict: If a concept X simultaneously satisfies both SubClassOf(X, ObjectSomeValuesFrom(r, Y)) and SubClassOf(X, ObjectAllValuesFrom(r, Z)), but does not declare Disjoint(Y, Z), then Y and Z are not necessarily mutually exclusive. Here, ObjectSomeValuesFrom and ObjectAllValuesFrom represent existence and universal restrictions, respectively.

[0144] Undeclared mutual exclusion condition for domains: If concept X satisfies SubClassOf(X, ObjectSomeValuesFrom(r,Y)) and attribute r has a global domain Z (i.e., SubClassOf(owl:Thing, ObjectAllValuesFrom (r,Z))), but Y and Z are not declared to be mutually exclusive, then the semantics of X cannot be considered mutually exclusive with Z. Here, owl:Thing represents the top concept defined in the OWL language, i.e., the parent concept of all concepts.

[0145] Unclosed domain mutual exclusion relationship: If concept X satisfies SubClassOf(X, ObjectSomeValuesFrom(r, Y)), and the domain of attribute r is Z (i.e., SubClassOf(ObjectSomeValuesFrom(r, owl:Thing),Z)), but X and Z are not declared to be mutually exclusive, then it indicates that X may be compatible with the domain Z.

[0146] Step 3: Design a multi-level prompting system that progresses from simple to complex, mainly including direct expression prompting words, domain-limiting prompting words, and structure-guided prompting words.

[0147] In this embodiment, direct-expressive prompts, through a direct and concise questioning approach, aim to stimulate the model's basic semantic discrimination capabilities. Their enhancement effect lies in forcing the model to directly extract and apply general rules regarding the mutual exclusion of concepts from parametric knowledge, laying the foundation for more complex reasoning and serving as the starting point for capability enhancement. For example, a prompt template could be designed as: "Are concepts A and B mutually exclusive? Please answer yes or no, and explain your reasoning."

[0148] In this embodiment, domain-specific prompts, by embedding specific domain background, guide the model to focus its judgments on a specific semantic space, thereby enhancing the domain relevance and accuracy of its discrimination results. This enhancement effect stems from the contextual constraint on the model's knowledge scope, effectively avoiding misjudgments caused by cross-domain semantic generalization and improving the model's applicability in professional scenarios. For example, a prompt template could be designed as: In [domain], are "Concept A" and "Concept B" mutually exclusive? Please answer only "yes" or "no" and explain your reasoning.

[0149] In this embodiment, structure-guided prompts are key to achieving enhanced capabilities. By explicitly embedding the ontological axioms and reasoning rules (such as hierarchical relationships and attribute constraints) upon which the positive and negative examples are constructed in this invention into the prompt content, the model is forced to perform structured logical reasoning rather than simple semantic matching. This design aims to enhance the model's logical consistency and explanatory depth, enabling the model to internalize the core discrimination logic of mutually exclusive relationships through repeated training, thereby achieving a fundamental improvement in capabilities from "intuitive judgment" to "rule-based reasoning." For example, a prompt template could be designed as follows: Given that "Concept C" and "Concept D" share a common parent class "Concept P," and all direct sub-concepts under "Concept P" are mutually exclusive. Are "Concept C" and "Concept D" mutually exclusive? Please answer "yes" or "no," and explain your reasoning.

[0150] Step 4: Perform automated testing.

[0151] In this step, the concept pairs in the above evaluation dataset are combined with the prompt words in the multi-level prompt word system, input into the LLMs to be evaluated, and the judgment results and reasoning basis of their output are obtained.

[0152] Automated test execution is achieved through the system's built-in test engine. The specific process is as follows: First, the engine combines concept pair samples with various prompt word templates to generate structured test cases. Then, through the configured RESTful API client, test cases under different prompt strategies are submitted in parallel to the target LLMs, and necessary calling parameters are encapsulated. Finally, the system captures the original responses of each model, and the parser uniformly extracts the binary judgment and inference text, and persists the structured results to provide a data foundation for subsequent evaluation.

[0153] Step 5: Multi-dimensional effect evaluation. Based on the judgment results and reasoning basis of the above output, an evaluation system is established from three dimensions: judgment accuracy, interpretation reliability, and cue word robustness, in order to quantify the performance of LLMs and verify the enhancement effect of enhanced cue words.

[0154] In this embodiment, the accuracy evaluation is based on the set of positive and negative concept pairs constructed in this invention, and a multi-index quantification model is used to assess the discrimination performance of mutual exclusion relationships between concepts, including the following indices:

[0155] 5a. Accuracy: The overall judgment accuracy of LLMs across all test samples is calculated using the following formula:

[0156] Accuracy = (TP + TN) / (TP + TN + FP + FN).

[0157] Here, Accuracy refers to the overall performance of the model. True Positive Examples (TP) are positive examples of mutually exclusive concept pairs correctly identified by the LLMs, True Negative Examples (TN) are negative examples of non-mutually exclusive concept pairs correctly identified by the LLMs, False Positive Examples (FP) are negative examples that the LLMs misclassified as mutually exclusive, and False Negative Examples (FN) are positive examples that the LLMs failed to classify as non-mutually exclusive. This metric comprehensively reflects the model's baseline discrimination capability.

[0158] 5b. Precision and Recall: Precision measures the confidence of LLMs in classifying results as "mutually exclusive".

[0159] Precision = TP / (TP + FP) ,

[0160] The low value indicates that LLMs are prone to false alarms, that is, they tend to incorrectly judge logically non-mutually exclusive negative instances (such as concept pairs with common instances) in this invention as mutually exclusive.

[0161] Recall assesses the ability of LLMs to cover true mutual exclusion relationships:

[0162] Recall = TP / (TP + FN),

[0163] The low value indicates that LLMs have serious underreporting, that is, they have failed to effectively identify the positive examples derived from various structural rules in this invention (such as positive examples generated by the mutual exclusion relationship inheritance strategy).

[0164] 5c, F1 score: As the harmonic mean of precision and recall, it is expressed as:

[0165] F1 = 2 * (Precision * Recall) / (Precision + Recall).

[0166] This metric is particularly suitable for evaluating the overall performance of LLMs on datasets where positive and negative examples may be imbalanced, as constructed in this invention. It balances the trade-off between avoiding false positives (high precision) and avoiding false negatives (high recall).

[0167] 5d. Bias: The following formula quantifies LLMs to determine whether a systematic bias exists:

[0168] Bias = |(TP+FN) / N - (TN+FP) / N| .

[0169] For example, if LLMs are overly inclined to give "mutually exclusive" judgments, it will lead to an increase in FP (presumably a specific type of judgment) and a decrease in FN (presumably a specific type of judgment), thereby increasing bias. This metric helps identify whether LLMs are influenced by potential biases in cue words or training data, ensuring the fairness of the evaluation.

[0170] In this embodiment, a reliability assessment is explained to deeply evaluate the reliability and understandability of the inference basis of LLMs output, and the analysis is conducted from two dimensions: general performance and specific risks.

[0171] 5e. General Assessment of Explanatory Quality: This indicator assesses the overall ability of LLMs to provide reasonable explanations for all correctly judged samples (including positive examples (TP) and negative examples (TN). The formula is as follows:

[0172] ,

[0173] For TP samples, "reasonable explanation" requires that the reasons provided correctly support the conclusion of "mutually exclusive concepts" (e.g., referencing the correct hierarchical structure or attribute constraints); for TN samples, the reasons must correctly support the conclusion of "non-mutually exclusive concepts" (e.g., indicating the existence of common instances or subclasses). All explanations must simultaneously meet the three core criteria of factual correctness, logical consistency, and semantic comprehensibility. The higher the value of this criterion, the more comprehensive and reliable the explanatory power of LLMs.

[0174] 5f. Specific test for interpretive hallucinations: This indicator specifically quantifies the frequency with which LLMs experience the specific hallucination phenomenon of "correct judgment but incorrect interpretation" when correctly identifying positive examples (TPs). The formula is as follows:

[0175] Explanatory hallucination rate = 1 - (number of reasonably explained TP samples / total number of TP samples).

[0176] For example, if an LLM correctly determines that "virus" and "bacteria" are mutually exclusive, but interprets it as "a virus is a type of bacteria" (a factual error), then this constitutes a hallucination. This indicator directly reflects the reliability of the LLM's interpretation in identifying mutual exclusion relationships; the higher the indicator value, the higher the risk that the LLM will produce fabricated facts or erroneous logic alongside its correct judgment.

[0177] In this embodiment, the robustness evaluation and enhancement effect verification of the prompt words are the core of verifying the effectiveness of the enhanced prompt word strategy of the present invention, including the following indicators:

[0178] 5g. Multi-prompt judgment consistency rate: This indicator measures the stability of LLMs in outputting the same judgment result for the same concept under different prompts. The formula is as follows:

[0179] Prompt Consistency = Number of prompts used for consistency checks / Total number of prompts.

[0180] Prompt Consistency refers to the consistency rate of prompts. For example, for the same concept pair, if LLMs output the same judgment (either "yes" or "no") under three types of prompts (direct expression, domain-restricted, and structure-guided), the consistency rate is 100%. The higher the value of this indicator, the more stable the judgment results of LLMs are, and the less affected they are by the form of the prompts.

[0181] 5h, Cue Sensitivity Index: This index quantifies the sensitivity of LLM judgment results to semantic changes in cue words using the principle of information entropy. Its formula is:

[0182] Sensitivity = -∑(p_i × log(p_i)),

[0183] Here, Sensitivity is the cue sensitivity index, where p_i represents the frequency of the i-th judgment result ("yes" or "no") under different cue words. When the LLMs output consistently under all cue words, the entropy value is 0, indicating complete stability; when the judgment results are randomly distributed, the entropy value is at its maximum, indicating extreme sensitivity. This index accurately reflects the robustness level of the model.

[0184] 5i. Enhancement Effect Verification: This verification uses paired experimental design to quantify the performance improvement brought by enhanced prompts. Its core metric is the enhancement gain coefficient (Gain).

[0185] Gain = (Metric_enhanced - Metric_baseline) / Metric_baseline × 100%,

[0186] Where `Metric_enhanced` represents the performance index of LLMs under enhanced prompts (domain-limited prompts and structure-guided prompts), and `Metric_baseline` represents the performance index of LLMs under baseline prompts (direct expression prompts). This calculation is applied to three dimensions: recognition accuracy (F1 score), interpretation quality (interpretation effectiveness), and robustness (prompt sensitivity index), and the statistical significance of the gain effect is confirmed by paired t-tests (p-value less than 0.05). A larger gain coefficient indicates a more significant optimization effect of the enhanced prompt strategy on the corresponding dimension.

[0187] Example 2

[0188] Based on the automated dataset construction process in Example 1, Example 2 further describes an optional manual review step for dataset quality, aiming to construct a "platinum" subset in the gold standard dataset, suitable for evaluation scenarios with high requirements for data quality.

[0189] This manual review process is implemented through a customized approach using a professional data annotation platform (such as Label Studio or Prodigy). To empower annotators to make accurate judgments, this data annotation platform is deeply integrated with underlying graph databases (such as Neo4j, used to store the concept hierarchy graph constructed in Example 1) and ontology tools (such as Protégé, used to visualize complex ontology axioms) through interfaces.

[0190] The specific review process is as follows:

[0191] The specific review process is as follows:

[0192] S1. Integrated Contextual Information Display: When an annotator encounters a pair of concepts awaiting review (such as "Golden Retriever" and "Persian Cat") on the platform interface, the system not only displays the concept names but also automatically retrieves and visualizes their hierarchical relationship paths and sibling concept list from the integrated Neo4j graph database. It also extracts key attribute definitions and logical axioms from the ontology. This provides annotators with unprecedentedly rich context, enabling them to base their judgments on deep, structured knowledge rather than intuition.

[0193] S2. Standardized Review Task Design: Within the annotation platform, structured review tasks are designed for each concept pair. For example, annotators are required not only to determine "whether they are mutually exclusive," but also to select the primary basis for their judgment from provided options (e.g., "based on an explicit owl:disjointWith declaration," "derived from sibling concepts," "there is a common instance, therefore they are not mutually exclusive," etc.), and can add free-text comments. This greatly standardizes the review criteria and provides structured data for subsequent analysis of annotation patterns.

[0194] S3. Multi-round review and arbitration mechanism: To ensure impartiality, each concept pair is assigned to at least two annotators for independent, back-to-back review. The platform automatically compares the review results; if the results are consistent, the sample is automatically approved. When disagreements arise, the system automatically marks the sample as a "disputed sample" and initiates an expert arbitration mechanism, where domain experts make a final ruling based on more comprehensive information. This multi-round review mechanism effectively eliminates individual subjective biases, ensuring high accuracy and logical consistency in the final sample set.

[0195] By introducing this optional human review process, the resulting evaluation benchmark is not only substantial in scale but also achieves near-imperfect reliability in quality, providing a solid data foundation for the accurate evaluation and capability diagnosis of cutting-edge models.

[0196] Example 3

[0197] This embodiment 3 provides an evaluation and enhancement device for the mutual exclusion recognition capability of LLM concepts, used to implement the evaluation and enhancement method for the mutual exclusion recognition capability of LLM concepts in embodiment 1 above. The device includes:

[0198] The preprocessing and knowledge acquisition module is used to automatically acquire raw data from knowledge sources, perform concept extraction, relation parsing, and text annotation integration to form a unified knowledge representation and construct a structured concept hierarchy graph. The knowledge sources include hierarchical recognition datasets with concept explanations and datasets with structural information. The datasets with structural information refer to datasets with clear concept hierarchy relationships and semantic connections between entities.

[0199] The evaluation dataset construction module is used to automatically construct an evaluation dataset containing positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs based on the concept hierarchy graph and / or the knowledge source.

[0200] The prompt word system design module is used to design a multi-level prompt word system, including direct expression prompt words, domain-limiting prompt words, and structure-guided prompt words;

[0201] An automated test execution module is used to combine concept pair samples from the evaluation dataset with each prompt word, input them into the LLMs to be evaluated, and obtain the output judgment results and reasoning basis; the concept pair samples include positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs;

[0202] The multi-dimensional effect evaluation module is used to evaluate the judgment results and reasoning basis, and to enhance the prompt words based on the evaluation results.

[0203] Furthermore, the multi-dimensional effect evaluation module includes establishing an evaluation index system for the accuracy of discrimination, an evaluation index system for the reliability of interpretation, and an evaluation index system for the robustness and enhancement effect of prompt words;

[0204] The accuracy evaluation index system includes:

[0205] Accuracy is calculated as follows:

[0206] Accuracy = (TP + TN) / (TP + TN + FP + FN);

[0207] Among them, TP refers to positive examples of mutually exclusive concept pairs correctly identified by LLMs, TN refers to negative examples of non-mutually exclusive concept pairs correctly identified by LLMs, FP refers to negative examples that LLMs misclassified as mutually exclusive, and FN refers to positive examples that LLMs failed to classify as non-mutually exclusive.

[0208] Precision is calculated as follows:

[0209] Precision = TP / (TP + FP);

[0210] Recall is calculated as follows:

[0211] Recall = TP / (TP + FN);

[0212] The F1 score is calculated as follows:

[0213] F1 = 2 * (Precision * Recall) / (Precision + Recall);

[0214] Bias is calculated as follows:

[0215] Bias = |(TP+FN) / N - (TN+FP) / N|;

[0216] The explanatory reliability assessment index system includes:

[0217] ;

[0218] Explanatory hallucination rate = 1 - (number of reasonably explainable TP samples / total number of TP samples);

[0219] The evaluation index system for the robustness and enhancement effect of the prompt words includes:

[0220] The Prompt Consistency is calculated as follows:

[0221] Prompt Consistency = Number of prompts used for consistency assessment / Total number of prompts;

[0222] The sensitivity index is calculated as follows:

[0223] Sensitivity = -∑(p_i × log(p_i)),

[0224] Where p_i represents the frequency of the i-th judgment result under different prompt words;

[0225] The gain factor Gain is calculated as follows:

[0226] Gain = (Metric_enhanced - Metric_baseline) / Metric_baseline × 100%,

[0227] Wherein, Metric_enhanced represents the performance index value of LLMs under enhanced prompts, and Metric_baseline represents the performance index value of LLMs under baseline prompts; the enhanced prompts refer to domain-qualified prompts and structure-guided prompts; the baseline prompts refer to direct expression prompts.

[0228] It is worth noting that this device embodiment corresponds to the above method embodiment. The implementation methods of the above method embodiments are all applicable to this device embodiment and can achieve the same or similar technical effects, so they will not be described in detail here.

[0229] Example 4

[0230] This embodiment 4 provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for evaluating and enhancing the mutual exclusion recognition capability of LLM concepts.

[0231] Example 5

[0232] This embodiment 5 provides a computing device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the above-described evaluation and enhancement method for LLM concept mutual exclusion recognition capability.

[0233] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0234] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0235] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0236] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0237] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for evaluating and enhancing the mutual exclusion recognition capability of LLM concepts, characterized in that, include: The system automatically acquires raw data from knowledge sources, performs concept extraction, relation parsing, and text annotation integration to form a unified knowledge representation and constructs a structured concept hierarchy graph. The knowledge sources include hierarchical recognition datasets with concept explanations and datasets with structural information. The datasets with structural information refer to datasets with clear concept hierarchy relationships and semantic connections between entities. Based on the concept hierarchy diagram and / or the knowledge source, an evaluation dataset containing positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs is automatically constructed. Design a multi-level prompting system, including direct expression prompts, domain-limiting prompts, and structure-guided prompts; The concept pair samples in the evaluation dataset are combined with each prompt word and input into the LLMs to be evaluated to obtain the output judgment results and reasoning basis; the concept pair samples include positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs; The judgment result and reasoning basis are evaluated, and the prompt words are enhanced based on the evaluation result.

2. The method for evaluating and enhancing the mutual exclusion recognition capability of LLMs according to claim 1, characterized in that, The aforementioned concept extraction, relation parsing, and text annotation integration refer to identifying and extracting independent concept units from a specified knowledge source through a data parser and natural language processing, parsing and establishing semantic relationships between concept units, and associating and fusing text descriptions or label annotations of the same concept unit obtained from different sources to form a machine-readable and structurally unified knowledge representation. The concept hierarchy diagram is a directed graph structure used to represent the hierarchical relationships between classes in a dataset; In the conceptual hierarchy diagram, each node represents a class, or conceptual unit, and the directed edges between nodes point from the child conceptual node to the parent conceptual node.

3. The method for evaluating and enhancing the mutual exclusion recognition capability of LLMs according to claim 2, characterized in that, Constructing the mutually exclusive concept pair of positive examples includes: Construct mutually exclusive concept pairs based on the aforementioned concept hierarchy diagram; Positive examples of mutually exclusive relation concepts declared through explicit semantic axioms are directly extracted from the original data; Based on known hierarchical and mutually exclusive relationship concepts, a new, implicit mutually exclusive relationship concept pair of positive examples is derived and generated through an automated rule engine.

4. The method for evaluating and enhancing the mutual exclusion recognition capability of LLMs according to claim 3, characterized in that, The construction of mutually exclusive concept pairs based on the concept hierarchy diagram includes: The structural features of concept nodes are calculated using graph algorithms on the concept hierarchy graph, and candidate concept pairs are sampled using heuristic rules. The sampled candidate concept pairs are mapped to vectors, and the similarity between the vectors is calculated. Candidate concept pairs with similarity below a set threshold are selected as positive examples of mutually exclusive concept pairs. For concept pairs A and B that are determined to have a mutual exclusion relationship, a graph traversal algorithm is used to automatically mark all sub-concepts of A and all sub-concepts of B as mutually exclusive, generating positive examples of mutually exclusive concept pairs.

5. The method for evaluating and enhancing the mutual exclusion recognition capability of LLMs according to claim 3, characterized in that, The process, based on known hierarchical and mutually exclusive relationship concept pairs, uses an automated rule engine to derive and generate new, implicit mutually exclusive relationship concept pairs as positive examples, including: Traverse each parent concept node in the concept hierarchy graph, pair all direct child concept nodes of the parent concept node in pairs, and automatically generate mutually exclusive concept pairs. Based on the declared mutually exclusive concept pairs A and B, obtain the sets of all descendant child concept nodes of A and B respectively, perform Cartesian product operation on the obtained child concept node sets, and mark all the resulting concept pairs as positive examples of mutually exclusive concept pairs; Traverse the attribute set of the original data, extract the list of multi-value domains for each attribute, and when the domain of an attribute is specified as at least two concepts at the same time, pair the specified concepts to generate mutually exclusive concept pairs. When two concepts X and Y are associated with two known mutually exclusive concept pairs through the same attribute, X and Y are automatically deduced to be positive examples of mutually exclusive concept pairs.

6. The method for evaluating and enhancing the mutual exclusion recognition capability of LLMs according to claim 2, characterized in that, Constructing the negative instance of the non-mutually exclusive concept pair includes: The text description of the concept unit is represented as a high-dimensional vector, and the similarity between the high-dimensional vectors is calculated. Concept pairs with similarity greater than a set threshold are selected as negative examples of non-mutually exclusive concept pairs. Traverse the concept hierarchy graph and mark all concept pairs that have direct or indirect hierarchical relationships as negative examples of non-mutually exclusive concept pairs; Based on the semantic structure and descriptive logic axioms explicitly defined in the original data, concept pairs that do not logically satisfy the mutual exclusion condition are automatically identified and constructed as negative examples of non-mutually exclusive concept pairs.

7. The method for evaluating and enhancing the mutual exclusion recognition capability of LLMs according to claim 1, characterized in that, The judgment results and reasoning basis are evaluated, including the establishment of an evaluation index system for the accuracy of the judgment, an evaluation index system for the reliability of the explanation, and an evaluation index system for the robustness and enhancement effect of the prompt words. The accuracy evaluation index system includes: Accuracy is calculated as follows: Accuracy = (TP + TN) / (TP + TN + FP + FN); Among them, TP refers to positive examples of mutually exclusive concept pairs correctly identified by LLMs, TN refers to negative examples of non-mutually exclusive concept pairs correctly identified by LLMs, FP refers to negative examples that LLMs misclassified as mutually exclusive, and FN refers to positive examples that LLMs failed to classify as non-mutually exclusive. Precision is calculated as follows: Precision = TP / (TP + FP); Recall is calculated as follows: Recall = TP / (TP + FN); The F1 score is calculated as follows: F1 = 2 * (Precision * Recall) / (Precision + Recall); Bias is calculated as follows: Bias = |(TP+FN) / N - (TN+FP) / N|; The explanatory reliability assessment index system includes: ； Explanatory hallucination rate = 1 - (number of reasonably explainable TP samples / total number of TP samples); The evaluation index system for the robustness and enhancement effect of the prompt words includes: The Prompt Consistency is calculated as follows: Prompt Consistency = Number of prompts used for consistency assessment / Total number of prompts; The sensitivity index is calculated as follows: Sensitivity = -∑(p_i × log(p_i)), Where p_i represents the frequency of the i-th judgment result under different prompt words; The gain factor Gain is calculated as follows: Gain = (Metric_enhanced - Metric_baseline) / Metric_baseline × 100%, Wherein, Metric_enhanced represents the performance index value of the LLMs model under enhanced prompts, and Metric_baseline represents the performance index value of the LLMs model under baseline prompts; the enhanced prompts refer to domain-limited prompts and structure-guided prompts; the baseline prompts refer to direct expression prompts.

8. An evaluation and enhancement device for the mutual exclusion recognition capability of LLM concepts, characterized in that, The apparatus for implementing the evaluation and enhancement method for LLM concept mutual exclusion recognition capability according to any one of claims 1 to 7, the apparatus comprising: The preprocessing and knowledge acquisition module is used to automatically acquire raw data from knowledge sources, perform concept extraction, relation parsing, and text annotation integration to form a unified knowledge representation and construct a structured concept hierarchy graph. The knowledge sources include hierarchical recognition datasets with concept explanations and datasets with structural information. The datasets with structural information refer to datasets with clear concept hierarchy relationships and semantic connections between entities. The evaluation dataset construction module is used to automatically construct an evaluation dataset containing positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs based on the concept hierarchy graph and / or the knowledge source. The prompt word system design module is used to design a multi-level prompt word system, including direct expression prompt words, domain-limiting prompt words, and structure-guided prompt words; An automated test execution module is used to combine concept pair samples from the evaluation dataset with each prompt word, input them into the LLMs to be evaluated, and obtain the output judgment results and reasoning basis; the concept pair samples include positive examples of mutually exclusive concept pairs and negative examples of non-mutually exclusive concept pairs; The multi-dimensional effect evaluation module is used to evaluate the judgment results and reasoning basis, and to enhance the prompt words based on the evaluation results.

9. The evaluation and enhancement device for LLMs concept mutual exclusion recognition capability according to claim 8, characterized in that, The multi-dimensional effect evaluation module includes establishing an evaluation index system for discrimination accuracy, an evaluation index system for interpretation reliability, and an evaluation index system for the robustness and enhancement effect of prompt words. The accuracy evaluation index system includes: Accuracy is calculated as follows: Accuracy = (TP + TN) / (TP + TN + FP + FN); Among them, TP refers to positive examples of mutually exclusive concept pairs correctly identified by LLMs, TN refers to negative examples of non-mutually exclusive concept pairs correctly identified by LLMs, FP refers to negative examples that LLMs misclassified as mutually exclusive, and FN refers to positive examples that LLMs failed to classify as non-mutually exclusive. Precision is calculated as follows: Precision = TP / (TP + FP); Recall is calculated as follows: Recall = TP / (TP + FN); The F1 score is calculated as follows: F1 = 2 * (Precision * Recall) / (Precision + Recall); Bias is calculated as follows: Bias = |(TP+FN) / N - (TN+FP) / N|; The explanatory reliability assessment index system includes: ； Explanatory hallucination rate = 1 - (number of reasonably explainable TP samples / total number of TP samples); The evaluation index system for the robustness and enhancement effect of the prompt words includes: The Prompt Consistency is calculated as follows: Prompt Consistency = Number of prompts used for consistency assessment / Total number of prompts; The sensitivity index is calculated as follows: Sensitivity = -∑(p_i × log(p_i)), Where p_i represents the frequency of the i-th judgment result under different prompt words; The gain factor Gain is calculated as follows: Gain = (Metric_enhanced - Metric_baseline) / Metric_baseline × 100%, Wherein, Metric_enhanced represents the performance index value of the LLMs model under enhanced prompts, and Metric_baseline represents the performance index value of the LLMs model under baseline prompts; the enhanced prompts refer to domain-limited prompts and structure-guided prompts; the baseline prompts refer to direct expression prompts.

10. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods in the evaluation and enhancement methods for LLM concept mutual exclusion recognition capabilities according to claims 1 to 7.

Images