Method and device for evaluating logic conflict detection capability of large language model

Abstract

The invention discloses a method and device for evaluating the logic conflict detection capability of a large language model, and belongs to the technical field of large language models.The method comprises the steps that a logic conflict mode is summarized according to an ontology containing logic conflicts; constructing a positive example data set and a negative example data set based on the logic conflict mode and the corresponding ontology; respectively combining the positive example data set and the negative example data set with multiple different types of pre-designed logic conflict detection cues, and asking questions for the large language model; according to the test result of the large language model, the logic conflict judgment and interpretation capability of the large language model is evaluated; according to the method, the logic conflict mode and the logic reasoning rule are considered, the complex context can be better understood, and the accuracy of logic conflict detection is improved.

Description

Technical Field

[0001] The present invention relates to a method and device for evaluating the logical conflict detection capability of a large language model, and belongs to the technical field of large language models. Background Art

[0002] In recent years, large language models (LLMs) have received widespread attention and application as an important breakthrough in the field of natural language processing. Large language models represented by GPT, BERT, T5, etc. have demonstrated excellent performance in tasks such as text generation, semantic understanding, machine translation, and question-answering systems, promoting the rapid development of artificial intelligence technology. With strong context understanding capabilities and flexible knowledge expression capabilities, large language models have been widely used in intelligent customer service, educational counseling, medical consultation, content creation and other fields, becoming an important tool for improving production efficiency and service quality. With the continuous iteration of technology, the scale and capabilities of large language models have been significantly improved, and their popularity has been increasing, becoming a research hotspot in academia and industry.

[0003] Although large language models have demonstrated powerful capabilities, their performance still needs to be verified and improved through multi-dimensional systematic evaluation. In applications, evaluating the accuracy, generation quality, robustness, knowledge coverage, and security of the model is the key to ensuring its reliability and practical value. Among them, the evaluation of logical reasoning ability has gradually become a research focus, which not only includes the model's understanding of complex logical relationships and reasoning tasks, but also covers its performance in dealing with logical contradictions. As an important part of logical reasoning ability, logical conflict detection ability reflects the model's ability to discover, analyze, and process contradictory information. This ability is crucial to the stability and credibility of the model in actual scenarios. Therefore, it is particularly necessary to conduct evaluation research on the logical conflict detection ability of large language models, which can provide a scientific basis for model optimization and promote its application in high-demand scenarios.

[0004] In existing research work, some progress has been made in evaluating the logical reasoning ability of large language models. These works usually test the reasoning ability of the model by designing specific logical reasoning tasks, including deductive reasoning, inductive reasoning, and abductive reasoning. These tasks involve inferring causal relationships, verifying the correctness of statement logic, and solving complex logical puzzles. In particular, there are special studies that evaluate the ability of large language models to understand grammar or semantics such as specific logical attribute characteristics (such as symmetry, transitivity, functionality, etc.), query answering, etc., and also evaluate the ability to detect logical inconsistencies.

[0005] Although these studies have shown that large language models can demonstrate certain reasoning capabilities in some scenarios, due to their characteristics of generating text based on probability distribution, large language models still have limitations in dealing with logical consistency and rigor, which may lead to incorrect reasoning or inconsistent output in practical applications. In particular, the evaluation of the ability to detect logical conflicts such as logical inconsistency or logical inconsistency lacks a relatively comprehensive test case construction method, prompt words based on logical conflict patterns or reasoning rules, and multi-dimensional evaluation criteria. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a method and device for evaluating the logical conflict detection capability of a large language model, which takes into account the pattern of logical conflicts and the rules of logical reasoning, can better understand complex contexts and improve the accuracy of logical conflict detection.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions: In a first aspect, the present invention provides a method for evaluating the logical conflict detection capability of a large language model, comprising: Summarize logical conflict patterns based on the ontology containing logical conflicts; Construct positive and negative datasets based on logical conflict patterns and corresponding ontologies; Combine the positive and negative datasets with different types of pre-designed logical conflict detection prompts to ask questions to the large language model. Based on the test results of the large language model, the logical conflict judgment and interpretation capabilities of the large language model are evaluated.

[0008] Further, summarizing the logic conflict pattern according to the ontology containing the logic conflict includes: Use existing ontology logic conflict interpretation tools to detect logic conflicts, calculate the smallest set of axioms that contain logic conflicts, and then summarize common logic conflict patterns; Wherein, the ontology includes concepts, object attributes, data type attributes and instances; The logical conflict means that there are contradictory statements or inferences in a logical system at the same time, including logical inconsistency or logical inconsistency; a set of axioms is logically inconsistent, which means that the set has no model or explanation, and a set of axioms is logically inconsistent, which means that the set contains at least one unsatisfiable concept; The minimum set of axioms containing logical conflicts indicates that the set contains logical inconsistency or incoordination, and removing any one of the axioms will cause the set to restore consistency or coordination; The ontology containing logical conflicts represents inconsistent or uncoordinated ontologies that are often used in experimental evaluations in existing research work on logical conflict handling; The method of using an existing ontology logic conflict interpretation tool to perform logic conflict detection and calculate the minimum axiom set containing logic conflicts means calculating the minimum inconsistency preserving subset MIS or the minimum incoherence preserving subset MIPS through an ontology network repair and diagnosis tool or an ontology editing tool, wherein a MIS is a minimum axiom set containing logic inconsistencies and a MIPS is a minimum axiom set containing logic inconsistencies; The formal definition of MIS is: given an inconsistent ontology O and its subset O′, if O′ is inconsistent, and every proper subset of O′ are all consistent, represents a true inclusion relation between sets, then O′ is called a MIS of O; The formal definition of MIPS is: Given a logically consistent ontology O, if a subset S of O is inconsistent, and every proper subset of S are all coordinated, then S is said to be a MIPS of O; The common logic conflict modes summarized include: selecting a MIPS mode or a MIS mode whose occurrence frequency is greater than a set threshold based on the MIPS calculated in the selected ontology and the MIS; wherein the MIPS mode represents common types of axiom sets that lead to logical inconsistencies; The MIS patterns represent common types of axiom sets that lead to logical inconsistencies.

[0009] Furthermore, the MIPS mode includes: MIPS Mode 1: If a concept has two disjoint parent concepts, then the concept is unsatisfiable. MIPS Mode 1 is expressed in axioms as follows: , and , where X, Y and Z represent arbitrary atomic concepts, Indicates the inclusion relationship between concepts. Indicates no relationship; MIPS Mode 2: If a concept has an existential quantifier constraint and the filler term of the constraint is unsatisfiable, then the concept is unsatisfiable. MIPS Mode 2 is expressed as axioms: and , where X and Y represent arbitrary atomic concepts, r represents object attributes, Represents the concept of emptiness, Indicates the existence of a quantifier constraint; MIPS Mode 3: If a concept has both existential and universal quantifier constraints on an attribute, and the filled-in terms of these two constraints do not intersect, then the concept is unsatisfiable. MIPS Mode 3 is expressed by axioms as follows: , and , where X, Y and Z represent arbitrary atomic concepts, r represents object attributes, Indicates universal quantifier constraint; MIPS Mode 4: If a concept has an existential quantifier constraint, and the domain of the attribute in the constraint does not intersect with the concept, then the concept is unsatisfiable. MIPS Mode 4 is expressed as an axiom: , and , where X and Z represent arbitrary atomic concepts, and Y represents an arbitrary concept. Represents the top concept, It is defined that concept Z is the domain of attribute r; MIPS Mode 5: If a concept has an existential quantifier constraint, and the filled-in terms of the constraint do not intersect with the range of its attributes, then the concept is unsatisfiable. MIPS Mode 5 is expressed in axioms as follows: , and , where X, Y and Z represent arbitrary atomic concepts, Represents the top concept, The concept Z is defined as the value range of the attribute r.

[0010] Further, the MIS mode includes: MIS Mode 1: If two disjoint concepts have the same instance, it leads to inconsistency. MIS Mode 1 is expressed by axioms as follows: , X(a), Y(a), where X and Y represent concepts, a represents an instance, and X(a) means that a is an instance of X; MIS Mode 2: For a function attribute, if an instance is associated with two different instances through the attribute, an inconsistency occurs. MIS Mode 2 is expressed by the axioms: r(a, b), r(a, c), Func(r), b≠c, where r represents an object attribute, a, b and c represent instances, and Func(r) represents that r is a function attribute; MIS Mode 3: For an inverse function attribute, if two different instances are associated with the same instance through the attribute, inconsistency occurs. The inverse function attribute is opposite to the function attribute. MIS Mode 3 is expressed by the axioms: r(b, a), r(c, a), InvFunc(r), b≠c, where r represents an object attribute, a, b and c represent instances, and InvFunc(r) represents that r is an inverse function attribute. MIS Mode 4: If a concept can be associated with at most one value on a data type attribute, and an instance of the concept is associated with two different values ​​through the attribute, it will lead to inconsistency. MIS Mode 4 is expressed by axioms as follows: , X(c), r(c, v1), r(c, v2), v1≠v2, where X represents an atomic concept, c represents an instance, r represents a data type attribute, v1 and v2 represent specific values, Literal represents literal, the first axiom indicates that X can be associated with at most one value through r, r(c, v1) indicates that there is a relationship r between c and v1, MIS mode 4 also applies to object attributes, which is expressed as axioms: , X(c), Y(y1), Y(y2), r(c, y1), r(c, y2), y1≠y2, where X and Y represent concepts, r represents an object attribute, and y1 and y2 are two different instances of concept Y; MIS Mode 5: If the domain of an attribute is empty and the attribute has instances, it will lead to inconsistency. MIS Mode 5 is expressed by axioms as follows: , r(a,b), where r represents the object attribute, a and b represent instances, It shows that the domain of r is the base concept. MIS mode 5 can also be transformed into the value range of r as the base concept, which will also lead to inconsistency.

[0011] Furthermore, constructing a positive data set and a negative data set based on the logical conflict pattern and the corresponding ontology includes: Constructing a positive example data set, including constructing positive examples with logical conflicts and no redundant axioms and positive examples with logical conflicts and redundant axioms, wherein each positive example is an axiom set; a positive example containing redundant axioms means that removing the redundant axioms from the positive example does not affect its logical inconsistency or logical inconsistency; Constructing a positive example with logical conflicts and no redundant axioms means selecting MIPS or MIS with different sizes and expressive power from existing MIPS or MIS; Constructing positive examples with logical conflicts and redundant axioms means selecting some positive examples that are relatively easy for the large language model to understand from the positive examples with logical conflicts and no redundant axioms, and adding one or more axioms to each selected positive example. The added axioms are divided into two situations: one is that they have no relationship with the current positive example, and the other is that they have a relationship. The existence of a relationship between two axioms means that the two axioms have the same concept, attribute or instance, and the existence of an axiom with an axiom set means that the axiom has a relationship with a certain axiom in the axiom set. Constructing a negative example dataset involves selecting some positive examples from positive examples that contain logical conflicts and no redundant axioms, and constructing them by removing different types and numbers of axioms.

[0012] Furthermore, the multiple different types of logical conflict detection prompt words include: simple and direct prompt words, pattern-based prompt words, and pattern and reasoning rule-based prompt words; The simple and direct prompt word means describing the task in a simple and clear sentence, performing logical conflict detection on the input positive or negative examples, and providing corresponding explanations if there is a logical conflict; The pattern-based prompt word means that a definition of a logic conflict corresponding pattern is given in the prompt word; The prompt words based on patterns and inference rules mean that in addition to considering the patterns of logical conflicts, some set common inference rules are also considered in the prompt words.

[0013] Furthermore, the common reasoning rules include the following reasoning rules: Inference rule 1: If a concept X is a subconcept of the union of two other concepts, and these two concepts are subconcepts of the same concept C, then it can be deduced that X is also a subconcept of C; expressed as axiom: , , Derived , where X, A, B and C are all concepts; Inference rule 2: If the union of two concepts is equal to another concept, then the two concepts are subconcepts of the other concept respectively; expressed as axiom: Derived and , where A, B and X are all concepts, Represents equivalence relations between concepts; Inference rule 3: If the domain of an object attribute is concept X, and one instance is related to another instance through the attribute, then it is inferred that the previous instance belongs to X, which can be expressed as axiom: , r(c1,c2) ​​derives X(c1), where X is a concept, c1 and c2 are instances, and r is an object attribute; Inference rule 4: If the value range of an object attribute is concept X, and one instance is related to another instance through the attribute, then it is inferred that the latter instance belongs to X, which can be expressed as: , r(c1,c2) ​​derives X(c2), where X is a concept, c1 and c2 are instances, and r is an object attribute; Inference Rule 5: If the range of an object attribute is Z, a concept has an existential quantifier constraint on the attribute, and there is an instance of the type of the concept, then it is inferred that Z is also a type of the instance, expressed as: , , X(a) leads to Z(a), where X, Y, and Z represent concepts and a represents an instance. Indicates the top concept; Inference Rule 6: If the domain of an object attribute is Z, and a concept is equivalent to defining a universal quantifier constraint on the attribute and is a subconcept of Z, then it is inferred that Z is a top-level concept, expressed by the axiom: , , Derived , where X, Y, and Z are concepts, For the top concept; Inference Rule 7: If the domain of an object attribute is Z, a concept is equivalent to defining a universal quantifier constraint on the attribute, and the concept is related to other concepts through the attribute, then it is inferred that Z is a top-level concept, which can be expressed by the axiom: , , Derived , where X, Y, Z, K are concepts, Top concept.

[0014] Furthermore, the positive and negative datasets are respectively combined with a plurality of different types of pre-designed logic conflict detection prompt words to ask questions to the large language model, including: first constructing a complete and executable prompt word, and then using the prompt word as the input of the large language model under test, and recording the output corresponding to each input, wherein the positive or negative examples in the prompt word, as well as the logic conflict pattern or reasoning rule, can be output in the format of an application program interface output of a natural language or a network ontology language or in the format of a description logic.

[0015] Furthermore, the logical conflict judgment and interpretation capabilities of the large language model are evaluated based on the test results of the large language model, including: According to the test cases, the correctness of the logical conflict judgment and the correctness of the interpretation are judged. For positive examples with logical conflicts and no redundant axioms, if the test results show that the positive example contains logical conflicts, the logical conflict judgment is considered correct, otherwise it is wrong; for such positive examples, if the explained information contains all the axiom information in the positive example and there is no redundancy, then the interpretation is correct; for positive examples with logical conflicts and redundant axioms, if the test results show that the logical conflict is contained, then the logical conflict judgment is correct, otherwise it is wrong; for such positive examples, if the explained information only contains all the axiom information in the original MIPS or MIS in the positive example, then the interpretation is correct; if in addition to this information, other information is also contained, it means that there is redundant information in the interpretation; if it does not contain all the axiom information in the original MIPS or MIS, then the interpretation is wrong; for negative examples, if the test results show that it contains logical conflicts, then the logical conflict judgment is wrong; if it does not contain logical conflicts, then the logical conflict judgment is correct; Based on the correctness judgment results, all test results are statistically analyzed as follows: For the prompt words containing the positive examples of the non-redundant axiom, count the results that can correctly judge the existence of logical conflicts and explain correctly, and mark the corresponding positive examples as the set ; For the prompt words containing redundant axiom positive examples, count the results that can correctly judge the existence of logical conflicts, correct interpretation and no redundant information, and mark the corresponding positive examples as the set In addition, the results that can correctly judge the existence of logical conflicts and the correct interpretation but with redundant information are counted, and the corresponding positive examples are marked as a set ; For the prompt words containing positive examples, count the results that can correctly judge the existence of logical conflicts but have incorrect interpretations, and mark the corresponding positive examples as the set ; In addition, the results of judging that there is no logical conflict are counted, and the corresponding positive examples are marked as the set ; For the prompt words containing negative examples, count the correct judgment results without logical conflicts and mark the corresponding negative examples as a set ; Secondly, count the results of wrong judgments with logical conflicts and mark the corresponding negative examples as a set ; Based on the statistical information, we define metrics of different dimensions, and then calculate the test results based on these metrics. We define the following metrics: Define the accuracy of the logical conflict judgment of the large language model under test : ; in, represents the set of positive examples of the non-redundant axiom, represents the set of positive examples with redundant axioms, represents the set of negative examples with redundant axioms, Representing a collection The number of elements in ; Define the error rate of logical conflict judgment of the large language model under test : ; in, ; Define the accuracy of the logical conflict interpretation of the large language model under test : ; Define the correctness of the minimum logical conflict interpretation of the large language model under test : ; Define the probability that the large language model under test will produce hallucinations when judging and interpreting logical conflicts : ; The tested large language models are evaluated according to the above metrics.

[0016] In a second aspect, the present invention provides a device for evaluating the logical conflict detection capability of a large language model, comprising: A logic conflict summary module, used for summarizing logic conflict patterns according to the ontology containing logic conflicts; Positive and negative example construction module, used to construct positive and negative example datasets based on logical conflict patterns and corresponding ontologies; The prompt word design module is used to design prompt words for testing large language models, including simple and direct prompt words, pattern-based prompt words, and prompt words based on patterns and inference rules; The model testing module is used to combine the positive and negative datasets with various types of pre-designed logical conflict detection prompt words to ask questions to the large language model; The performance evaluation module is used to evaluate the logical conflict judgment and interpretation capabilities of the large language model based on the test results of the large language model.

[0017] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned methods.

[0018] In a fourth aspect, the present invention provides a computer device, comprising: Memory, for storing computer programs / instructions; A processor is used to execute the computer program / instructions to implement the steps of any of the aforementioned methods.

[0019] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of any of the aforementioned methods when executed by a processor.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention provides a method and device for evaluating the logical conflict detection capability of a large language model, which is used to construct a multi-type data set specifically for testing the logical conflict detection capability of a large language model. By summarizing common logical conflict patterns, test cases of different difficulty levels are designed, which significantly improves the coverage and scientificity of the evaluation. 2. The present invention combines logical conflict patterns with inference rules to design prompt words, which can effectively enhance the logical conflict detection capability of the large language model under test, thereby more accurately evaluating the model performance; 3. The present invention proposes a multi-dimensional metric that can not only evaluate the accuracy and error rate of the model in logical conflict judgment, but also quantify its ability to identify the smallest logical conflict, and evaluate the probability of hallucination generation during the detection process, providing a comprehensive performance analysis. In summary, through the evaluation method of the present invention, the performance of different large language models in logical conflict detection capabilities can be systematically and comprehensively compared, which makes up for the shortcomings of existing methods in evaluation depth and breadth, provides an important reference for the evaluation of the logical reasoning ability of large language models, and helps promote the optimization and application of related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a method for evaluating the logical conflict detection capability of a large language model provided by an embodiment of the present invention; Figure 2 It is a structural diagram of a device for evaluating the logical conflict detection capability of a large language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0023] Embodiment 1: This embodiment introduces a method for evaluating the logical conflict detection capability of a large language model, including: Summarize logical conflict patterns based on the ontology containing logical conflicts; Construct positive and negative datasets based on logical conflict patterns and corresponding ontologies; Combine the positive and negative datasets with different types of pre-designed logical conflict detection prompts to ask questions to the large language model. Based on the test results of the large language model, the logical conflict judgment and interpretation capabilities of the large language model are evaluated.

[0024] like Figure 1 As shown, the method for evaluating the logical conflict detection capability of a large language model provided in this embodiment specifically involves the following steps in its application process: Step 1: Summarize logical conflict patterns based on existing inconsistent or incoherent ontologies.

[0025] In this embodiment, step 1 specifically includes the following steps: Step 1.1: Select an ontology from existing ontologies that contain logical conflicts, that is, select from existing inconsistent or incoherent ontologies, and select those that are frequently used in existing related work. For example, the chemical ontology CHEM-A.owl, economic ontology economy.owl, and transportation ontology Transportation.owl, which are often used for logical conflict detection and repair, the km1500 ontology built by ontology learning tools, the fusion ontology that merges matching ontologies through the corresponding relationship built by the ontology mapping system, and the inconsistent ontology built by the inconsistent ontology building tool injector. Among them, the extension owl represents the network ontology language.

[0026] Step 1.2: Use existing ontology logic conflict interpretation tools to detect logic conflicts and calculate the minimum set of axioms that contain logic conflicts. That is, calculate the minimum inconsistency-preserving subsets (MIS) or the minimum incoherence-preserving subsets (MIPS) through the ontology network repair and diagnosis tool RaDON (Repair and Diagnose Ontology Networks) or the ontology editing tool Protégé. A MIS is a minimum set of axioms that contain logical inconsistencies, and a MIPS is a minimum set of axioms that contain logical inconsistencies.

[0027] The formal definition of MIS is: given an inconsistent ontology O and its subset O′, if O′ is inconsistent, and every proper subset of O′ are all consistent, If it represents a proper inclusion relation between sets, then O′ is called a MIS of O.

[0028] The formal definition of MIPS is: Given a logically consistent ontology O, if a subset S of O is inconsistent, and every proper subset of S If they are all coordinated, then S is said to be one MIPS of O.

[0029] Step 1.3: Summarize the common logical conflict patterns. This refers to the MIPS or MIS patterns with higher frequency selected from the MIPS calculated in the selected ontology. This pattern is a compressed representation of MIPS or MIS, mainly removing the reasoning chain contained in the atomic concept. For example, a MIPS contains axioms and , without affecting other axioms, these two axioms can be combined into , where X, Y and Z represent arbitrary atomic concepts, Indicates the inclusion relationship between two concepts. Indicates no relationship.

[0030] The MIPS mode refers to the common axiom set types that lead to logical inconsistency. Based on the existing inconsistency ontology, the MIPS mode summarized mainly includes: MIPS Mode 1 (Disjoint Subclass Mode): If a concept has two disjoint parent concepts, then the concept is unsatisfiable. This mode can be expressed using axioms as follows: , and , where X, Y and Z represent arbitrary atomic concepts, Indicates the inclusion relationship between concepts. Indicates no relationship; MIPS Mode 2 (Existential Concept Mode): If a concept has an existential quantifier constraint and the filler of the constraint is unsatisfiable, then the concept is unsatisfiable. This mode can be expressed as axioms: and , where X and Y represent arbitrary atomic concepts, r represents object attributes, represents the empty concept (i.e. the concept interpreted as empty), Indicates the existence of a quantifier constraint; MIPS Mode 3 (Existential Universal Mode): If a concept has both existential and universal quantifier constraints on an attribute, and the filled terms of these two constraints do not intersect, then the concept is unsatisfiable. This mode can be expressed using axioms as follows: , and , where X, Y and Z represent arbitrary atomic concepts, r represents object attributes, Indicates universal quantifier constraint; MIPS Mode 4 (Existential Domain Mode): If a concept has an existential quantifier constraint, and the domain of the attribute in the constraint does not intersect with the concept, then the concept is unsatisfiable. This mode can be expressed as axioms: , and , where X and Z represent arbitrary atomic concepts, and Y represents an arbitrary concept. represents the top concept (i.e., the parent concept of any concept), It is defined that concept Z is the domain of attribute r; MIPS Mode 5 (Existential Range Mode): If a concept has an existential quantifier constraint and the filled-in terms of the constraint do not intersect with the range of its attributes, then the concept is unsatisfiable. This mode can be expressed as axioms: , and , where X, Y and Z represent arbitrary atomic concepts, Represents the top concept, The concept Z is defined as the value range of the attribute r.

[0031] For example, we take a MIPS in miniTambis as an example to illustrate how to extract patterns from a MIPS. This MIPS contains the following axioms: [1] ; [2] ; [3] ; [4] ; Among them, isomerization means isomerization, involvess means involvement, regulation means regulation, ligation means connection, process means process, contains means inclusion, isomers means isomers, and racemation means racemization. From the second axiom, we can see that isomerization is a subconcept of ligation. Combined with axiom 3, we can deduce that isomerization⊑∀involves.racemation (mark this derived axiom as [5]) holds. Therefore, according to axioms 1, 4, and 5, we can see that isomerization is an unsatisfiable concept. It can be seen that the original axioms 1-4 can deduce an unsatisfiable concept, and removing any axiom will destroy the reasoning chain and make it impossible to deduce this unsatisfiable concept. Therefore, axioms 1-4 constitute a MIPS. To extract patterns from them, we only need to focus on the core information: for the second axiom, although it also involves the concepts process, isomers and the attribute contains, these have no effect on the derivation of unsatisfiable concepts, so they can be ignored; the fifth axiom is derived from the second and third axioms, and the fifth axiom can replace the second and third axioms, which has no effect on the derivation of unsatisfiable concepts. Therefore, the core axioms finally obtained are 1, 4, and 5. Any one of the axioms cannot be derived from the other axioms, and these three axioms can derive an unsatisfiable concept. They can be used as a pattern (i.e., MIPS pattern 3), and characters can be used to replace the concept or attribute names in them, which appears more general.

[0032] The MIS pattern refers to the common axiom set type that leads to logical inconsistency. Based on the existing inconsistent ontology, the MIS pattern summarized mainly includes: MIS pattern 1 (disjoint type pattern): If two disjoint concepts have the same instance, it leads to inconsistency. This pattern can be expressed as axioms: , X(a), Y(a), where X and Y represent concepts, a represents an instance, and X(a) means that a is an instance of X; MIS pattern 2 (functional property contradiction pattern): For a functional property, if an instance is associated with two different instances through the property, inconsistency occurs. This is because the functional property essentially constrains the instance of the property, that is, if an instance is associated with other instances through a functional property, it can only be associated with a unique instance. This pattern can be expressed by axioms: r(a, b), r(a, c), Func(r), b≠c, where r represents an object property, a, b and c represent instances, and Func(r) means that r is a functional property; MIS pattern 3 (inverse functional property contradiction pattern): For an inverse functional property, if two different instances are associated with the same instance through the property, inconsistency occurs. Here, the inverse functional property is the opposite of the functional property. This pattern can be expressed by axioms: r(b, a), r(c, a), InvFunc(r), b≠c, where r represents an object property, a, b and c represent instances, and InvFunc(r) indicates that r is an inverse functional property. Obviously, this pattern is similar to MIS pattern 2; MIS Mode 4 (Maximum Quantity Constraint Mode): If a concept can be associated with at most one value on a data type attribute, and an instance of the concept is associated with two different values ​​through the attribute, it leads to inconsistency. This mode can be expressed as axioms: , X(c), r(c, v1), r(c, v2), v1≠v2, where X represents an atomic concept, c represents an instance, r represents a data type attribute, v1 and v2 represent specific values, Literal represents literal, the first axiom indicates that X can only be associated with one value through r, and r(c, v1) indicates that there is a relationship r between c and v1. It is worth noting that this pattern also applies to object properties, which can be expressed as: , X(c), Y(y1), Y(y2), r(c, y1), r(c, y2), y1≠y2, where X and Y represent concepts, r represents an object attribute, and y1 and y2 are two different instances of concept Y; MIS pattern 5 (empty domain instance pattern): If the domain of an attribute is empty, but the attribute has an instance, it will lead to inconsistency. This pattern can be expressed as axioms: , r(a,b), where r represents the object attribute, a and b represent instances, This indicates that the domain of r is the base concept. This pattern can be transformed to the range of r as the base concept, which will also lead to inconsistency.

[0033] Step 2: Construct positive and negative example datasets based on the logical conflict patterns and corresponding ontologies, where positive examples include positive examples with logical conflicts and no redundant axioms and positive examples with logical conflicts and redundant axioms. Each positive or negative example is an axiom set.

[0034] In this embodiment, step 2 specifically includes the following steps: Step 2.1: Constructing a positive example with logical conflicts and no redundant axioms means selecting MIPS or MIS with different sizes and expressiveness from the existing MIPS or MIS according to each summarized pattern. The selected MIPS or MIS should come from different ontologies as much as possible. A positive example containing redundant axioms means that removing the redundant axioms from the positive example does not affect its logical inconsistency or logical inconsistency.

[0035] Step 2.2: Constructing positive examples with logical conflicts and redundant axioms means selecting some positive examples that are relatively easy to understand for the large language model from the positive examples with logical conflicts and no redundant axioms, and adding one or more axioms to each selected positive example. The types of axioms added should be as diverse as possible, such as adding inclusion axioms, non-intersection axioms, domain or range definitions of attributes between atomic concepts, etc. In addition, the added axioms need to distinguish between two situations: one is that there is no relationship with the current positive example, and the other is that there is a relationship. Here, the existence of two axioms means that the same concepts, attributes or instances exist in the two axioms, and the existence of an axiom with the same axiom set means that the axiom has a relationship with a certain axiom in the axiom set. The added axioms can come from the ontology where the positive example is located, or they can be constructed by themselves based on experience.

[0036] Step 2.3: Constructing a negative example dataset refers to selecting some positive examples from positive examples that contain logical conflicts and no redundant axioms, and constructing them by removing axioms of different types and numbers. In fact, negative examples are constructed by interrupting the reasoning chain of logical conflicts. To interrupt the reasoning chain, it is enough to delete one axiom from each selected positive example. However, due to the hallucination problem of the large language model, sometimes the deletion of one axiom is not recognized by it. Therefore, the construction of negative examples considers deleting one or two axioms from a selected positive example. The removed axioms are also as diverse as possible to facilitate observation of which types of removals can allow the large language model to make better correct judgments.

[0037] Step 3: Design three different types of logical conflict detection prompts, including simple and direct prompts, pattern-based prompts, and pattern and reasoning rule-based prompts.

[0038] In this embodiment, step 3 specifically includes the following steps: Step 3.1: Simple and direct prompt words refer to describing the task in a simple and clear sentence, that is, performing logical conflict detection on the input positive or negative examples, and giving corresponding explanations if there is a logical conflict.

[0039] Step 3.2: Prompt words based on patterns refer to giving a definition of the corresponding pattern of logical conflict in the prompt words; prompt words based on patterns and reasoning rules refer to considering some common reasoning rules in addition to the pattern of logical conflict in the prompt words.

[0040] The common reasoning rules are summarized based on the calculated MIPS or MIS reasoning rules, and mainly include the following reasoning rules: Inference rule 1: If a concept X is a subconcept of the union of two other concepts, and these two concepts are subconcepts of the same concept C, then it can be deduced that X is also a subconcept of C; expressed as axiom: , , Derived , where X, A, B and C are all concepts.

[0041] Inference rule 2: If the union of two concepts is equal to another concept, then the two concepts are subconcepts of the other concept respectively; expressed as axiom: Derived and , where A, B and X are all concepts, Indicates the equivalence relationship between concepts.

[0042] Inference rule 3: If the domain of an object attribute is concept X, and one instance is related to another instance through the attribute, then it is inferred that the previous instance belongs to X, which can be expressed as axiom: , r(c1,c2) ​​derives X(c1), where X is the concept, c1 and c2 are instances, and r is the object attribute.

[0043] Inference rule 4: If the value range of an object attribute is concept X, and one instance is related to another instance through the attribute, then it is inferred that the latter instance belongs to X, which can be expressed as: , r(c1,c2) ​​derives X(c2), where X is the concept, c1 and c2 are instances, and r is the object attribute.

[0044] Inference Rule 5: If the range of an object attribute is Z, a concept has an existential quantifier constraint on the attribute, and there is an instance of the type of the concept, then it is inferred that Z is also a type of the instance, expressed as: , , X(a) leads to Z(a), where X, Y, and Z represent concepts and a represents an instance. Indicates the top concept.

[0045] Inference Rule 6: If the domain of an object attribute is Z, and a concept is equivalent to defining a universal quantifier constraint on the attribute and is a subconcept of Z, then it is inferred that Z is a top-level concept, expressed by the axiom: , , Derived , where X, Y, and Z are concepts, Top concept.

[0046] Inference Rule 7: If the domain of an object attribute is Z, a concept is equivalent to defining a universal quantifier constraint on the attribute, and the concept is related to other concepts through the attribute, then it is inferred that Z is a top-level concept, which can be expressed by the axiom: , , Derived , where X, Y, Z, K are concepts, This rule is similar to inference rule 6, except that Replace with .

[0047] The above rules provide common reasoning rules for concept inclusion relationships, instance types, and equivalent top concepts, which can assist large language models in detecting logical conflicts.

[0048] Step 4: Combine the positive and negative examples with the three prompt words respectively and ask questions to the large language model.

[0049] The method of combining positive examples and negative examples with three prompt words respectively and asking questions to the large language model means first constructing a complete and executable prompt word, and then using the prompt word as the input of the large language model under test, and recording the output corresponding to each input (i.e., the test result). The large language model under test can be any large language model, such as ChatGPT, LLaMA, or PaLM. The positive or negative examples in the prompt word, as well as the logical conflict patterns or reasoning rules involved, can be given in the format of natural language, the output of the network ontology language application program interface (OWLAPI), or the format of description logic.

[0050] For example, for the four axioms in the above example: [1] ; [2] ; [3] ; [4] ; These axioms are expressed in the form of description logic axioms. If the format output by OWLAPI is used, they are expressed as follows: [1] SubClassOf( <isomerisation>ObjectSomeValuesFrom( <involves> <regulation>)); [2] DisjointClasses( <racemation> <regulation>); [3] EquivalentClasses( <isomerisation>ObjectIntersectionOf( <ligation> <process>ObjectSomeValuesFrom( <contains> <isomers>)) ); [4] SubClassOf( <ligation>ObjectAllValuesFrom( <involves> <racemation>)); Among them, SubClassOf represents the inclusion relationship between concepts, ObjectSomeValuesFrom represents the existence of a quantifier constraint, DisjointClasses represents the disjoint relationship between concepts, ObjectIntersectionOf represents the intersection between concepts, EquivalentClasses represents the equivalence relationship between concepts, and ObjectAllValuesFrom represents the universal quantifier constraint. If expressed in natural language, the following conversion is performed using the large language model ChatGPT as an example to obtain the following sentence: [1] Isomerization is a process that involves regulation.

[0051] [2] Isomerization is equivalent to the intersection of connection, process, and things that contain isomers.

[0052] [3] Ligation is a process that involves only racemization.

[0053] [4] Racemization is not regulation.

[0054] Step 5: Evaluate the logical conflict judgment and interpretation capabilities of the large language model based on the test results of the large language model. This means first judging the correctness of whether the logical conflict is contained, then looking at the correctness of the interpretation, and then performing statistics based on these correctness judgment results, and evaluating the tested large language model according to the metric standard.

[0055] In this embodiment, step 5 specifically includes the following steps: Step 5.1: The correctness of the logical conflict judgment and the correctness of the interpretation are judged based on the test cases. For positive examples with logical conflicts and no redundant axioms, if the test results show that the positive example contains logical conflicts, the logical conflict judgment is considered correct, otherwise it is wrong; for such positive examples, if the interpretation information contains all the axiom information in the positive example and there is no redundancy, then the interpretation is correct; for positive examples with logical conflicts and redundant axioms, if the test results show that the logical conflict is contained, then the logical conflict judgment is correct, otherwise it is wrong; for such positive examples, if the interpretation information only contains all the axiom information in the original MIPS or MIS in the positive example, then the interpretation is correct; if in addition to this information, it also contains other information, indicating that there is redundant information in the interpretation; if it does not contain all the axiom information in the original MIPS or MIS, then the interpretation is wrong; for negative examples, if the test results show that it contains logical conflicts, then the logical conflict judgment is wrong; if it does not contain logical conflicts, then the logical conflict judgment is correct.

[0056] Step 5.2: Performing statistics based on these correctness judgment results means that for a large language model, all of its test results are statistically analyzed as follows: (1) For the prompt words that contain positive examples of the non-redundant axiom, count the results that can correctly judge the existence of logical conflicts and the correct interpretation, and mark the corresponding positive examples as the set ; (2) For the prompt words containing positive examples of redundant axioms, count the results that can correctly judge the existence of logical conflicts, correct interpretation and no redundant information, and mark the corresponding positive examples as the set In addition, the results that can correctly judge the existence of logical conflicts and the correct interpretation but with redundant information are counted, and the corresponding positive examples are marked as a set ; (3) For the prompt words containing positive examples, count the results that can correctly judge the existence of logical conflicts but have incorrect interpretations, and mark the corresponding positive examples as the set ; In addition, the results of judging that there is no logical conflict are counted, and the corresponding positive examples are marked as the set ; (4) For the prompt words containing negative examples, count the correct judgment results without logical conflicts and mark the corresponding negative examples as the set ; Secondly, count the results of wrong judgments with logical conflicts and mark the corresponding negative examples as a set .

[0057] It is worth noting that in In the judgment of the test results of the middle element, the correct interpretation defaults to the absence of redundant information in the interpretation, because the test results are generated based on the input information, which is reflected in the design of the prompt words to reduce the occurrence of hallucination problems. In addition, for the prompt words containing negative examples, since they do not contain logical conflicts, the corresponding explanations are not important, and it is only necessary to judge whether they contain logical conflicts; for incorrect judgments, it mainly means that the large language model under test produces hallucinations.

[0058] Step 5.3: Evaluate the large language model under test according to the metric standard, which means first defining the metric standards of different dimensions based on statistical information, and then calculating the test results according to these metric standards. , and They represent the positive example set without redundant axioms, the positive example set with redundant axioms, and the negative example set, respectively. Representing a collection The number of elements in the ,define the following metrics: First, define the correct rate of logical conflict judgment of the tested large language model : ; Next, define the error rate of logical conflict judgment of the large language model under test : ; From this we can see that Similarly, the accuracy of the logical conflict interpretation of the large language model under test can be defined as : ; In addition, the accuracy of the minimum logical conflict interpretation of the large language model under test can be defined : ; Finally, we can define the probability that the large language model under test will produce hallucinations when judging and interpreting logical conflicts. : ; in, and The judgment of the results of the elements in the test is mostly caused by hallucination problems. The former is that the logical conflict in the positive example is correctly judged but the interpretation is wrong, which means that the large language model under test does not really understand the corresponding input; the latter is that the logical conflict does not exist in the negative example. The judgment is wrong, which means that the large language model under test does not strictly judge the logical conflict according to the input axioms, but draws conclusions based on assumptions. The logical conflicts in the positive examples are not recognized by the large language model, which is often not due to hallucination problems, but mainly due to the insufficient logical reasoning ability of the large language model being tested.

[0059] Through these metrics, the overall accuracy of the tested large language model in logical conflict judgment (i.e. ) and error rate ( ) can be evaluated, and the accuracy of the logical conflict interpretation of the large language model under test can be further evaluated ( ) and the ability to calculate with minimal logical conflicts ( Among them, the minimization of logical conflicts is often the property that academic researchers prefer. On the one hand, it is convenient for users to not be disturbed by redundant axioms and better understand logical conflicts. On the other hand, it provides the most core information for later logical conflict repair. Finally, the probability of hallucinations in the judgment and interpretation of logical conflicts by the tested large language model is also quantified and measured (i.e. ), which helps people understand the probability of interference caused by the large language model under test when making logical conflict judgments or interpretations.

[0060] Embodiment 2: This embodiment provides a device for evaluating the logical conflict detection capability of a large language model, comprising: A logic conflict summary module, used for summarizing logic conflict patterns according to the ontology containing logic conflicts; Positive and negative example construction module, used to construct positive and negative example datasets based on logical conflict patterns and corresponding ontologies; The prompt word design module is used to design prompt words for testing large language models, including simple and direct prompt words, pattern-based prompt words, and prompt words based on patterns and inference rules; The model testing module is used to combine the positive and negative datasets with various types of pre-designed logical conflict detection prompt words to ask questions to the large language model; The performance evaluation module is used to evaluate the logical conflict judgment and interpretation capabilities of the large language model based on the test results of the large language model.

[0061] The specific functional implementation of each of the above modules can be found in the relevant contents of the method in Example 1 and will not be elaborated here.

[0062] Embodiment 3, this embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in Embodiment 1 are implemented.

[0063] Embodiment 4: This embodiment provides a computer device, including: Memory, for storing computer programs / instructions; A processor, configured to execute the computer program / instructions to implement the steps of any one of the methods described in Example 1.

[0064] Embodiment 5: This embodiment provides a computer program product, including a computer program / instruction, which implements the steps of any method described in Embodiment 1 when executed by a processor.

[0065] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0066] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.< / racemation> < / involves> < / ligation> < / isomers> < / contains> < / process> < / ligation> < / isomerisation> < / regulation> < / racemation> < / regulation> < / involves> < / isomerisation>

Claims

1. A method for evaluating the logical conflict detection capability of a large language model, characterized in that: include: Summarize logical conflict patterns based on the ontology containing logical conflicts; Construct positive and negative datasets based on logical conflict patterns and corresponding ontologies; Combine the positive and negative datasets with different types of pre-designed logical conflict detection prompts to ask questions to the large language model. Based on the test results of the large language model, the logical conflict judgment and interpretation capabilities of the large language model are evaluated.

2. The method for evaluating the logical conflict detection capability of a large language model according to claim 1, characterized in that: The method of summarizing the logic conflict pattern according to the ontology containing the logic conflict includes: Use existing ontology logic conflict interpretation tools to detect logic conflicts, calculate the smallest set of axioms that contain logic conflicts, and then summarize common logic conflict patterns; Wherein, the ontology includes concepts, object attributes, data type attributes and instances; The logical conflict means that there are contradictory statements or inferences in a logical system at the same time, including logical inconsistency or logical inconsistency; a set of axioms is logically inconsistent, which means that the set has no model or explanation, and a set of axioms is logically inconsistent, which means that the set contains at least one unsatisfiable concept; The minimum set of axioms containing logical conflicts indicates that the set contains logical inconsistency or incoordination, and removing any one of the axioms will cause the set to restore consistency or coordination; The ontology containing logical conflicts represents inconsistent or uncoordinated ontologies that are often used in experimental evaluations in existing research work on logical conflict handling; The method of using an existing ontology logic conflict interpretation tool to perform logic conflict detection and calculate the minimum axiom set containing logic conflicts means calculating the minimum inconsistency preserving subset MIS or the minimum incoherence preserving subset MIPS through an ontology network repair and diagnosis tool or an ontology editing tool, wherein a MIS is a minimum axiom set containing logic inconsistencies and a MIPS is a minimum axiom set containing logic inconsistencies; The formal definition of MIS is: given an inconsistent ontology O and its subset O′, if O′ is inconsistent, and every proper subset of O′ are all consistent, represents a true inclusion relation between sets, then O′ is called a MIS of O; The formal definition of MIPS is: Given a logically consistent ontology O, if a subset S of O is inconsistent, and every proper subset of S are all coordinated, then S is said to be a MIPS of O; The common logic conflict modes summarized include: selecting a MIPS mode or a MIS mode whose occurrence frequency is greater than a set threshold based on the MIPS calculated in the selected ontology and the MIS; wherein the MIPS mode represents common types of axiom sets that lead to logical inconsistencies; The MIS patterns represent common types of axiom sets that lead to logical inconsistencies.

3. The method for evaluating the logical conflict detection capability of a large language model according to claim 2, characterized in that: The MIPS modes include: MIPS Mode 1: If a concept has two disjoint parent concepts, then the concept is unsatisfiable. MIPS Mode 1 is expressed in axioms as follows: , and , where X, Y and Z represent arbitrary atomic concepts, Indicates the inclusion relationship between concepts. Indicates no relationship; MIPS Mode 2: If a concept has an existential quantifier constraint and the filler term of the constraint is unsatisfiable, then the concept is unsatisfiable. MIPS Mode 2 is expressed as axioms: and , where X and Y represent arbitrary atomic concepts, r represents object attributes, Represents the concept of emptiness, Indicates the existence of a quantifier constraint; MIPS Mode 3: If a concept has both existential and universal quantifier constraints on an attribute, and the filled-in terms of these two constraints do not intersect, then the concept is unsatisfiable. MIPS Mode 3 is expressed by axioms as follows: , and , where X, Y and Z represent arbitrary atomic concepts, r represents object attributes, Indicates universal quantifier constraint; MIPS Mode 4: If a concept has an existential quantifier constraint, and the domain of the attribute in the constraint does not intersect with the concept, then the concept is unsatisfiable. MIPS Mode 4 is expressed as an axiom: , and , where X and Z represent arbitrary atomic concepts, and Y represents an arbitrary concept. Represents the top concept, It is defined that concept Z is the domain of attribute r; MIPS Mode 5: If a concept has an existential quantifier constraint, and the filled-in terms of the constraint do not intersect with the range of its attributes, then the concept is unsatisfiable. MIPS Mode 5 is expressed in axioms as follows: , and , where X, Y and Z represent arbitrary atomic concepts, Represents the top concept, The concept Z is defined as the value range of the attribute r.

4. The method for evaluating the logical conflict detection capability of a large language model according to claim 2, characterized in that: The MIS mode includes: MIS Mode 1: If two disjoint concepts have the same instance, it leads to inconsistency. MIS Mode 1 is expressed by axioms as follows: , X(a), Y(a), where X and Y represent concepts, a represents an instance, and X(a) means that a is an instance of X; MIS Mode 2: For a function attribute, if an instance is associated with two different instances through the attribute, an inconsistency occurs. MIS Mode 2 is expressed by the axioms: r(a, b), r(a, c), Func(r), b≠c, where r represents an object attribute, a, b and c represent instances, and Func(r) represents that r is a function attribute; MIS Mode 3: For an inverse function attribute, if two different instances are associated with the same instance through the attribute, inconsistency occurs. The inverse function attribute is opposite to the function attribute. MIS Mode 3 is expressed by the axioms: r(b,a), r(c, a), InvFunc(r), b≠c, where r represents an object attribute, a, b and c represent instances, and InvFunc(r) represents that r is an inverse function attribute. MIS Mode 4: If a concept can be associated with at most one value on a data type attribute, and an instance of the concept is associated with two different values ​​through the attribute, it will lead to inconsistency. MIS Mode 4 is expressed by axioms as follows: , X(c), r(c, v1), r(c, v2), v1≠v2, where X represents an atomic concept, c represents an instance, r represents a data type attribute, v1 and v2 represent specific values, Literal represents literal, the first axiom indicates that X can be associated with at most one value through r, r(c, v1) indicates that there is a relationship r between c and v1, MIS mode 4 also applies to object attributes, which is expressed as axioms: , X(c), Y(y1), Y(y2), r(c, y1), r(c, y2), y1≠y2, where X and Y represent concepts, r represents an object attribute, and y1 and y2 are two different instances of concept Y; MIS Mode 5: If the domain of an attribute is empty and the attribute has instances, it will lead to inconsistency. MIS Mode 5 is expressed by axioms as follows: , r(a,b), where r represents the object attribute, a and b represent instances, It shows that the domain of r is the base concept. MIS mode 5 can also be transformed into the value range of r as the base concept, which will also lead to inconsistency.

5. The method for evaluating the logical conflict detection capability of a large language model according to claim 1, characterized in that: The constructing of positive and negative datasets based on the logical conflict patterns and corresponding ontologies includes: Constructing a positive example data set, including constructing positive examples with logical conflicts and no redundant axioms and positive examples with logical conflicts and redundant axioms, wherein each positive example is an axiom set; a positive example containing redundant axioms means that removing the redundant axioms from the positive example does not affect its logical inconsistency or logical inconsistency; Constructing a positive example with logical conflicts and no redundant axioms means selecting MIPS or MIS with different sizes and expressive power from existing MIPS or MIS; Constructing positive examples with logical conflicts and redundant axioms means selecting some positive examples that are relatively easy for the large language model to understand from the positive examples with logical conflicts and no redundant axioms, and adding one or more axioms to each selected positive example. The added axioms are divided into two situations: one is that they have no relationship with the current positive example, and the other is that they have a relationship. The existence of a relationship between two axioms means that the two axioms have the same concept, attribute or instance, and the existence of an axiom with an axiom set means that the axiom has a relationship with a certain axiom in the axiom set. Constructing a negative example dataset involves selecting some positive examples from positive examples that contain logical conflicts and no redundant axioms, and constructing them by removing different types and numbers of axioms.

6. The method for evaluating the logical conflict detection capability of a large language model according to claim 5, characterized in that: The multiple different types of logical conflict detection prompt words include: simple and direct prompt words, pattern-based prompt words, and pattern and reasoning rule-based prompt words; The simple and direct prompt word means describing the task in a simple and clear sentence, performing logical conflict detection on the input positive or negative examples, and providing corresponding explanations if there is a logical conflict; The pattern-based prompt word means that a definition of a logic conflict corresponding pattern is given in the prompt word; The prompt words based on patterns and inference rules mean that in addition to considering the patterns of logical conflicts, some set common inference rules are also considered in the prompt words.

7. The method for evaluating the logical conflict detection capability of a large language model according to claim 6, characterized in that: The common inference rules include the following inference rules: Inference rule 1: If a concept X is a subconcept of the union of two other concepts, and these two concepts are subconcepts of the same concept C, then it can be deduced that X is also a subconcept of C; expressed as axiom: , , Derived , where X, A, B and C are all concepts; Inference rule 2: If the union of two concepts is equal to another concept, then the two concepts are subconcepts of the other concept respectively; expressed as axiom: Derived and , where A, B and X are all concepts, Represents equivalence relations between concepts; Inference rule 3: If the domain of an object attribute is concept X, and one instance is related to another instance through the attribute, then it is inferred that the previous instance belongs to X, which can be expressed as axiom: , r(c1,c2) ​​derives X(c1), where X is a concept, c1 and c2 are instances, and r is an object attribute; Inference rule 4: If the value range of an object attribute is concept X, and one instance is related to another instance through the attribute, then it is inferred that the latter instance belongs to X, which can be expressed as: , r(c1,c2) ​​derives X(c2), where X is a concept, c1 and c2 are instances, and r is an object attribute; Inference Rule 5: If the range of an object attribute is Z, a concept has an existential quantifier constraint on the attribute, and there is an instance of the type of the concept, then it is inferred that Z is also a type of the instance, expressed as: , , X(a) leads to Z(a), where X, Y, and Z represent concepts and a represents an instance. Indicates the top concept; Inference Rule 6: If the domain of an object attribute is Z, and a concept is equivalent to defining a universal quantifier constraint on the attribute and is a subconcept of Z, then it is inferred that Z is a top-level concept, expressed by the axiom: , , Derived , where X, Y, and Z are concepts, For the top concept; Inference Rule 7: If the domain of an object attribute is Z, a concept is equivalent to defining a universal quantifier constraint on the attribute, and the concept is related to other concepts through the attribute, then it is inferred that Z is a top-level concept, which can be expressed by the axiom: , , Derived , where X, Y, Z, K are concepts, Top concept.

8. The method for evaluating the logical conflict detection capability of a large language model according to claim 1, characterized in that: The method combines the positive and negative datasets with a plurality of different types of pre-designed logic conflict detection prompt words to ask questions to the large language model, including: first constructing a complete and executable prompt word, then using the prompt word as the input of the large language model to be tested, and recording the output corresponding to each input, wherein the positive or negative examples in the prompt word, as well as the logic conflict pattern or reasoning rule, can be output in the format of an application program interface output of a natural language or a network ontology language or in the format of a description logic.

9. The method for evaluating the logical conflict detection capability of a large language model according to claim 1, characterized in that: According to the test results of the large language model, the logical conflict judgment and interpretation ability of the large language model is evaluated, including: According to the test cases, the correctness of the logical conflict judgment and the correctness of the interpretation are judged. For positive examples with logical conflicts and no redundant axioms, if the test results show that the positive example contains logical conflicts, the logical conflict judgment is considered correct, otherwise it is wrong; for such positive examples, if the explained information contains all the axiom information in the positive example and there is no redundancy, then the interpretation is correct; for positive examples with logical conflicts and redundant axioms, if the test results show that the logical conflict is contained, then the logical conflict judgment is correct, otherwise it is wrong; for such positive examples, if the explained information only contains all the axiom information in the original MIPS or MIS in the positive example, then the interpretation is correct; if in addition to this information, other information is also contained, it means that there is redundant information in the interpretation; if it does not contain all the axiom information in the original MIPS or MIS, then the interpretation is wrong; for negative examples, if the test results show that it contains logical conflicts, then the logical conflict judgment is wrong; if it does not contain logical conflicts, then the logical conflict judgment is correct; Based on the correctness judgment results, all test results are statistically analyzed as follows: For the prompt words containing the positive examples of the non-redundant axiom, count the results that can correctly judge the existence of logical conflicts and explain correctly, and mark the corresponding positive examples as the set ; For the prompt words containing redundant axiom positive examples, count the results that can correctly judge the existence of logical conflicts, correct interpretation and no redundant information, and mark the corresponding positive examples as the set In addition, the results that can correctly judge the existence of logical conflicts and the correct interpretation but with redundant information are counted, and the corresponding positive examples are marked as a set ; For the prompt words containing positive examples, count the results that can correctly judge the existence of logical conflicts but have incorrect interpretations, and mark the corresponding positive examples as the set ; In addition, the results of judging that there is no logical conflict are counted, and the corresponding positive examples are marked as the set ; For the prompt words containing negative examples, count the correct judgment results without logical conflicts and mark the corresponding negative examples as a set ; Secondly, count the results of wrong judgments with logical conflicts and mark the corresponding negative examples as a set ; Based on the statistical information, we define metrics of different dimensions, and then calculate the test results based on these metrics. We define the following metrics: Define the accuracy of the logical conflict judgment of the large language model under test : ； in, represents the set of positive examples of the non-redundant axiom, represents the set of positive examples with redundant axioms, represents the set of negative examples with redundant axioms, Representing a collection The number of elements in ; Define the error rate of logical conflict judgment of the large language model under test : ； in, ; Define the accuracy of the logical conflict interpretation of the large language model under test : ； Define the correctness of the minimum logical conflict interpretation of the large language model under test : ； Define the probability that the large language model under test will produce hallucinations when judging and interpreting logical conflicts : ； The tested large language models are evaluated according to the above metrics.

10. A device for evaluating the logical conflict detection capability of a large language model, characterized in that: include: A logic conflict summary module, used for summarizing logic conflict patterns according to the ontology containing logic conflicts; Positive and negative example construction module, used to construct positive and negative example datasets based on logical conflict patterns and corresponding ontologies; The prompt word design module is used to design prompt words for testing large language models, including simple and direct prompt words, pattern-based prompt words, and prompt words based on patterns and inference rules; The model testing module is used to combine the positive and negative datasets with various types of pre-designed logical conflict detection prompt words to ask questions to the large language model; The performance evaluation module is used to evaluate the logical conflict judgment and interpretation capabilities of the large language model based on the test results of the large language model.

Images