Evidence-based evaluation method, device and medium for model output content

Abstract

The application relates to the technical field of scientific base model, and discloses a method and device for evidence-based evaluation of model output content, electronic equipment and a medium. The method for evidence-based evaluation of model output content comprises the following steps: obtaining original text of a model answer output by a scientific base model; constructing a field knowledge package based on data of a subject database; performing disassembly processing on the original text and the field knowledge package to obtain an atomic proposition set; obtaining evidence from the subject database based on the atomic proposition set to generate a proposition-evidence association package; generating an evidence-based scoring package according to the atomic proposition set and the proposition-evidence association package, wherein the evidence-based scoring package comprises a final fidelity score and evidence conflict entropy of each atomic proposition; and generating a corresponding evidence-based report according to the original text and the evidence-based scoring package. According to the method for evidence-based evaluation of model output content, the controversy or disagreement between evidences can be quantitatively expressed.

Description

Technical Field

[0001] This application relates to the field of scientific foundation model technology, and in particular to an evidence-based evaluation method, apparatus, electronic device and medium for model output content. Background Technology

[0002] The Scientific Foundation Model (SciFM) is a general-purpose scientific model pre-trained with large-scale, high-quality scientific data for the scientific field, providing a unified intelligent foundation for scientific research tasks across various disciplines.

[0003] In related technologies, when there is academic controversy among multiple sources of retrieved evidence, there is a lack of quantitative conflict assessment mechanisms for the model answers output by scientific foundation models. Summary of the Invention

[0004] This application provides an evidence-based evaluation method, apparatus, electronic device, and medium for model output content, which can quantify the disputes or disagreements between pieces of evidence.

[0005] In a first aspect, embodiments of this application provide an evidence-based evaluation method for model output content, comprising: obtaining the original text of the model answer output by a scientific foundation model; constructing a domain knowledge package based on data from a subject database, the domain knowledge package including an entity weight table, a relation matrix, and term classification information, the entity weight table including terms in the domain knowledge package and term scores for each term, the relation matrix including the association strength value between every two terms; decomposing and processing the original text and the domain knowledge package to obtain a set of atomic propositions; obtaining evidence from the subject database based on the set of atomic propositions to generate a proposition evidence association package, the proposition evidence association package including each atomic proposition in the set of atomic propositions and the optimal evidence set associated with each atomic proposition; generating an evidence-based scoring package based on the set of atomic propositions and the proposition evidence association package, the evidence-based scoring package including the final fidelity score and evidence conflict entropy for each atomic proposition; and generating a corresponding evidence-based report based on the original text and the evidence-based scoring package.

[0006] According to some of the foregoing embodiments of the first aspect of this application, the step of constructing a domain knowledge package based on data from a subject database includes: when the domain distribution ratio of candidate terms in the subject database is greater than a first threshold, determining that the candidate terms are terms included in the domain knowledge package, and calculating the term score of the candidate terms; generating an entity weight table based on all the terms determined to be included in the domain knowledge package and the corresponding term scores; calculating the association strength value between every two terms in the entity weight table, and generating a relationship matrix.

[0007] According to some of the foregoing embodiments of the first aspect of this application, the step of calculating the term score of the candidate term includes: calculating the terminology degree, degree of freedom, and cohesion degree of the candidate term; normalizing and weighting the terminology degree, degree of freedom, and cohesion degree of the candidate term to obtain the term score of the candidate term.

[0008] According to some of the foregoing embodiments of the first aspect of this application, calculating the terminology degree, freedom degree, and cohesion degree of the candidate terms includes: calculating the terminology degree of the candidate terms using the following formula: ; Where 'a' represents a candidate term. The terminology level of the candidate terms. The phrase length of the candidate term. The frequency of occurrence of the candidate terms. For a longer set of phrases containing the candidate terms; The right neighbor information entropy and left neighbor information entropy of the candidate terms are calculated using the following formula: ; ; in, The right neighbor information entropy of the candidate term. The left-neighbor information entropy of the candidate term. For the word elements that appear after the candidate word, In order to follow up on the candidate terms when they appear The conditional probability; The degrees of freedom of the candidate terms are calculated using the following formula: ; in, The degrees of freedom of the candidate terms; The cohesion of the candidate terms is calculated using the following formula: ; in, The cohesion of the candidate terms. The probability of the candidate term appearing. For individual words that make up a candidate term Independent probability of occurrence.

[0009] According to some of the foregoing embodiments of the first aspect of this application, the step of calculating the association strength value between every two terms in the entity weight table and generating a relationship matrix includes: calculating the association strength value between every two terms in the entity weight table using the following formula: ; in, For the entry Related to the entry The correlation strength value between them For a collection of domain texts, For the entry The probability of appearing in the domain text For the entry The probability of appearing in the domain text For the entry Related to the entry The probability of them appearing together in the domain text.

[0010] According to some of the aforementioned embodiments of the first aspect of this application, the term classification information of the domain knowledge package includes entities, operators, attributes, or constraints; the step of decomposing the original text and the domain knowledge package to obtain a set of atomic propositions includes: generating a semantic graph based on the original text and the domain knowledge package; constructing an operator set from the terms in the domain knowledge package whose term classification information is an operator, locating the core predicate node in the semantic graph through the operator set; and performing a subgraph search in the semantic graph with the core predicate node as the central source point to obtain a result containing a core triple and a total weight loss. The minimum connected subgraph with the least cost is obtained, and the character offset interval corresponding to the minimum connected subgraph is obtained. The core triple includes a subject node, a predicate node, and an object node. In the minimum connected subgraph, the subject node, predicate node, object node, quantized value or state node, and unit or constraint node are extracted and used as the subject, scientific relation operator, object, quantized value or state, and unit or constraint of the atomic proposition in sequence. The atomic proposition is type-labeled to obtain the atomic proposition type label value. Based on the atomic proposition, the atomic proposition type label value, and the character offset interval, the atomic proposition set is obtained.

[0011] According to some of the foregoing embodiments of the first aspect of this application, the step of obtaining evidence from the subject database based on the atomic proposition set and generating a propositional evidence association package includes: recalling evidence from the subject database based on the atomic proposition set to obtain a candidate evidence set, the candidate evidence set including a plurality of candidate evidence; calculating the cross-modal alignment score of each candidate evidence based on the atomic proposition type tag value, the scientific relation operator, the quantification value or state of the atomic proposition; calculating the final comprehensive score of the candidate evidence based on the cross-modal alignment score; sorting the candidate evidence in the candidate evidence set in descending order according to the final comprehensive score, taking the top preset number of candidate evidence in the sorted set, mapping the original structured data from the subject database respectively to obtain the corresponding optimal evidence, and forming an optimal evidence set.

[0012] According to some of the foregoing embodiments of the first aspect of this application, generating an evidence-based scoring package based on the set of atomic propositions and the propositional evidence association package includes: calculating the degree of support of each of the best pieces of evidence for the corresponding atomic proposition; calculating the contribution weight of each of the best pieces of evidence under the corresponding atomic proposition; calculating the weighted average support and evidence conflict entropy of all the best pieces of evidence under the atomic proposition based on the degree of support and the contribution weight; applying a nonlinear penalty to the weighted average support using the evidence conflict entropy to obtain the corresponding final fidelity score, thereby obtaining the evidence-based scoring package.

[0013] According to some of the foregoing embodiments of the first aspect of this application, the atomic proposition set includes the atomic propositions, the atomic proposition type marker values ​​corresponding to the atomic propositions, and character offset intervals; the step of generating a corresponding evidence-based report based on the original text and the evidence-based scoring package includes: obtaining atomic proposition mapping metadata for each atomic proposition in the atomic proposition set; obtaining an optimal evidence path set based on the optimal evidence set in the proposition evidence association package; dividing the original text into multiple semantic segments according to the character offset intervals; and performing mapping processing on the semantic segments, wherein, if the semantic segments... If the position range of a segment falls within one of the character offset ranges, the semantic segment is mapped to the atomic proposition corresponding to the character offset range, thus confirming that the semantic segment is successfully anchored to the atomic proposition; otherwise, the semantic segment is mapped to an empty set. For semantic segments successfully anchored to the atomic proposition, a corresponding source tag is generated based on the evidence-based scoring package and the optimal evidence path set. Based on the evidence conflict entropy under the atomic proposition, the dispute indexing operator corresponding to the semantic segment is determined. An evidence-based report is obtained based on the semantic segment, the source tag, and the dispute indexing operator.

[0014] Secondly, embodiments of this application provide an evidence-based evaluation device for model output content, comprising: a model answer receiving module for acquiring the original text of the model answer output by a scientific foundation model; a domain knowledge package construction module for constructing a domain knowledge package based on data from a subject database, the domain knowledge package including an entity weight table, a relation matrix, and term classification information, the entity weight table including terms in the domain knowledge package and a term score for each term, the relation matrix including the association strength value between every two terms; and a proposition decomposition module for decomposing the original text and the domain knowledge package. The system comprises: a set of atomic propositions; an evidence retrieval module, used to obtain evidence from the subject database based on the set of atomic propositions and generate a propositional evidence association package, the propositional evidence association package including each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition; a quantitative evaluation module, used to generate an evidence-based scoring package based on the set of atomic propositions and the propositional evidence association package, the evidence-based scoring package including the final fidelity score and evidence conflict entropy of each atomic proposition; and an evidence-based report generation module, used to generate a corresponding evidence-based report based on the original text and the evidence-based scoring package.

[0015] Thirdly, embodiments of this application provide an electronic device, which includes a memory and a processor. The memory is communicatively connected to the processor. The memory stores instructions, and the processor invokes the instructions in the memory to cause the electronic device to execute an evidence-based evaluation method for model output content according to any of the foregoing embodiments of the first aspect of this application.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed by a processor, implement an evidence-based evaluation method for model output content according to any of the foregoing embodiments of the first aspect of this application.

[0017] The evidence-based evaluation method for model output content according to embodiments of this application includes: obtaining the original text of the model answer output by a scientific foundation model; constructing a domain knowledge package based on data from a subject database, the domain knowledge package including an entity weight table, a relation matrix, and term classification information, the entity weight table including terms in the domain knowledge package and the term score for each term, and the relation matrix including the association strength value between every two terms; decomposing and processing the original text and the domain knowledge package to obtain a set of atomic propositions; obtaining evidence from the subject database based on the set of atomic propositions to generate a propositional evidence association package, the propositional evidence association package including each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition; generating an evidence-based scoring package based on the set of atomic propositions and the propositional evidence association package, the evidence-based scoring package including the final fidelity score and evidence conflict entropy for each atomic proposition; and generating a corresponding evidence-based report based on the original text and the evidence-based scoring package. The evidence-based scoring package includes evidence conflict entropy, which quantifies the controversy or disagreement between evidence, making it easier for users to perceive the degree of evidence controversy behind the conclusion. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 This is a flowchart of one embodiment of an evidence-based evaluation method for the model output content of this application; Figure 2 This is a flowchart illustrating the process of constructing a domain knowledge package based on data from a subject database, as one embodiment of the evidence-based evaluation method for the model output content of this application. Figure 3 This is a flowchart illustrating the process of decomposing the original text and domain knowledge package to obtain a set of atomic propositions in one embodiment of the evidence-based evaluation method for the model output content of this application. Figure 4 This is a flowchart illustrating how evidence is obtained from a subject database based on a set of atomic propositions to generate a propositional evidence association package, according to one embodiment of the evidence-based evaluation method for the model output content of this application. Figure 5 This is a flowchart illustrating how an evidence-based evaluation method for the model output content of this application generates an evidence-based scoring package based on a set of atomic propositions and a propositional evidence association package, in one embodiment. Figure 6 This is a flowchart of generating a corresponding evidence-based report based on the original text and evidence-based scoring package, according to one embodiment of the evidence-based evaluation method for the model output content of this application. Figure 7This is a schematic diagram of the structure of an embodiment of an evidence-based evaluation device for the model output content according to this application; Figure 8 This is a schematic diagram of the hardware structure of an embodiment of the electronic device according to this application. Detailed Implementation

[0020] The technical solutions in the embodiments (or "implementations") of this application will be clearly and completely described herein with reference to the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0021] If the embodiments of this application contain terms relating to directional indications or positional relationships (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the attached figures); if the specific posture changes, the directional indications or positional relationships will also change accordingly. Furthermore, the terms "first" and "second" used in the embodiments of this application are only for descriptive convenience and should not be construed as indicating or implying relative importance.

[0022] This application provides an evidence-based evaluation method for model output content. Figure 1 This is a flowchart of one embodiment of the evidence-based evaluation method for the model output content according to this application. The evidence-based evaluation method for the model output content includes steps S110 to S160.

[0023] In step S110, the original text of the model answer output by the scientific base model is obtained.

[0024] In step S120, a domain knowledge package is constructed based on the data in the subject database. The domain knowledge package includes an entity weight table, a relation matrix, and term classification information. The entity weight table includes terms in the domain knowledge package and the term score for each term. The relation matrix includes the association strength value between every two terms.

[0025] Figure 2 This is a flowchart illustrating the process of constructing a domain knowledge package based on data from a subject database, according to one embodiment of the evidence-based evaluation method for the model output content of this application. In some embodiments, step S120 of constructing the domain knowledge package based on data from a subject database includes steps S121 to S123.

[0026] In step S121, when the domain distribution ratio of candidate terms in the subject database is greater than the first threshold, the candidate terms are determined to be terms included in the domain knowledge package, and the term score of the candidate terms is calculated.

[0027] In some embodiments, calculating the term score of a candidate term includes: calculating the terminology degree, freedom degree, and cohesion degree of the candidate term; normalizing and weighting the terminology degree, freedom degree, and cohesion degree of the candidate term to obtain the term score of the candidate term.

[0028] Specifically, in some embodiments, calculating the terminology degree, freedom degree, and cohesion degree of candidate terms includes the following steps: The terminology degree of candidate terms is calculated using the following formula: ; Where 'a' represents a candidate term. For the terminology level of candidate terms, The phrase length of the candidate term. The frequency of occurrence of candidate terms. This is a longer set of phrases that contain candidate terms.

[0029] The context of candidate terms in the subject database, including the set of left neighbor terms. and the set of right neighbor words .

[0030] The right neighbor information entropy and left neighbor information entropy of candidate terms are calculated using the following formula: ; ; in, The right neighbor information entropy of the candidate term. The left-neighbor information entropy of the candidate terms. These are the tokens that appear after the candidate terms. To closely follow when candidate terms appear The conditional probability.

[0031] The degrees of freedom of candidate terms are calculated using the following formula: ; in, The degree of freedom for candidate terms.

[0032] Degrees of freedom The higher the value, the better the candidate term. The more free the boundaries.

[0033] The cohesion of candidate terms is calculated using the following formula: ; in, The cohesion of candidate terms, The probability of a candidate term appearing. For individual words that make up a candidate term Independent probability of occurrence. Cohesion. The larger the value, the more likely these words in the candidate terms are to form a fixed expression.

[0034] In some embodiments, the terminology degree, freedom degree, and cohesion degree of candidate terms are normalized and weighted and fused to obtain the term score of the candidate terms, including: calculating the term score of the candidate terms using the following formula: ; in, Rate the candidate terms. , , The terminology level of the candidate terms Degrees of freedom Cohesion The scores after Min-Max normalization ensure that the dimensions are consistent and all belong to the same category. , , , Let these be the scoring weight coefficients corresponding to terminology degree, degree of freedom, and cohesion, respectively, and satisfy the following conditions: .

[0035] As mentioned earlier, when the domain distribution ratio of candidate terms in the subject database is greater than the first threshold, the candidate terms are determined to be terms included in the domain knowledge package, and the term score of the candidate terms is calculated.

[0036] In this embodiment, the domain distribution ratio of candidate terms is calculated according to the following formula. : ; in, This represents the probability of a candidate term appearing in a subject database. This represents the probability of a candidate term appearing in a general large-scale corpus. This is the first preset constant, which is a very small positive number to prevent the denominator from being zero.

[0037] if This indicates the candidate term. It is unique to this discipline; if This indicates the candidate term. It is a general term. When Calculate the term score for the candidate term. .

[0038] In step S122, an entity weight table is generated based on all terms identified as belonging to the domain knowledge package and their corresponding term scores.

[0039] In some embodiments, the entity weight table ,in For the entry, .

[0040] In some embodiments, the term classification information of the domain knowledge package includes entities, operators, attributes, or constraints.

[0041] For example, based on the statistical position of the term in the syntactic structure, terms can be classified as: entities. Operator Attributes or constraints .entity It usually appears in the subject or object position, and the entry score is... Extremely high; operator It typically appears in the verb position and has a high frequency in general corpora but has a specific meaning in this field; attributes or constraints. It usually appears in the position of a modifier and is often accompanied by a number or unit.

[0042] In step S123, the association strength value between every two terms in the entity weight table is calculated to generate a relationship matrix.

[0043] In some embodiments, the step S123 of calculating the association strength value between every two terms in the entity weight table and generating the relationship matrix includes: calculating the association strength value between every two terms in the entity weight table using the following formula: ; in, For the entry Related to the entry The correlation strength value between them For a collection of domain texts, For the entry The probability of appearing in the domain text For the entry The probability of appearing in the domain text For the entry Related to the entry The probability of them appearing together in the domain text.

[0044] Relationship matrix Each value in Association strength value The higher the value, the greater the prior probability that there is a strong semantic edge between the two nodes in the semantic graph that is subsequently processed.

[0045] like Figure 1 In step S130, the original text and domain knowledge package are decomposed and processed to obtain a set of atomic propositions.

[0046] In some embodiments, in step S130, the input object includes: the original text of the model answer output by the scientific foundation model. Original text It is a collection A sequence of lexical units: Domain knowledge package, which includes an entity weight table. And relation matrix .

[0047] In some embodiments, the output object in step S130 includes: a set of atomic propositions. .

[0048] Each atomic proposition Defined as a standard semantic tuple: ; in, As the main body, For scientific relation operators (such as " "That is, positive correlation," "That is, it belongs to", (that is, approximate) As an object, To quantify values ​​or states, For units or constraints.

[0049] Atomic proposition type flag value Determined in the following ways: ; That is, if the quantified value or state is not an empty set, meaning the atomic proposition includes specific numerical calculations, then the atomic proposition type marker value is... This indicates a proposition containing numerical values; if the quantified value or state is an empty set, that is, the atomic proposition does not include specific numerical calculations, but only describes properties, relationships, or existence, then the atomic proposition type flag value is... This indicates a proposition without a numerical value, or a descriptive proposition.

[0050] Character offset range For example: .

[0051] Figure 3 This is a flowchart illustrating the process of decomposing the original text and domain knowledge package to obtain a set of atomic propositions, as described in one embodiment of the evidence-based evaluation method for the model output content of this application. As mentioned above, the term classification information of the domain knowledge package includes entities, operators, attributes, or constraints. In some embodiments, step S130, which involves decomposing the original text and domain knowledge package to obtain a set of atomic propositions, includes steps S131 to S135.

[0052] In step S131, a semantic graph is generated based on the original text and the domain knowledge package.

[0053] In some embodiments, the semantic graph is a directed acyclic graph. Specifically, generating a semantic graph involves the following processes: Perform node set Generation: Utilizing entity items (terms) from the Lexicon entity weight table. For the original text Perform multi-pattern matching. If multiple tokens are identified to form a high-weight term (…), then… If it collapses into a single node, then it will be reduced to a single node. Assign weights to each node. .like For the matched entity, then ,otherwise .

[0054] Perform edge set Generation and Correction: Generating Initial Edge Sets Using the Basic Dependency Parser Then, using the relation matrix... Make corrections: If the initial edge set Middle node There are grammatical paths between them, and the relation matrix... middle Then the original weight of this edge. Perform linear augmentation: ; in, For syntax weight function, This represents the field strength coefficient.

[0055] In step S132, an operator set is constructed from the terms whose classification information in the domain knowledge package is used as operators. The core predicate node is located in the semantic graph using the operator set. This step achieves the extraction of logical operator kernels.

[0056] In step S133, using the core predicate node as the central source point, a subgraph search is performed in the semantic graph to obtain the minimum connected subgraph containing the core triples and having the minimum total weight loss. The character offset interval corresponding to the minimum connected subgraph is then obtained. The core triples include the subject node, the predicate node, and the object node; that is, the core triples are... .in, For the main node, For predicate nodes, As an object.

[0057] Find the minimum connected subgraph that contains the core triples and has the minimum total weight loss based on the following formula: ; in, It is a connected subgraph.

[0058] When obtaining the character offset interval corresponding to the minimum connected subgraph, the atomic proposition... Corresponding connected subgraph Its character offset range as follows: ; in, For nodes In the original text The starting or ending character position in the text.

[0059] In step S134, in the minimum connected subgraph, the subject node, predicate node, object node, quantized value or state node, and unit or constraint node are extracted and used as the subject, scientific relation operator, object, quantized value or state, and unit or constraint of the atomic proposition in turn.

[0060] In some embodiments, in the determined minimum connected subgraph In the middle, along Finding the highest path and This involves extracting the main and secondary components; and traversing the minimum connected subgraph. All non-core nodes that are retained in the process are used to extract quantized values ​​or states. Units or constraints .

[0061] Therefore, the atomic proposition can be obtained. It is defined as a standard semantic tuple: .

[0062] In step S135, the atomic propositions are type-marked to obtain atomic proposition type mark values, and a set of atomic propositions is obtained based on the atomic propositions, atomic proposition type mark values, and character offset intervals.

[0063] Atomic proposition type flag value Determined in the following ways: ; That is, if the quantified value or state is not an empty set, meaning the atomic proposition includes specific numerical calculations, then the atomic proposition type marker value is... This indicates a proposition containing numerical values; if the quantified value or state is an empty set, that is, the atomic proposition does not include specific numerical calculations, but only describes properties, relationships, or existence, then the atomic proposition type flag value is... This indicates a proposition without a numerical value, or a descriptive proposition.

[0064] Character offset range For example: .

[0065] Atomic Propositions .

[0066] In step S140, evidence is obtained from the subject database based on the set of atomic propositions to generate a propositional evidence association package. The propositional evidence association package includes each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition.

[0067] In some embodiments, the input object in step S140 includes: a set of atomic propositions. .

[0068] In some embodiments, the output object in step S140 includes: propositional evidence association package. Among them, the optimal set of evidence This includes propositions concerning atoms. The first retrieved 1 optimal piece of evidence, each optimal piece of evidence Includes: optimal evidence text context Optimal evidence structured data This refers to the corresponding JSON structured numerical data or structured tabular data, metadata. (e.g., unique identifiers of digital objects, source weights, paths, etc.).

[0069] Figure 4 This is a flowchart illustrating how, in one embodiment of the evidence-based evaluation method for the model output content of this application, evidence is obtained from a subject database based on a set of atomic propositions to generate a propositional evidence association package. In some embodiments, step S140, which involves obtaining evidence from a subject database based on a set of atomic propositions to generate a propositional evidence association package, includes steps S141 to S144.

[0070] In step S141, based on the set of atomic propositions, a recall is performed from the subject database to obtain a candidate evidence set, which includes several candidate pieces of evidence.

[0071] From subject databases A recall was conducted to obtain a candidate evidence set. To eliminate the difference in units, semantic similarity and keyword scores are normalized. The boundary is determined by the evidence recall score. Definition: ; in, , is the Min-Max normalization function. This refers to the semantic similarity between the text of an atomic proposition and the text in a subject database, such as cosine similarity. BM25 is an existing Best Matching 25 keyword retrieval ranking algorithm. It serves as the recall scoring weighting coefficient, controlling the balance between semantics and keyword retrieval.

[0072] Candidate evidence set It is obtained from the following relationship: ; in, This is the recall threshold.

[0073] In step S142, the cross-modal alignment score of each candidate piece of evidence is calculated based on the atomic proposition type marker value, scientific relation operator, quantization value or state of the atomic proposition.

[0074] against Numerical constraints and relationships Calculate evidence Alignment : like If Indicates propositions without numerical values, and aligns them. for: ; like If This indicates a proposition containing numerical values, with alignment. for: ; in, Candidate evidence The set of values ​​extracted from; This is a Boolean indicator function; if the value... Satisfy logical relationship (like If the value is 1, then the value is 1; otherwise, the value is 0. The local standard deviation of the field containing the candidate evidence (describing the dispersion of the data); This is a protection constant; For the constraint matching function, if Units or constraints Units or constraints related to candidate evidence If there is a conflict (such as with temperature), the score will drop significantly.

[0075] In step S143, the final comprehensive score of the candidate evidence is calculated based on the cross-modal alignment score.

[0076] In some embodiments, candidate evidence is calculated according to the following formula. Final overall score Ensure all portions are in The interval has a weight sum of 1: ; in, , , For the comprehensive score weighting parameters, satisfying ; The impact factor or authority score of the evidence source for candidate evidence; The base of the natural logarithm ensures ; This is the normalization function.

[0077] In step S144, the candidate evidence in the candidate evidence set is sorted in descending order according to the final comprehensive score. The top preset number of candidate evidence in the sorted set are taken and mapped from the original structured data of the subject database to obtain the corresponding optimal evidence and form the optimal evidence set.

[0078] In some embodiments, the candidate evidence set According to the final composite score, it decreased Sort in descending order, take the first few. One candidate piece of evidence: ; in, This is the preset minimum confidence threshold.

[0079] For each selected candidate piece of evidence By mapping raw structured data from subject databases, the corresponding optimal evidence can be obtained. : ; in, A function to map physical addresses from a subject database index back to the original structured data (JSON object); The impact factor or authority score of the evidence source for candidate evidence; This refers to the physical index or storage path of the candidate evidence in the subject database.

[0080] Propositional Evidence Related Package Among them, the optimal set of evidence This includes propositions concerning atoms. The first retrieved 1 optimal piece of evidence, each optimal piece of evidence Includes: optimal evidence text context Optimal evidence structured data This refers to the corresponding JSON structured numerical data or structured tabular data, metadata. (e.g., unique identifiers of digital objects, source weights, paths, etc.).

[0081] like Figure 1 In step S150, an evidence-based scoring package is generated based on the set of atomic propositions and the proposition evidence association package. The evidence-based scoring package includes the final fidelity score and evidence conflict entropy for each atomic proposition.

[0082] In some embodiments, the input object in step S150 includes: propositional evidence association package. Preset rigor coefficient .

[0083] In some embodiments, the output object in step S150 includes: an evidence-based scoring package. ,in, For the atomic proposition The final loyalty score, For the atomic proposition Evidence conflict entropy.

[0084] Figure 5 This is a flowchart illustrating the process of generating an evidence-based scoring package based on a set of atomic propositions and a propositional evidence association package, according to one embodiment of the evidence-based evaluation method for the model output content of this application. In some embodiments, step S150 of generating the evidence-based scoring package based on the set of atomic propositions and the propositional evidence association package includes steps S151 to S152.

[0085] In step S151, the degree of support for the corresponding atomic proposition is calculated for each optimal piece of evidence.

[0086] As mentioned earlier, each optimal piece of evidence Includes: optimal evidence text context Optimal evidence structured data That is, the corresponding JSON structured numerical data or structured tabular data, the best evidence metadata. (e.g., unique identifiers of digital objects, source weights, paths, etc.).

[0087] For each optimal piece of evidence Utilizing its optimal evidence text context and optimal evidence structured data Calculate its relation to atomic propositions level of support .

[0088] like If This represents propositions without numerical values, primarily relying on textual semantic logic to represent atomic propositions. The text description serves as a premise, and the best evidence is presented. Optimal Evidence Text Context As a hypothesis, the input is an external Natural Language Inference (NLI) model, and the support level is calculated. : ; The NLI model outputs a probability distribution with three dimensions: [Support, Neutral, Oppose], taking the normalized probability value of the "Support" dimension.

[0089] like If For propositions containing numerical values, the degree of support is defined when the semantics are consistent and the numerical logic is true. The calculation logic is as follows: ; in, For symbolic verification function, symbolic verification function The definition is as follows: ; in, To obtain the best evidence Optimal Evidence Structured Data The set of values ​​extracted from it. This is the third preset constant. To obtain the numerical set of the best evidence The value in Quantitative values ​​of atomic propositions As a scientific relation operator that satisfies atomic propositions The comparison. Explanation of the above formula: If the numerical set of the optimal evidence... There is at least one value in it. Its quantitative value relative to atomic propositions Scientific relation operators that satisfy atomic propositions Then the symbol verification function Take 1; if the scientific relation operator of the atomic proposition is 1. yes" That is, "approximate", then the symbol verification function If the numerical set of optimal evidence All values ​​in Quantitative values ​​of atomic propositions The relationship between these operators and the scientific relationship to the atomic proposition is crucial. If there is a contradiction, then the symbol verification function... Take 0.

[0090] In step S152, the contribution weight of each optimal piece of evidence under the corresponding atomic proposition is calculated.

[0091] To balance the original retrieval score and source authority of the best evidence, the contribution weight of each piece of best evidence under the current atomic proposition is calculated. The weight satisfies : ; in, This is the final comprehensive score for the best evidence; Metadata derived from the best evidence The authority constant.

[0092] In step S153, based on the degree of support and contribution weight, the weighted average support and evidence conflict entropy of all the best evidence under the atomic proposition are calculated.

[0093] The weighted average support of all the best evidence for the atomic proposition is calculated using the following formula. : ; Based on weighted average support Calculate the entropy of evidence conflict Used to quantify discrepancies between pieces of evidence: ; in, This is the fourth preset constant, which is a very small offset to prevent... .

[0094] Evidence Conflict Entropy When the weighted average support When (support and opposition each account for half), the evidence conflict entropy Reaching the maximum value This represents a highly controversial issue.

[0095] In step S154, the weighted average support is non-linearly penalized using the evidence conflict entropy to obtain the corresponding final fidelity score, thereby obtaining the evidence-based scoring package.

[0096] Using the entropy of evidence conflict Weighted average support A non-linear penalty is applied to obtain the final loyalty score. In some embodiments, a rigor coefficient is used. To control the intensity of punishment: ; When the evidence is highly consistent ( ), final loyalty score When there is a significant conflict in the evidence ( (Larger), final loyalty score It will be quickly pulled down.

[0097] Evidence-based scoring package ,in, For the atomic proposition The final loyalty score, For the atomic proposition Evidence conflict entropy.

[0098] like Figure 1 In step S160, a corresponding evidence-based report is generated based on the original text and the evidence-based scoring package.

[0099] Figure 6 This is a flowchart illustrating the generation of a corresponding evidence-based report based on the original text and evidence-based scoring package in one embodiment of the evidence-based evaluation method for the model output content of this application. As mentioned above, the atomic proposition set includes atomic propositions, atomic proposition type marker values ​​corresponding to the atomic propositions, and character offset ranges. In some embodiments, step S160, which generates the corresponding evidence-based report based on the original text and evidence-based scoring package, includes steps S161 to S167.

[0100] In step S161, the atomic proposition mapping metadata for each atomic proposition in the atomic proposition set is obtained. Atomic proposition mapping metadata .

[0101] In step S162, the optimal evidence path set is obtained based on the optimal evidence set in the propositional evidence association package. It originates from the optimal evidence metadata of the optimal evidence set. .

[0102] In step S163, the original text is divided into multiple semantic segments according to the character offset interval.

[0103] In step S164, the semantic segment is mapped. If the position range of the semantic segment is contained within one of the character offset ranges, the semantic segment is mapped to the atomic proposition corresponding to the character offset range, and the semantic segment is successfully anchored to the atomic proposition. Otherwise, the semantic segment is mapped to an empty set.

[0104] In some embodiments, the original text Based on character offset range Divided into semantic fragments .

[0105] In some embodiments, a mapping function is defined. This ensures that every generated sentence can find its corresponding original atomic proposition: ; in, The first in the evidence-based report A semantic fragment; semantic fragments Position coordinates within the entire evidence-based report.

[0106] In step S165, for semantic segments successfully anchored to atomic propositions, corresponding source tracing tags are generated based on the evidence-based scoring package and the optimal evidence path set.

[0107] For semantic fragments that are successfully anchored to atomic propositions Using evidence-based scoring packages and the optimal set of evidence paths Matching the corresponding traceability tags : ; Among them, the high-confidence evidence path library The construction logic is as follows: ; in, This is the minimum confidence threshold.

[0108] In step S166, the dispute indexing operator for the corresponding semantic segment is determined based on the evidence conflict entropy under the atomic proposition. The dispute indexing operator is used to identify whether there is scientific disagreement at the textual position of the semantic segment.

[0109] In some embodiments, the dispute indexing operator is determined by the following relationship: : ; in, This is the preset conflict threshold. Explanation of the above formula: When... hour, That is, determining that there is scientific disagreement at the textual location of the semantic segment indicates a significant divergence of opinions on the atomic proposition based on different pieces of evidence; otherwise, This means that there is no significant scientific disagreement regarding the textual position of the semantic segment, i.e., there is no dispute.

[0110] In step S167, an evidence-based report is obtained based on semantic fragments, source tags, and dispute indexing operators.

[0111] In one example, evidence-based reporting It is a document stream containing enhanced metadata tags, which consists of a sequence of triples: .

[0112] The evidence-based evaluation method for model output content according to embodiments of this application includes: obtaining the original text of the model answer output by a scientific foundation model; constructing a domain knowledge package based on data from a subject database, the domain knowledge package including an entity weight table, a relation matrix, and term classification information, the entity weight table including terms in the domain knowledge package and the term score for each term, and the relation matrix including the association strength value between every two terms; decomposing and processing the original text and the domain knowledge package to obtain a set of atomic propositions; obtaining evidence from the subject database based on the set of atomic propositions to generate a propositional evidence association package, the propositional evidence association package including each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition; generating an evidence-based scoring package based on the set of atomic propositions and the propositional evidence association package, the evidence-based scoring package including the final fidelity score and evidence conflict entropy for each atomic proposition; and generating a corresponding evidence-based report based on the original text and the evidence-based scoring package. The evidence-based scoring package includes evidence conflict entropy, which quantifies the controversy or disagreement between evidence, making it easier for users to perceive the degree of evidence controversy behind the conclusion.

[0113] The evidence-based evaluation method for model output content according to the above embodiments of this application solves the problem of imperceptible logical gaps between semantics and numerical values ​​in related technologies. An atomic proposition includes a subject, a scientific relation operator, an object, a quantified value or state, a unit, or a constraint. An atomic proposition set includes atomic propositions, atomic proposition type marker values, and character offset intervals. Evidence is obtained from a subject database based on the atomic proposition set, generating a propositional evidence association package, including: recalling candidate evidence from the subject database based on the atomic proposition set, the candidate evidence set including several candidate pieces of evidence; calculating the cross-modal alignment score for each candidate piece of evidence based on the atomic proposition type marker value, scientific relation operator, quantified value, or state; calculating the final comprehensive score of the candidate evidence based on the cross-modal alignment score; sorting the candidate evidence in the candidate evidence set in descending order of the final comprehensive score, taking the top preset number of candidate pieces of evidence from the sorted list, mapping them to the original structured data from the subject database to obtain the corresponding optimal evidence, and forming an optimal evidence set. Rigor verification is achieved through numerical alignment of the cross-modal alignment score. The above approach not only requires that the evidence be semantically consistent with the atomic proposition, but also requires that the structured data in the evidence mathematically satisfy the tuple constraints of the atomic proposition. If there is a logical inconsistency in the numerical values, the support level of the evidence will be greatly reduced.

[0114] The evidence-based evaluation method for the model output content according to the above embodiments of this application generates a corresponding evidence-based report based on the original text and the evidence-based scoring package. This includes: obtaining atomic proposition mapping metadata for each atomic proposition in the atomic proposition set; obtaining the optimal evidence path set based on the optimal evidence set in the proposition evidence association package; dividing the original text into multiple semantic segments based on character offset intervals; mapping the semantic segments, wherein if the position interval of a semantic segment is contained within one of the character offset intervals, the semantic segment is mapped to the atomic proposition corresponding to the character offset interval, confirming that the semantic segment is successfully anchored to the atomic proposition; otherwise, the semantic segment is mapped to an empty set; for semantic segments successfully anchored to atomic propositions, generating corresponding source tracing tags based on the evidence-based scoring package and the optimal evidence path set; determining the dispute indexing operator for the corresponding semantic segment based on the evidence conflict entropy under the atomic proposition; and obtaining the evidence-based report based on the semantic segment, source tracing tags, and dispute indexing operator. In the above scheme, through domain ontology and coordinate anchoring, users can know which piece of evidence a specific value or limiting condition in the final generated text originates from, achieving precise source tracing.

[0115] This application also provides an evidence-based evaluation device for model output content. Figure 7 This is a schematic diagram of one embodiment of the evidence-based evaluation device for model output content according to this application. The evidence-based evaluation device 100 for model output content includes a model answer receiving module 110, a domain knowledge package construction module 120, a proposition decomposition module 130, an evidence retrieval module 140, a quantitative evaluation module 150, and an evidence report generation module 160.

[0116] The model answer receiving module 110 communicates with the scientific foundation model 200. The domain knowledge package construction module 120 communicates with the subject database 300. The proposition decomposition module 130 communicates with the model answer receiving module 110, the domain knowledge package construction module 120, and the evidence-based report generation module 160. The evidence-based retrieval module 140 communicates with the proposition decomposition module 130, the subject database 300, and the evidence-based report generation module 160. The quantitative evaluation module 150 communicates with the evidence-based retrieval module 140 and the evidence-based report generation module 160. The evidence-based report generation module 160 communicates with the quantitative evaluation module 150, the evidence-based retrieval module 140, and the proposition decomposition module 130.

[0117] The model answer receiving module 110 is used to obtain the original text of the model answer output by the scientific pedestal model 200.

[0118] The domain knowledge package construction module 120 is used to construct a domain knowledge package based on the data in the subject database 300. The domain knowledge package includes an entity weight table, a relation matrix, and term classification information. The entity weight table includes terms in the domain knowledge package and the term score for each term. The relation matrix includes the association strength value between every two terms.

[0119] The proposition decomposition module 130 is used to decompose and process the original text and domain knowledge package to obtain a set of atomic propositions.

[0120] The evidence retrieval module 140 is used to obtain evidence from the subject database 300 based on the set of atomic propositions and generate a propositional evidence association package. The propositional evidence association package includes each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition.

[0121] The quantitative evaluation module 150 is used to generate an evidence-based scoring package based on the set of atomic propositions and the propositional evidence association package. The evidence-based scoring package includes the final fidelity score and evidence conflict entropy for each atomic proposition.

[0122] The evidence-based report generation module 160 is used to generate corresponding evidence-based reports based on the original text and the evidence-based scoring package.

[0123] According to the evidence-based evaluation device for the model output content of this application embodiment, the evidence retrieval module 140 is used to obtain evidence from the subject database 300 based on the atomic proposition set, and generate a propositional evidence association package. The propositional evidence association package includes each atomic proposition in the atomic proposition set and the optimal evidence set associated with each atomic proposition. The quantitative evaluation module 150 is used to generate an evidence-based scoring package based on the atomic proposition set and the propositional evidence association package. The evidence-based scoring package includes the final fidelity score and the evidence conflict entropy for each atomic proposition. The evidence conflict entropy includes evidence conflict entropy, which quantitatively expresses the controversy or disagreement between evidence, making it easier for users to perceive the degree of evidence controversy behind the conclusion.

[0124] This application also provides an electronic device. Figure 8 This is a schematic diagram of the hardware structure of an embodiment of the electronic device according to this application. The electronic device includes a memory 910 and a processor 920. The memory 910 and the processor 920 are communicatively connected. The memory 910 stores instructions, and the processor 920 calls the instructions in the memory 910 to cause the electronic device to execute an evidence-based evaluation method for the model output content according to any of the foregoing embodiments of this application.

[0125] The evidence-based evaluation method for the model output includes: obtaining the original text of the model's answers output by the scientific foundation model; constructing a domain knowledge package based on data from a subject database, the domain knowledge package including an entity weight table, a relation matrix, and term classification information; the entity weight table including terms in the domain knowledge package and the term score for each term, and the relation matrix including the association strength value between every two terms; decomposing and processing the original text and the domain knowledge package to obtain a set of atomic propositions; obtaining evidence from the subject database based on the set of atomic propositions to generate a proposition evidence association package, which includes each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition; generating an evidence-based scoring package based on the set of atomic propositions and the proposition evidence association package, which includes the final fidelity score and evidence conflict entropy for each atomic proposition; and generating a corresponding evidence-based report based on the original text and the evidence-based scoring package.

[0126] Specifically, the processor 920 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0127] Memory 910 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 910 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 910 may include removable or non-removable (or fixed) media. Where appropriate, memory 910 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 910 may be the non-volatile memory described above. In a particular embodiment, memory 910 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0128] In one example, the electronic device may also include a communication interface 930 and a bus 940. The processor 920, memory 910, and communication interface 930 are connected via the bus 940 and communicate with each other.

[0129] The communication interface 930 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0130] Bus 940 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 940 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0131] Furthermore, in conjunction with the evidence-based evaluation method for model output content in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores instructions that, when executed by a processor, implement the evidence-based evaluation method for model output content in any of the foregoing embodiments of this application.

[0132] The evidence-based evaluation method for the model output includes: obtaining the original text of the model's answers output by the scientific foundation model; constructing a domain knowledge package based on data from a subject database, the domain knowledge package including an entity weight table, a relation matrix, and term classification information; the entity weight table including terms in the domain knowledge package and the term score for each term, and the relation matrix including the association strength value between every two terms; decomposing and processing the original text and the domain knowledge package to obtain a set of atomic propositions; obtaining evidence from the subject database based on the set of atomic propositions to generate a proposition evidence association package, which includes each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition; generating an evidence-based scoring package based on the set of atomic propositions and the proposition evidence association package, which includes the final fidelity score and evidence conflict entropy for each atomic proposition; and generating a corresponding evidence-based report based on the original text and the evidence-based scoring package.

[0133] This application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0134] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0135] It should be noted that the technical solutions or features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; all modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An evidence-based evaluation method for model output content, characterized in that, include: Obtain the raw text of the model's answer output by the scientific foundation model; Based on data from a subject database, a domain knowledge package is constructed. The domain knowledge package includes an entity weight table, a relation matrix, and term classification information. The entity weight table includes terms in the domain knowledge package and a term score for each term. The relation matrix includes the association strength value between every two terms. The original text and the domain knowledge package are decomposed and processed to obtain a set of atomic propositions, and each atomic proposition is defined as a semantic tuple. ,in, As the main body, For scientific relation operators, As an object, To quantify values ​​or states, As a unit or constraint, the set of atomic propositions includes the atomic proposition, the atomic proposition type marker value corresponding to the atomic proposition, and the character offset range; Evidence is obtained from the subject database based on the set of atomic propositions, and a propositional evidence association package is generated. The propositional evidence association package includes each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition. Based on the set of atomic propositions and the proposition evidence association package, an evidence-based scoring package is generated, which includes the final fidelity score and evidence conflict entropy for each atomic proposition. Based on the original text and the evidence-based scoring package, a corresponding evidence-based report is generated; The step of obtaining evidence from the subject database based on the atomic proposition set and generating a proposition evidence association package includes: Based on the set of atomic propositions, a recall is performed from the subject database to obtain a candidate evidence set, which includes several candidate pieces of evidence. Calculate the cross-modal alignment score for each candidate piece of evidence based on the atomic proposition type marker value, the scientific relation operator, the quantization value or state; The final comprehensive score of the candidate evidence is calculated based on the cross-modal alignment score; The candidate evidence in the candidate evidence set is sorted in descending order according to the final comprehensive score. The top preset number of candidate evidence in the sorted set are then mapped from the subject database to the original structured data to obtain the corresponding optimal evidence and form the optimal evidence set.

2. The evidence-based evaluation method for the model output content according to claim 1, characterized in that, The domain knowledge package, constructed from the data in the subject database, includes: When the domain distribution ratio of candidate terms in the subject database is greater than a first threshold, the candidate terms are determined to be terms included in the domain knowledge package, and the term score of the candidate terms is calculated. An entity weight table is generated based on all the terms identified as included in the domain knowledge package and the corresponding term scores. Calculate the association strength value between every two terms in the entity weight table to generate a relationship matrix.

3. The evidence-based evaluation method for the model output content according to claim 2, characterized in that, The calculation of the term score for the candidate terms includes: Calculate the terminology degree, freedom degree, and cohesion degree of the candidate terms; The terminology degree, degree of freedom, and degree of cohesion of the candidate terms are normalized and weighted and fused to obtain the term score of the candidate terms.

4. The evidence-based evaluation method for the model output content according to claim 3, characterized in that, The calculation of the terminology degree, freedom degree, and cohesion degree of the candidate terms includes: The terminology degree of the candidate terms is calculated using the following formula: ； Where 'a' represents a candidate term. The terminology level of the candidate terms. The phrase length of the candidate term. The frequency of occurrence of the candidate terms. For a longer set of phrases containing the candidate terms; The right neighbor information entropy and left neighbor information entropy of the candidate terms are calculated using the following formula: ； ； in, The right neighbor information entropy of the candidate term. The left-neighbor information entropy of the candidate term. For the word elements that appear after the candidate word, In order to follow up on the candidate terms when they appear The conditional probability; The set of left-neighbor words of the candidate term in the context of the subject database. The set of right-neighbor words of the candidate term in the context of the subject database; The degrees of freedom of the candidate terms are calculated using the following formula: ； in, The degrees of freedom of the candidate terms; The cohesion of the candidate terms is calculated using the following formula: ； in, The cohesion of the candidate terms. The probability of the candidate term appearing. For individual words that make up a candidate term Independent probability of occurrence.

5. The evidence-based evaluation method for the model output content according to claim 2, characterized in that, The step of calculating the association strength value between every two terms in the entity weight table and generating a relationship matrix includes: The association strength value between every two terms in the entity weight table is calculated using the following formula: ； in, For the entry Related to the entry The correlation strength value between them For a collection of domain texts, For the entry The probability of appearing in the domain text For the entry The probability of appearing in the domain text For the entry Related to the entry The probability of them appearing together in the domain text.

6. The evidence-based evaluation method for the model output content according to claim 1, characterized in that, The term classification information of the domain knowledge package includes entities, operators, attributes, or constraints; The process of disassembling the original text and the domain knowledge package yields a set of atomic propositions, including: A semantic graph is generated based on the original text and the domain knowledge package; An operator set is constructed from the terms whose term classification information is an operator in the domain knowledge package, and the core predicate node is located in the semantic graph through the operator set; Using the core predicate node as the central source point, a subgraph search is performed in the semantic graph to obtain the minimum connected subgraph containing the core triple and having the minimum total weight loss, and the character offset interval corresponding to the minimum connected subgraph is obtained. The core triple includes the subject node, the predicate node, and the object node. In the minimum connected subgraph, the subject node, predicate node, object node, quantized value or state node, and unit or constraint node are extracted and used as the subject, scientific relation operator, object, quantized value or state, and unit or constraint of the atomic proposition, respectively. The atomic propositions are type-labeled to obtain atomic proposition type label values. A set of atomic propositions is obtained based on the atomic propositions, the atomic proposition type label values, and the character offset intervals.

7. The evidence-based evaluation method for the model output content according to claim 1, characterized in that, The step of generating an evidence-based scoring package based on the set of atomic propositions and the propositional evidence association package includes: Calculate the degree of support for the corresponding atomic proposition for each of the best pieces of evidence; Calculate the contribution weight of each of the optimal pieces of evidence under the corresponding atomic proposition; Based on the support level and the contribution weight, calculate the weighted average support and evidence conflict entropy of all the best evidence under the atomic proposition; The evidence conflict entropy is used to apply a non-linear penalty to the weighted average support to obtain the corresponding final fidelity score, thereby obtaining the evidence-based scoring package.

8. The evidence-based evaluation method for the model output content according to claim 7, characterized in that, The step of generating a corresponding evidence-based report based on the original text and the evidence-based scoring package includes: Obtain the atomic proposition mapping metadata for each atomic proposition in the set of atomic propositions; Based on the optimal evidence set in the propositional evidence association package, the optimal evidence path set is obtained; The original text is divided into multiple semantic segments based on the character offset interval; The semantic segment is mapped, wherein if the position range of the semantic segment is contained within one of the character offset ranges, the semantic segment is mapped to the atomic proposition corresponding to the character offset range, and the semantic segment is successfully anchored to the atomic proposition; otherwise, the semantic segment is mapped to an empty set. For the semantic fragment successfully anchored to the atomic proposition, a corresponding source tracing label is generated based on the evidence-based scoring package and the optimal evidence path set; Based on the evidence conflict entropy under the atomic proposition, determine the dispute indexing operator corresponding to the semantic segment; An evidence-based report is obtained based on the semantic fragment, the source tag, and the dispute indexing operator.

9. An evidence-based evaluation device for model output content, characterized in that, include: The model answer receiving module is used to obtain the raw text of the model answer output by the scientific foundation model; The domain knowledge package construction module is used to construct a domain knowledge package based on data from a subject database. The domain knowledge package includes an entity weight table, a relation matrix, and term classification information. The entity weight table includes terms in the domain knowledge package and a term score for each term. The relation matrix includes the association strength value between every two terms. The proposition decomposition module is used to decompose the original text and the domain knowledge package to obtain a set of atomic propositions, whereby each atomic proposition is defined as a semantic multi-dimensional group. ,in, As the main body, For scientific relation operators, As an object, To quantify values ​​or states, As a unit or constraint, the set of atomic propositions includes the atomic proposition, the atomic proposition type marker value corresponding to the atomic proposition, and the character offset range; The evidence-based retrieval module is used to obtain evidence from the subject database based on the set of atomic propositions and generate a propositional evidence association package, wherein the propositional evidence association package includes each atomic proposition in the set of atomic propositions and the optimal set of evidence associated with each atomic proposition; The quantitative evaluation module is used to generate an evidence-based scoring package based on the set of atomic propositions and the proposition evidence association package. The evidence-based scoring package includes the final fidelity score and evidence conflict entropy for each atomic proposition. The evidence-based report generation module is used to generate a corresponding evidence-based report based on the original text and the evidence-based scoring package. The step of obtaining evidence from the subject database based on the atomic proposition set and generating a proposition evidence association package includes: Based on the set of atomic propositions, a recall is performed from the subject database to obtain a candidate evidence set, which includes several candidate pieces of evidence. Calculate the cross-modal alignment score for each candidate piece of evidence based on the atomic proposition type marker value, the scientific relation operator, the quantization value, or the state of the atomic proposition; The final comprehensive score of the candidate evidence is calculated based on the cross-modal alignment score; The candidate evidence in the candidate evidence set is sorted in descending order according to the final comprehensive score. The top preset number of candidate evidence in the sorted set are then mapped from the subject database to the original structured data to obtain the corresponding optimal evidence and form the optimal evidence set.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being communicatively connected to the processor, and the memory storing instructions. The processor invokes the instructions in the memory to cause the electronic device to execute the evidence-based evaluation method for the model output content according to any one of claims 1 to 8.

11. A computer-readable storage medium storing instructions thereon, characterized in that, When the instruction is executed by the processor, it implements the evidence-based evaluation method for the model output content according to any one of claims 1 to 8.

Images