Large language model generation text evaluation method and device, equipment and storage medium

Abstract

The invention discloses a large language model generation text evaluation method and device, equipment and a storage medium. Comprising the following steps: performing statement splitting on a large language model generation text to obtain a plurality of text statements; extracting subjects of the text statements; knowledge nodes matched with the subjects are extracted from a preset graph database, the preset graph database is constructed based on professional knowledge corpora of the knowledge field corresponding to the large language model generation text, the preset graph database comprises at least one knowledge node, and each knowledge node comprises at least one piece of knowledge information; obtaining knowledge information matched with the corresponding text statement in each matched knowledge node; inputting each text statement and the corresponding matched knowledge information into a preset evaluation model to obtain a statement evaluation result of each text statement; and calculating an overall evaluation result of the large language model generation text based on the statement evaluation result of each text statement. By means of the mode, efficient and accurate evaluation on the large language model generation text can be achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for evaluating text generated by a large language model. Background Technology

[0002] With the development of artificial intelligence technology, Large Language Models (LLMs) have been widely used in natural language processing, text generation, and question answering. However, when generating text, they may have problems such as factual errors, logical contradictions, and semantic ambiguities, making it difficult to ensure factual consistency and logical rigor. There is an urgent need for efficient and accurate evaluation of the generated results to identify potential errors in a timely manner and reduce decision-making risks.

[0003] Currently, the evaluation methods for text generated by large language models mainly include manual inspection methods, rule-based judgment methods, calculation methods based on automatic evaluation metrics, and evaluation methods based on pre-trained models. Among these methods, manual inspection involves professionals reading text content generated by large language models in the relevant field and issuing quality assessment reports. However, this method suffers from high professional requirements, high talent costs, low evaluation efficiency, and strong subjectivity, resulting in inconsistent evaluation results and making it unsuitable for large-scale text evaluation. Rule-based judgment methods verify text compliance according to preset grammatical rules or keyword matching logic. However, this method relies on manually defined rules, leading to insufficient coverage of complex semantic scenarios and low evaluation accuracy. Methods based on automatic evaluation metrics quantify text similarity using metrics such as bilingual evaluation substitutes, summary-oriented recall-oriented evaluation, or explicit ranking translation evaluation metrics to assess the surface-level matching degree between the generated and reference texts. However, this approach has limitations in assessing deep semantic logic and factual consistency, making it difficult to reflect the true semantic quality and factual accuracy of the text. Evaluation methods based on pre-trained models utilize pre-trained language models and their automatic evaluation metrics to calculate the semantic similarity between the generated and reference texts, achieving automated text quality assessment. However, because pre-trained models rely on large-scale labeled data and have limited adaptability to domain knowledge, they struggle to accurately detect factual errors in specific domains.

[0004] Therefore, existing evaluation methods are unable to efficiently and accurately achieve a comprehensive assessment of the factual consistency and logical rigor of text generated by large language models. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for evaluating text generated by large language models, aiming to solve the technical problem of poor evaluation accuracy of existing methods for text generated by large language models.

[0006] The first aspect of this invention provides a method for evaluating text generated by a large language model. The method includes: splitting the text generated by the large language model into multiple text sentences; extracting the subject of each text sentence; extracting knowledge nodes from a preset graph database that match each subject; obtaining knowledge information from each matched knowledge node that matches the corresponding text sentence; inputting each text sentence and its corresponding matched knowledge information into a preset evaluation model to obtain a sentence evaluation result for each text sentence; and calculating an overall evaluation result for the text generated by the large language model based on the sentence evaluation results for each text sentence. The preset graph database is constructed based on professional knowledge corpus corresponding to the knowledge domain of the text generated by the large language model. The preset graph database includes at least one knowledge node, which serves as a carrier of entities in the preset graph database. Each knowledge node corresponds one-to-one with an entity, and each knowledge node includes at least one piece of knowledge information.

[0007] Optionally, in a first implementation of the first aspect of the present invention, the construction step of the preset graph database includes: acquiring the professional knowledge corpus corresponding to the knowledge domain of the text generated by the large language model; extracting entities from the professional knowledge corpus, and the relations and attributes corresponding to each entity; mapping each entity to a knowledge node, mapping each relation to an edge of the corresponding knowledge node, and mapping each attribute to the attribute information of the corresponding knowledge node, to obtain a knowledge graph structure; based on the knowledge graph structure, determining at least one subject-predicate-object triple corresponding to each knowledge node, and using each triple as the knowledge information of the corresponding knowledge node; and storing the triple in a structured manner to obtain the preset graph database.

[0008] Optionally, in a second implementation of the first aspect of the present invention, the step of extracting the subject of each of the text statements includes: performing part-of-speech tagging on the text statements to identify a first candidate subject in the text statements that conforms to the part of speech of the subject; performing syntactic analysis on the text statements to generate a syntactic tree structure of the text statements, and determining a second candidate subject of the text statements based on the syntactic tree structure; and selecting the subject of the text statements from the first candidate subject and the second candidate subject based on preset grammar rules.

[0009] Optionally, in a third implementation of the first aspect of the present invention, the step of extracting knowledge nodes from a preset graph database that match each of the subjects includes: performing format standardization processing on the entities of the subjects and each knowledge node in the preset graph database; calculating the matching degree between the subjects and the entities of the knowledge nodes in the preset graph database, and determining whether there are any entities in the preset graph database that match the subject object described by the subject; if there are entities in the preset graph database that match the subject object described by the subject, then the corresponding knowledge node is taken as the matched knowledge node; if there are no entities in the preset graph database that match the subject object described by the subject, then the similarity between the subject object described by the subject and the entities of each knowledge node is calculated, and the knowledge node corresponding to the entity with the highest similarity is taken as the matched knowledge node.

[0010] Optionally, in a fourth implementation of the first aspect of the present invention, obtaining the knowledge information in each of the matched knowledge nodes that matches the corresponding text statement includes: converting the text statement and each knowledge information in the matched knowledge nodes into vector form respectively; calculating the vector distance between each knowledge information in vector form and the corresponding text statement in vector form; and selecting the first preset number of knowledge information in the matched knowledge nodes that has the smallest vector distance to the corresponding text statement as the matched knowledge information.

[0011] Optionally, in a fifth implementation of the first aspect of the present invention, the step of splitting the large language model-generated text into multiple text sentences includes: detecting natural delimiters in the large language model-generated text; and splitting the large language model-generated text into multiple text sentences based on the natural delimiters.

[0012] Optionally, in a sixth implementation of the first aspect of the present invention, the large language model generates text based on a first preset large language model; the preset evaluation model is constructed based on a second preset large language model; the first preset large language model and the second preset large language model are different models; the sentence evaluation result is the sentence correctness parameter output by the preset evaluation model, and the overall evaluation result is the accuracy of the text generated by the large language model.

[0013] A second aspect of the present invention provides a large language model-generated text evaluation device, comprising: a text splitting module for splitting the large language model-generated text into multiple text sentences; a subject extraction module for extracting the subject of each text sentence; a node matching module for extracting knowledge nodes from a preset graph database that match each subject; an information acquisition module for acquiring knowledge information from each matched knowledge node that matches the corresponding text sentence; a sentence evaluation module for inputting each text sentence and the corresponding matched knowledge information into a preset evaluation model to obtain a sentence evaluation result for each text sentence; and a text evaluation module for calculating an overall evaluation result of the large language model-generated text based on the sentence evaluation results of each text sentence; wherein the preset graph database is constructed based on professional knowledge corpus of the knowledge domain corresponding to the large language model-generated text, the preset graph database includes at least one knowledge node, the knowledge node is the carrier of entities in the preset graph database, the knowledge node corresponds one-to-one with the entity, and each knowledge node includes at least one knowledge information.

[0014] A third aspect of the present invention also provides a computer device, the computer device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the computer device to perform the large language model generating text evaluation method as described above.

[0015] A fourth aspect of the present invention also provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the large language model-generated text evaluation method as described above.

[0016] This invention provides a method, apparatus, device, and storage medium for evaluating text generated by a large language model. First, the text generated by the large language model is segmented into multiple text sentences. Each text sentence can be used as an independent evaluation unit, enabling refined and localized evaluation, which facilitates more accurate discovery and location of local errors in the generated text. Then, by extracting the subject of each text sentence and extracting knowledge nodes matching each subject from a preset graph database, knowledge information matching the corresponding text sentence can be obtained from each matching knowledge node. This allows for precise extraction of knowledge information related to the text sentence. Furthermore, leveraging the structured storage and efficient retrieval characteristics of the graph database, key knowledge can be quickly located, ensuring the accuracy, authority, and applicability of the knowledge, thus providing strong support for the evaluation of text sentences. Finally, by inputting each text statement and its corresponding matching knowledge information into a preset evaluation model, the evaluation results of each text statement are obtained. Based on the evaluation results of each text statement, the overall evaluation result of the text generated by the large language model is calculated. This enables a comprehensive evaluation from the local to the overall level. The results obtained in this way can objectively reflect the overall quality and local problems of the generated text, achieving efficient and accurate evaluation of the text generated by the large language model. It can effectively assess the factual consistency and logical rigor of the text generated by the large language model. Moreover, the evaluation process is highly automated, reducing human intervention, and can quickly adapt to different domain knowledge systems. It has good generalization ability and is suitable for large-scale high-accuracy evaluation needs, which is conducive to promoting the large-scale application of large language models in high-risk scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating one embodiment of the text evaluation method generated by a large language model in this invention. Figure 2 for Figure 1 Step 101 in the embodiment is a flowchart of an embodiment; Figure 3 for Figure 1 Step 102 in the embodiment is a flowchart of an embodiment; Figure 4 for Figure 1 Step 103 in the embodiment is a flowchart of an embodiment; Figure 5 for Figure 1 A flowchart of a previous embodiment preceding step 103 in this example; Figure 6 for Figure 1 Step 104 in the embodiment is a flowchart of an embodiment; Figure 7 This is a schematic diagram of the functional modules of an embodiment of the large language model-generated text evaluation device of the present invention; Figure 8 This is a schematic diagram of the functional modules of a computer device according to one embodiment of the present invention. Detailed Implementation

[0018] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] For ease of understanding, the specific process of the large language model-generated text evaluation method in this embodiment of the invention is described below. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the large language model-generated text evaluation method of the present invention. In this embodiment, the large language model-generated text evaluation method includes: 101. Segment the text generated by the large language model to obtain multiple text sentences; In this embodiment, the large language model-generated text is coherent and logical text content automatically generated by a large-scale pre-trained language model by analyzing input prompts and predicting subsequent word sequences. This large language model-generated text is typically structured text presented in paragraphs or bullet points, usually including a series of text statements expressing independent semantics, used to describe the attributes of a single entity, or to describe the relationships between multiple entities.

[0020] In this embodiment, since text generated by large language models is typically lengthy and semantically complex, it is difficult to directly treat it as a whole for semantic understanding, association matching, and content localization. Therefore, to make the text generated by large language models more structured and parsable, this invention splits the text generated by large language models into multiple text sentences, each serving as an independent evaluation unit. Specifically, the splitting of the text generated by large language models can be achieved through natural delimiter segmentation, sentence boundary recognition based on grammatical rules, or other text splitting tools. The text sentences obtained through this splitting process will participate as independent units in subsequent subject extraction, knowledge matching, and indicator evaluation, improving the granularity and accuracy of the evaluation and facilitating the accurate identification of local problems in the generated text.

[0021] Optionally, see Figure 2 In some embodiments, the text generated by the large language model is split into multiple text sentences, which can be performed as follows: 1011. Detect natural segmentation characters in text generated by large language models; 1012. Based on natural segmentation, split the text generated by a large language model into multiple text sentences.

[0022] In this optional embodiment, a method for segmenting text generated by a large language model is specifically described. Natural delimiters refer to symbols used to identify sentence boundaries in natural language text, such as periods, question marks, exclamation marks, paragraph numbers, and bullet points. These natural delimiters typically possess clear semantic pause characteristics, and their appearance usually indicates the end of a sentence or a logical separation. Based on their boundary marking function, text generated by a large language model can be segmented into multiple text sentences for use in subsequent evaluation stages such as subject extraction and knowledge matching.

[0023] 102. Extract the subject of each text sentence; In this embodiment, the subject is the main component in a text statement that carries the core semantics, i.e., the executor of the action described by the statement, the object being described, or the core discussion entity. For example, in the text statement "large language model generates text," the executor of the action, the object being described, and the core discussion entity are all "large language model," and therefore it can be used as the subject of this text statement. By extracting the subject, the focus of each text statement can be clarified, and the subject can serve as the core identifier for knowledge matching in subsequent processing steps. It can also quickly index the corresponding graph database entity nodes for accurate knowledge association and relationship verification. Specifically, subject extraction can be achieved through methods such as part-of-speech tagging, dependency parsing, or named entity recognition models. At the same time, during the subject extraction process, the referential relationship of compound subjects or implicit subjects can be clarified through the referential resolution technique. Furthermore, after the subject is extracted, the reasonableness of the extraction result can be verified, such as by performing a preliminary match with known subject texts in the subject database to filter out invalid or incorrect subjects.

[0024] Optionally, see Figure 3 In some embodiments, extracting the subject of each text statement can be performed as follows: 1021. Perform part-of-speech tagging on the text sentences to identify the first candidate subject in the text sentences that matches the part of speech of the subject; 1022. Perform syntactic analysis on the text statement to generate the syntactic tree structure of the text statement, and determine the second candidate subject of the text statement based on the syntactic tree structure; 1023. Based on preset grammar rules, select the subject of the text statement from the first candidate subject and the second candidate subject.

[0025] This optional embodiment specifically describes a fusion method for extracting subjects from text sentences through part-of-speech tagging, syntactic analysis, and preset grammatical rules. Part-of-speech tagging (POS tagging) is a text data processing method that assigns a grammatical part-of-speech label to each word in the text to identify the grammatical role of words in a sentence. Specifically, it can use POS tagging tools to tag each word in the text sentence to identify words with noun characteristics, i.e., nouns, pronouns, or noun phrases, which are usually used as subjects, as the first candidate subjects for subsequent screening. Syntactic parsing is a natural language processing technique that performs in-depth analysis of the grammatical structure of sentences to reveal the hierarchical relationships and syntactic functions between words. Specifically, it can use a syntactic parser to generate a syntactic tree (Dependency Parse) of the sentence. The system uses a tree structure to locate core noun phrases in the subject-predicate structure, extracting syntactic subject candidates to obtain a second candidate subject. Predefined grammatical rules are a set of rules used to filter and verify candidate subjects. These rules can be set according to the characteristics of the subject. For example, based on the characteristic that the subject is usually a noun phrase and a core child node under the root node of the subject-predicate structure in the syntactic tree, the final subject can be selected from the first and second candidate subjects, choosing a phrase that simultaneously satisfies part-of-speech and syntactic conditions. Since a text statement has only one subject or multiple parallel subjects, if multiple candidate subjects or inconsistent candidate results exist during the subject selection process, conflict resolution can be achieved through predefined priority rules, such as setting syntactic analysis results to take precedence over part-of-speech tagging results, ensuring the uniqueness and accuracy of the final subject. The subject of the text statement determined through the above method can be verified by combining part-of-speech, syntax, and rules, ensuring the accuracy and robustness of subject extraction and facilitating accurate matching with knowledge entities in the graph database.

[0026] 103. Extract knowledge nodes from the preset graph database that match each subject; wherein, the preset graph database is constructed based on the professional knowledge corpus of the knowledge domain corresponding to the text generated by the large language model. The preset graph database includes at least one knowledge node. The knowledge node is the carrier of the entity in the preset graph database. The knowledge node corresponds one-to-one with the entity. Each knowledge node includes at least one knowledge information. In this embodiment, the graph database is a data management system designed for efficient storage and retrieval of graph data, using nodes and edges as basic storage units. It achieves efficient knowledge association retrieval and semantic reasoning through a topological structure where nodes represent entities and edges represent relationships between entities. The pre-defined graph database in this invention is a domain knowledge benchmark library pre-built based on professional knowledge corpora corresponding to the knowledge domains generated by the large language model. It provides a reliable knowledge reference for evaluating the factual and logical validity of the text generated by the large language model. Specifically, the professional knowledge corpus can be high-quality knowledge sources that have undergone rigorous screening, such as professional literature, industry standards, and authoritative databases, possessing high authority and accuracy. This professional knowledge corpus covers knowledge information such as core concepts, entity attributes, relationships, and factual descriptions within the domain. By extracting entities, identifying relationships, and extracting attributes from this knowledge information, and performing structured mapping according to the structural characteristics of the graph database, knowledge nodes with clear semantic boundaries can be obtained. Each knowledge node corresponds one-to-one with an entity. As the carrier of entities, these knowledge nodes can represent the core entities of the domain knowledge and their inherent logical relationships, providing verifiable factual evidence. By using these knowledge nodes as the basic storage units in the graph database, the corresponding graph database can have efficient entity association query capabilities, thus constructing the preset graph database, which includes at least one knowledge node. Each knowledge node contains specific knowledge information such as the definition, characteristics, and relationships of entities.

[0027] In this embodiment, by using rule-based pattern matching extraction, semantic embedding-based vector matching, and graph structure-based neighborhood traversal, knowledge nodes matching each subject can be extracted from the pre-built preset graph database. This enables precise binding of the subject in the text statement with structured knowledge entities, thereby establishing a semantic association between the text and domain knowledge. Subsequently, the authoritative knowledge information contained in the determined knowledge node can be used to verify the factual consistency of the text statement, thus achieving a reliability evaluation of the text generated by the large language model.

[0028] Optionally, see Figure 4 In some embodiments, extracting knowledge nodes from a preset graph database that match each subject can be performed as follows: 1031. Standardize the format of entities in the subject and each knowledge node in the preset graph database; 1032. Calculate the matching degree between the subject and the entities of the knowledge nodes in the preset graph database, and determine whether there are entities of the knowledge nodes in the preset graph database that match the main object described by the subject. 1033. If an entity with a knowledge node in the preset graph database matches the subject object described by the subject, then the corresponding knowledge node is taken as the matching knowledge node. 1034. If there is no entity in the preset graph database that matches the subject object described by the subject, then calculate the similarity between the subject object described by the subject and the entities of each knowledge node, and take the knowledge node corresponding to the entity with the highest similarity as the matched knowledge node.

[0029] In this optional embodiment, a method is provided for extracting knowledge nodes that match the subject from a preset graph database. To achieve accurate matching between the subject and the entities of each knowledge node, the entities of the subject and knowledge nodes can first undergo standardization processing to unify their format. For example, this can be done by removing stop words to purify the text, replacing synonyms with standardized terminology, or mapping abbreviations to full names to a unified expression format. This identifies the main object described by the subject, i.e., the entity the subject points to or variants of the entity with the same semantic meaning (such as different translations, abbreviations, or spelling variations of the entity). After standardization, a preliminary matching degree calculation is performed to determine if there are any completely identical knowledge nodes. This matching degree calculation can be based on character comparison covering common entity variants, which can quickly locate nodes that match the main object described by the subject, reducing the computational load of subsequent similarity calculations and improving matching efficiency. If the entity of a knowledge node matches the main object described by the subject, it means that the two are semantically identical, and this node can be directly used as the matching result to ensure fast and accurate matching. If the entity of the subject and the knowledge node do not match the main object described by the subject, it indicates a difference in expression. In this case, cross-expression matching needs to be achieved through semantic similarity calculation to quantify the degree of semantic association. Specifically, this similarity calculation can be performed using methods such as semantic embedding, cosine similarity calculation, or Euclidean distance. The knowledge node with the highest similarity is then selected as the matching knowledge node, thus achieving cross-expression semantic matching. This method of extracting knowledge nodes that match the subject balances matching efficiency and semantic coverage, achieving accurate extraction of knowledge nodes.

[0030] Optionally, see Figure 5 In some embodiments, the construction steps of the preset graph database include: 1001. Obtain professional knowledge corpus corresponding to the knowledge domain of the text generated by the large language model. 1002. Extract entities from the professional knowledge corpus, as well as the corresponding relationships and attributes of each entity. 1003. Map each entity to a knowledge node, each relation to an edge of the corresponding knowledge node, and each attribute to the attribute information of the corresponding knowledge node to obtain the knowledge graph structure. 1004. Based on the knowledge graph structure, determine at least one subject-predicate-object triple corresponding to each knowledge node, and use each triple as the knowledge information of the corresponding knowledge node; 1005. Structure and store each triplet to obtain a preset graph database.

[0031] This optional embodiment specifically describes the processing steps for constructing a preset graph database. Since this preset graph database is constructed from professional knowledge corpora, the professional knowledge corpora can be acquired before construction. Specifically, they can be collected from academic literature, industry standards, or authoritative databases, and high-quality corpora can be accurately acquired through methods such as manual screening, automated crawling, or API calls. These acquired professional knowledge corpora typically contain a large number of core entities in the domain. These entities can represent the core concepts and logical relationships of domain knowledge, and therefore can be used for knowledge graph structure construction through entity extraction, relationship identification, and attribute extraction. Specifically, an entity can be a concrete concept, proper noun, or abstract entity used to refer to core elements within the domain, while its corresponding relationship refers to the connection between the entity and other entities, and its corresponding attribute refers to the description of the characteristics possessed by the entity itself.

[0032] In this optional embodiment, once the entities and their corresponding relationships and attributes are determined, a structured mapping can be performed to obtain the corresponding knowledge graph structure. In this graph structure, entities are mapped to knowledge nodes, which serve as the basic units and carriers of the corresponding entities. Relationships between entities are mapped to connections between nodes, acting as directed edges representing semantic associations between entities, enabling logical connections and path reasoning. Attributes corresponding to entities are mapped to their respective nodes; these attributes do not participate in the graph's topological connections but serve as descriptive information for the nodes, supplementing the entity's feature details and quantitative indicators. After defining each entity as a knowledge node and associating relationships and attributes, the corresponding knowledge graph structure can be determined according to the graph database's topological requirements. This knowledge graph structure is organized in a node-edge-attribute manner, reflecting the inherent logical connections and hierarchical structure of domain knowledge, facilitating subsequent knowledge retrieval and semantic reasoning, and achieving structured knowledge storage and efficient relational querying.

[0033] In this optional embodiment, the knowledge information in the pre-defined graph database to be constructed is specifically stored in the form of subject-predicate-object (SPO) triplets. The subject-predicate-object structure is a standardized semantic representation structure. The subject refers to the executor of the action or the core object of discussion; the predicate refers to the semantic relationship between the subject and the object; and the object is the recipient of the action or the object being described. This structure can clearly express semantic logic, presenting knowledge units in a concise triplet form. This allows the triplets to achieve atomic representation of knowledge, facilitating machine understanding and computation. Specifically, these triplets can be obtained by traversing the relationships and attributes between nodes based on the node and edge information in the aforementioned knowledge graph structure, thus achieving the structuring and computability of knowledge. The subject-predicate-object triplets can accurately describe the semantic relationships between entities, providing standardized units for subsequent knowledge matching and fact verification, improving the accuracy and interpretability of the evaluation. Meanwhile, this end-to-end processing flow from corpus to graph database can ensure the authority and accuracy of knowledge, realize the structured accumulation of domain knowledge, and the preset graph database obtained through this processing flow can provide a reliable knowledge benchmark for the evaluation of text generated by large language models, provide an objective reference for fact consistency verification and logical rationality analysis, and effectively support the automated evaluation of text generated by large language models.

[0034] 104. Obtain the knowledge information that matches the corresponding text statement in each matched knowledge node; In this embodiment, after extracting the subjects of each text statement generated by the large language model and extracting knowledge nodes matching each subject from the knowledge node set of the preset graph database, knowledge information matching the corresponding text statement can be further obtained from each matching knowledge node. This knowledge information refers to the standardized knowledge units contained in the knowledge nodes, used to represent the attributes, relationships, and factual descriptions of entities. For example, based on the knowledge information corresponding to the knowledge node constructed through the aforementioned steps 1001-1005, the corresponding knowledge information is the triplet of the subject-predicate-object structure stored therein. This knowledge information covers the core facts such as the definition, characteristics, and relationships of entities, and can provide authoritative references for the factual verification of text statements. By obtaining knowledge information, after clarifying the association between the subject of the text statement and the knowledge nodes of the graph database, the semantic content of the text statement can be accurately compared with the authoritative knowledge units, thereby realizing the automated verification of the factual consistency of the generated text and laying the foundation for subsequent evaluation index calculation.

[0035] Optionally, see Figure 6 In some embodiments, the knowledge information that matches the corresponding text statement in each matched knowledge node is obtained, which can be performed according to the following steps: 1041. Convert the text statements and the knowledge information in the matched knowledge nodes into vector form respectively; 1042. Calculate the vector distance between each piece of knowledge information in vector form and its corresponding text statement in vector form; 1043. Select the first preset number of knowledge information items from the knowledge information of the matched knowledge nodes that have the smallest vector distance to the corresponding text statement as the matched knowledge information.

[0036] In this optional embodiment, quantitative matching of text statements and knowledge information is achieved through vector semantic similarity matching. Specifically, the vector form conversion of text statements and knowledge information in knowledge nodes can be implemented using pre-trained word vector models or sentence vector models. This vector form conversion transforms text semantics into dense vectors in a high-dimensional space, enabling more accurate capture of deep semantic features. By calculating vector distances (such as cosine distance or Euclidean distance) and quantitatively comparing similarity rankings, the core reference knowledge information used for subsequent evaluation can be determined. Specifically, this invention selects a pre-set number of knowledge information items as the core basis for the matching results. This pre-set number can be set and adjusted based on factors such as the knowledge information density of knowledge nodes, the semantic complexity of text statements, or system computing power limitations. This avoids excessive redundant information due to an excessively large pre-set number, increasing subsequent processing overhead, or insufficient pre-set number, leading to the omission of key knowledge information and affecting the comprehensiveness of the matching. This ensures the relevance and representativeness of the knowledge information, providing a reliable basis for verifying factual consistency.

[0037] 105. Input each text statement and its corresponding matching knowledge information into the preset evaluation model to obtain the statement evaluation results for each text statement; 106. Based on the sentence evaluation results of each text sentence, calculate the overall evaluation result of the text generated by the large language model.

[0038] In this embodiment, a hierarchical evaluation of the factual consistency and logical rationality of the text generated by the large language model is specifically adopted, proceeding from local to overall. First, each text sentence is evaluated independently until all sentences have been compared with the knowledge information. Then, an overall evaluation result for the entire text is calculated based on the evaluation results of all sentences. Specifically, the sentence evaluation result can be a quantitative score of indicators such as the correctness, completeness, or relevance of the text sentence. By inputting the aforementioned text sentences and matching knowledge information into a preset evaluation model, the sentence-level evaluation result is output. This preset evaluation model is based on a deep learning classification model or a pre-trained language model. Specifically, it can be pre-trained using labeled data (text sentence-knowledge information pairs and their matching labels), enabling the model to automatically determine the degree of matching between the text sentences and the knowledge information, thereby outputting a sentence-level factual consistency score. Alternatively, this preset evaluation model can be based on a different large language model from the knowledge domain corresponding to the text generated by the large language model. By inputting prompt words, it is guided to complete the evaluation task and output a binary judgment result or a normalized score result.

[0039] In this embodiment, the overall evaluation result can specifically be a comprehensive indicator such as accuracy, recall, or F1 score. It can be calculated based on different strategies using the sentence evaluation results, such as calculating the overall accuracy reflected by the correctness of each text sentence through mean calculation or weighted average calculation, determining overall logical consistency through majority voting, and generating a defect distribution report through error type statistics. This sentence evaluation result reflects the factual accuracy of the generated text at a local level, providing a basis for locating specific erroneous sentences. The overall evaluation result comprehensively reflects the global quality of the generated text, providing quantitative feedback for the optimization and iteration of the large language model. This evaluation method enables a comprehensive assessment of the text generated by the large language model from micro to macro levels. Based on the evaluation results, it can provide precise directions for improving the quality of the generated text, achieving interpretability evaluation of the text generated by the large language model, and helping to improve the reliability and credibility of the large language model in professional fields.

[0040] Optionally, in some embodiments, the large language model generates text based on a first preset large language model; the preset evaluation model is constructed based on a second preset large language model; the first preset large language model and the second preset large language model are different models; the sentence evaluation result is the sentence correctness parameter output by the preset evaluation model, and the overall evaluation result is the accuracy of the text generated by the large language model.

[0041] In this optional embodiment, the technical selection of the first preset large language model and the second preset large language model is further specified. The preset evaluation model adopts an independent large language model that is different from the knowledge domain corresponding to the generated text. For example, the first preset large language model can be a large language model with strong text generation capabilities, while the second preset large language model can be a large language model with better logical reasoning and fact-checking capabilities. This application does not impose specific restrictions on this, as long as the first preset large language model and the second preset large language model use different model architectures or training data.

[0042] In this optional embodiment, the evaluation results of text sentences and the overall evaluation results of text generated by the large language model are respectively assessed using a correctness parameter and the overall accuracy reflected by the comprehensive sentence correctness parameter. The accuracy parameter is a core indicator measuring the degree of matching between the text sentence and knowledge information. It can be a binary parameter (e.g., 0 for incorrect matching and 1 for correct matching), or a probability value or a normalized continuous score of that probability value (e.g., 0.85 for an 85% probability of correct matching). The quantitative characteristics of the correctness parameter and accuracy rate enable objective comparison of evaluation results, providing a unified standard for performance comparison of different generated texts or model versions. Compared to other indicators that can be used for partial or overall evaluation of text quality, it focuses more on the essential requirement of factual consistency, which helps avoid semantically similar but factually incorrect texts from obtaining high scores, thus improving the relevance of evaluation results to practical application needs.

[0043] This embodiment first splits the text generated by the large language model into multiple text sentences using the method described above. This allows each text sentence to be treated as an independent evaluation unit, enabling refined and localized evaluation, which facilitates more accurate discovery and location of local errors in the text generated by the large language model. Subsequently, by extracting the subject of each text sentence and extracting knowledge nodes matching each subject from a pre-defined graph database, the embodiment obtains knowledge information matching the corresponding text sentence from each matching knowledge node. This allows for precise extraction of knowledge information related to the text sentence. Furthermore, leveraging the structured storage and efficient retrieval characteristics of the graph database, key knowledge can be quickly located, ensuring the accuracy, authority, and applicability of the knowledge, thus providing strong support for the evaluation of the text sentences. Finally, by inputting each text statement and its corresponding matching knowledge information into a preset evaluation model, the evaluation results of each text statement are obtained. Based on the evaluation results of each text statement, the overall evaluation result of the text generated by the large language model is calculated. This enables a comprehensive evaluation from the local to the overall level. The results obtained in this way can objectively reflect the overall quality and local problems of the generated text, achieving efficient and accurate evaluation of the text generated by the large language model. Moreover, this evaluation process has a high degree of automation, reduces manual intervention, can quickly adapt to different domain knowledge systems, has good generalization ability, is suitable for large-scale high-accuracy evaluation needs, and is conducive to promoting the large-scale application of large language models in high-risk scenarios.

[0044] To perform the corresponding steps in the above method embodiments and various possible implementations, an implementation of a large language model-generated text evaluation device is provided below. Please refer to... Figure 7 , Figure 7 This is a functional module diagram of an embodiment of the large language model-generated text evaluation device of the present invention. In this embodiment, the large language model-generated text evaluation device 200 includes: a text splitting module 201, used to split the large language model-generated text into multiple text sentences; a subject extraction module 202, used to extract the subject of each text sentence; a node matching module 203, used to extract knowledge nodes from a preset graph database that match each subject; an information acquisition module 204, used to acquire knowledge information from each matched knowledge node that matches the corresponding text sentence; a sentence evaluation module 205, used to input each text sentence and its corresponding matching knowledge information into a preset evaluation model to obtain the sentence evaluation result of each text sentence; and a text evaluation module 206, used to calculate the overall evaluation result of the large language model-generated text based on the sentence evaluation results of each text sentence; wherein, the preset graph database is constructed based on professional knowledge corpus of the knowledge domain corresponding to the large language model-generated text, the preset graph database includes at least one knowledge node, the knowledge node is the carrier of entities in the preset graph database, the knowledge node corresponds one-to-one with the entity, and each knowledge node includes at least one piece of knowledge information.

[0045] Optionally, in some embodiments, the text splitting module 201 is further configured to: detect natural delimiters in the text generated by the large language model; and split the text generated by the large language model into multiple text sentences based on the natural delimiters.

[0046] Optionally, in some embodiments, the subject extraction module 202 is further configured to: perform part-of-speech tagging on the text statement to identify a first candidate subject in the text statement that conforms to the part of speech of the subject; perform syntactic analysis on the text statement to generate a syntactic tree structure of the text statement, and determine a second candidate subject of the text statement based on the syntactic tree structure; and select the subject of the text statement from the first candidate subject and the second candidate subject based on preset grammatical rules.

[0047] Optionally, in some embodiments, the node matching module 203 is further configured to: perform format standardization processing on the entity of each knowledge node in the subject and the preset graph database; calculate the matching degree between the entity of the subject and the entity of the knowledge node in the preset graph database, and determine whether there is an entity of a knowledge node in the preset graph database that matches the subject object described by the subject; if there is an entity of a knowledge node in the preset graph database that matches the subject object described by the subject, then the corresponding knowledge node is taken as the matched knowledge node; if there is no entity of a knowledge node in the preset graph database that matches the subject object described by the subject, then the similarity between the subject object described by the subject and the entity of each knowledge node is calculated, and the knowledge node corresponding to the entity with the highest similarity is taken as the matched knowledge node.

[0048] Optionally, in some embodiments, the large language model-generated text evaluation device 200 further includes a graph database module (not shown in the figure). This graph database module is specifically used for: obtaining professional knowledge corpus corresponding to the knowledge domain of the large language model-generated text; extracting entities from the professional knowledge corpus, as well as the relations and attributes corresponding to each entity; mapping each entity to knowledge nodes; mapping each relation to edges of the corresponding knowledge nodes; mapping each attribute to attribute information of the corresponding knowledge nodes, thereby obtaining a knowledge graph structure; based on the knowledge graph structure, determining at least one subject-predicate-object triple corresponding to each knowledge node, and using each triple as the knowledge information of the corresponding knowledge node; and storing each triple in a structured manner to obtain a preset graph database.

[0049] Optionally, in some embodiments, the information acquisition module 204 is further configured to: convert each knowledge information in the text statement and the matched knowledge node into vector form respectively; calculate the vector distance between each knowledge information in vector form and the corresponding text statement in vector form; and select the first preset number of knowledge information with the smallest vector distance to the corresponding text statement from the knowledge information of the matched knowledge node as the matched knowledge information.

[0050] Optionally, in some embodiments, the large language model generates text based on a first preset large language model; the preset evaluation model is constructed based on a second preset large language model; the first preset large language model and the second preset large language model are different models; the sentence evaluation result is the sentence correctness parameter output by the preset evaluation model, and the overall evaluation result is the accuracy of the text generated by the large language model.

[0051] Since the embodiments of the device part correspond to the embodiments of the above method, the description of the large language model generating text evaluation device 200 provided in the embodiments of the present invention should refer to the above method embodiments. The embodiments of the present invention will not be described again here, but have the same beneficial effects as the above large language model generating text evaluation method.

[0052] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing computer-readable instructions, which, when executed by the processor, cause the processor to perform the large language model-generated text evaluation method as described above. Figure 8 This is a functional block diagram of a computer device 300 provided in an embodiment of the present invention. The computer device 300 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 300. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the computer device 300.

[0053] Computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331. Those skilled in the art will understand that... Figure 8 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0054] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the large language model-generated text evaluation method as described above.

[0055] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated module or unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0056] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating text generated by a large language model, characterized in that, include: The text generated by the large language model is split into multiple text sentences. Extract the subject of each of the text statements; Extract knowledge nodes from the preset graph database that match each of the subjects; Obtain the knowledge information in each of the matched knowledge nodes that matches the corresponding text statement; Each of the text statements and the corresponding matching knowledge information are input into a preset evaluation model to obtain the statement evaluation results of each of the text statements; Based on the evaluation results of each of the text statements, the overall evaluation result of the text generated by the large language model is calculated; The preset graph database is constructed based on the professional knowledge corpus of the knowledge domain corresponding to the text generated by the large language model. The preset graph database includes at least one knowledge node, which is the carrier of the entity in the preset graph database. There is a one-to-one correspondence between the knowledge node and the entity, and each knowledge node includes at least one knowledge information.

2. The method for evaluating text generated by a large language model according to claim 1, characterized in that, The steps for constructing the preset graph database include: Obtain the professional knowledge corpus corresponding to the knowledge domain of the text generated by the large language model; Extract entities from the professional knowledge corpus, as well as the relationships and attributes corresponding to each entity; Each entity is mapped to a knowledge node, each relation is mapped to an edge of the corresponding knowledge node, and each attribute is mapped to the attribute information of the corresponding knowledge node, thus obtaining the knowledge graph structure. Based on the knowledge graph structure, at least one subject-predicate-object triplet corresponding to each knowledge node is determined, and each triplet is used as the knowledge information of the corresponding knowledge node. The triples are structured and stored to obtain the preset graph database.

3. The method for evaluating text generated by a large language model according to claim 1, characterized in that, The extraction of the subject of each of the text statements includes: The text statement is tagged with part-of-speech tags to identify the first candidate subject in the text statement that matches the part of speech of the subject; The text statement is parsed to generate a syntactic tree structure, and a second candidate subject is determined based on the syntactic tree structure. Based on preset grammar rules, the subject of the text statement is selected from the first candidate subject and the second candidate subject.

4. The method for evaluating text generated by a large language model according to claim 1, characterized in that, The extraction of knowledge nodes from the preset graph database that match each of the subjects includes: The subject and the entities of each knowledge node in the preset graph database are subjected to format standardization processing; The matching degree of the subject is calculated with the entities of the knowledge nodes in the preset graph database to determine whether there are entities of the knowledge nodes in the preset graph database that match the subject object described by the subject. If an entity with a knowledge node in the preset graph database matches the subject object described by the subject, then the corresponding knowledge node is taken as the matched knowledge node. If no entity in the preset graph database matches the subject object described by the subject, then the similarity between the subject object described by the subject and the entities of each knowledge node is calculated, and the knowledge node corresponding to the entity with the highest similarity is taken as the matched knowledge node.

5. The method for evaluating text generated by a large language model according to claim 1, characterized in that, The step of obtaining the knowledge information that matches the corresponding text statement in each of the matched knowledge nodes includes: Convert the text statement and each knowledge information in the matched knowledge node into vector form respectively; Calculate the vector distance between each piece of knowledge information in vector form and the corresponding text statement in vector form; The first preset number of knowledge information items with the smallest vector distance to the corresponding text statement from the knowledge information of the matched knowledge node are taken as the matched knowledge information.

6. The method for evaluating text generated by a large language model according to claim 1, characterized in that, The process of splitting the text generated by the large language model into multiple text sentences is described, including: Detect natural segmentation characters in the text generated by the large language model; The large language model generates text that is split into multiple text sentences based on the natural segmentation.

7. The method for evaluating text generated by a large language model according to any one of claims 1-6, characterized in that, The large language model generates text based on a first preset large language model; the preset evaluation model is constructed based on a second preset large language model; the first preset large language model and the second preset large language model are different models; the sentence evaluation result is the sentence correctness parameter output by the preset evaluation model, and the overall evaluation result is the accuracy of the text generated by the large language model.

8. A large language model-generated text evaluation device, characterized in that, include: The text splitting module is used to split the text generated by the large language model into multiple text sentences. The subject extraction module is used to extract the subject of each of the text statements. The node matching module is used to extract knowledge nodes from the preset graph database that match each of the subjects. The information acquisition module is used to acquire knowledge information that matches the corresponding text statement in each of the matched knowledge nodes; The statement evaluation module is used to input each of the text statements and the corresponding matching knowledge information into a preset evaluation model to obtain the statement evaluation results of each of the text statements; The text evaluation module is used to calculate the overall evaluation result of the text generated by the large language model based on the evaluation results of each of the text sentences. The preset graph database is constructed based on the professional knowledge corpus of the knowledge domain corresponding to the text generated by the large language model. The preset graph database includes at least one knowledge node, which is the carrier of the entity in the preset graph database. There is a one-to-one correspondence between the knowledge node and the entity, and each knowledge node includes at least one knowledge information.

9. A computer device, characterized in that, The computer device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the computer device to execute the large language model generating text evaluation method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the large language model text generation evaluation method as described in any one of claims 1-7.

Images